Two Gates Are Closing on AI Web Scraping (opens in new tab)

Google narrowed developer access to its web-search tools in January, while Cloudflare documented broader controls for blocking or challenging AI crawlers. Together, those changes have made ai web scraping more constrained at both the search layer and the site-access layer. The squeeze is practical, not abstract. Google’s changes affect how developers get URLs and search results at scale; Cloudflare’s controls affect whether bots can fetch the pages behind those URLs. For agent workflows that ...