Adweek reports, that Cloudflare is set to deny access by multi-purpose crawlers to many sites in their content delivery network by default, particularly those that serve to gain data for training generative AI, which stands to block Google's access to up to one-fifth of the World Wide Web.
The default will be set for new accounts and sites on Cloudflare's free tier. The move will affect Apple and Bing, but especially Google, which has used a single crawler to index sites both for their search and their AI training. In the past this would have meant economic suicide for commercial sites, but over time they've been getting so much less traffic from Google that the tradeoff doesn't seem as bad.
The default will be set for new accounts and sites on Cloudflare's free tier. The move will affect Apple and Bing, but especially Google, which has used a single crawler to index sites both for their search and their AI training. In the past this would have meant economic suicide for commercial sites, but over time they've been getting so much less traffic from Google that the tradeoff doesn't seem as bad.
/s3/static.nrc.nl/wp-content/uploads/2026/07/09154855/ad2d2ef3-f295-4e58-99f2-93327a18e734.jpg)
/s3/static.nrc.nl/wp-content/uploads/2026/07/09085531/090726VER_2035083021_verhorenWillemsen.jpg)
