Cloudflare, a major internet infrastructure provider powering about 20% of global web traffic, has rolled out a significant policy change: as of July 2025, it is blocking AI web crawlers from scraping content by default unless website owners explicitly permit them. This marks a turning point in the landscape of AI model training and content creator rights.
Hello and welcome back to AI SIX, the podcast that demystifies AI in just six minutes a day, and inspires you by others’ experiences in our Saturday long-listen episode.
By default, AI crawlers (bots used by companies such as OpenAI, Google, Anthropic, Meta, and more to collect content for large language models) are now blocked from accessing any website using Cloudflare’s services unless permission is granted by the site owner.
This shift means AI companies can no longer freely scrape huge swathes of the Internet for data without direct consent or compensation for the content creators.
The Cloudflare story is big AI news for three reasons.
- It immediately reduces the amount of freely accessible web data for AI training. Since Cloudflare powers a large share of the world’s websites, a major portion of public internet content is now off-limits to AI by default unless explicit permission is given, forcing a fundamental change in how companies acquire data for large models.
- It shifts economic power and control back to content creators and publishers. With mechanisms like Pay Per Crawl and default blocking, creators can now decide who uses their work and potentially earn compensation, correcting the previous “free-for-all” era dominated by unconsented AI scraping.
- It signals a new era of AI industry regulation and sustainability. By using its infrastructure, Cloudflare has set a precedent, pushing the web towards a consent-based, permission-first model for AI data collection, influencing other infrastructure providers and likely reshaping AI business models industry-wide
Cloudflare’s move is a response to several industry pressures:
- Content creators and publishers have grown increasingly frustrated with AI firms using their work, sometimes in massive volumes, without consent or compensation.
- Until now, AI bots would often ignore robots.txt and scrape content freely, whereas search engines at least sent some traffic back to the creators, allowing ad revenue; the new paradigm left original creators out of the loop.
- Cloudflare positioned this change as necessary to safeguard a sustainable, “free and vibrant internet” that doesn’t allow value extracted from creators to bypass them.
How Does Cloudflare’s Blocking Work?
- New domains signing up for Cloudflare are prompted to allow or block AI crawlers as part of the setup process.
- Existing customers now have a one-click option in the dashboard to block all known AI bots (including those identifying as AI crawlers for training, inference, or search purposes).
- For known AI crawlers, Cloudflare uses advanced behavioral analysis and machine learning to detect even those attempting to disguise themselves by spoofing user agents.
- The new setting is “permission-first”: AI companies must clearly declare their purpose and obtain explicit consent before crawling. If they want access for training, inference, or search, they must specify which one.
Introducing Pay Per Crawl
- Cloudflare has debuted a “Pay Per Crawl” system, allowing participating publishers to set terms (including pricing) for AI companies to access their content.
- This creates a permissioned, potentially compensated ecosystem for content access, opening the door to new monetization streams for original creators.
Industry Impact
- Magnitude: Because Cloudflare powers about a fifth of the Internet, its default blocking now removes a substantial portion of online content from being freely available to AI trainers.
- Publisher Participation: Major publishers and content sites,including Gannett, BuzzFeed, The Atlantic, Time, the Associated Press, and Stack Overflow have endorsed or joined Cloudflare’s initiative.
- For AI Companies: GenAI firms must now negotiate for access, pay for it, or look elsewhere, making unauthorised scraping of content substantially harder and riskier.
What Does It Mean For the Web?
- Empowerment for creators and publishers: Shifts the balance of control and value, letting content owners decide how (and whether) their work powers AI systems.
- Sustainable digital ecosystem: Could channel more money and credit toward originators, not just aggregators or AI model providers.
- Changing AI business models: AI companies face increased costs or friction; smaller AI projects could be particularly impacted due to loss of unrestricted access.
| Change | Previous State | New State (Post-Cloudflare) |
| AI bots access (default) | Open by default | Blocked by default |
| Site owner control | Difficult, not granular | One-click block, granular control |
| Monetization for content creators | Mostly absent | Introduced via “Pay Per Crawl” |
| Respect for robots.txt by AI bots | Often ignored | Cloudflare enforces, goes beyond |
| Impact on AI training data availability | Nearly unrestricted | ~20% of web now restricted |
Cloudflare’s new blocking policy is a major internet governance event, ending the era of free-for-all AI web scraping and pushing toward a consent-based, compensated model. It is seen as empowering publishers and shifting the debate on how the benefits of AI, which relies fundamentally on huge volumes of human-created content, are distributed across the web economies.
About the Podcast Host
Joanne Sweeney is a specialist in public-sector marketing and digital communications. She’s the author of Public Sector Marketing Pro and the creator of the Professional Diploma in AI for Public Sector Marketing. With years of experience in training, consulting, and advocating for digital transformation, Joanne provides insight and guidance for government agencies navigating the digital landscape. She is also the host of the award-winning Public Sector Marketing Show podcast.
