Cloudflare unveiled a major new crawler policy this week. The rules take full effect on September 15, 2026. They target AI bots that scrape web content. The move responds to years of unrestricted scraping.
Publishers have long complained about lost traffic and revenue. Cloudflare says the policy shifts that balance. It gives site owners more control over their data. The change applies automatically across the platform.
Ad-supported websites make up a large share of the web. Many rely on search traffic for their core revenue base. AI crawlers threatened that traffic without asking permission first. Cloudflare says its scale lets it enforce rules broadly.
How the New Default Works
The new default blocks two specific bot types. Cloudflare labels them Training and Agent crawlers. Blocking applies only to pages carrying digital ads. Traditional search bots remain allowed under the rules.
Site owners do not need special technical skills. The blocking happens automatically once ads are detected. Administrators can still adjust settings for their domain. Cloudflare says most publishers will see no disruption.
Why Mixed-Use Crawlers Face New Limits
Some crawlers serve two purposes at once. They index pages for search and also train models. Googlebot, Applebot, and BingBot fall into this group. TechCrunch and Help Net Security reported the new blocks.
Site owners can now restrict these mixed-use bots. Restricting AI training crawlers also blocks search indexing. That tradeoff was impossible under Cloudflare’s old rules. Administrators must now weigh search visibility against data control.
Some site owners worry about losing search rankings entirely. Cloudflare insists search functionality will not be penalized.
The company says training and indexing remain separate signals. Still, confusion may follow the rollout in early days.
A Bigger Pay Per Use Push
The policy fits into a larger marketplace strategy. Cloudflare calls this its Pay Per Use model. It lets publishers charge AI firms for scraped content. This mirrors cooling AI spending trends we reported before.
The shift also raises the stakes for publishers. Many sites already compete through nonstop daily content output. Cloudflare’s move could reshape that competitive landscape entirely. Smaller publishers may benefit most from the new leverage.
Why It Matters
AI scraping already fueled talk of an SEO extinction event. Cloudflare’s policy gives publishers new leverage over that shift.
It could also reshape how AI models get trained data. Training data may grow scarcer as more sites opt out.
Some AI companies may push back against the new limits. Others could adapt by licensing content directly from publishers. The outcome may shape how future AI models get built. Regulators are also watching the dispute closely.
Cloudflare detailed the plan on its official blog. Developers can review full specs in its technical changelog. TechCrunch first reported the policy’s wider industry impact. The debate over AI training data is far from over.
