Cloudflare Gives All Customers Granular Controls Over AI Crawlers—and a Path to Charge Them
Summary: On July 2, Cloudflare deployed a new AI traffic management layer available to all plans—including Free—letting website owners selectively block or allow AI crawlers by purpose rather than by blanket rule.
Key Points
- Three-way crawler classification: Search (allowed by default), Agent, and Training bots can each be controlled independently per zone or per page section
- BotBase: an enterprise searchable catalog of verified bots with classifications and behavioral data, giving administrators visibility into what's hitting their sites
- September 15 default change: all new domains will auto-block Training and Agent bots on ad-monetized pages; existing customers can opt out before that date via Security settings
- Pay Per Crawl evolves to Pay Per Use: content creators can receive compensation based on how AI agents actually consume their content, not just on access
Why It Matters
Cloudflare CEO Matthew Prince confirmed that bot traffic has for the first time exceeded human traffic. By turning AI crawler management into a free, infrastructure-level feature, Cloudflare is effectively standardizing how publishers defend and monetize their content against AI companies—shifting leverage from AI labs to website owners at global scale.