Cloudflare launches Pay Per Crawl for AI bots
Cloudflare lets publishers charge AI crawlers per request via HTTP 402. From September 15, agent and training bots are blocked by default on new domains.
Cloudflare announced Pay Per Crawl on July 1, a system letting publishers charge AI bots per HTTP request using the 402 status code and Web Bot Auth key pairs. Publishers set flat per-request prices; crawlers either discover pricing from a 402 response or declare intent with price headers upfront. Cloudflare acts as merchant of record. The system is in private beta, with a public waitlist open.
Alongside Pay Per Crawl, Cloudflare published its one-year retrospective on AI traffic and announced three new crawler control tiers: search, agent, and training. Starting September 15, 2026, new Cloudflare domains and all free-tier sites will block agent and training bots by default on ad-monetized pages. Search crawlers remain allowed. Existing customers can opt out before the cutover.
Cloudflare’s data puts more than 50% of internet traffic as non-human as of mid-2026, with AI training crawlers accounting for 52% of all crawler requests, up from 22% in spring 2025. Ceramic.ai and You.com are listed as early Pay Per Crawl partners for content compensation.
Why this matters for builders
Two populations of operators are directly affected. First, anyone running AI agents that browse the web for research, RAG, or competitor monitoring will encounter 402 paywalls at scale once the ecosystem adopts Pay Per Crawl, adding a per-request cost to web access. Second, anyone running a content site on Cloudflare needs to decide their default before September 15 or accept the new block-by-default behavior.
For the agent-side, the practical response is to audit your web-fetch calls now and identify which sources would block you under the new defaults. The guide on MCP integrations covers building resilient tool calls, and the same resilience applies when a previously-free URL starts returning 402.