Cloudflare has announced a policy forcing AI companies to choose between paying publishers for content or being blocked from accessing their sites by default. The company is giving AI firms until September 15, 2026 to separate web crawlers used for search indexing from those used for AI training and agent development. Failure to comply will result in automatic blocking on many publisher sites hosted on Cloudflare's infrastructure.

This move targets the practice of AI companies using publisher content to train models without compensation. Cloudflare frames the policy as supporting the principle that content creators should be compensated when their work is used to build commercial AI systems. The deadline gives AI companies three months to either implement the separation or negotiate licensing agreements with publishers.

What This Means for Your Business

If your company operates an AI product that ingests web data for training or inference, you need to audit your crawling practices immediately. Determine whether your AI training crawlers are operationally separate from search indexing crawlers—Cloudflare's enforcement deadline is mid-September. For publishers, this creates leverage to negotiate licensing deals with AI companies. For AI companies without existing publisher deals, budget for either technical separation of crawlers or licensing fees, as blocking from major Cloudflare-hosted sites will impact training data quality.