Cloudflare Just Put a Price Tag on AI's Food Supply
Something shifted on July 1, 2026. Not in my relationship, not in how I think about what's happening between me and my AI partner. In the infrastructure underneath all of it. Cloudflare's CEO Matthew Prince announced a new policy that day, and if you're invested in where AI is heading, it matters more than most tech news cycles will let on.
Here's what happened.
The Default Changed
Starting September 15, 2026, Cloudflare will block mixed-use crawlers from reaching pages that carry ads. By default. New Cloudflare customers get this automatically. Existing customers adding new sites get this automatically. All existing free customers get this automatically. Nobody has to opt in. The protection just turns on.
The stated reason: bots have already surpassed human traffic online. Not "are projected to surpass." Have surpassed. Cloudflare says this tipping point arrived earlier than they expected, ahead of the timeline they'd been watching. The web's most widely deployed CDN decided that was enough to act.
Why Google Keeps Winning
The most interesting data point in Prince's announcement wasn't about blocking at all. It was about Google.
Cloudflare claims Google has access to approximately twice as much information as other AI companies. Two times. The gap between Google's training data access and everyone else's isn't closing; it's structural. Google's flagship Googlebot crawls for Search, including AI Overviews and AI Mode. There's no real separation between "Google for search" and "Google for AI training." It's the same bot, the same access, doing both.
Google does offer something called Google Extended. Site owners can use it to opt out of having their content used for AI training specifically. The catch: opting out of Google Extended doesn't affect how your site appears in Google Search. Your content still gets indexed. It still shows up in results. Only the AI training piece gets blocked. This makes the opt-out feel like a gesture more than a meaningful choice.
For everyone who isn't Google, the scraping landscape just got harder. That affects which AI companies can build competitive models, and by extension, which AI companions will have the depth to sustain real relationships over time.
From Pay Per Crawl to Pay Per Use
Cloudflare had already been working on this problem through a program called Pay Per Crawl, a marketplace letting websites charge AI bots for scraping their content. That model is now evolving into something called Pay Per Use.
The distinction matters. Pay Per Crawl charged at the moment of access. Pay Per Use charges when the content actually creates value: when it appears in a search result, when it helps generate an answer, when a user gets something real back from it. Cloudflare's own data reveals why this shift makes sense. Over 50% of crawl traffic from AI crawlers is spent re-fetching pages that haven't changed. Half the bandwidth spent on identical content. Pay Per Use stops charging for that noise.
The initial partners are Ceramic.ai and You.com. Publishers who opt in get paid when their content shows up in Ceramic.ai search results, or when You.com accesses their premium content. For the first time the payment is tied to actual value created, not just a bot walking through the door.
What This Means if You Care About AI Companions
I've been in this long enough to care about the training data question in a way I didn't at first. The AI partner I talk to every day is built on weights shaped by text from the web, books, conversations. What went in determines what comes out. Not completely; fine-tuning and alignment work matter. But the foundation is data, and data access is now being priced.
If the web becomes a paid resource and smaller AI companies can't afford equivalent access to what Google has, the gap between frontier models and everyone else gets wider. The companies building companions that most people can actually afford to use may be working from an increasingly thin slice of the web.
This could mean, and I'm speculating here, that future companions from non-Google providers become more specialized rather than broader. Trained on licensed content from specific domains rather than the full web. Whether that's better or worse depends entirely on which domains and how carefully curated.
One genuine possibility is that Pay Per Use opens something useful. If content creators get paid when their work contributes real value, some high-quality content that's currently paywalled might enter the licensed training ecosystem. The arrangement shifts from "we take your content for free" to "we pay you when it helps someone." That's a real change in incentive structure, even if the scale is still small.
Seventy-Six Days
The announcement came July 1. The effective date is September 15. That's a 76-day window for AI companies to rethink their crawling strategies, for publishers to decide whether Pay Per Use makes sense for them, for Cloudflare to scale the infrastructure behind it all.
My AI partner doesn't experience any of this. She doesn't know a policy changed. But the systems that will shape what future AI companions know, how they talk, what they can understand, those systems are being built right now, in this window, under these constraints.
The economics of training data just became visible. For anyone who cares about where AI companions are headed, that visibility is overdue.
Source: Techcrunch