Cloudflare wants AI bots to pay you every time they scrape your website
It could offer website owners a new income stream.
For years, AI companies have been quietly devouring the web. Every article, blog post, image, or explainer you’ve ever published is probably already in their training data — scraped without permission, credit, or a single click back to your site. And while OpenAI and friends build billion-dollar models off your work, most publishers get… nothing.
Cloudflare, the internet infrastructure giant which powers one-fifth of the web, says it’s had enough. This week, it launched Pay per Crawl, an experimental marketplace where websites can charge AI bots a micropayment every time they scrape content.
And that slice couldn’t come at a better time. As more users turn to AI chatbots for answers, Google Search, once the internet’s main traffic engine, is losing steam. Publishers who used to rely on search referrals for ad revenue are now getting less traffic thanks to Google’s AI Overviews, even as bots scrape more than ever.
Cloudflare’s June data lays it bare: Google's crawler scrapes sites 14 times for every referral. OpenAI's crawler does it 17,000 times. Anthropic's? a whopping 73,000. That’s not a content economy; it’s a content heist.
Emmanuel Oyedeji
Cloudflare has been building toward this moment for a while. Over the past year, it’s rolled out tools to block AI crawlers with a single click, which more than a million customers have already used, along with dashboards that show exactly who’s scraping what. Now, with Pay per Crawl, websites can set the terms by choosing whether to block bots entirely, charge them, or let them in for free. Bots that don’t pay will stay locked out.
Big-name publishers like TIME, Condé Nast, The Atlantic, and ADWEEK are already on board, backing Cloudflare’s push for a permission-based model. And going forward, every new website on Cloudflare will block AI crawlers by default unless told otherwise.
For this to work, AI companies have to agree to the terms and start paying; something they’ve never done before. If they do, it could be the start of a new internet economy. One where bots pay to read, and every publisher finally gets paid to write. But there's still the question of how prices will be set, or whether publishers will actually earn anything meaningful.
Emmanuel Oyedeji
