Cloudflare accuses Perpelexity AI of sneaking around to scrape website content
This isn't the first time the AI company has been accused of dipping its hands in the data jar.
Last month, Cloudflare posed a question: “Why are AI bots allowed to eat the internet for free?” In answering that question, the company then introduced Pay per crawl, which requires AI companies to pay website owners every time they scrape content.
Not more than a month later, Cloudflare publicly accused Perplexity of scraping websites that had not consented to be scraped. According to Cloudflare, Perplexity’s crawler hid behind fake user agents, impersonated Google Chrome, rotated through shadow IP addresses, and accessed content that was explicitly blocked.
Consequently, Perplexity’s bots were delisted from its trusted list.
If you’re wondering whether this was some random misunderstanding, it wasn’t. Cloudflare set up test domains, specifically hidden from search engines, and Perplexity’s crawlers still found their way in. The AI company was actively working around restrictions, and we're not talking about a few one-off hits. This was happening across tens of thousands of domains and millions of requests per day.
Kelechi Edeh
Perplexity, of course, denied everything, claiming the crawler in question wasn’t even theirs. But it’s hard to ignore the growing pile of allegations. This isn’t the first time Perplexity’s been caught with its hand in the data jar. In 2024, Wired accused the company of plagiarism.
All of this plays into a much bigger moment for the internet. As AI-powered search tools like Perplexity grow in popularity, traditional search engines like Google are slowly losing dominance. And while users might love getting AI-generated summaries instantly, the websites that fuels those answers are being hollowed out. No traffic. No credit. No compensation.
Cloudflare holds a significant position, putting away the era of free scraping. With Pay per Crawl, they’re giving websites a way to either block AI bots entirely, charge them for access, or let them in for free. Going forward, every new site using Cloudflare will block AI bots by default, a reversal of the open-access era that trained GPT, Claude, and Grok.
As for Perplexity, the damage may already be done. The company’s bots are now lumped in with untrusted traffic, making it harder to crawl sites undetected. And if Cloudflare’s defenses hold up, Perplexity’s AI search engine may start looking a lot less smart without the data it’s been quietly scraping.
Louis Eriakha
