Perplexity accused of scraping web sites that explicitly blocked AI scraping

[ad_1] AI startup Perplexity is crawling and scraping content material from web sites which have explicitly indicated they don’t wish to be scraped, in keeping with web infrastructure supplier Cloudflare.

On Monday, Cloudflare printed analysis saying it noticed the AI startup ignore blocks and conceal its crawling and scraping actions. The community infrastructure large accused Perplexity of obscuring its id when making an attempt to scrape net pages “in an try to avoid the web site’s preferences,” Cloudflare’s researchers wrote.

AI merchandise like these supplied by Perplexity depend on gobbling up massive quantities of information from the web, and AI startups have lengthy scraped textual content, pictures, and movies from the web many occasions with out permission to make their merchandise work. In current occasions, web sites have tried to battle again by utilizing the online commonplace Robots.txt file, which tells engines like google and AI corporations which pages could be listed and which shouldn’t, efforts which have seen combined outcomes up to now.

Perplexity seems to be willingly circumventing these blocks by altering its bots “consumer agent,” that means a sign that identifies an internet site customer by their machine and model kind; in addition to altering their autonomous system networks, or ASN, primarily a quantity that identifies massive networks on the web, in keeping with Cloudflare.

“This exercise was noticed throughout tens of hundreds of domains and tens of millions of requests per day. We have been in a position to fingerprint this crawler utilizing a mix of machine studying and community alerts,” learn Cloudflare’s publish.

Perplexity spokesperson Jesse Dwyer dismissed Cloudflare’s weblog publish as a “gross sales pitch,” including in an electronic mail to TechCrunch that the screenshots within the publish “present that no content material was accessed.” In a follow-up electronic mail, Dwyer claimed the bot named within the Cloudflare weblog “isn’t even ours.”

Cloudflare mentioned it first seen the habits after its prospects complained that Perplexity was crawling and scraping their websites, even after they added guidelines on their Robots file and for particularly blocking Perplexity’s identified bots. Cloudflare mentioned it then carried out assessments to test and confirmed that Perplexity was circumventing these blocks.

Techcrunch occasion

San Francisco | October 27-29, 2025

“We noticed that Perplexity makes use of not solely their declared user-agent, but additionally a generic browser meant to impersonate Google Chrome on macOS when their declared crawler was blocked,” in keeping with Cloudflare.

The corporate additionally mentioned that it has de-listed Perplexity’s bots from its verified record and added new strategies to dam them.

Cloudflare has just lately taken a public stance in opposition to AI crawlers. Final month, Cloudflare introduced the launch of a market permitting web site homeowners and publishers to cost AI scrapers who go to their websites. Cloudflare’s chief government Matthew Prince sounded the alarm on the time, saying AI is breaking the enterprise mannequin of the web, significantly publishers. Final yr, Cloudflare additionally launched a free device to stop bots from scraping web sites to coach AI.

This isn't the primary time Perplexity is accused of scraping with out authorization.

Final yr, information retailers, resembling Wired, alleged Perplexity was plagiarizing their content material. Weeks later, Perplexity’s CEO Aravind Srinivas was unable to right away reply when requested to offer the corporate’s definition of plagiarism throughout an interview with TechCrunch’s Devin Coldewey on the Disrupt 2024 convention.