Cloudflare just caught Perplexity impersonating Chrome to scrape blocked sites
What's the story
Perplexity, a company involved in AI-powered search, has been accused of circumventing website scraping restrictions. The allegation comes from a new report by Cloudflare, which says Perplexity's bots are "stealth crawling" sites by disguising their identity to bypass robots.txt files and firewalls. Robots.txt is a standard file that websites use to communicate whether or not their content can be scraped by web crawlers.
Bot behavior
Despite block, Perplexity bots scraped content from unindexed website
Perplexity's official web crawling bots are "PerplexityBot" and "Perplexity-User." However, Cloudflare's tests showed that even when these specific bots were blocked by robots.txt files, Perplexity was still able to display content from a new, unindexed website. This behavior was also observed on sites with specific Web Application Firewall (WAF) rules that restricted web crawlers.
Bypass tactics
Cloudflare theorizes on how Perplexity is bypassing restrictions
Cloudflare theorizes that Perplexity may be circumventing these restrictions by using "a generic browser intended to impersonate Google Chrome on macOS" when its normal bots are blocked by robots.txt. The company's undisclosed crawler was also found rotating through IP addresses not listed in Perplexity's official IP range, allowing it to bypass firewalls. Cloudflare also observed the same behavior with autonomous system numbers (ASNs) used by Perplexity's crawler across tens of thousands of domains and millions of requests per day.
Past practices
Perplexity previously blamed 3rd-party web crawlers for scraping
In the past, multiple websites reported that Perplexity continued to scrape their content despite robots.txt prohibiting it. The company had then blamed the third-party web crawlers it was using at the time. To make amends for its previous actions, Perplexity partnered with several publishers to share revenue generated from ads displayed alongside their content.
Action taken
Cloudflare has removed Perplexity's bots from its verified bots list
In light of the allegations, Cloudflare has removed Perplexity's bots from its list of verified bots. The company has also introduced a method to identify and block Perplexity's stealth crawler from accessing its customers' content. This move comes as part of an ongoing effort to combat unauthorized web scraping practices by AI companies like Perplexity.