Perplexity allegedly used stealth mode to scrape web data
Cloudflare researchers have called out Perplexity, an AI company, for sneaking around website blocks and scraping data from sites that clearly said "no bots allowed."
Even when websites used tools like robots.txt and banned Perplexity's official bots, the company allegedly disguised its software as regular browsers and switched up its IP addresses to keep scraping content.
Cloudflare has rolled out new security rules
To double-check, Cloudflare tested with private sites that weren't even indexed or open to bots. Still, Perplexity managed to pull detailed info from them.
When Cloudflare blocked Perplexity's access, the quality of its answers dropped a lot—showing it relies on this kind of data.
In response, Cloudflare has rolled out new security rules and smarter bot detection using machine learning to stop stealthy crawlers like this for all their users—even those on free plans.