How Cloudflare is helping website owners tackle AI crawlers
What's the story
Cloudflare, a leading internet infrastructure provider, has announced that it will block known artificial intelligence (AI) web crawlers by default. The move is aimed at preventing these bots from "accessing content without permission or compensation." The company will also ask website owners if they want to allow AI scrapers, and give some publishers the option to implement a "Pay Per Crawl" fee.
Monetization initiative
'Pay Per Crawl' program
The "Pay Per Crawl" program lets publishers set a price for AI scrapers to access their content. AI companies can then view this pricing and decide whether to register for the "Pay Per Crawl" fee or not. However, this option is currently available only for a select group of leading publishers and content creators.
Publisher backing
Restrictions on AI crawlers backed by major publishers
In March, Cloudflare introduced a feature that sends the web-crawling bots into an "AI Labyrinth" to prevent unauthorized scraping. Major publishers and online platforms such as The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora have backed the new restrictions on AI crawlers. This comes as websites prepare for a future where more people discover information through AI chatbots instead of traditional search engines.
Collaboration
AI companies have to state their purpose
Cloudflare is also working with AI companies to verify their crawlers and let them "clearly state their purpose," such as whether they're using the content for training, inference, or search. This information will be reviewed by website owners who can decide which crawlers to allow.