Cloudflare lets sites block AI crawlers with one click

Enlarge (credit: Moor Studio | DigitalVision Vectors)

Cloudflare announced new tools Monday that it claims will help end the era of endless AI scraping by giving all sites on its network the power to block bots in one click.

That will help stop the firehose of unrestricted AI scraping, but, perhaps even more intriguing to content creators everywhere, Cloudflare says it will also make it easier to identify which content that bots scan most, so that sites can eventually wall off access and charge bots to scrape their most valuable content. To pave the way for that future, Cloudflare is also creating a marketplace for all sites to negotiate content deals based on more granular AI audits of their sites.

These tools, Cloudflare's blog said, give content creators "for the first time" ways "to quickly and easily understand how AI model providers are using their content, and then take control of whether and how the models are able to access it."