The age of the AI scraping free-for-all may be coming to an end. At least if Cloudflare gets its way.
Web crawlers have trawled the internet for information for decades. Without them, people would lose vitally important online tools, from Google Search to the Internet Archive’s invaluable digital preservation work. But the AI boom has produced a corresponding boomlet in AI-focused web crawlers, and these bots scrape web pages with a frequency that can mimic a DDoS attack, straining servers and knocking websites offline.
According to a report from the content licensing platform Tollbit, which offers its own marketplace for publishers to negotiate with AI companies over bot access, AI scraping is still on the rise—including scraping that ignores robots.txt. Tollbit found that over 26 million scrapes ignored the protocol in March 2025 alone.
Artificial Intelligence Bots Algorithms
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Cloudflare will now block AI crawlers by defaultCloudflare has announced that it will now block AI web crawlers by default for new customers. It’s also introducing a new “Pay Per Crawl” fee that will let some publishers make AI companies pay to scrape their content.
Read more »
Web giant Cloudflare to block AI bots from scraping content by defaultStarting Tuesday, every new web domain that signs up to Cloudflare will be given the option to allow — or block — AI crawlers.
Read more »
How creepy crawlers of all kinds respond to extreme Texas heatSome bugs are seeking refuge from the heat in human habitats, while others thrive in rising temperatures.
Read more »
Cloudflare CEO says people aren't checking AI chatbots' source linksMariella Moon has been a night editor for Engadget since 2013, covering everything from consumer technology and video games to strange little robots that could operate on the human body from the inside one day. She has a special affinity for space, its technologies and its mysteries, though, and has interviewed astronauts for Engadget.
Read more »
Sky-high housing prices spur San Jose charter school defaultDowntown College Prep, a three-campus charter school operator in San Jose, defaulted on $34 million of municipal bonds earlier this month.
Read more »
Trump administration sues Maryland federal judges over order blocking removal of immigrantsThe Trump administration is suing federal judges in Maryland over an order that blocks the immediate removal of any detained immigrant who requests a court hearing.
Read more »