Josh has worked a freelance writer for the past ten years, writing news and features focusing on the gaming, science, and tech industries. He has covered big events like E3, CES, and a slew of other smaller press events oriented around the latest consumer technology and gadgets.
A new report from Cloudflare claims that Perplexity has been scraping content from websites that have opted to block AI web scrapers. The company says that Perplexity's continued attempts to hide its crawling activity has led to diminished trust from websites that have opted out of sharing their content with AI companies like Perplexity.
That particular file — for those who haven't run a website — is responsible for relaying a website owner's preferences to bots. And since Perplexity has supposedly been ignoring the preferences set by users, Cloudflare says it has delisted the company as a verified bot and has added additional measures to its services to block the stealthy crawling attempts.
Cloudflare says that it found that when met with blockage, Perplexity resorted to using an undeclared user-agent intended to mimic Google Chrome on macOS. This undeclared crawler then utilized multiple IP addresses not listed in Perplexity's official IP range and would rotate through those IPs as it ran into blockages from the robots.txt file for certain pages.
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Perplexity is allegedly scraping websites it's not supposed to, againFind the latest technology news and expert tech product reviews. Learn about the latest gadgets and consumer tech products for entertainment, gaming, lifestyle and more.
Read more »
Cloudflare says Perplexity’s AI bots are ‘stealth crawling’ blocked sitesA new report from Cloudflare accuses Perplexity of ignoring robots.txt files and concealing the identity of its AI crawlers to get past websites’ restrictions.
Read more »
Perplexity CEO says AI tools cut development time from days to hoursBusiness Insider tells the global tech, finance, stock market, media, economy, lifestyle, real estate, AI and innovative stories you want to know.
Read more »
Perplexity launches AI-powered Comet browser, and there’s a big reason to stay awayPerplexity launched its own browser called Comet, and I have a big reason to avoid it. Spoiler, it’s not the price.
Read more »
Perplexity launches AI-powered web browser for select group of subscribersPerplexity AI launched a new artificial intelligence-powered web browser called Comet
Read more »
Perplexity just launched an AI web browserPerplexity has launched Comet, a web browser that uses Perplexity as its default search engine and comes with a built-in AI assistant.
Read more »