Bots are overwhelming websites with their hunger for AI data(theregister.com)
theregister.com
Bots are overwhelming websites with their hunger for AI data
https://www.theregister.com/2025/06/17/bot_overwhelming_websites_report/
14 comments
This is an ever growing problem.
The model of the web host paying for all bandwidth was somewhat aligned with traditional usage models, but the wave of scrapping for training data is disrupting this logic.
I remember reading, about 10 years ago?, of how backend website communications (ads and demographic data sharing) had surpassed the bandwidth consumed by actual users. But even in this case, the traffic was still primarily linked to the website hosts.
Whereas with the recent scrapping frenzy the traffic is purely client side, and not initiated by actual website users, and not particularly beneficial to the website host.
One has to wonder what percentage of web traffic now is generated by actual users, versus host backend data sharing, and the mammoth new wave of scrapping.
The model of the web host paying for all bandwidth was somewhat aligned with traditional usage models, but the wave of scrapping for training data is disrupting this logic.
I remember reading, about 10 years ago?, of how backend website communications (ads and demographic data sharing) had surpassed the bandwidth consumed by actual users. But even in this case, the traffic was still primarily linked to the website hosts.
Whereas with the recent scrapping frenzy the traffic is purely client side, and not initiated by actual website users, and not particularly beneficial to the website host.
One has to wonder what percentage of web traffic now is generated by actual users, versus host backend data sharing, and the mammoth new wave of scrapping.
> Some of the bots identify themselves, but some don't. Either way, the respondents say that robots.txt directives – voluntary behavior guidelines that web publishers post for web crawlers – are not currently effective at controlling bot swarms.
Is anybody tracking the IP ranges of bots or anything similar that's reliable?
It seems like they're taking the "what are you gonna do about it" approach to this.
Edit: Yes [1]
[1] https://github.com/FabrizioCafolla/openai-crawlers-ip-ranges
Is anybody tracking the IP ranges of bots or anything similar that's reliable?
It seems like they're taking the "what are you gonna do about it" approach to this.
Edit: Yes [1]
[1] https://github.com/FabrizioCafolla/openai-crawlers-ip-ranges
I think that some of big tech already said that they don't respect robots in what is prohibited. It is highly probable that ordinary bots also does not respect robots.txt.
GPTBot certainly doesn't - I added a blanket disallow for it several months ago and in the last 5 days, it's done 22k requests (rate-limited to a max of 5 req/minute, all proxied to iocaine[0].)
[0] https://iocaine.madhouse-project.org
[0] https://iocaine.madhouse-project.org
Many bots use residential IP proxy networks, so they come from the same IPs that humans use
The additional bad outcome is that all content can go behind logins, and paywalls. What then? You will have to provide data, email in every corner of the web to lo in.
There are also good crawlers that search for sites, like Google, or marginalia which gives your page recognizibility. If you lock everything from the web, we'll it disappears from the web.
There are also good crawlers that search for sites, like Google, or marginalia which gives your page recognizibility. If you lock everything from the web, we'll it disappears from the web.
Information is valuable; we just weren't charging for it. AI is just bringing the market for knowledge back into equilibrium.
It looks more like information is valuable in aggregate.
That's a point that's often overlooked. I suspect many of the "amazing insights." LLM events only happen because training sets encompass an extensive range of knowledge and can arrive at conclusions previously unseen.
One could reasonably claim that the value of AI systems and very large training sets is not that it is an approach to AGI, but that it makes finding previously unseen connections possible.
One could reasonably claim that the value of AI systems and very large training sets is not that it is an approach to AGI, but that it makes finding previously unseen connections possible.
Who's doing this at such a high volume? Most of the data is static enough that there isn't value in frequent crawls, crawls are (probably) more expensive than caching, and small shops and hobbyists don't have the resources to move the needle.
I think it was wikipedia recently that was pissed because bots were crawling the site instead of using the already available datasets.
AI bots don't care about caches. That's one of the big issues
AI bots don't care about caches. That's one of the big issues
While catchy that headline kind of misses the point. It should be "Corporations are overwhelming websites with their hunger for AI data". They're the ones doing it and corporations are by far the most damaging non-human persons (especially since they are formed nowadays to abstract away liability for the damage they cause).
This is not some new enemy "bots". This is the same old non-human legal persons that polluted our physical world repeating things in the digital. Bots run by actual human persons are not the problem.
This is not some new enemy "bots". This is the same old non-human legal persons that polluted our physical world repeating things in the digital. Bots run by actual human persons are not the problem.
I'm not sure that's true. As hardware gets cheaper, you're going to see more and more people wanting to build+deploy their own personal LLMs to avoid the guardrails/censorship (or just the cost) of the commercial ones, and that means scraping the internet themselves. I suspect the amount of scraping that's coming from individuals or small projects is going to increase dramatically in the months/years to come.
What's the solution here? Metered usage based on network traffic that gets shared with the website owners?
Otherwise everything moves behind a paywall?
Otherwise everything moves behind a paywall?
For now the solution is proof-of-work systems like Anubis combined with cookie-based rate limiting: you get throttled if your session cookie indicates you scraped here before, and if you throw the cookie out you get the POW challenge again. I don't know how long this will continue to work, but for my site at least it seems to be holding back the deluge, for the moment.
>Otherwise everything moves behind a paywall?
Basically. Paywalls and private services. Do things that are anti-scale, because things meant for consumption at scale will inevitably draw parasites.
Basically. Paywalls and private services. Do things that are anti-scale, because things meant for consumption at scale will inevitably draw parasites.
ai.robots.txt contains a big list of AI crawlers to block, either through robots.txt or via server rules:
https://github.com/ai-robots-txt/ai.robots.tx
https://github.com/ai-robots-txt/ai.robots.tx
This actually blocks a lot more than just AI crawlers. You shouldn’t use this without reviewing it in detail so that you understand what you are actually blocking.
For instance, it includes ChatGPT-User. This is not a crawler. This is used when a ChatGPT user pastes a link in and asks ChatGPT about the contents of the page.
One of the entries is facebookexternalhit. When you share a link on Facebook, Threads, WhatsApp, etc., this is the user-agent Meta uses to fetch the OpenGraph metadata to display things like the title and thumbnail.
Skimming through the list, I see a bunch of things like this. Not every non-browser fetch is an AI crawler!
For instance, it includes ChatGPT-User. This is not a crawler. This is used when a ChatGPT user pastes a link in and asks ChatGPT about the contents of the page.
One of the entries is facebookexternalhit. When you share a link on Facebook, Threads, WhatsApp, etc., this is the user-agent Meta uses to fetch the OpenGraph metadata to display things like the title and thumbnail.
Skimming through the list, I see a bunch of things like this. Not every non-browser fetch is an AI crawler!
Your link is missing the t at the end of .txt. You should be able to edit it though.
This is going to drive all blogs to GitHub gists and such.
RIP internet. It will soon make no sense to share something with the world unless you're in for profit. But who's gonna pay for it?
[deleted]
[deleted]
I thought everything was going well after that, until suddenly it started getting even worse. I started realizing that instead of one IP hitting the site a hundred times per second, it was now hundreds of IP's hitting the site Slightly below the Throttling threshold I had set up.