Crawlers
"Crawling" a website is the process of visiting every link on a website and building a database of its contents. This might sound weird, until you realize that the entire concept of search engines depends on this. Obviously, this has the potential to disable a website by overwhelming it with crawler traffic, especially if the crawler is from a massive company like Google. To work around this, the internet has come up with a two-fold solution.
The first part is rate-limiting - if there is too much traffic from a single source, the server blocks further requests and informs the client that it may try again after a specified amount of time. Reputed crawlers like Google's tend to stay within the enforced rate-limits, because they have an interest in maintaining goodwill.
The second part is a file called robots.txt - this is a file placed in every website that declares restrictions for each client on accessing the site. Clients are identified by a "user-agent" value in the header, and the server uses the user-agent value to enforce the restrictions declared in the robots.txt file. Reputed crawlers tend to respect this too, so as not to poison the well that everyone drinks out of.
Predatory Practices Of Data Mining Companies
The machine learning industry has always had a tendency to compile datasets with utter disregard - verging on mild contempt - for data ownership rights. Collecting data with questionable consent, using datasets for commercial uses when licenses permit only academic usage, and other such violations. With AI companies attempting to collect data from pages on the internet to train their models, some are choosing to violate the above norms.
Xe Iaso is a developer whose Gitea server was overwhelmed by the AmazonBot crawler, leading to site instability and downtime. He tried to stop it with all the normal defenses - setting limits in robots.txt, blocking user-agents associated with crawlers, filtering out IP addresses with suspicious traffic patterns - but this crawler seems to be configured to sidestep these protections. The crawler has been observed to spoof its user-agent and use IP addresses from residential areas.
The larger issue is that AI-affiliated crawlers bypass existing conventions and tend to overwhelm the servers of projects with limited resources or manpower. It's almost as if goodwill is irrelevant to AI companies with 12 figures worth of capital.
Solutions
Restrictions
Iaso wanted to maintain public access to his server, but was soon forced to hide it behind a VPN. He soon brought it back for public access, this time with a protection called Anubis. This first forces the client to find a solution (out of many possible solutions) to a mathematical problem via trial-and-error, which takes about 1 million attempts at the default difficulty. Once the client sends a solution back which is verified by Anubis, it is given a cookie with a lifespan of 1 week, that grants it permission to access the server for a week. Crawlers, that tend to be made up of a bunch of computers, typically don't share these cookies with each other, so it is expensive for them to have to pass through Anubis every time.
Counterattacks
Nepenthes, named for a carnivorous plant that eats anything it traps, is an "AI tarpit". On pages that are restricted from crawlers in robots.txt, there is an entry point into the tarpit. This is a page containing links to other pages within the tarpit, with each of these pages linking to even more pages. Each of these pages is generated algorithmically, so it appears to never change, and is filled with meaningless drivel. A human going to this page would quickly spot that they've stumbled into something useless. A greedy data scraper, however, would lap it all up, focusing all its efforts in traversing this exponential tree of webpages filled with junk engineered to poison the LLM.