AI Crawlers: The Vampiric Leech Sucking Websites Dry

Hello everyone. Let’s talk about the infestation plaguing the internet today – no, not influencer reels or “growth hacks” promising six figures in passive income if you scream into TikTok for long enough. I’m talking about AI crawlers and fetchers, those parasitic bots unleashed into cyberspace by the new Silicon Valley messiahs – Meta, OpenAI, and their band of merry exploiters. And boy, if Fastly’s report is accurate, it’s less about “the open web” and more about GTA V’s online lobby chaos – you log in, and within two seconds the whole thing implodes under the strain of nonsense packets being lobbed at the servers.

The Problem: An Internet Choked With Bots

Now, let’s examine the core issue here. According to Fastly’s data, AI crawlers are responsible for 80% of AI bot traffic, while fetchers account for the other 20%. The crucial point? This isn’t charming background traffic. We’re not talking about a friendly spider politely visiting your website once a week. We’re talking thousands of requests per minute. That’s the server equivalent of being asked if you’ve tried Microsoft Clippy’s “helpful tips” 39,000 times in a row. Unsurprisingly, this isn’t healthy.

The stats are mind-numbingly telling: Meta alone is behind 52% of crawler traffic, Google standing at 23%, and OpenAI at 20%. Yes, that triumvirate of “tech saviors” account for a choking 95% of crawler chaos. What remains is a thin slice reserved for Anthropic and some bottom feeders like Perplexity, who, apparently, can’t even respect the sanctity of a robots.txt file. Translation: the single digital equivalent of a “please do not disturb” sign – and they still burst in like drunk students at 3 AM looking for pizza.

Fetchers: The Internet’s Unsleeping Vampires

If crawlers are annoying mosquitos, fetchers are giant bats with bloodshot eyes. They pounce when prompted by user requests. And my word, when they pounce, they hit harder than a Dark Souls boss ambush. Fastly records OpenAI alone accounting for 98% of fetch traffic. When a single fetcher runs 39,000 requests a minute, that’s less fetching, more full-on DoS attack dressed in “innovation” cosplay.

Here’s the medical analogy: think of a patient whose immune system is under constant attack from a flu virus that never ever tires. That’s your webserver under OpenAI’s fetchers. It’s feverish, it’s wheezing, and it’s about to flatline unless someone cuts off the infection at the source. And who’s paying the medical bills? Not Meta, not OpenAI, but everyone running a halfway functional site online. Yes, congratulations, dear webmaster – you’re now an involuntary organ donor for Silicon Valley’s new religion.

Ethics? Responsibility? Don’t Make Me Laugh

Fastly’s researcher insisted that AI companies should at least honor robots.txt and publish IP addresses so sysadmins can know what’s hitting them. That’s like suggesting a burglar leave a copy of their driver’s license on your kitchen counter. Laudable idea, doomed in practice. Perplexity already ignores robots.txt, because apparently some people think “please respect the rules” is an optional side quest rather than a main mission. This isn’t Clippy politely asking – this is Doom demons crawling through your vents while the developer shrugs and says, “eh, emergent gameplay.”

And let’s not forget the complete hypocrisy here. These AI bots claim they’re in service of “progress” and “democratizing knowledge.” What they’re actually doing is draining cultural commons dry, chewing up community-driven data while contributing zip back. This is less utopia and more Strip Mining Simulator 2025. The AI bubble defenders wax lyrical about “value creation.” I call it necromancy – reviving the corpse of human creativity to shuffle around spewing uncanny valley drivel while human authors quietly hemorrhage relevance.

The AI Bubble: Endgame Boss Fight

Here’s the kicker – the only thing people expect will stop this is the bubble popping. Xe Iaso, the developer behind Anubis (a countermeasure against invasive crawlers), outright said it: only an AI crash will curb this addiction. Games industry players already know this story – it’s like MMOs in the 2000s. Everyone thought WoW’s revenue model was bottomless, then came copycats like Warhammer Online, Tabula Rasa, and we all know how that ended: a digital graveyard littered with half-baked experiments that never reached endgame.

The truth is simple: AI fetchers and crawlers are unsustainable. They’re expensive, they’re abusive, and they’re breaking the very infrastructure they’re built on. Yet, hyped-up VCs are still pouring billions into it, convinced they’re funding digital angels instead of code-guzzling parasites. The same story plays out in every bubble: tulips, dot-com, crypto, NFTs. Spoiler alert for those skipping to the credits – this movie doesn’t end well.

Regulation: Cue Boss Music

Whenever chaos emerges, the cry for regulation follows. Xe Iaso demanded existential fines, reparations, the works. Imagine a government actually telling Meta it has to pay hospital bills for the servers it’s frying – glorious fantasy, like Half-Life 3. But let’s not get carried away. Governments regulating AI scraping is the equivalent of a level one warrior with a broken sword trying to tank a dragon. Their record at containing tech giants is laughable at best. The only way regulators might successfully intervene here is if they accidentally copy-paste the laws meant for the tobacco industry.

Cloudflare’s Tollbooth: Because Paywalls Solve Everything

Fastly’s rival, Cloudflare, is testing “tollbooths” for bots – essentially charging AI companies for slurping content. Do you hear that slapping sound? That’s every site owner being milked by both sides – first bots rob them blind, then vendors say, “We can protect you… for a fee.” It’s a dystopian racket, like Resident Evil’s Merchant turning up after Mr. X wastes you, whispering: “What are ya buyin?” Except here, the item is basic survival from unregulated AI looting, wrapped in a monthly subscription plan.

Conclusion: A Sickness in the System

Let’s not mince words: the entire ecosystem is sick. Crawlers are out of control. Fetchers are practically denial-of-service bots with lipstick. Companies like Meta and OpenAI are offloading the costs onto countless smaller sites while happily swallowing their data. It’s theft with extra steps, dressed up in marketing fluff about “AI innovation.” A doctor’s diagnosis? The patient – the open web – is on life support, and without intervention, the parasites win.

The future? Either regulation comes in swinging like a surprise DLC boss fight, or the AI hype bubble bursts under its own absurd weight. Until then, consider hosting a website like running a survival horror campaign – barricade your doors, set up traps, and pray the monster runs out of venture capital before it eats you alive.

Verdict: Bad. Very bad. The open web isn’t thriving; it’s being drained. Unless dramatic action is taken, don’t be surprised when the lights flicker, and the house of cards finally collapses flat.

And that, ladies and gentlemen, is entirely my opinion.

Source: https://www.theregister.com/2025/08/21/ai_crawler_traffic/