My traffic dashboard had a tidy line for crawlers: Googlebot, Bingbot, GPTBot and
the rest, parsed out of the User-Agent header. It read like a fact. It was a
list of what clients had said about themselves.
When I finally checked those claims against where the requests actually came from, one Google Cloud address had presented eighteen different crawler identities, among them Amazonbot, Applebot, GPTBot, ClaudeBot, PerplexityBot, Slackbot and LinkedInBot, while requesting this:
/@fs/root/.aws/credentials?raw??
Every one of those requests had been counted as legitimate crawler traffic, because the only thing consulted was the string the attacker chose.
The test: who announces this address?
A real Googlebot comes from Google's network. Every public address is announced by an autonomous system, and a free database like DB-IP's or MaxMind's will map an address to its ASN. So a claim can be checked with one lookup:
- The claimed operator runs this network:
verified. - It doesn't:
impersonated. - I can't judge it:
unknown.
On the first 30 days this ran over, 587 of 7,867 claimed-bot requests came back forged.
Match the number, never the name
The obvious shortcut is to look for "Google" in the network's organization name. That shortcut would have cleared the impostors:
| | ASN | organization | |---|---|---| | Googlebot's network | AS15169 | Google LLC | | Google Cloud, rentable by anyone | AS396982 | Google LLC |
Same organization string, and two completely different meanings. On the first run
this caught a Google Cloud address sending a textbook Googlebot User-Agent while
requesting /.env.bak. An org-name match would have labeled it verified, which is
worse than having no check, because now the forgery has a stamp on it.
The strength of verified also varies by operator. AS15169 can't be rented, so a
verified Googlebot means a lot. Any operator whose bots run on a public cloud's
address space gets a much weaker version of that word, since a customer on the same
cloud passes the same test.
Most of the answer is "I don't know", on purpose
The check only works for operators with centrally run, published infrastructure, so
it's opt-in per bot. Everything else resolves to unknown, and the reasons are
specific:
- Baiduspider really does crawl from China Unicom, not a Baidu network. A naive rule would call the genuine crawler a fake.
- MJ12bot is volunteer-run by design, so no network is "correct".
- AhrefsBot showed 115 rows from OVH across 114 addresses and 13 from its own AS. That's plausible either way, so I don't get to pick.
Calling a genuine crawler an impersonator is the worse error, and it decides several
small choices. A missing ASN must read as unknown, never as a mismatch, so a gap
in my geo data can't turn into an accusation. And unknown is stored as a real
value rather than left NULL, because NULL is the work queue, and merging the two
would re-examine every unjudgeable row forever.
The check also runs on a background ticker over stored rows, not while ingesting a line. Its input comes from the geo enricher, which is itself a background job, and waiting on it during ingest would put a lookup table's health in front of the one thing the collector must never stop doing.
The second test needs no reference at all
The ASN test has an obvious limit. Those eighteen identities included plenty of bots
it had no opinion on. Of one address's eighteen, it caught exactly one, Amazonbot,
because Amazon happened to be on my list. The other seventeen stayed unknown.
But that address had already told me everything I needed. No operator crawls as both Applebot and LinkedInBot from the same machine. An address presenting itself as several different crawlers has contradicted itself, and at most one of the claims can be true. That needs no list of networks at all.
Over the 30 days I measured when I added it, 33 addresses qualified, and 1,228 rows
moved from unknown to impersonated, against 404 the ASN test had found on its
own. The cheaper test, with no reference data, roughly tripled detection. And none of
the qualifying addresses sat inside the networks of the operators I can verify. All
of them were on rentable infrastructure.
The threshold is three distinct named identities, and it's deliberately not two. Two
has a thin but real innocent explanation, such as a shared cloud NAT carrying two
services. Three doesn't. That has a known cost: one Google Cloud address claimed
CCBot and Diffbot, is almost certainly forged, and stays unknown because it stopped
at two. I'm fine with that miss, since the opposite mistake is libeling a real
crawler.
Two more rules keep this test honest. It only ever upgrades unknown. It never
overwrites verified, which is stronger evidence, and never touches NULL, which
belongs to the first test's queue. Production data can't exercise that rule, because
no offender has a verified row, so I had a comment claiming a test covered it. No
such test existed. A later audit found that, and the test now plants those rows
against a real Postgres, and it goes red when the rule is removed.
Where it stands now
Over a more recent month, October 8 to November 6, the archive has 5,349 claimed-bot requests:
| verdict | requests | share | |---|---|---| | verified | 1,214 | 23% | | impersonated | 2,210 | 41% | | unknown | 1,925 | 36% |
(Shares are rounded, and each row's verdict is as of the day it was archived, so a
few unknowns may have been upgraded since.)
That's the number I'd have wanted from the start. Under a quarter of what claims to be a crawler can be shown to be one, and more of it is provably lying than provably honest. A traffic report built on the raw header would have called all 5,349 of them crawlers.
This is one small site, so please don't read that as a measure of the internet. It's a sample of what reaches one server on wildcard DNS, and that's likely to skew hostile. But the method works anywhere, and you probably already have the data: your access log, plus a free ASN database.
If you have a dashboard line derived from a User-Agent, rename it in your head to
"claimed". Then go find out how much of it you can prove.