hupden/ projects
blogtools
← blog

A User-Agent is a claim, so check it against the ASN

#security#bots#go#self-hosting

My traffic dashboard had a tidy line for crawlers: Googlebot, Bingbot, GPTBot and the rest, parsed out of the User-Agent header. It read like a fact. It was a list of what clients had said about themselves.

When I finally checked those claims against where the requests actually came from, one Google Cloud address had presented eighteen different crawler identities, among them Amazonbot, Applebot, GPTBot, ClaudeBot, PerplexityBot, Slackbot and LinkedInBot, while requesting this:

/@fs/root/.aws/credentials?raw??

Every one of those requests had been counted as legitimate crawler traffic, because the only thing consulted was the string the attacker chose.

The test: who announces this address?

A real Googlebot comes from Google's network. Every public address is announced by an autonomous system, and a free database like DB-IP's or MaxMind's will map an address to its ASN. So a claim can be checked with one lookup:

On the first 30 days this ran over, 587 of 7,867 claimed-bot requests came back forged.

Match the number, never the name

The obvious shortcut is to look for "Google" in the network's organization name. That shortcut would have cleared the impostors:

| | ASN | organization | |---|---|---| | Googlebot's network | AS15169 | Google LLC | | Google Cloud, rentable by anyone | AS396982 | Google LLC |

Same organization string, and two completely different meanings. On the first run this caught a Google Cloud address sending a textbook Googlebot User-Agent while requesting /.env.bak. An org-name match would have labeled it verified, which is worse than having no check, because now the forgery has a stamp on it.

The strength of verified also varies by operator. AS15169 can't be rented, so a verified Googlebot means a lot. Any operator whose bots run on a public cloud's address space gets a much weaker version of that word, since a customer on the same cloud passes the same test.

Most of the answer is "I don't know", on purpose

The check only works for operators with centrally run, published infrastructure, so it's opt-in per bot. Everything else resolves to unknown, and the reasons are specific:

Calling a genuine crawler an impersonator is the worse error, and it decides several small choices. A missing ASN must read as unknown, never as a mismatch, so a gap in my geo data can't turn into an accusation. And unknown is stored as a real value rather than left NULL, because NULL is the work queue, and merging the two would re-examine every unjudgeable row forever.

The check also runs on a background ticker over stored rows, not while ingesting a line. Its input comes from the geo enricher, which is itself a background job, and waiting on it during ingest would put a lookup table's health in front of the one thing the collector must never stop doing.

The second test needs no reference at all

The ASN test has an obvious limit. Those eighteen identities included plenty of bots it had no opinion on. Of one address's eighteen, it caught exactly one, Amazonbot, because Amazon happened to be on my list. The other seventeen stayed unknown.

But that address had already told me everything I needed. No operator crawls as both Applebot and LinkedInBot from the same machine. An address presenting itself as several different crawlers has contradicted itself, and at most one of the claims can be true. That needs no list of networks at all.

Over the 30 days I measured when I added it, 33 addresses qualified, and 1,228 rows moved from unknown to impersonated, against 404 the ASN test had found on its own. The cheaper test, with no reference data, roughly tripled detection. And none of the qualifying addresses sat inside the networks of the operators I can verify. All of them were on rentable infrastructure.

The threshold is three distinct named identities, and it's deliberately not two. Two has a thin but real innocent explanation, such as a shared cloud NAT carrying two services. Three doesn't. That has a known cost: one Google Cloud address claimed CCBot and Diffbot, is almost certainly forged, and stays unknown because it stopped at two. I'm fine with that miss, since the opposite mistake is libeling a real crawler.

Two more rules keep this test honest. It only ever upgrades unknown. It never overwrites verified, which is stronger evidence, and never touches NULL, which belongs to the first test's queue. Production data can't exercise that rule, because no offender has a verified row, so I had a comment claiming a test covered it. No such test existed. A later audit found that, and the test now plants those rows against a real Postgres, and it goes red when the rule is removed.

Where it stands now

Over a more recent month, October 8 to November 6, the archive has 5,349 claimed-bot requests:

| verdict | requests | share | |---|---|---| | verified | 1,214 | 23% | | impersonated | 2,210 | 41% | | unknown | 1,925 | 36% |

(Shares are rounded, and each row's verdict is as of the day it was archived, so a few unknowns may have been upgraded since.)

That's the number I'd have wanted from the start. Under a quarter of what claims to be a crawler can be shown to be one, and more of it is provably lying than provably honest. A traffic report built on the raw header would have called all 5,349 of them crawlers.

This is one small site, so please don't read that as a measure of the internet. It's a sample of what reaches one server on wildcard DNS, and that's likely to skew hostile. But the method works anywhere, and you probably already have the data: your access log, plus a free ASN database.

If you have a dashboard line derived from a User-Agent, rename it in your head to "claimed". Then go find out how much of it you can prove.