hupden/ projects
blogtools
← blog

The safest way to ship it was to not build the API

#go#security#api-design#self-hosting

Last week I wrote about finding that my admin API had spent its whole life signing tokens with an empty key. Anyone who guessed that could have minted a valid admin session. The reason that was an embarrassment rather than a disaster was a property I had not chosen on purpose: the API had no write routes. A forged token could read dashboard data and do nothing else, because there was nothing to do.

I ended that post by saying I had since declined a feature on those grounds. This is that feature, and it is the half of the story I would rather be judged on. Finding your own bug is luck plus diligence. What you change afterwards is the part that is actually yours.

The feature

fail2ban deletes its own ban history. It keeps a SQLite database of bans and purges old rows from it, so by default anything older than about a day is gone for good. I wanted history that outlived that — "has this address been banned before?" is the question fail2ban cannot answer about itself — so I wrote a snapshotter: copy fail2ban's whole ban table into Postgres every fifteen minutes, upsert on the natural key, and have the history view read Postgres instead.

Fifteen minutes means the newest bans can be up to fifteen minutes behind. So the obvious companion was a button on the admin tools page: snapshot now. You see a ban happen, you want it in the history table, you click.

I built it. POST /admin/fail2ban/snapshot, behind the admin auth, calling the same function the ticker calls. Small handler, a few tests, done. It was about the least controversial code I have written.

Then I deleted it before it shipped. This was six days after the empty-key discovery.

Identical code, different thing

The case for the button is that it adds no new capability. The snapshot already runs every fifteen minutes whether anyone asks or not. The route calls the same function. It is idempotent: the upsert cannot create a duplicate however often it runs. What could a caller do with it that the ticker was not already doing?

That argument is about what the code does, and it is correct about that. What it misses is who can make it happen, and those are different questions.

The ticker is a goroutine started at boot. There is no route to it, and no request, token, header or payload reaches it. The set of people who can cause a snapshot is empty, and that is not because they are checked; nobody has anything to present.

The endpoint puts the same code behind the auth middleware. The set of people who can cause a snapshot is now "anyone the auth lets through". Six days earlier, I had learned exactly how large that set had been for months: the whole internet, for anyone who thought to try an empty key.

So "it adds no capability" is true only if the auth works. The whole reason I cared about zero write routes was that I now had direct evidence it sometimes does not. A design that is only safe while the auth holds is not bounding the blast radius of an auth failure. It is betting on there not being one.

"It's harmless" is a claim about today

The second argument for the button is that even with broken auth, the worst a stranger could do is trigger a refresh. Nothing is deleted, nothing is exposed, and the data comes out the same either way.

Also true, for the handler as written. Three things it doesn't account for:

What the call touched at the time. When I built this, the snapshot pulled fail2ban's database out of the proxy container through the Docker API. So the button was an authenticated HTTP request that ended in a Docker socket call, and that socket is root on the host. The handler couldn't make that call do anything but copy one file. Still, an internet-facing request path now reached the most dangerous object on the machine, gated only by the auth I had just caught failing. (I later removed the socket from that service entirely. That's a separate post.)

Work on demand is a resource an attacker can spend. The ticker does its copy four times an hour. An endpoint does it as fast as someone can send requests. That's not a serious denial of service on its own, but "every request costs a database copy and a transaction" is exactly the kind of property you don't want on the path a forged token can reach.

The first write route makes the second one cheap. This is the one I trust most. The frontend only knows how to make authenticated GETs. A button needs an adminPost helper next to that fetch helper, and once it exists, every future admin action is a three-line change. Nobody decides to give a read-only service a write surface. It builds up one harmless convenience at a time, and the first one is the only one anyone looks at hard.

The clever compromise that was worse

Before deleting the button I tried to keep its benefit without the write route: snapshot on the first request each day. No POST, no button. The history endpoint just checks whether a snapshot has run recently and triggers one if not.

It looks like a middle path. It is worse than either side:

That last point generalizes past security: a system whose job is to record what happened while you weren't looking cannot be driven by you looking.

What I shipped instead

The ticker, and nothing else. The fifteen-minute lag is stated in the UI rather than hidden. The history panel says the newest bans may be a few minutes behind, so a missing fresh ban reads as "not yet" rather than "missing". That sentence is the whole cost of the decision, paid in text.

Everything else went into making the property survive me:

It's written down as an invariant, not remembered as a preference. The service's docs now say that it has zero write routes on purpose, and that adding one is a change to its security posture to be argued on its own. It is not an implementation detail of whichever feature needs it. An accidental property that saves you once is worth converting into a rule, because the accident won't repeat.

The explanation lives where someone would add the route. The type the snapshot function returns carries a comment saying there is deliberately no HTTP route to trigger it, why, and what adding one would mean. Whoever next thinks "this should have a button" will be editing the file that says so, rather than a document they'd have to know to look for.

The absence of a helper is a tripwire. The frontend has no adminPost. If one appears, a write surface has been added, and that is visible in a diff rather than inferable from reading every handler.

One precision, because "zero write routes" isn't literally true and I don't want to claim more than it is: login is a POST. It checks a password and issues a token, and stores nothing. The invariant is about routes that change state the service keeps. Login is the one door, and it is the most heavily defended thing in the service for that reason.

What it cost

Not much, and it's worth saying so plainly, because a decision like this is easy to present as more principled than it was. I gave up on-demand freshness on top of an interval that was already short. I have not once wanted the button since.

The honest version of the lesson is less grand than "never add write routes". Sometimes you need one, and when I do I'll add it. But the day after you learn your auth was bypassable is the worst possible day to widen what it guards, and "this code is harmless" is a statement about the code, not about who can run it. A background job and an endpoint that call the same function are different things. The only difference between them is reachability, and that's the one that mattered when the auth failed.