hupden/ projects
blogtools

Blog

Notes from running a small self-hosted stack — nginx, fail2ban, Docker, Go, and the things that broke.

RSS

A User-Agent is a claim, so check it against the ASN

My logs counted forged crawlers as crawlers because the only evidence was a string the client chose. Checking the announcing network caught a Google Cloud address claiming to be eighteen different bots, and a second test that needs no reference data found three times more. Real numbers, and why most of the answer is still "unknown".

#security#bots#go#self-hosting

I anonymized my visitors and kept the attackers

A log archive made my privacy policy false the day it shipped. Truncating addresses fixed that, except for the copy one column over, the rows that had to keep their address, and the copies nobody was thinking of. What the rule keys on, and why it can't be the user agent.

#privacy#gdpr#postgres#self-hosting

The table format kept the IPs I'd scrubbed

Rewrite a column in an Apache Iceberg table and the old values survive in three places: old snapshots, the metadata tables, and — after you expire the snapshots — the manifests themselves, compressed so grep can't see them. Measured, with the two commands that actually clear it.

#iceberg#privacy#data-engineering#spark

Postgres 18 moved PGDATA, and my warning about it was wrong twice

The Postgres 18 image moved its data directory, so an old-style bind mount at /var/lib/postgresql/data no longer holds your database. I documented that trap during my upgrade. Re-testing it for this post, the current image refuses to start, and the version that did lose data didn't lose it the way I said.

#postgres#docker#self-hosting#upgrades

The docker-socket-proxy everyone recommends wouldn't have saved me

The standard fix for "my service needs the Docker socket" is a filtering proxy. Mine needed one exec call, and allowing that meant allowing container creation, which is host root again. What worked was moving the reads off the request path and mounting nothing at all.

#docker#security#self-hosting#containers

Sorting the page you fetched isn't sorting the data

A capped API plus a client-side sort gives confident, wrong answers: "worst repeat offender" quietly becomes "worst among the newest 200". The fix is ORDER BY before LIMIT, and since ORDER BY can't be a bind parameter, a whitelist map.

#go#postgres#sql#api-design

The safest way to ship it was to not build the API

A "refresh now" button needed my API's first write route, six days after I found out its auth had been decorative for months. I built the route, then deleted it. A background ticker and an endpoint can run identical code and still not be the same thing.

#go#security#api-design#self-hosting

My auth was signing production tokens with an empty key

HMAC is perfectly well defined for a zero-length key. So an unset JWT_SECRET does not crash anything — it signs valid tokens, verifies them, and looks completely healthy. Mine had been doing that in production, and what found it was a startup check I added for a hypothetical.

#go#security#jwt#self-hosting

Behavioral fail2ban jails beat signature lists

A jail that matches known-bad paths is always one week behind whatever scanners are trying now. A jail that counts 404s per IP never needs updating. Here is the filter, the two delivery traps that made it silently do nothing, and the blind spot it took me two months to notice.

#fail2ban#nginx#security#self-hosting

My agent context file was 60KB and loaded on every session

Every fact I had ever written down about this server was being read into context before a single line of work happened — whether the task was a CSS tweak or a database migration. The fix was not deciding what mattered. Everything in there mattered.

#ai-assisted#documentation#self-hosting

The reminder that never fired

I automated a recurring security-audit reminder, and it was broken from the first commit. It ran on schedule for 35 days, found nothing every time, and exited 0 — which is exactly what it looks like when there is genuinely nothing to report.

#automation#shell#observability#ai-assisted

fail2ban's client only shows you the present

I wanted to know which addresses had been banned most often. fail2ban-client can't tell you — it reports current state and a couple of counters that reset on restart. The history is in its SQLite database, and by default that database is throwing it away after a day.

#fail2ban#sqlite#postgres#self-hosting

FROM scratch Go images have no /tmp

os.CreateTemp failed in production with "no such file or directory" — on the temp directory itself. The tests passed because go test runs on a host that has a /tmp. Nothing in the Go toolchain can see this bug.

#go#docker#containers

Four days of nothing, and every health check stayed green

My log collector went silent for four days at a logrotate boundary. The container was up, the process was running, its own logs kept ticking, and it had ingested exactly zero rows. An open file descriptor follows the inode, not the path.

#go#observability#logrotate#self-hosting

fail2ban kept banning my own git pushes

Git's smart-HTTP protocol always sends one unauthenticated request first and expects a 401 back. SWAG's bundled nginx-unauthorized jail counts any 401 site-wide — so pushing to my own private repos banned my own IP, twice in two days.

#fail2ban#nginx#git#self-hosting