I have a small admin dashboard for the server this site runs on, and I wanted to add one panel to it: the addresses fail2ban has banned most often. Not who is banned right now — who keeps coming back.
It seemed like a five-minute job. fail2ban ships a perfectly good CLI, the data obviously exists, and I already had somewhere to put it.
It was not a five-minute job, and the reason is more interesting than the panel.
What the client actually tells you
Here is what fail2ban-client status <jail> gives you:
Status for the jail: nginx-botsearch
|- Filter
| |- Currently failed: 0
| |- Total failed: 286
| `- File list: /config/log/nginx/access.log
`- Actions
|- Currently banned: 3
|- Total banned: 37
`- Banned IP list: 198.51.100.14 203.0.113.7 203.0.113.90
(Those addresses are from the RFC 5737 documentation ranges — the example.com
of IP space, reserved so they can never belong to anyone. No real client address
appears anywhere on this site.)
Read that as a data source rather than as a status page and it is thinner than it looks.
Banned IP list is the set of addresses banned at this instant. An address
whose ban expired an hour ago is simply not there. Currently banned is the
size of that list. And the two Total counters are process-local counters —
they count since fail2ban started, so a restart, an image update, or a config
reload sets them back to zero.
Most importantly: there is not a single timestamp anywhere in that output. Not when an address was banned, not for how long, not how many times before. Every question I actually wanted to ask was a question about time, and the client's entire vocabulary is the present tense.
The history exists, it just isn't exposed
fail2ban keeps state in SQLite, and the interesting table is bans:
SELECT jail, ip, timeofban, bantime, bancount FROM bans;
That is exactly the panel I wanted. timeofban is a Unix epoch, bantime is
how long that ban was meant to last, and bancount is how many times fail2ban
has banned that address before — which is the field that drives escalating ban
times, if you have those switched on.
So the data was there. The problem became how to read it from a different container without doing something stupid.
The part I want to argue about: how to reach the file
My first instinct was the obvious one, and it was wrong:
docker exec swag sqlite3 /config/fail2ban/fail2ban.sqlite3 "SELECT ..."
That works. It is also a service on the public internet acquiring the ability
to run commands inside the one container that terminates TLS and proxies every
other service on the box. To get docker exec you need the Docker socket, and
the Docker socket is not "an API for container management", it is root on the
host with extra steps — anything that can create a container can create a
privileged one with the host's filesystem mounted inside it.
For a dashboard panel. Trading unrestricted host root for a leaderboard is not a trade.
So the second version copied the database out over Docker's archive API — the
same mechanism as docker cp, read-only, no exec, no shell in the proxy
container. Better. Still needed the socket.
The third version is the one I should have written first, and I didn't see it for weeks because I was thinking about containers instead of files. fail2ban's database was already on a host bind mount. It had to be — that is what makes ban state survive a container restart in the first place. The "inside a container" framing was doing no work at all. The file was sitting on the host filesystem the entire time.
volumes:
- ~/swag-config/fail2ban:/f2b/swag:ro
- /var/lib/fail2ban:/f2b/host:ro
Two read-only bind mounts, no Docker access of any kind. And because this was the last thing on that service still reaching for the socket, removing it let me take the socket mount off the service entirely — which is a much bigger security win than the panel was ever worth.
The general version, which I now believe fairly strongly: when something is awkward to reach inside a container, check whether it is already on the host before you reach for the Docker API. Persistent state usually is, because persistence and containers are in tension and bind mounts are how that gets resolved. The container boundary is often a framing you inherited rather than a wall you have to climb.
Don't read the live database
One thing that is not optional: copy the file before you read it.
func copyFail2banDBFromMount(dbPath string) (string, func(), error) {
tmpDir, err := os.MkdirTemp("", "fail2ban-*")
if err != nil {
return "", nil, err
}
cleanup := func() { os.RemoveAll(tmpDir) }
dst := filepath.Join(tmpDir, filepath.Base(dbPath))
// The main database, plus any journal/WAL sidecars sitting beside it.
sidecars, _ := filepath.Glob(dbPath + "-*")
for _, src := range append([]string{dbPath}, sidecars...) {
if err := copyFile(src, filepath.Join(tmpDir, filepath.Base(src))); err != nil {
cleanup()
return "", nil, fmt.Errorf("copying %s: %w", src, err)
}
}
return dst, cleanup, nil
}
Two separate reasons, and I only knew about one of them going in.
The mount is read-only, and SQLite may need to write even to read. If the writing process left a hot journal behind, a reader has to replay it to reach a consistent view, and it cannot do that on a read-only filesystem. You get an error from what you thought was a pure read.
And fail2ban is actively writing to that file while you read it. Copying keeps any torn state inside one tick: the read fails, the tick logs it, and the next tick re-reads the whole table from scratch. The snapshot carries no state between runs, which is what makes a bad copy a non-event rather than a corruption.
The dbPath + "-*" glob is the part that is easy to leave out. -journal,
-wal and -shm sit beside the database, and a copy of the main file without
them can look corrupt if a transaction was in flight when you took it.
There were two fail2bans
Wiring this up reminded me of something I had set up years earlier and then completely forgotten about: this server runs two entirely separate fail2ban installations, and I put both of them there.
SWAG's, inside the proxy container, watching nginx logs. And a package install
on the host itself, watching /var/log/auth.log for SSH — one of the first
things I configured when I got the machine, and then never thought about again,
precisely because it had been quietly doing its job ever since. Neither knows
the other exists — separate configs, separate databases, separate iptables
chains. fail2ban-client reaches whichever one you happen to be standing next
to.
They are complementary rather than redundant: the containerised one structurally cannot see SSH, because it only has nginx's logs mounted. But it does mean any table collecting "my ban history" is collecting from two sources, and jail names are not globally unique. So the instance is part of the key, not an annotation:
PRIMARY KEY (instance, jail, ip, banned_at)
Without that column, two sources with a same-named jail would silently merge into one row instead of erroring. I would rather a schema make that impossible than rely on remembering it.
That key also makes the whole snapshot idempotent — re-running it updates rather than duplicates, which is what lets it run on a timer without any coordination.
The ending I wasn't expecting
I ran the first snapshot, and it found 34 bans.
Thirty-four felt low for a server that gets scanned around the clock, so I looked at the oldest row. It was 23 hours and 48 minutes old.
fail2ban has a setting called dbpurgeage, and its default is 1d. It was not
storing my ban history. It was storing a rolling 24-hour window and deleting
everything behind it — and it had been doing that since the day the server was
built, with no output, because purging old rows is not an error, it is the
feature working as configured.
So the panel I had been trying to build was never going to work, and not for any of the reasons I had spent a week on. The client couldn't show me history because there was no history to show. Everything before yesterday had already been deleted, permanently, and no amount of clever plumbing was going to get it back.
Raising dbpurgeage fixes it going forward. But the durable fix is that the
snapshot writes into Postgres on a timer, where retention is my decision rather
than a default I never read. fail2ban's database is now a source, not the
record.
There is a sequel to this: dbpurgeage turned out to be quietly breaking
escalating ban times too, because the escalation logic counts an address's prior
bans by asking that same database — and it had been forgetting them after a day.
That one deserves its own post.
What I'd take away from it
The thing I got wrong was not a plumbing decision, it was an assumption I never examined: a tool having a database does not mean the tool is keeping your data. fail2ban's SQLite file is working state — what it needs to enforce bans correctly right now. That it happens to look like a history table is a coincidence of implementation, and the default retention is set for the job it is actually doing, which is not archival.
fail2ban-client status is honest about this, in retrospect. Its whole output
is the present tense because the present tense is what fail2ban is for. I was
the one reading a status command as a query interface.
And the smaller lesson, which cost me more time: I spent days solving "how do I get data out of a container" when the real question was "where does this file live." The answer was: on the host, in a directory I had configured myself, long before any of this came up.