My admin API used to have /var/run/docker.sock mounted into it. I knew what
that meant, and so does anyone who has read a Docker security page: access to
the Docker socket is root on the host. Anyone who can talk to it can start a
privileged container with the host's filesystem mounted and do whatever they
like. The service was internet-facing and parsed hostile input, so any future
remote-code bug in it wouldn't be a bug in one service. It would be the whole
machine, in one step.
It sat as an accepted risk for a while, because what the service actually did with the socket was modest and read-only:
GET /containers/json, to list containers for the dashboardGET /containers/{id}/stats, for CPU and memoryGET /containers/{id}/archive, to copy fail2ban's SQLite database out of the reverse-proxy container- an exec of
fail2ban-client statusinside that container, to read the current jail state
When I finally fixed it, I reached for the standard answer, and it turned out not to fit.
The standard answer
If you search "container needs docker socket securely", you'll be pointed at a
socket proxy, most often Tecnativa's docker-socket-proxy. It's an HAProxy
sidecar that holds the real socket and exposes a filtered TCP endpoint. Your
service talks to the proxy instead of the socket, and you enable only the API
sections you need via environment variables:
environment:
CONTAINERS: 1
POST: 0
For a lot of services this is genuinely good advice. A dashboard or a reverse
proxy that only needs to list containers gets CONTAINERS=1, POST=0, and
is limited to reads. I'd still recommend it for that.
My plan was exactly that, plus whatever the fail2ban exec needed. That's where it came apart.
What one exec costs
An exec over the Docker API is two POSTs:
POST /containers/{id}/exec → create the exec instance
POST /exec/{id}/start → run it
So the proxy needs CONTAINERS=1, EXEC=1 and POST=1. The question is what
else those switches open, and the answer is in the proxy's own HAProxy config:
http-request deny unless METH_GET || { env(POST) -m bool }
...
http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers } { env(CONTAINERS) -m bool }
http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/exec } { env(EXEC) -m bool }
Two things fall out of that:
CONTAINERSis a prefix match on/containers. Enabling it enables every endpoint under that path: list, inspect, stats, and alsocreate,start,kill,delete.POSTis one global switch. It isn't "POST for the exec section". It lifts the GET-only rule for every section you've enabled.
Put those together, and allowing my one fail2ban-client status call allows
POST /containers/create followed by POST /containers/{id}/start. Create a
container with Privileged: true and / bind-mounted, start it, and you're
root on the host, which is precisely the escalation the proxy was there to
prevent. The filtering is by API section × HTTP method, not by path, and the
section I needed a single write in is the one containing the most dangerous
write there is.
The proxy would have added a container to maintain, and narrowed the risk far less than its presence would suggest to anyone reading the compose file later. That second part bothers me more than the first. A mitigation that looks thorough and isn't makes the next person stop looking.
"Use a proxy with per-path rules, then"
There are socket proxies that filter by regex per method and path, and with one
of them you could allow POST /containers/<proxy-container>/exec and nothing
else. That closes the create hole. It still doesn't get you what you want,
because of where the command lives.
The path of an exec request names the container. The command is in the JSON
body: Cmd, and alongside it User and Privileged. A path filter never looks
at the body. So "allow exec in the proxy container" isn't "allow
fail2ban-client status". It's "allow any command, as any user, in the
container that terminates TLS for every site on the box and holds the private
key for its certificate". The narrowest rule a path filter can express is still
arbitrary code execution in the second most sensitive place on the machine.
At that point I stopped trying to filter the socket. The problem wasn't that the filter was too coarse. It was that a request path could reach the socket at all.
What worked: get the reads off the request path
Every one of those four calls was a read, and none of them needed to happen at request time. The dashboard doesn't need container stats computed at the moment you load the page. It needs them from the last minute or so.
So the reads moved to the host, as two scheduled jobs:
- A timer every minute curls the socket for the container list and per-container
stats, and writes the verbatim Engine API responses to a directory. I
mean verbatim: I used the API, not
docker ps --format json, because the CLI output has a different shape (JSON Lines, truncated IDs, pre-formatted memory strings), and matching the API byte for byte meant the service's decode structs and CPU arithmetic didn't change at all. - A timer every fifteen minutes runs
fail2ban-client statusfor each jail and writes the output to a file.
The service mounts that directory read-only and reads files. The
archive call disappeared entirely: fail2ban's database was already sitting on
a host bind mount the whole time, so the service now mounts that read-only
too and copies from it. I had been pulling a file out through the Docker API
that was available as a plain file.
Then the socket mount was removed from the service with nothing put in its place. No proxy, no sidecar. The exec helper, the stream demuxer for Docker's 8-byte frame headers, the tar extraction for the archive endpoint: all deleted. The service got smaller.
The new failure mode
This design has a cost, and it's the one I'd have ignored if I hadn't been bitten by it before: a dead timer doesn't produce an error. It produces data that looks current. If the export job stops, the files stay, and a dashboard reading them keeps showing a healthy-looking container list indefinitely. I've had a log collector ingest nothing for four days while every health check stayed green, and I wasn't going to build that bug on purpose.
So:
- The exporters write to a temp file in the same directory and
mvit into place.renameis atomic, so a reader never sees a half-written file, and a failed run leaves the previous file alone instead of replacing it with garbage. - The file's mtime is the sample time. No timestamp is written into the payload, so the two can't disagree.
- The service refuses files older than a max age and fails the request, rather than serving them. The sample time also goes out on the wire, so the UI can say how old the numbers are.
A broken exporter now shows up as a dashboard panel that errors, not one that lies. That trade is explicit: widening the max age trades directly against it.
What this was, and what it wasn't
This is the part most write-ups about the Docker socket skip, and I think it matters most.
The timer still talks to the socket. It runs as a normal user in the docker
group, and that user has sudo on the box anyway. So nothing here reduced how
much privilege exists on the machine, or how many things can reach the socket.
If you count processes with socket access, the number didn't go down.
It was never a privilege reduction. It was a reachability reduction.
The service takes requests from the internet, so a bug in its parsing or auth is a foothold for an unauthenticated stranger, and while it held the socket, that foothold was host root. The timer takes input from nobody. It answers no requests and parses nothing anyone outside can send it. Reaching it requires a shell on the box, and someone with a shell as that user has already won, socket or no socket.
What mattered was never how many things can touch the socket. It was which side of the trust boundary they sit on. A proxy keeps the socket reachable from the request path and tries to narrow what the request can say to it. Moving the work off the request path means no request can reach it at all, and that turned out to be both simpler and stronger.
If your service needs the socket for reads that can tolerate being a minute old, you probably don't need a proxy. You need a cron job and a read-only mount. And if it needs the socket for a write, a filter in front of it is protecting less than it appears to, so it's worth checking exactly what that switch turns on.