hupden/ projects
blogtools
← blog

The docker-socket-proxy everyone recommends wouldn't have saved me

#docker#security#self-hosting#containers

My admin API used to have /var/run/docker.sock mounted into it. I knew what that meant, and so does anyone who has read a Docker security page: access to the Docker socket is root on the host. Anyone who can talk to it can start a privileged container with the host's filesystem mounted and do whatever they like. The service was internet-facing and parsed hostile input, so any future remote-code bug in it wouldn't be a bug in one service. It would be the whole machine, in one step.

It sat as an accepted risk for a while, because what the service actually did with the socket was modest and read-only:

When I finally fixed it, I reached for the standard answer, and it turned out not to fit.

The standard answer

If you search "container needs docker socket securely", you'll be pointed at a socket proxy, most often Tecnativa's docker-socket-proxy. It's an HAProxy sidecar that holds the real socket and exposes a filtered TCP endpoint. Your service talks to the proxy instead of the socket, and you enable only the API sections you need via environment variables:

environment:
  CONTAINERS: 1
  POST: 0

For a lot of services this is genuinely good advice. A dashboard or a reverse proxy that only needs to list containers gets CONTAINERS=1, POST=0, and is limited to reads. I'd still recommend it for that.

My plan was exactly that, plus whatever the fail2ban exec needed. That's where it came apart.

What one exec costs

An exec over the Docker API is two POSTs:

POST /containers/{id}/exec     → create the exec instance
POST /exec/{id}/start          → run it

So the proxy needs CONTAINERS=1, EXEC=1 and POST=1. The question is what else those switches open, and the answer is in the proxy's own HAProxy config:

http-request deny unless METH_GET || { env(POST) -m bool }
...
http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers } { env(CONTAINERS) -m bool }
http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/exec } { env(EXEC) -m bool }

Two things fall out of that:

  1. CONTAINERS is a prefix match on /containers. Enabling it enables every endpoint under that path: list, inspect, stats, and also create, start, kill, delete.
  2. POST is one global switch. It isn't "POST for the exec section". It lifts the GET-only rule for every section you've enabled.

Put those together, and allowing my one fail2ban-client status call allows POST /containers/create followed by POST /containers/{id}/start. Create a container with Privileged: true and / bind-mounted, start it, and you're root on the host, which is precisely the escalation the proxy was there to prevent. The filtering is by API section × HTTP method, not by path, and the section I needed a single write in is the one containing the most dangerous write there is.

The proxy would have added a container to maintain, and narrowed the risk far less than its presence would suggest to anyone reading the compose file later. That second part bothers me more than the first. A mitigation that looks thorough and isn't makes the next person stop looking.

"Use a proxy with per-path rules, then"

There are socket proxies that filter by regex per method and path, and with one of them you could allow POST /containers/<proxy-container>/exec and nothing else. That closes the create hole. It still doesn't get you what you want, because of where the command lives.

The path of an exec request names the container. The command is in the JSON body: Cmd, and alongside it User and Privileged. A path filter never looks at the body. So "allow exec in the proxy container" isn't "allow fail2ban-client status". It's "allow any command, as any user, in the container that terminates TLS for every site on the box and holds the private key for its certificate". The narrowest rule a path filter can express is still arbitrary code execution in the second most sensitive place on the machine.

At that point I stopped trying to filter the socket. The problem wasn't that the filter was too coarse. It was that a request path could reach the socket at all.

What worked: get the reads off the request path

Every one of those four calls was a read, and none of them needed to happen at request time. The dashboard doesn't need container stats computed at the moment you load the page. It needs them from the last minute or so.

So the reads moved to the host, as two scheduled jobs:

The service mounts that directory read-only and reads files. The archive call disappeared entirely: fail2ban's database was already sitting on a host bind mount the whole time, so the service now mounts that read-only too and copies from it. I had been pulling a file out through the Docker API that was available as a plain file.

Then the socket mount was removed from the service with nothing put in its place. No proxy, no sidecar. The exec helper, the stream demuxer for Docker's 8-byte frame headers, the tar extraction for the archive endpoint: all deleted. The service got smaller.

The new failure mode

This design has a cost, and it's the one I'd have ignored if I hadn't been bitten by it before: a dead timer doesn't produce an error. It produces data that looks current. If the export job stops, the files stay, and a dashboard reading them keeps showing a healthy-looking container list indefinitely. I've had a log collector ingest nothing for four days while every health check stayed green, and I wasn't going to build that bug on purpose.

So:

A broken exporter now shows up as a dashboard panel that errors, not one that lies. That trade is explicit: widening the max age trades directly against it.

What this was, and what it wasn't

This is the part most write-ups about the Docker socket skip, and I think it matters most.

The timer still talks to the socket. It runs as a normal user in the docker group, and that user has sudo on the box anyway. So nothing here reduced how much privilege exists on the machine, or how many things can reach the socket. If you count processes with socket access, the number didn't go down.

It was never a privilege reduction. It was a reachability reduction.

The service takes requests from the internet, so a bug in its parsing or auth is a foothold for an unauthenticated stranger, and while it held the socket, that foothold was host root. The timer takes input from nobody. It answers no requests and parses nothing anyone outside can send it. Reaching it requires a shell on the box, and someone with a shell as that user has already won, socket or no socket.

What mattered was never how many things can touch the socket. It was which side of the trust boundary they sit on. A proxy keeps the socket reachable from the request path and tries to narrow what the request can say to it. Moving the work off the request path means no request can reach it at all, and that turned out to be both simpler and stronger.

If your service needs the socket for reads that can tolerate being a minute old, you probably don't need a proxy. You need a cron job and a read-only mount. And if it needs the socket for a write, a filter in front of it is protecting less than it appears to, so it's worth checking exactly what that switch turns on.