hupden/ projects
blogtools
← blog

Postgres 18 moved PGDATA, and my warning about it was wrong twice

#postgres#docker#self-hosting#upgrades

In July I took this site's database from Postgres 16 to 18. It's a dump and restore, not an in-place upgrade: the on-disk format changes between majors, and the official image doesn't ship both versions' binaries, which pg_upgrade needs. I rehearsed it locally against a real production dump, gated it on every table's row count matching exactly, and then ran it on the server. It went fine.

Along the way I added one line to the compose file and a comment explaining it. This post started as a write-up of that comment. While writing it, I tested the comment, and it turned out to be wrong in two separate ways. The line itself is still right.

What changed in the 18 image

Up to 16, the official image stored its cluster at /var/lib/postgresql/data, and everyone's compose file, mine included, bind-mounted a host directory there:

volumes:
  - ./data:/var/lib/postgresql/data

From 18, the image changed two things. The default PGDATA is now /var/lib/postgresql/18/docker, a major-version-specific directory, which is the layout pg_ctlcluster expects and what makes a future pg_upgrade --link possible. And the image declares its VOLUME one level up, at /var/lib/postgresql.

So an unchanged compose file mounts your host directory at a path the new image no longer uses. My fix was to pin the old path back explicitly:

environment:
  PGDATA: /var/lib/postgresql/data

And the comment I wrote above it said, in short: without this, the bind mount points at a directory Postgres never writes to, the cluster silently lands in the container's writable layer, the restore looks completely successful, and the database disappears the next time the container is recreated.

I caught the change by reading the new image's environment before running anything, which is the right way round. But it means I never ran the failure. The consequence in that comment was reasoned, not observed, and it sat in my repo for two months reading exactly like something I had seen happen.

What actually happens today

I ran the unmodified compose file, with the old bind mount and no PGDATA, against the current postgres:18-alpine. It doesn't silently do anything. It refuses to start:

Error: in 18+, these Docker images are configured to store database data in a
       format which is compatible with "pg_ctlcluster" (specifically, using
       major-version-specific directory names).  This better reflects how
       PostgreSQL itself works, and how upgrades are to be performed.

       See also https://github.com/docker-library/postgres/pull/1259

       Counter to that, there appears to be PostgreSQL data in:
         /var/lib/postgresql/data (unused mount/volume)

That happened with an empty host directory, which is exactly the situation during a dump-and-restore upgrade, where the old cluster has been moved aside. The entrypoint doesn't just look for an old cluster. If PGDATA is the new default and anything is mounted at the old path, it stops.

If you found this post by searching that error, you have two fixes:

  1. Keep your layout: set PGDATA: /var/lib/postgresql/data, as above. Your existing mount works unchanged.
  2. Adopt the new one: mount your directory at /var/lib/postgresql instead, and let the cluster live in 18/docker beneath it. This is what upstream recommends, because the next major's directory can then sit alongside it on the same filesystem, which pg_upgrade --link needs.

Either way, if you're coming from 16 or earlier, you still need a real upgrade, because a 16 cluster won't start on an 18 binary. The error isn't asking you to move files. It's telling you your mount and the image disagree.

I kept option 1. At this database's size, the next major will be a dump and restore again, so the --link benefit is worth little to me, and not changing the host layout was worth more.

So was the trap ever real?

Yes, and for exactly the image I use. The entrypoint's upstream history tells the story:

So for roughly six months, postgres:18-alpine with an old-style empty bind mount did exactly the silent thing. I didn't want to take that from reading a diff, so I reproduced it: the current image, with that one check reverted to its BusyBox-only form.

rows: 1000   data_directory: /var/lib/postgresql/18/docker
host bind dir entries: 0

The database works. Queries succeed, row counts match, and nothing is written to the directory you think holds your data.

Where my comment was wrong

First: it isn't the writable layer. The image declares VOLUME /var/lib/postgresql, so Docker creates an anonymous volume there, and the cluster at 18/docker lands inside it. docker inspect on the reproduction shows it plainly:

bind   -> /var/lib/postgresql/data
volume -> /var/lib/postgresql

That sounds like a pedantic distinction. It isn't, because of the second one.

Second: it doesn't disappear "the next time the container is recreated". It depends on how it's recreated:

--- up --force-recreate
rows: 1000
--- config change, then up
 Container pgdata-test-db-1 Recreated
rows: 1000
--- down; up
ERROR:  relation "t" does not exist

Compose carries anonymous volumes across a recreate. docker compose up --force-recreate, and a plain up after editing the service's config, both replaced the container and kept the data. docker compose down removes the container, and the next up creates a new one with a fresh anonymous volume. The old one isn't deleted. It's orphaned, sitting in docker volume ls with a random name, until someone runs docker volume prune as routine cleanup.

That makes the real trap worse than the one I described. The obvious post-upgrade sanity test is "restart it and check the data is still there", and this configuration passes it: a restart and even a forced recreate both keep the data. It only fails on the down/up you do weeks later during unrelated maintenance, when you're no longer thinking about the upgrade. And the evidence is recoverable for exactly as long as nobody tidies up.

The check that would have caught it anyway

The upgrade script has one check I'm glad I wrote before understanding any of this. After the restore, and after the row counts match, it looks for PG_VERSION on the host, in the bind-mounted directory, not inside the container.

Row counts prove the data exists somewhere. They're measured through the database, and the database is happy wherever its files are. A file check on the host proves the data is where you are going to back it up from and where the next container will look. That's the check that fails under the silent variant, whatever the image does or doesn't detect. It would have caught this regardless of my wrong explanation, which is the point of checking outcomes rather than mechanisms.

The lesson I'd actually take

My comment had the right conclusion, keep this line, and a wrong reason. That combination is easy to leave alone, because nothing breaks: the line stays, the database is fine, and nobody ever tests the explanation.

But a wrong reason gets reused. Someone reading "writable layer" goes looking in docker diff and finds nothing. Someone reading "disappears on recreate" tests with --force-recreate, sees the data survive, and concludes the warning is outdated and the line can go. Wrong reasons are how correct lines get deleted.

So: when you write down why a line is load-bearing, test the why, not just the line. And if you only reasoned it, say so in the comment, because a reasoned consequence and an observed one read identically two months later. Testing mine took ten minutes and a throwaway compose file. The comment now says what actually happens, and which half of it I watched happen.