This site's privacy policy said full server logs are deleted after 30 days, and it leaned on that for its answer to GDPR erasure requests: you don't have to ask, they age out.
Then I wanted long-term traffic history, so I wrote a script that exports each completed day to a gzipped CSV before the purge reaches it, and keeps those files indefinitely. It worked well. It also made a published commitment false the moment it ran, because the export carried every full client address. A security sweep caught that the same day, and nothing about the script had failed. It did exactly what I asked, and what I asked contradicted a page I had written.
The fix sounds like one line: truncate the addresses on the way out. Getting it right took more than that.
Reduce only the copy that outlives the promise
IPv4 goes to its /24 and IPv6 to its first three groups, so 203.0.113.47 becomes
203.0.113.0. That happens on export, not in the database. The live tables keep
full addresses, because the dashboard and ban correlation need them and they age
out at 30 days anyway. Only the copy that survives the purge has to be reduced.
The address was also one column over
The first version truncated the address columns and looked finished. It wasn't.
My parser stores the entire nginx error line in message, then also extracts the
client, server and request into their own columns. So every error row holds the
client address twice:
client_addr 192.0.2.44
message *1 open() "/x" failed (2: No such file or directory),
client: 192.0.2.44, server: _, request: "GET /x HTTP/1.1"
Truncating client_addr alone leaves the full address in plain text right next to
the column you just cleaned, and any check that looks at address columns passes.
The message column now gets its own regex pass.
Who keeps a full address, and why
Not everything should be truncated. A security archive that can't recognize a returning attacker isn't much use, so requests that were probing rather than visiting keep their full address.
The basis is GDPR Recital 49, which names processing "strictly necessary and proportionate for ensuring network and information security" as a legitimate interest. It's not any idea that a scanner forfeits its rights. That distinction does real work, because it forces the rule to track evidence of probing rather than a guess about who's on the other end. (This is my reading as an operator, not legal advice.)
So the test looks only at what was requested:
- The Host isn't one this server serves, which covers requests to the bare IP and every name the wildcard DNS record lets through.
- The hostname has the shape of subdomain enumeration.
- The path only exists on software this site doesn't run:
/.env,/wp-login.phpand the like.
Two tempting inputs are deliberately left out.
The user agent. My classifier has a scanner UA list right there, and it's the
obvious thing to use. But a UA is a claim the client makes about itself. When I
measured it, 1,340 rows were flagged as scanners on the UA alone, and a developer
running curl against a blog post is not evidence of anything. Keying on the UA
would retain someone's full address on the strength of a string, and a real
attacker could get truncated just by sending a browser's.
The stored flags. The database already had is_scanner and
is_unconfigured_host columns that looked like exactly this test. At the time, the
classifier stopped at its first matching rule, so a curl hit on the bare IP was
stored as a scanner and not as an unconfigured host. Keying on the flag would have
missed 6,213 rows, 7.3% of all traffic and overwhelmingly raw-IP probing, which is
the traffic the exemption exists for. Deriving the answer from the host column
caught them.
"Exception" is the wrong word for it
I expected the exemption to be small. It isn't. Across the archive so far, from
mid-July to mid-October, about 70% of 267,604 access log rows keep a full
address, ranging from 54% to 78% by month. (Counted by address shape, so a real
address that happens to end in .0 reads as truncated. The true figure is, if
anything, slightly higher.)
That's a consequence of running wildcard DNS on the open internet: most of what reaches the server was never looking for this site. It's still worth saying plainly, because "visitors are truncated, with an exception for attackers" sounds like a narrow carve-out, and by row count the exception is the majority. The policy now describes the rule and what it catches rather than implying it's rare.
The failure direction also matters. The list of hosts I serve is duplicated in the export script, so a new subdomain I forget to add there would keep full addresses for its real visitors. That's the safer way for it to drift, and it's still wrong.
And "anonymized" in the title is doing more work than it's entitled to. A truncated address next to a timestamp, a path and a full user agent is a reduced record, not an anonymous one in any strict sense. What truncation removes is the direct identifier. I'd rather say that here than have the headline imply more.
Retrofitting found a gap in the fix
Days already archived needed rewriting, so the script got a mode that re-exports them from the database, and refuses to replace a file with fewer rows than it already holds, since a day past the purge would come back empty and destroy its only copy.
It ran and reported nothing kept, which reads as "all done". It wasn't. The mode walked a lookback calendar rather than the archive directory, and the window stopped two days short of the oldest file, so those two days were never visited and the guard never had a reason to speak. I caught it by comparing the output against the known oldest archive, and re-ran it while those days were still in the database.
The copies I wasn't thinking about
The archive was the copy I knew about. Then a month later:
The database dumps. A dump from July 30 still held 68,267 access log rows with full addresses going back to July 11, and dumps outlive the purge by design. Fixing that took four separate changes, because each route was individually reasonable: scheduled dumps included the log tables, the fetch script synced them down, the restore script kept its own copy, and a dump with logs lingered on the server until rotation reached it. Log tables are now excluded by default, and a dump that includes them says so in its filename, which the sync refuses.
The analytics workbench. I keep a local copy of the logs in Iceberg to learn the format, and the table format turned out to hold onto values in places I hadn't expected.
The restore copy that survived the fix. This one is the embarrassing one, because I only found it while writing this series. Restoring production locally pulls a dump with logs by design, and the next restore overwrites it. No next restore ever came. So a dump with 88,641 access log rows sat on my dev machine, in a gitignored directory that felt safe, past the 30 days the policy promises. The dump and the restored database are gone now.
That last one is the real lesson. I'd audited this exact problem, written down every route, and closed all four. The rule "the next run wipes it" quietly assumed there would be a next run. A retention promise isn't a property of a table. It's a claim about every copy that exists, including the ones that get created after you made it, and it needs rechecking whenever you add something that copies things.