I have a rule for the four small application repos I run: every so often, stop and read the whole thing for security problems rather than only reviewing each change as it lands. The trigger is commit count — once the four repos have collectively moved about 25 commits past the last audit, do another one.
That rule exists because of what an audit found the first time it ran, which was a service signing production admin tokens with an empty key. So I care about it firing.
A rule that depends on someone remembering to count commits is not a rule, it is an intention. So I automated it: a hook that runs at the start of each working session, reads the last audit's baseline commit SHAs out of a log file, counts how far each repo has moved, and speaks up if the total crosses the threshold.
That was on the 25th of July. On the 29th of August I asked a question I had not thought to ask before — has that thing ever actually gone off? — and the answer was no. Not once. Not in 35 days. It had never worked at all.
Where the baselines live
Each audit appends an entry to a log file, and every entry carries a machine-readable line so the hook doesn't have to parse prose:
<!-- baseline: portal-web=a1b2c3d portal-api=4e5f6a7 log-collector=b8c9d0e botcheck-api=1f2a3b4 audited=2026-08-29 -->
Entries are newest first, so the hook wants the first such line in the file. That is what it went looking for:
line=$(grep -m1 'baseline:' "$LOG")
And then, per repo:
sha=$(printf '%s\n' "$line" | grep -oE "${name}=[0-9a-fA-F]+" | head -1 | cut -d= -f2)
if [ -z "$sha" ]; then
continue
fi
Read those two snippets together and the bug is already visible. It took me five weeks and a direct question to see it.
The file documented its own format
Near the top of that log file, above every real entry, is a section explaining how to add one. It contains the format:
<!-- baseline: portal-web=<sha> portal-api=<sha> log-collector=<sha> botcheck-api=<sha> audited=YYYY-MM-DD -->
grep -m1 'baseline:' stops at the first match, and the first match is the
documentation, sitting above the data by construction — because instructions go
at the top of a file and the newest entry goes below them.
So the hook read the template. Then, for each repo, it looked for
portal-web= followed by hex characters, and found portal-web=<sha>, where
the placeholder is not hex. No match. Empty. continue.
Four repos, four continues. The running total stayed at zero. Zero is less
than the threshold, so the hook exited quietly, having done exactly what it
would do on a day when there was nothing to report.
The parser could not tell the difference between the data and the file's description of the data — and the description was guaranteed to come first.
Why five weeks of nothing looked correct
This is the part I keep turning over.
A reminder's normal output is silence. That is the whole design: it stays quiet until you need to hear from it. Which means the failure state and the healthy state are byte-identical from outside. Exit code 0, no output, no log line, no mention anywhere.
There was no moment where I could have noticed. I did not ignore a warning sign; there was no sign. Every single day, the system reported precisely what I expected it to report, and it was wrong every single time.
And it was not a near-miss. Going back through the history afterwards:
- On the 9th of August the four repos hit exactly 25 commits since the previous baseline. The threshold is 25. Silence.
- On the 29th of August they hit 26. Silence.
Both audits happened anyway. Working out why took me longer than finding the bug, and it is the real answer to how five weeks went by.
The rule was written down in two places. One was the hook. The other was a paragraph in my project instructions telling whoever is working in these repos to count the commits themselves and raise it if the total is over the threshold. I built the hook precisely because I did not trust the paragraph — a rule that depends on someone remembering is an intention, as I said at the top.
The paragraph won. Both times, the arithmetic was done by hand by whoever was working in the repos that day, following that written instruction, and the audit got done. Not by me from memory, and not by the hook. The outcome was correct on both occasions, produced entirely by the path I had considered unreliable, while the path I trusted contributed nothing at all.
That is why there was no symptom to notice. A dead component whose job is also being done by a live one does not produce a visible failure. It produces a correct outcome and a false attribution. I never thought "good, the hook caught that" in so many words — but I certainly never thought "the hook didn't catch that", because nothing about the result looked any different from the result I was expecting.
So the cost is not that the reminder was absent. An absent reminder leaves you appropriately nervous. The cost is that I had two mechanisms for one job, one of them was dead, and the working one kept the outcome correct just often enough that the dead one was never missed. Redundancy is supposed to be what saves you. It is also what hides which of the two is doing the work.
The self-referential bit
I want to state this plainly, because it is the reason I bothered writing it up.
The thing that broke was the component whose entire job was noticing that something had been left too long. It was the one piece of the system specifically responsible for catching neglect, and it was itself neglected, for five weeks, and no other part of the system could possibly have told me — because its only means of communication is an occasional message, and "never sends one" is indistinguishable from "nothing to send."
A watchdog's failure mode is the one thing a watchdog cannot watch.
The fix is two changes, and the first is the boring one
The obvious repair is to make the pattern skip the template:
line=$(grep -m1 -E '<!-- baseline:.*portal-web=[0-9a-fA-F]{7,}' "$LOG")
Anchoring on a real 7+ character hex SHA for the first repo skips the placeholder and still matches every genuine entry. Fine. That fixes today's bug.
It does not fix the class of bug, which is that the script had a dozen ways to produce zero and every one of them exited silently. So the second change is a policy, applied to every path:
Silence must mean "not due". It must never mean "couldn't tell."
Concretely, everything that cannot produce a real number now speaks up instead of exiting:
if [ -z "$line" ]; then
nudge "Security-audit cadence check FAILED: no usable baseline line found, so
the commit count could not be computed. Treat the cadence as unknown and
therefore due."
fi
and, per repo:
if [ -z "$sha" ]; then
# A repo missing from an otherwise valid line is the same silent undercount
# the template bug caused, one repo at a time. Count it as due rather than
# as zero.
details="$details ${name}:no-baseline"
total=$((total + THRESHOLD))
continue
fi
That continue is the same line as before. What changed is that it no longer
adds nothing — a repo it cannot measure now counts as a full threshold's worth
of drift, which forces the nudge. Same for a baseline SHA that isn't an
ancestor of HEAD, which means history was rewritten or the SHA is stale: not
measurable, therefore due.
There is exactly one silent exit left, and it is annotated as such: if jq
isn't installed there is no way to emit a message at all, because emitting the
message is what jq is for. I couldn't close that one, so I wrote down that it
is open.
The general principle is not new — fail closed, treat unknown as unsafe. What I had not internalized is that it applies to reminders and monitors just as much as to authentication, and it is much easier to get wrong there, because the "safe" direction feels like it should be "don't bother the user."
How to test something whose normal output is nothing
I had tested this hook, in the sense that I ran it and it didn't crash. That test would have passed on every day of its broken life.
What I do now is feed it the state it is supposed to complain about, which turns out to be four separate cases:
- Current state. 0 commits since the last audit — must be silent. Confirms I haven't made it nag constantly.
- A known-overdue baseline. The 9th-of-August entry, which should produce a count of 27 — must nudge, with the right number.
- A log file containing only the template line. This is precisely the bug as shipped, preserved as a test case. It must now nudge with an explicit failure message rather than exiting 0.
- A valid line with one repo removed. Must nudge, and must name the repo it couldn't measure rather than quietly counting three out of four.
Cases 3 and 4 are the ones that matter, and they are the ones I would never have written before this. They are not testing whether the feature works. They are testing whether the absence of the feature is detectable, which for anything whose output is silence is the only interesting question.
A monitor you have never seen fire is a monitor you have not tested. If you can't remember watching it go off, make it go off.
The part about the robot
I should say where this code came from, because I think it is the most useful thing in this post and glossing over it would be dishonest.
I write a lot of this infrastructure with an AI agent. Not as an experiment — as the normal way I work. This hook was one of those: I described the cadence rule, and got back a clean, commented, sensibly structured shell script that looked exactly like what I'd asked for.
And it was. That is the thing. The script was well-formed, readable, correctly handled the cases it considered, and did precisely what it described itself as doing. Nothing about it was sloppy. If you had shown me it in review — and I did review it — I would have approved it, because I did.
What it lacked was consequence. Nobody had run it against a real log file over five weeks and noticed it never said anything. That is not a thing the authoring step can supply, by either of us. The model could not know that the file it was parsing would document its own format above the data, because that section was written by a different session on a different day for a human reader. I could have known, and didn't look.
I have started thinking about it this way: AI collapses the cost of writing code. It does nothing to the cost of consequence — of code having been run, in production, long enough for the gap between "it ran" and "it worked" to show up. And an enormous share of what actually breaks infrastructure lives in that gap. Everything expensive I have had to fix on this server was code that ran perfectly and did the wrong thing quietly: a log tailer that read the wrong inode for four days, a database restore that reported complete success into a directory that would evaporate on the next restart, an auth service happily signing tokens with an empty key.
In every one of those, the code was correct in the sense the author meant. And generating code that is correct in the sense the author meant is exactly what these tools are outstanding at.
There is a second turn to this that I did not see coming. The thing that kept the cadence on track through those five weeks was also an agent — a different session, reading that paragraph in the project instructions and dutifully doing the arithmetic by hand, every time. So the same class of tool wrote the broken automation and then quietly covered for it. That is not irony, it is the mechanism: the compensation is exactly what removed the symptom.
And neither side could have caught it, because catching it meant holding two facts together — the hook exists and the count was worked out by hand again — and being suspicious of the pair. Each session saw only its own half, and each half was working.
So this is not a story about an AI writing bad code. The code was fine. It is a story about the one review question that no amount of reading the code will answer, which a human asked five weeks late:
Has this ever actually fired?
I now ask that about anything I've built whose job is to tell me something. Not "is it running" — running was never in doubt. Has it ever spoken? And if it has not, is that because there has been nothing to say, or because it has lost the ability to say it?
Most of the time the answer is fine. It took one occasion for it not to be.