Every reverse proxy on the public internet gets walked. Not attacked in any
targeted sense — walked, by something with a wordlist, checking whether you left
/.env or /.git/config or /credentials.json lying around. It arrives within
hours of the DNS record existing and it never stops.
The standard answer is a fail2ban jail matching a list of known-bad paths.
SWAG — the nginx-plus-fail2ban container I run at the edge — bundles one called
nginx-botsearch and enables it by default. It works. It is also, structurally,
always slightly out of date, because it is a list, and the thing it describes
changes without telling you.
So I added a second jail next to it that matches no paths at all.
The premise
A scanner walking a wordlist has one property that no amount of rotating its paths can hide: almost everything it asks for does not exist. It generates 404s in volume, quickly, from one address.
A real visitor does not. A browser session on a working site produces a handful of 404s at most — a missing favicon, a stale bookmark, a dead link — spread over minutes. The gap between those two behaviors is enormous and it does not depend on which paths are fashionable this month.
So: count plain 404s per IP over a short window. Ban above a threshold. That is
the whole idea, and the useful part is what it doesn't contain. There is no
list to maintain. A scanner that switches from /.env to /.env.bak to
/config/database.yml to whatever is next trips exactly the same rule, because
the rule was never about the paths.
The filter
The one detail you have to get right is matching your own log format. Mine is a
custom log_format, not nginx's combined, so the fields are in different
places:
$remote_addr - $remote_user [$time_local] "$method $uri $proto" $status
$bytes $req_time $upstream_time "$referer" "$ua" $host $ssl_proto $ssl_cipher
The filter, in filter.d/nginx-404-scan.conf:
[Definition]
failregex = ^<HOST> \S+ \S+ \[.*\] "\S+ \S+ \S+" 404\b
ignoreregex =
<HOST> is fail2ban's own token for the address to ban — it expands to a
pattern that handles both IPv4 and IPv6 and captures the result. The rest is
deliberately loose: three opaque tokens for the request line, because the whole
point is not to care what was requested, and then 404 anchored with \b so it
cannot also match a 4041ms timing field further along the line.
Against a matching log line:
203.0.113.7 - - [10/Jul/2026:14:23:01 -0700] "GET /.ssh/id_rsa HTTP/1.1" 404 ...
Before enabling anything, run it against your actual log, not against a line you invented:
docker exec swag fail2ban-regex /config/log/nginx/access.log \
/config/fail2ban/filter.d/nginx-404-scan.conf
fail2ban-regex prints how many lines matched and how many were ignored. This
is the step where you discover your log format is not what you thought it was,
and it costs about four seconds. I wrote the filter against an example line
first and it did match — but "matches the line I made up" is a test of my
typing, not of the filter.
The two ways it silently did nothing
Both of these are delivery problems rather than logic problems, and both share a failure mode: config that looks applied, passes validation, and has no effect.
The jail file went where nothing reads it. I dropped the jail stanza into
/config/fail2ban/jail.d/, which is where you would put it. LinuxServer's
images bridge configuration out of /config into the container's real
/etc/fail2ban on start — that is the whole convention — so this seemed
obviously right.
It bridges jail.local, filter.d and action.d. It does not bridge jail.d.
The jail simply never appeared. Not an error, not a warning; the directory
fail2ban actually reads stayed exactly as it was built into the image, which I
confirmed by looking at its mtime — stale from image build day, while
jail.local and filter.d updated live in the same container. A full
fail2ban-client restart changed nothing, because there was nothing to load.
The fix is an explicit bind mount in the compose file, mapping the host's
jail.d directory onto /etc/fail2ban/jail.d read-only. After that, edits to
files inside it need only a fail2ban-client reload; only changing the mount
itself needs a container recreate.
The general lesson, which cost me more than this one instance: what a
container image bridges from its config volume is specific, undocumented in
detail, and version-dependent. jail.local copies from /config. In the same
image, fail2ban.local copies from the image, so editing the /config copy is
discarded on every start — exactly inverted, in the same directory, for reasons
that make sense only as history. Check where your file actually lands before
assuming an edit took effect. A silent no-op is the normal failure here, not the
exotic one.
The jail was configured to ban by email. My first draft of the stanza set an
explicit action copied from a fail2ban example — one of the variants that bans
and sends mail with a whois lookup and the matched log lines attached.
Nothing runs an SMTP server on this box. That action would have errored every time a ban fired. It had not fired yet, which is the only reason I found it by reading rather than by discovering months later that a jail I believed in had been throwing errors instead of banning.
The fix is to set no action at all and take fail2ban's plain ban-only default.
Worth stating as a rule: do not copy an action line out of an example
config. The defaults ban. The named variants in every tutorial add
notification paths that assume infrastructure you probably do not have, and a
broken action is a jail that counts correctly and then fails at the one moment
it matters.
On thresholds, and why I am not printing mine
The jail needs a count and a window — how many 404s, over how long.
I am deliberately not publishing the values I use, and I would suggest you don't either. A threshold is a rate limit, and publishing a rate limit tells anyone who cares exactly how to stay underneath it. There is no upside; the numbers are not the interesting part of the technique.
How to pick them: think about the widest gap between the two behaviors rather than about a "safe" number. A browser session, even a badly broken one, does not produce double-digit 404s within a couple of minutes. A wordlist scanner produces that in about a second. Anywhere in that gap works, and the gap is wide enough that precision is not required — which is the real strength of a behavioral rule, and the reason it survives scanners changing tactics.
One honest caveat about my own: I picked them by reasoning, before I had any production data, and I have not gone back to check them against a few weeks of real traffic. That is a piece of homework I have not done. The jail has been banning real scanners since the day it went live, so the threshold is clearly not too high — but "it fires" is not the same evidence as "it does not fire on anyone legitimate," and I should measure the second one.
It complements the signature jail, it does not replace it
I run both, and the framing that finally made sense to me is that they fail in opposite directions.
The signature jail matches on the first request, if that request is on its
list. Someone who asks for exactly one known-bad path and leaves is caught by
nginx-botsearch and invisible to a volume-based rule.
The behavioral jail catches the pattern, whatever the paths are, but needs several requests before it has a pattern to see. It is structurally incapable of reacting to a single probe.
Neither subsumes the other, they cost nothing to run together, and their overlap is not wasted work — it is two independent reasons to ban the same address.
The blind spot
This is the part I would want to read in someone else's post about this, so here it is.
A behavioral jail is not actually behavioral. It is behavioral within one response status, and mine counts 404s. Anything a scanner does that produces a different status code is invisible to it, no matter how obviously it is scanning.
I had exactly that hole and did not see it for two months. The edge had a misconfigured default server, so a request for any hostname I had not explicitly configured — a wildcard DNS record means every guess resolves — got handed to the main site and answered 200. Enumeration of hostnames was, by volume, around a fifth of everything in my logs, and every single one of those requests was a success as far as the jail was concerned. Not under the threshold. Uncounted. The rule that was supposed to catch scanning had a large category of scanning routed around it, and there was no symptom, because a jail that is not matching looks identical to a jail with nothing to match.
Fixing the default server did not fix that half, either. The catch-all now completes the TLS handshake, logs the request, and closes the connection without responding — nginx's non-standard 444. So those requests are recorded and correctly labeled, and they are still not 404s, so the 404 jail still does not see them. That is now a deliberate gap rather than an accidental one, but it is a gap.
There is a related trap in that fix worth flagging, since it is the same idea one level down. The other obvious way to write that catch-all is to refuse the TLS handshake outright. It is cleaner and it is cheaper. It also means no HTTP request ever exists, so nothing reaches the access log at all — I measured this on a local replica of the edge before shipping: three rejected requests, zero log lines. That would have deleted a fifth of my traffic data and made every hostname-guessing scanner permanently unbannable, because fail2ban cannot ban what nginx never logged. The variant that logs and then closes keeps the evidence.
So the honest summary of the technique: counting behavior instead of matching signatures buys you a rule that does not decay, and that is a real and underrated win. What it does not buy you is coverage. You still chose a signal — mine is "the response was a 404" — and everything that avoids your signal is outside your jail, whether or not it is obviously an attack. Worth knowing which one you picked, and worth occasionally asking what is on the other side of it.