Building a resilient web server
Early in the year, we encountered a number of challenges, one of the most disruptive was a sustained wave of automated attacks hammering several of the web servers we manage. The volume of traffic overwhelmed server resources, slowing websites to a crawl and frequently triggering 504 Gateway Timeout errors. With uptime and user experience at risk, we needed a robust solution to detect and block malicious traffic in real time, before it could do further damage.
We tackled the issue using Fail2Ban, focusing on a targeted use case - detecting and banning Apache “bogus FCGI” error patterns that weren’t covered by default filters.
Understanding the problem
Apache logs showed repeated error lines like:
[Wed Apr 09 16:22:34.597182 2025] [proxy_fcgi:error] [pid 14590:tid 139993576597248] [client 00.000.00.00:00000] AH01068: Got bogus version 111These attacks weren't direct login attempts or spam bots. They triggered resource-heavy proxy errors. Apache would still respond to these requests, wasting precious server cycles.
Why default Fail2Ban filters weren't enough
Out of the box, Fail2Ban includes filters for 404s, authentication failures, and common bad bots. However, there was no built-in filter matching the specific FCGI attack pattern above.
Building a custom filter
We created a custom filter file `/etc/fail2ban/filter.d/apache-bogus-fcgi.conf`.
Minimal structure:
[Definition]
failregex = \[proxy_fcgi:error\](?: \[pid \d+:tid \d+\])?\s+\[client <HOST>:\d+\]\s+AH01068:\s+Got\sbogus\sversion\s+\d+
ignoreregex =Note:
- Escape the square brackets `[` and `]`
- Use `<HOST>` to identify the attacking IP
- Allow optional `pid` and `tid` fields
Setting up the jail
Then we defined a jail in `/etc/fail2ban/jail.local`:
[apache-bogus-fcgi]
enabled = true
filter = apache-bogus-fcgi
action = iptables[name=Apache-bogus-fcgi, port=http, protocol=tcp]
logpath = /var/www/html/system/[domain]/logs/error_log
findtime = 300
bantime = 86400
maxretry = 1This jail watches the Apache error log and bans after a single match.
The debugging process: why it failed initially
Despite having the regex correctly structured, `fail2ban-regex` kept reporting:
Failregex: 0 totalWhich means: Fail2Ban *did not match* any lines.
The real breakthrough came when we manually validated the regex using **grep** directly on live data.
grep -P "\[proxy_fcgi:error\]\s+\[pid \d+:tid \d+\]\s+\[client \d+\.\d+\.\d+\.\d+:\d+\]\s+AH01068:\s+Got\sbogus\sversion\s+\d+" /var/www/html/system/[domain]/logs/error_logOnce `grep` succeeded in matching, we were confident the regex was correct.
**Key lesson:** Always test regex separately against live log files with `grep -P` when fine-tuning Fail2Ban filters.
Commands you will use regularly
Check Fail2Ban status
sudo fail2ban-client status
Check jail-specific status
sudo fail2ban-client status apache-bogus-fcgi
Restart Fail2Ban after filter/jail changes
sudo systemctl restart fail2ban
Test a filter before going live
sudo fail2ban-regex /path/to/logfile /path/to/filter.conf
Reload Fail2Ban without full restart
sudo fail2ban-client reload
What a successful result looks like
After applying the correct filter and restarting Fail2Ban, a healthy jail looked like this:
Status for the jail: apache-bogus-fcgi
|- Filter
| |- Currently failed: 3
| |- Total failed: 3
| `- File list: /var/www/html/system/[domain]/logs/error_log
`- Actions
|- Currently banned: 48
|- Total banned: 48
`- Banned IP list: 00.000.00.00 00.000.00.00 ...IPs have been obfuscated.
This meant Fail2Ban had automatically identified and banned attackers within seconds.
The wrap
Building effective protections often requires going beyond documentation and default tooling. In our case, relying solely on fail2ban-regex was misleading, as it failed to match log entries even when the regex was correct. The breakthrough came from testing directly against live logs with grep, which validated our pattern and gave us confidence in the solution.
Key takeaway, when fine-tuning Fail2Ban filters, don’t rely purely on syntax checks — test against real traffic. It’s the fastest way to catch subtle issues.
If your server is under pressure from unusual requests, slowdowns, or 504 errors, it’s worth investing the time to craft precise, targeted filters. That extra effort pays off in performance, uptime, and a better experience for everyone.