Intro
I’ve sung the praises of fail2ban as a modern way to shutdown those annoying probes of your cloud server. I recently got to work with a Redhat v 7.4 system, so much newer than my old CentOS 6 server. And fail2ban failed even to work! Instead of the usual extensive debugging I just wrote my own. I’m sharing it here.
The details
I have a bare-bones RHEL 7.4 system. A yum search fail2ban does not find that package. Supposedly you simply need to add the EPEL repository to make that package available but the recipe on how to do that is not obvious. So I got the source for fail2ban and built it. Although it runs, you gotta build a local jail to block ssh attempts and that’s where it fails. So instead of going down that rabbit hole – I was already too deep, I decided to heck with it and I’m building my own.
All I really wanted was to ban IPs which are hitting my sshd server endlessly, often once per second or more. I take it personally.
RHEL 7 has a new firewall concept, firewalld. It’s all new to me and I don’t want to go down that rabbit hole either, at least not right down. So I rely on that old standard of mine: cut off an attacker by making an invalid route to his IP address, along the lines of
$ route add ‐host
And voila, they can no longer establish a TCP connection. It’s not quite as good as a firewall rule because their source UDP packets could still get through, but come on, we don’t need to be purists. And furthermore, in practice it produces the desired behaviour: stops the ssh dictionary attacks cold.
I knocked tghis out in one night, avoiding the rabbit hole of “fixing” fail2ban. So I had to use the old stuff I know so well, perl and stupid little tricks. I call drjfail2ban.
#!/bin/perl # suppress IPs with failed logins # DrJ - 2017/10/07 $DEBUG = 0; $sleep = 30; $cutoff = 3; $headlines = 60; @goodusers =("drjohn1","user57"); %blockedips = (); while(1) { # $time = `date +%Y%m%d%H%M%S`; main(); sleep($sleep); } sub main() { if ($DEBUG) { for $ips (keys %blockedips) { print "blocked ip: $ips " } } # man last shows what this means: -i forces IP to be displayed, etc. open(LINES,"last -$headlines -i -f /var/log/btmp|") || die "Problem with running last -f btmp!!\n"; # output: #ubnt ssh:notty 185.165.29.197 Sat Oct 7 19:30 gone - no logout while(<LINES>) { ($user,$ip) = /^(\S+)\s+\S+\s+(\S+)/; print "user,ip: $user,$ip\n" if $DEBUG; next if $blockedips{$ip}; #we can't handle hostnames right now next if $ip =~ /[a-z]/i; $candidateips{$ip} += 1; $bannedusers{$ip} = $user; } for (keys %candidateips) { $ip = $_; # allow my usual source IPs without blocking... next if $ip =~ /^(50\.17\.188\.196|51\.29\.208\.176)/; next if $blockedips{$ip}; $usr = $bannedusers{$ip}; $ipct = $candidateips{$ip}; print "ip, usr, ipct: $ip, $usr, $ipct\n" if $DEBUG; # block $block = 1; for $gu (@goodusers) { print "gu: $gu\n" if $DEBUG; $block = 0 if $usr eq $gu; } if ($block) { # more tests: persistence of attempt $hitcnt = $candidateips{$ip}; if ($hitcnt < $cutoff) { # do not block and reset counter for next go-around print "Not blocking ip $ip and resetting counter\n" if $DEBUG; $candidateips{$ip} = 0; } else { $blockedips{$ip} = 1; print "Blocking ip $ip with hit count $hitcnt at " . `date`; # prevent further communication... system("route add -host $ip gw 127.0.0.1"); } } #print "route add -host $ip gw 127.0.0.1\n"; } close(LINES); } # end main function |
Highlights from the program
The comments are pretty self-explanatory. Just a note about the philosophy. I fear making a goof and locking myself out! So I was conservative and try to not do any blocking if the source IP matches one of my favored source IPs, or if the user matches one of my usual usernames like drjohn1. I use obscure userids and the hackers try the stupid stuff like root, admin, etc. So they may be dictionary attacking the password, but they certainly aren’t dictionary attacking the username!
I don’t mind wiping the slate clean of all created routes after sever reboot so I only plan to run this from the command line. To make it persistent until the next reboot you just run it from the root account like so (let’s say we put it in /usr/local/sbin):
$ nohup /usr/local/sbin/drjfail2ban > /var/log/drjfail2ban &
And it just sits there and runs, even after you log out.
Results
Since it hasn’t been running for long I can provide a partial log file as of this publication.
Blocking ip 103.80.117.74 with hit count 6 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 89.176.96.45 with hit count 5 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 31.162.51.206 with hit count 3 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 218.95.142.218 with hit count 6 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 202.168.8.54 with hit count 5 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 13.94.29.182 with hit count 4 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 40.71.185.73 with hit count 4 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 77.72.85.100 with hit count 13 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 201.180.104.63 with hit count 7 at Sun Oct 8 17:34:43 CEST 2017 SIOCADDRT: File exists Blocking ip 121.14.27.58 with hit count 4 at Sun Oct 8 17:40:43 CEST 2017 Blocking ip 36.108.234.99 with hit count 6 at Sun Oct 8 17:47:13 CEST 2017 Blocking ip 185.165.29.69 with hit count 6 at Sun Oct 8 18:02:43 CEST 2017 Blocking ip 190.175.40.195 with hit count 6 at Sun Oct 8 19:05:43 CEST 2017 Blocking ip 139.199.167.21 with hit count 4 at Sun Oct 8 19:29:13 CEST 2017 Blocking ip 186.60.67.51 with hit count 5 at Sun Oct 8 20:49:14 CEST 2017 |
And what my route table looks like currently:
$ netstat ‐rn|grep 127.0.0.1
Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 2.177.217.155 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 13.94.29.182 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 31.162.51.206 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 36.108.234.99 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 37.204.23.84 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 40.71.185.73 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 42.7.26.15 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 46.6.60.240 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 59.16.74.234 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 77.72.85.100 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 89.176.96.45 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 103.80.117.74 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 109.205.136.10 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 113.195.145.13 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 118.32.27.85 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 121.14.27.58 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 139.199.167.21 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 162.213.39.235 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 176.50.95.41 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 176.209.89.99 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 181.113.82.213 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 185.165.29.69 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 185.165.29.197 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 185.165.29.198 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 185.190.58.181 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 186.57.12.131 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 186.60.67.51 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 190.42.185.25 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 190.175.40.195 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 193.201.224.232 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 201.180.104.63 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 201.255.71.14 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 202.100.182.250 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 202.168.8.54 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 203.190.163.125 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 213.186.50.82 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 218.95.142.218 127.0.0.1 255.255.255.255 UGH 0 0 0 lo 221.192.142.24 127.0.0.1 255.255.255.255 UGH 0 0 0 lo |
Here’s a partial listing of the many failed logins, just to keep it real:
... root ssh:notty 190.175.40.195 Sun Oct 8 19:05 - 19:28 (00:23) root ssh:notty 190.175.40.195 Sun Oct 8 19:05 - 19:05 (00:00) root ssh:notty 190.175.40.195 Sun Oct 8 19:05 - 19:05 (00:00) root ssh:notty 190.175.40.195 Sun Oct 8 19:05 - 19:05 (00:00) root ssh:notty 190.175.40.195 Sun Oct 8 19:05 - 19:05 (00:00) root ssh:notty 190.175.40.195 Sun Oct 8 19:05 - 19:05 (00:00) admin ssh:notty 185.165.29.69 Sun Oct 8 18:02 - 19:05 (01:02) admin ssh:notty 185.165.29.69 Sun Oct 8 18:02 - 18:02 (00:00) admin ssh:notty 185.165.29.69 Sun Oct 8 18:02 - 18:02 (00:00) admin ssh:notty 185.165.29.69 Sun Oct 8 18:02 - 18:02 (00:00) root ssh:notty 185.165.29.69 Sun Oct 8 18:02 - 18:02 (00:00) root ssh:notty 185.165.29.69 Sun Oct 8 18:02 - 18:02 (00:00) root ssh:notty 185.165.29.69 Sun Oct 8 18:02 - 18:02 (00:00) root ssh:notty 36.108.234.99 Sun Oct 8 17:47 - 18:02 (00:15) root ssh:notty 36.108.234.99 Sun Oct 8 17:47 - 17:47 (00:00) root ssh:notty 36.108.234.99 Sun Oct 8 17:47 - 17:47 (00:00) root ssh:notty 36.108.234.99 Sun Oct 8 17:47 - 17:47 (00:00) root ssh:notty 36.108.234.99 Sun Oct 8 17:47 - 17:47 (00:00) root ssh:notty 36.108.234.99 Sun Oct 8 17:46 - 17:47 (00:00) ubuntu ssh:notty 121.14.27.58 Sun Oct 8 17:40 - 17:46 (00:06) ubuntu ssh:notty 121.14.27.58 Sun Oct 8 17:40 - 17:40 (00:00) aaaaaaaa ssh:notty 121.14.27.58 Sun Oct 8 17:40 - 17:40 (00:00) aaaaaaaa ssh:notty 121.14.27.58 Sun Oct 8 17:40 - 17:40 (00:00) root ssh:notty 206.71.63.4 Sun Oct 8 17:34 - 17:40 (00:06) root ssh:notty 206.71.63.4 Sun Oct 8 17:34 - 17:34 (00:00) root ssh:notty 89.176.96.45 Sun Oct 8 16:15 - 17:34 (01:19) root ssh:notty 89.176.96.45 Sun Oct 8 16:15 - 16:15 (00:00) root ssh:notty 89.176.96.45 Sun Oct 8 16:15 - 16:15 (00:00) root ssh:notty 89.176.96.45 Sun Oct 8 16:15 - 16:15 (00:00) ... |
Before running drjfail2ban it was much more obnoxious, with the same IP hitting my server every second or so.
Conclusion
I found it easier to roll my own than battle someone else’s errors. It’s kind of fun for me to create these little scripts. I don’t care if anyone else uses them. I will refer to this post myself and probably re-use it elsewhere!
References and related
In an earlier time, I was singing the praises of fail2ban on CentOS.