Distributed SSH Brute Force Attempts, part 3

I wanted to be able to harvest the log data that the brute force attempts are generating, so I've decided to not move the SSH listening port. I'm also logging each event within my firewall logs. The particular machine I'm seeing the attacks on is a FreeBSD box (I should've mentioned that earlier) and I'm using PF as the firewall of choice.

I'd like to show you my listing of blocked IPs. I've been actively gathering them since approximately 11/17/2008. I was unhappy that I hadn't noticed the distributed attempts until November and wanted more trending data, so I reached into my SSH logs and parsed the files present with a very dirty script that added each unique IP to a PF table that is designed to block such activity. The script is below:

[root@delly ~]# cat IPscript
#/bin/bash

cd /var/log
bunzip2 pflog.*

cat /var/log/auth.log | grep sshd | grep -i 'invalid user' > /tmp/auth_IP_list_1
bzcat /var/log/auth.log.[01234567].bz2 | grep sshd| grep -i 'invalid user' >> /tmp/auth_IP_list_1
sed '/Failed keyboard-interactive/d' /tmp/auth_IP_list_1 > /tmp/auth_IP_list_2
awk '{print $10}' /tmp/auth_IP_list_2 > /tmp/auth_IP_list_3

tcpdump -nettttr /var/log/pflog > /tmp/fw_IP_list_1
tcpdump -nettttr /var/log/pflog.0 >> /tmp/fw_IP_list_1
awk '{print $9}' /tmp/fw_IP_list_1 > /tmp/fw_IP_list_2
nawk -F. '{print $1, $2, $3, $4}' /tmp/fw_IP_list_2 > /tmp/fw_IP_list_3
sed 's/ /./g' /tmp/fw_IP_list_3 > /tmp/fw_IP_list_4

cat /tmp/fw_IP_list_4 >> /tmp/auth_IP_list_4
cat /tmp/auth_IP_list_4 | sort -rn | uniq > /tmp/auth_IP_list_5
cat /tmp/auth_IP_list_5 | grep -v '64.62.231.220' > /tmp/auth_IP_list_6
cat /tmp/auth_IP_list_6 | grep -v '66.160.141.30' > /tmp/auth_IP_list_7
cat /tmp/auth_IP_list_7 | grep -v '10.150.1' > /tmp/auth_IP_list_8
cat /tmp/auth_IP_list_8
cat /tmp/auth_IP_list_8 | wc -l

pfctl -t bruteforce -T add -f /tmp/auth_IP_list_8

rm -rf /tmp/fw_IP_list_* /tmp/auth_IP_list_*

I apologize for the lack of comments in the script...as I said, it was an extremely dirty hack that required me to learn a tad of sed and awk. The script outputs the following file: http://wigglit.ath.cx/txt/bruteforce_IPtable

My logs don't actually go back that far. The FW logs go back to 14 Dec. The SSH service logs go back to 26 Nov. I probably have captured IPs reaching back to the beginning of November, though.

My FW policy prevents anything from entering the network unless specifically allowed (default deny policy). The reason I wanted to track the IPs was because my Denyhosts configuration wasn't catching most of these and it was working fine before this new trend occurred. I'm a security consultant and a researcher at heart, so I thought that tracking this would be cool. My home router has ports 22, 443, and 3306 exposed to the wild for the FreeBSD box. The FW only allows certain IPs in on those ports, though, so nothing will get in. The activity that doesn't get immediately blocked is blocked by the bruteforce_IPtable script that I run daily. The script is flawless and I will begin to have it run hourly via a cronjob. I'll also have it send an updated list to my website daily.

The IPs within my block table number 565. The script parsed 115 from the logs tonight, but only added 1 IP. The norm is usually 2-3 daily. The rest of the IPs are from logs over the last month and a half.

Oh yeah, I've another script does a daily copy of the IPs that are added to the table, so I can at least quickly determine (using 'diff') what was added on a certain day. I can create a script that will show me what was added daily for a given timespan (day/week/month).

Distributed SSH Brute Force Attempts, part 2

OK, so I'm still curious about the distributed SSH brute force attempts I've been seeing.

I wanted to check all my logs and not just the most recent logfile. Looking at my /var/log directory:

-su-2.05b# ls | grep auth
auth.log
auth.log.0.bz2
auth.log.1.bz2
auth.log.2.bz2
auth.log.3.bz2
auth.log.4.bz2
auth.log.5.bz2
auth.log.6.bz2
auth.log.7.bz2

I tailed the auth.log file and grabbed a random IP:

Nov 24 21:53:59 delly sshd[75490]: Invalid user bryan from 170.56.255.20
Nov 24 21:54:00 delly sshd[75490]: error: PAM: authentication error for illegal user bryan from 170.56.255.20
Nov 24 21:54:00 delly sshd[75490]: Failed keyboard-interactive/pam for invalid user bryan from 170.56.255.20 port 43229 ssh2

I then wanted to check all the auth.log.* files, but was curious as to how I could check compressed files. I found that there's a command called bzgrep that allows one to grep compressed files, so I used the following command and came up with quite a few hits for the referenced IP over seven (7) log files:

-su-2.05b# bzgrep '170.56.255.20' auth.log.*

The results show 2-3 instances of log entries per login attempt, so I wanted to isolate each instance without having to use arcane sed and sort commands, so I used the following:

-su-2.05b# bzgrep '170.56.255.20' auth.log.* | grep 'Invalid user' | wc -l
19

So, this particular IP generated 19 log entries between 22 and 24 Nov.

That's not particularly good. I'd love to create a script that would break down all of these IPs' unique login attempts (and possibly block them). Sounds like a project, no? :)

I've a bit of time, so I did the following (non-scripted). I cat'd the auth.log file and collected a screens-worth of data:

-su-2.05b# cat auth.log | less
Nov 24 15:00:00 delly newsyslog[74001]: logfile turned over due to size>100K
Nov 24 15:00:49 delly sshd[74014]: Invalid user brand from 218.80.215.198
Nov 24 15:00:50 delly sshd[74014]: error: PAM: authentication error for illegal user brand from 218.80.215.198
Nov 24 15:00:50 delly sshd[74014]: Failed keyboard-interactive/pam for invalid user brand from 218.80.215.198 port 19051 ssh2
Nov 24 15:02:25 delly sshd[74017]: Invalid user brandee from 83.19.224.11
Nov 24 15:02:26 delly sshd[74017]: error: PAM: authentication error for illegal user brandee from dum11.internetdsl.tpnet.pl
Nov 24 15:02:26 delly sshd[74017]: Failed keyboard-interactive/pam for invalid user brandee from 83.19.224.11 port 50163 ssh2
Nov 24 15:03:48 delly sshd[74020]: Invalid user brandee from 194.224.118.61
Nov 24 15:03:48 delly sshd[74020]: error: PAM: authentication error for illegal user brandee from 194.224.118.61
Nov 24 15:03:48 delly sshd[74020]: Failed keyboard-interactive/pam for invalid user brandee from 194.224.118.61 port 6345 ssh2
Nov 24 15:05:21 delly sshd[74026]: Invalid user brandee from 90.176.233.222
Nov 24 15:05:22 delly sshd[74026]: error: PAM: authentication error for illegal user brandee from 222.233.broadband9.iol.cz
Nov 24 15:05:22 delly sshd[74026]: Failed keyboard-interactive/pam for invalid user brandee from 90.176.233.222 port 46108 ssh2
Nov 24 15:06:42 delly sshd[74029]: Invalid user branden from 125.77.106.246
Nov 24 15:06:42 delly sshd[74029]: error: PAM: authentication error for illegal user branden from 125.77.106.246
Nov 24 15:06:42 delly sshd[74029]: Failed keyboard-interactive/pam for invalid user branden from 125.77.106.246 port 46495 ssh2
Nov 24 15:10:59 delly sshd[74035]: Invalid user brandi from 122.224.128.222
Nov 24 15:10:59 delly sshd[74035]: error: PAM: authentication error for illegal user brandi from 122.224.128.222
Nov 24 15:10:59 delly sshd[74035]: Failed keyboard-interactive/pam for invalid user brandi from 122.224.128.222 port 42253 ssh2
Nov 24 15:12:27 delly sshd[74051]: Invalid user brandi from 59.125.200.51
Nov 24 15:12:28 delly sshd[74051]: error: PAM: authentication error for illegal user brandi from 3w.upcc.com.tw
Nov 24 15:12:28 delly sshd[74051]: Failed keyboard-interactive/pam for invalid user brandi from 59.125.200.51 port 14046 ssh2
Nov 24 15:15:23 delly sshd[74057]: Invalid user brandice from 62.112.222.88
Nov 24 15:15:24 delly sshd[74057]: error: PAM: authentication error for illegal user brandice from 3e70de58.adsl.enternet.hu
Nov 24 15:15:24 delly sshd[74057]: Failed keyboard-interactive/pam for invalid user brandice from 62.112.222.88 port 42127 ssh2
Nov 24 15:16:49 delly sshd[74060]: Invalid user brandice from 218.80.215.198
Nov 24 15:16:50 delly sshd[74060]: error: PAM: authentication error for illegal user brandice from 218.80.215.198
Nov 24 15:16:50 delly sshd[74060]: Failed keyboard-interactive/pam for invalid user brandice from 218.80.215.198 port 57929 ssh2
Nov 24 15:18:11 delly sshd[74063]: Invalid user brandice from 65.203.231.41
Nov 24 15:18:11 delly sshd[74063]: error: PAM: authentication error for illegal user brandice from 65.203.231.41
Nov 24 15:18:11 delly sshd[74063]: Failed keyboard-interactive/pam for invalid user brandice from 65.203.231.41 port 38395 ssh2
Nov 24 15:19:43 delly sshd[74066]: Invalid user brandie from 123.14.10.64
Nov 24 15:19:44 delly sshd[74066]: error: PAM: authentication error for illegal user brandie from 123.14.10.64
Nov 24 15:19:44 delly sshd[74066]: Failed keyboard-interactive/pam for invalid user brandie from 123.14.10.64 port 4925 ssh2
Nov 24 15:21:07 delly sshd[74072]: Invalid user brandie from 200.170.141.134
Nov 24 15:21:07 delly sshd[74072]: error: PAM: authentication error for illegal user brandie from 200-170-141-134.static.ctbctelecom.com.br
Nov 24 15:21:07 delly sshd[74072]: Failed keyboard-interactive/pam for invalid user brandie from 200.170.141.134 port 39979 ssh2
Nov 24 15:22:44 delly sshd[74088]: Invalid user brandie from 80.51.31.84
Nov 24 15:22:44 delly sshd[74088]: error: PAM: authentication error for illegal user brandie from 80.51.31.84
Nov 24 15:22:44 delly sshd[74088]: Failed keyboard-interactive/pam for invalid user brandie from 80.51.31.84 port 39453 ssh2
Nov 24 15:24:02 delly sshd[74091]: Invalid user brandon from 200.157.176.13
Nov 24 15:24:03 delly sshd[74091]: error: PAM: authentication error for illegal user brandon from 200.157.176.13
Nov 24 15:24:03 delly sshd[74091]: Failed keyboard-interactive/pam for invalid user brandon from 200.157.176.13 port 54638 ssh2

I then checked every IP for unique log entries within all of my ssh logs:

-su-2.05b# bzgrep '218.80.215.198' auth.log.* | grep 'Invalid user' | wc -l
6

-su-2.05b# bzgrep '218.80.215.198' auth.log.* | grep 'Invalid user'
auth.log.0.bz2:Nov 24 10:55:25 delly sshd[73084]: Invalid user bjorn from 218.80.215.198
auth.log.1.bz2:Nov 23 23:22:28 delly sshd[70023]: Invalid user bahari from 218.80.215.198
auth.log.3.bz2:Nov 23 11:18:54 delly sshd[66908]: Invalid user archibald from 218.80.215.198
auth.log.4.bz2:Nov 23 04:11:52 delly sshd[65051]: Invalid user amy from 218.80.215.198
auth.log.5.bz2:Nov 22 23:20:53 delly sshd[63465]: Invalid user alize from 218.80.215.198
auth.log.7.bz2:Nov 22 01:07:10 delly sshd[57652]: Invalid user claire from 218.80.215.198

-su-2.05b# bzgrep '83.19.224.11' auth.log.* | grep 'Invalid user' | wc -l
8
-su-2.05b# bzgrep '194.224.118.61' auth.log.* | grep 'Invalid user' | wc -l
11
-su-2.05b# bzgrep '90.176.233.222' auth.log.* | grep 'Invalid user' | wc -l
0
-su-2.05b# bzgrep '125.77.106.246' auth.log.* | grep 'Invalid user' | wc -l
9
-su-2.05b# bzgrep '122.224.128.222' auth.log.* | grep 'Invalid user' | wc -l
8
-su-2.05b# bzgrep '59.125.200.51' auth.log.* | grep 'Invalid user' | wc -l
5
-su-2.05b# bzgrep '62.112.222.88' auth.log.* | grep 'Invalid user' | wc -l
10
-su-2.05b# bzgrep '218.80.215.198' auth.log.* | grep 'Invalid user' | wc -l
6
-su-2.05b# bzgrep '65.203.231.41' auth.log.* | grep 'Invalid user' | wc -l
15

-su-2.05b# bzgrep '65.203.231.41' auth.log.* | grep 'Invalid user'
auth.log.0.bz2:Nov 24 14:56:28 delly sshd[73982]: Invalid user bran from 65.203.231.41
auth.log.1.bz2:Nov 23 22:32:01 delly sshd[69793]: Invalid user azra from 65.203.231.41
auth.log.1.bz2:Nov 24 02:04:43 delly sshd[70680]: Invalid user bartholemew from 65.203.231.41
auth.log.1.bz2:Nov 24 04:07:45 delly sshd[71474]: Invalid user beck from 65.203.231.41
auth.log.2.bz2:Nov 23 19:02:03 delly sshd[68866]: Invalid user aurora from 65.203.231.41
auth.log.2.bz2:Nov 23 20:16:18 delly sshd[69213]: Invalid user avi from 65.203.231.41
auth.log.3.bz2:Nov 23 10:08:33 delly sshd[66592]: Invalid user april from 65.203.231.41
auth.log.3.bz2:Nov 23 10:24:43 delly sshd[66657]: Invalid user aquila from 65.203.231.41
auth.log.3.bz2:Nov 23 11:22:12 delly sshd[66933]: Invalid user archie from 65.203.231.41
auth.log.4.bz2:Nov 23 04:22:52 delly sshd[65094]: Invalid user anahid from 65.203.231.41
auth.log.4.bz2:Nov 23 05:32:46 delly sshd[65407]: Invalid user andra from 65.203.231.41
auth.log.5.bz2:Nov 22 23:00:09 delly sshd[63393]: Invalid user alisha from 65.203.231.41
auth.log.6.bz2:Nov 22 12:41:17 delly sshd[60534]: Invalid user abraham from 65.203.231.41
auth.log.6.bz2:Nov 22 16:14:07 delly sshd[61564]: Invalid user africa from 65.203.231.41
auth.log.7.bz2:Nov 22 11:48:33 delly sshd[60289]: Invalid user aaralyn from 65.203.231.41

-su-2.05b# bzgrep '123.14.10.64' auth.log.* | grep 'Invalid user' | wc -l
19
-su-2.05b# bzgrep '200.170.141.134' auth.log.* | grep 'Invalid user' | wc -l
6
-su-2.05b# bzgrep '80.51.31.84' auth.log.* | grep 'Invalid user' | wc -l
3
-su-2.05b# bzgrep '200.157.176.13' auth.log.* | grep 'Invalid user' | wc -l
4

So, someone appears to have a pool of compromised machines and is using each one in a scaled SSH brute force attack, based on the referenced user accounts being bruteforced. I'm seeing more of this than standard, blatant SSH BF attempts. I'll be checking Denyhosts' website to see if they've a resolution on how to track and ban such activity.