Wednesday, March 22, 2017

Some Fail2ban Success

I've been playing with Fail2ban jail configurations since the last post and I think I've got my setup running close to perfect.

In my last post, I mentioned that I wanted Fail2ban to block non-ssh traffic.  This was difficult to get working because there aren't all that many explanations on the inner workings of this tool.  The readmes aren't exactly descriptive.  With a lot of web searches I got things working.

The jail list shows that I've enabled the following filters: 

root@linode:/var/log# fail2ban-client status
|- Number of jail:      15
`- Jail list:   apache, apache-multiport, apache-noscript, apache-overflows, courier-auth, courier-smtp, dropbear, mysqld-auth, php-url-fopen, postfix, postfix-sasl, sasl, ssh-ddos, sshd, xinetd-fail

Of them, I've seen traffic blocked from apache-noscript, apache-overflows, ssh-ddos, and sshd.

The rest of the filters have not captured any logs, but that just means conditions haven't been met to block/log.  In fact, I've only seen one apache-overflows alert trigger.

What I've been doing is trying to correlate the Fail2ban log entries to the service logs (ie, an alert is generated against the apache-noscript filter and I grep the apache logs for the IP to see what occurred.

Here's an example Fail2ban alert:

root@linode:/var/log# cat /var/log/fail2ban.log | grep 'script' | grep 'Ban'
2017-03-23 00:00:00,322 fail2ban.actions        [26381]: NOTICE  [apache-noscript] Ban

Here's the Apache log entries for that IP:

root@linode:/var/log# cat apache2/access.log | grep - - [22/Mar/2017:18:13:56 +0000] "GET //wp-includes/registration-functions.php HTTP/1.1" 500 185 "-" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" - - [22/Mar/2017:23:59:59 +0000] "GET //wall_login.php?login=cmd HTTP/1.1" 404 510 "-" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko"

Here's how the apache-noscript section looks within my jail.local file:


enabled  = true
port     = http,https
filter   = apache-noscript
logpath  = /var/log/apache2/error.log
maxretry = 1
findtime = 60
bantime  = -1

You see two log entries.  In this case, the filter is looking for more than one violation in a 60 second timeframe.  Violators are banned indefinitely.

The logs look hokey when comparing against the apache-noscript configuration within the jail.local file, but it's correct.  The logs look like this attack occurred after the offending IP connected to the Apache server twice within five hours and was banned at midnight on the second attempt.  That's not what happened.  The logs are deceiving.  The attacks (defined by maxretry) must occur within the findtime value.  Since the maxretry is 1 and the findtime is 60, a ban occurred when the offending IP tried a consecutive attack within 60 seconds (at midnight).  Apache only logged the first attempt (at midnight).  After the second attempt occurred, a ban was set before Apache could log the attempt.

The ssh-ddos filter discovers distributed attacks relating to brute-forcing of SSH connections.  There are also many other filters relating to ssh but they're pretty much redundant in that they block the same activity, so if I have several of them enabled, I end up with redundant alerts in my log file.  I've turned off the ones that generate duplicate alerts.

I also need to back up my configuration files so that I don't have to experiment with and tune the setup if I happen to lose my configuration files later and have to reinstall Fail2ban.  That would suck.

Thursday, January 12, 2017

Ubuntu 16.04, Fail2ban and Postfix...Ugh...

So, I've been trying to get Fail2ban working with Postfix.

It has been a bit of a hassle and I'm still not sure if I've got it working properly.

First, when I edit jail.conf to enable the postfix configuration, Fail2ban stops working when I add a ports listing.

Second, I've got it running without errors but can see that Fail2ban isn't blocking incoming bruteforcing attempts on Postfix.  I can see the attacks happening in the mail logs but can't see Fail2ban blocking them.  The Postfix jail is showing when I run "fail2ban-client status".

I've a crapload of studying up to do, as I just found the man pages for fail2ban-client.

I need to configure for FTP and HTTP as well.  SSH is already done.

UPDATE (1/15/2017) - I now have Fail2ban working with more than just SSH.  I'm running it to monitor Apache and Xinetd, as well as MySQL and php-url-fopen attacks.  But I'm stills struggling with getting it to track Postfix brute-forcing attempts.

Saturday, November 26, 2016

Slackware Box Stopped Working - Upgaded It and Now Slackware Won't Run

A few months ago, my Slackware box died.  It would no longer boot up (no error POST beeps, no BIOS bootup screen).  The system fans wouldn't even activate as they normally would.

I thought it was the power supply, so I replaced it.  Afterward, the system would begin the bootup process, I could hear the fans, but still couldn't see the BIOS bootup messages or access the setup screen.  I then replaced the motherboard.

It was previously running a Pentinum D 820 and Intel-based motherboard.  I replaced it with a spare, an Asus M4N98TD EVO running an AMD Phenom II X6 X1100T.

After then swap of the motherboards, I was able to get to the BIOS but the system would no longer boot up it's instance of Slackware (v14.1).  It would attempt to boot up Slackware but would run into a kernel panic condition:

slackware 14.1 kernel panic - not syncing: VFS: Unable to mount root fs on unknown block (8,3)

Instead of troubleshooting, I figured that I'd just do an install on that same system's spare hard drive.  I installed onto that drive and ran into the same message.

I then wiped both drives and focused on installing on just one drive.  I did this twice.  I ended up with the same error message both times.

 I then decided to do some research, as this was something I have never experienced in the past.

This page describes what I'm experiencing.  I'll highlight the relevant details:

In case your kernel does not include the driver for your root filesystem, or a driver for your SATA bus, or other stuff that is only built as modules, your kernel will panic if it boots and can not access the necessary disks, partitions and/or files. Typically, this looks like
VFS: Cannot open root device "802" or unknown-block (8,2)
Please append a correct "root=" boot option
Kernel Panic-not syncing: VFS: unable to mount root fs on unknown block(8,2)
and this means you will have to build an initrd or “Initial Ram Disk” containing the required modules. The location of the initrd is then added in the appropriate section of /etc/lilo.conf so that the kernel can find it when it boots, and is able to load the drivers it needs to access your disks.

The problem is, I can't follow the described steps because the system can't mount the drive, so I can't use the necessary tools to build an initrd, and can't edit the /etc/lilo.conf file (it's on the partition that won't mount).  mkinitrd is not on the install/rescue disk (well, it's not accessible as a command).

I'm to the point that I'm about to ask for assistance at, but will try to run Slackware 14.1 in VMware to see if I can build the files and put them on the partition I can access on the physical Slackware machine.  I'd still run into the problem of being able to edit /etc/lilo.conf, though (that file resides on the / partition, which can't be accessed until I'm able to fix the issue.  Or, I can find a Slackware-based live CD that'll have the proper files and drivers that will mount the partitions and allow me to make the needed fixes.

Hopefully I'll be able to fix this issue without too much hoop-jumping.

UPDATE:  It is working now.  It's been up the last few days with no load.  I'm still trying to find out why it wasn't working, but it certainly has to do with the partition layout I opted to use, which is weird because I was using the same partition scheme I've been using for years (a /boot partiton, a swap partition, and a / partition).  I ended up using a swap partition and / partition (no dedicated /boot since it appears that may've been the issue...I may be able to add it after the fact).

Tuesday, November 24, 2015

IPTables and SSH - Resetting Brute Force Traffic, UPDATE

I showed how to determine IPs that were bruteforcing in my last blog entry, and how to block the majority of it (the posts are here and here).

In checking my logs over the last week, I see some activity that's making it through the firewall.  To be honest, I doubt I'll be able to filter ALL the bruteforcing using the rules I've currently in place.  I did tweak them a bit, changing the 4th and 5th rules' threshold to 60 seconds (vs 30 seconds).  It helped some but then I saw one particular IP blast through the firewall's ruleset like it was designed to go low and slow.

Here are the IPs that have been bruteforcing port 22 this week, as well as their hit counts:
root@linode:~/ssh_brute_force_logs# sed 's/S.*=//' combined1 | uniq -c
IP was the most prevalent by far, with IP coming in second.

I'm to the point where I should create a script that would determine any IP with a hit count of, for example, 1000, and block them.  Or I can just use opensource products like fail2ban (remember, denyhosts isn't working because tcpwrapper support isn't compiled into inetd.conf on Ubuntu 14.04 LTS.).  So, I've installed fail2ban and will watch to see how it blocks things with the default setting before making changes to the configuration.  The cool thing about fail2ban is that it will watch and block more than just SSH activity.

Am I worrying too much about this?  Some would say, YES!  I'm doing this for learning purposes and also to get a better understanding of how such bruteforcing is evolving, but I don't really want to have to deal with a box that has been hacked, either.  Every admin has the responsiblity of doing their utmost to ensure their machines aren't hacked...that means that the admin needs to be proactive in placing defenses to deter or slow down attacks.


Already, I see the following in the fail2ban logs:

2015-11-24 22:33:10,453 fail2ban.actions: WARNING [ssh] Ban
2015-11-24 22:33:10,458 fail2ban.actions: WARNING [ssh] Ban
2015-11-24 22:33:10,462 fail2ban.actions: WARNING [ssh] Ban
2015-11-24 22:33:10,466 fail2ban.actions: WARNING [ssh] Ban
2015-11-24 22:43:11,085 fail2ban.actions: WARNING [ssh] Unban
2015-11-24 22:43:11,090 fail2ban.actions: WARNING [ssh] Unban
2015-11-24 22:43:11,094 fail2ban.actions: WARNING [ssh] Unban
2015-11-24 22:43:11,098 fail2ban.actions: WARNING [ssh] Unban

Fail2ban banned and then unbanned the IPs 10 minutes later (default settings).

What I need to study is how Fail2ban is implementing the bans.  Also, do I want to outright permanently ban these that even feasible?  We'll see.

UPDATE 2 - 12/24/2016:

I checked on things.  I checked today's bans since I've been out of touch with the server and Linux in general (been doing work-related things but nothing close to this type of stuff).  I didn't even remember how log Fail2ban was blocking but I see I changed the default from 10 minutes to 7 days.  I was too lazy to check the logs, so what I did was pick an IP from the ban logs and grep for it.  I saw it banned and saw it unbanned 7 days later.

The logs go as far back as 11/20.  The trend seems to be mid teens for Nov, although I saw several spikes to the mid twenties.  I'm just checking to see what the norm is, so that as I begin to get back into the swing of things, I have a baseline to compare.  It would be nice if I could archive these files somehow.

Tuesday, November 17, 2015

IPTables and SSH - Resetting Brute Force Traffic

In my prior blog post, I described an issue with brute force attempts of my Postfix server.  Due to the weird way xinetd was handling services and me not wanting to spend inordinate amounts of time tuning xinetd's tcpwrapper functionality, I uninstalled xinetd.  I will continue to rely on Postfix to reject the traffic.

That does me no good when it comes to SSH, though.  With Slackware, I rely on either tcpwrapper or packages that leverage tcpwrappers.  This works well when it comes to blocking SSH, but I can't use that method on my Ubuntu server without substantial rebuilding (building by source, which I want to avoid).

My solution?  Leverage IPtables.

I wanted to use a different method than maintaining block lists.  They work but I wanted to learn something new.  I found this.

I added the following rules to the firewall:

I added the following rules:
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -m recent --rcheck --seconds 30 --hitcount 4 --rttl --name SSH --rsource -j REJECT --reject-with tcp-reset
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -m recent --rcheck --seconds 30 --hitcount 3 --rttl --name SSH --rsource -j LOG --log-prefix "SSH brute force "
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -m recent --update --seconds 30 --hitcount 3 --rttl --name SSH --rsource -j REJECT --reject-with tcp-reset
-A INPUT -i eth0 -p tcp -m tcp -s --dport 22 -j ACCEPT
The first rule tells the system:

TCP packets are going to come in, that will attempt to establish an SSH connection.  Mark them as SSH.  Pay attention to the source of the packet.

The second rule says:

If a packet attempting to establish an SSH connection comes, and it's the fourth packet to come from the same source in thirty seconds, just reject it with prejudice and stop thinking about it.

The third and fourth rules mean:

If an SSH connection packet comes in, and it's the third attempt from the same guy in thirty seconds, log it to the system log once, then immediately reject it and forget about it.

The 5th rule allows SSH traffic from a certain domain to pass into the environment.

It has cut down my syslogs significantly.  I'm pretty sure the firewall isn't suffering, because all it's doing is tracking the source IPs that are coming in on destination port 22.  I've not noticed the server suffering under any significant load.  Some IPs are still making it through, but I'm counting between 10-20 IP entries in a 24 hour period.  I can tell they're making it through because the traffic is hitting my clean-up rule.  I've had 8 make it through all the rules before being blocked by the clean-up rule.

root@linode:/var/log# grep -i 'dpt=22' syslog | grep -i 'clean-up rule'

Nov 17 07:03:20 linode kernel: Clean-up Rule - BLOCKED: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:84:78:ac:0d:a6:41:08:00 SRC= LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=32204 DPT=22 WINDOW=0 RES=0x00 RST URGP=0
Nov 17 07:05:13 linode kernel: Clean-up Rule - BLOCKED: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:84:78:ac:0d:79:c1:08:00 SRC= DST= LEN=40 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=46595 DPT=22 WINDOW=0 RES=0x00 RST URGP=0
Nov 17 07:10:17 linode kernel: Clean-up Rule - BLOCKED: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:84:78:ac:0d:a6:41:08:00 SRC= DST= LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=32203 DPT=22 WINDOW=0 RES=0x00 RST URGP=0
Nov 17 11:19:15 linode kernel: Clean-up Rule - BLOCKED: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:84:78:ac:0d:a6:41:08:00 SRC= DST= LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=49995 DPT=22 WINDOW=0 RES=0x00 RST URGP=0
Nov 17 16:55:46 linode kernel: Clean-up Rule - BLOCKED: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:84:78:ac:0d:a6:41:08:00 SRC= DST= LEN=40 TOS=0x00 PREC=0x00 TTL=116 ID=25601 DF PROTO=TCP SPT=16895 DPT=22 WINDOW=16591 RES=0x00 ACK FIN URGP=0
Nov 17 16:55:46 linode kernel: Clean-up Rule - BLOCKED: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:84:78:ac:0d:a6:41:08:00 SRC= DST= LEN=140 TOS=0x00 PREC=0x00 TTL=116 ID=25772 DF PROTO=TCP SPT=16895 DPT=22 WINDOW=16591 RES=0x00 ACK PSH FIN URGP=0
Nov 17 19:17:40 linode kernel: Clean-up Rule - BLOCKED: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:84:78:ac:0d:79:c1:08:00 SRC= DST= LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=32205 DPT=22 WINDOW=0 RES=0x00 RST URGP=0
Nov 17 19:24:37 linode kernel: Clean-up Rule - BLOCKED: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:84:78:ac:0d:79:c1:08:00 SRC= DST= LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=32202 DPT=22 WINDOW=0 RES=0x00 RST URGP=0

root@linode:/var/log# grep -i 'dpt=22' syslog | grep -i 'clean-up rule' | wc -l

Thursday, November 12, 2015

IPs hammering Postfix

So, I've been lax in checking my logs lately, but decided to check them last night on my Linode server, which is running Ubuntu.

I'm running PSAD but for some reason, it stopped reporting scanning IPs to me. I've fixed that but it won't account for the last few months of PSAD not working.

What I'm seeing in the syslogs are connection attempts to my Postfix mail server. The server appears to have automatically dropped the traffic. I see no indication that Iptables tried to block the traffic, but this is probably due to needing to have Postfix open to the internet.

I can either rely on Postfix to continue to kill these connection attempts or I can add them to the firewall. First, I've to compile a listing of unique IPs (there were many duplicates as well as other connection information that needed to be removed).

The first thing I did was filter any logs not relevant to Postfix and redirect that output to a file. In fact what I did was grep the syslogs using the following filter: 'disconnect from unknown', which showed me every instance of Postfix disconnecting an IP from communication:

root@linode:/var/log# grep 'disconnect from unknown' syslog > /root/postfix_drops

Nov 12 05:44:12 linode postfix/smtpd[15130]: disconnect from unknown[]
Nov 12 05:46:10 linode postfix/smtpd[15176]: disconnect from unknown[]
Nov 12 05:57:11 linode postfix/smtpd[15366]: disconnect from unknown[]
Nov 12 06:03:48 linode postfix/smtpd[15424]: disconnect from unknown[]
Nov 12 06:05:49 linode postfix/smtpd[15467]: disconnect from unknown[]
Nov 12 06:17:03 linode postfix/smtpd[15677]: disconnect from unknown[]
Nov 12 06:17:20 linode postfix/smtpd[15677]: disconnect from unknown[]
Nov 12 06:21:59 linode postfix/smtpd[15861]: disconnect from unknown[]
Nov 12 06:22:20 linode postfix/smtpd[15861]: disconnect from unknown[]
Nov 12 06:23:41 linode postfix/smtpd[15861]: disconnect from unknown[]
Nov 12 06:35:21 linode postfix/smtpd[16335]: disconnect from unknown[]
Nov 12 06:48:00 linode postfix/smtpd[16516]: disconnect from unknown[]

The small snippet above shows that there are repeat offenders.  That'll need to be sorted out but we'll do that later.  Right now, all I need are the IPs.  I've to remove everything else.

I then use the following to remove a good bit of the logs:

root@linode:/var/log# nawk '{print $8}' /root/postfix_drops > /root/postfix_drops_2

The result is:


Note that everytime I'm making a change to the output, I'm also redirecting the output to a new file.

I then remove all instances of 'unknown' from the file (I had to add spaces between the the pointed brackets and the word "unknown", otherwise neither will render...remove the spaced for the code to work properly):

root@linode:/var/log# sed 's/\< unknown \>//g' /root/postfix_drops_2 > /root/postfix_drops_3


All that's left are removing the brackets:

root@linode:/var/log# sed 's/\[//g;s/\]//g' /root/postfix_drops_3 > /root/postfix_drops_4

I then sort the file while also only wanting to see unique IPs:

root@linode:/var/log# sort -u /root/postfix_drops_4

I started out with 394 lines and ended up with 94 unique IPs, across two syslog files.  I still have to do the archived syslog files.  I also saw that the 94 IPs show many IPs that are in sequence, which means that the culprits are trying to distribute the attacks over a range of IPs (I've posted similar attacks when analyzing SSH logs).

The resulting log file is here.

So, now, I can either be happy with how Postfix is handling the traffic or add the IPs to my firewall ruleset.  This was more of an exercise in seeing how well I could quickly assess the nature of the traffic in question and I'm happy (it was simple enough, although I'm out of practice in doing such things).


1. I scrubbed the archived syslog files and the total, including those IPs I already scrubbed, is 959, and when sorted uniquely, there are 301 IPs.

2.  I added the first batch of IPs to /etc/hosts.deny for tcpwrappers to block.  I also found that I couldn't use tcpwrappers since I didn't compile postfix with tcpwrapper support, so I installed xinetd (which can wrap services inside itself with some configuration).  I'll monitor to see if the IPs will be blocked and if it doesn't work, I'll add the IPs to the firewall's ruleset.

Saturday, May 16, 2015

Linux - User Account Management, Part II

In my post titled, "Linux - User Account Management, Part I," I talked about how to check a user's account for expiration.  I set a user account to expire on 5/15/2015.  It is now 5/16/2015.  The user account has expired.  When I attempt to log into this account, it shows as expired:
ron@slackbox:~$ su nor
Your login has expired.  Contact the system administrator.
 To re-enable the account, I will use either of the following commands:
usermod -e yyyy-mm-dd username
chage -E yyyy-mm-dd username 
When running the chage command, there's no message or prompt after execution. I'm able to log back in without issue.

Next, we'll learn how to set the number of days until a password change is required.  We'll set the password to expire for 30 days:
ron@slackbox:~$ron@slackbox:~$ sudo chage -M 30 nor
ron@slackbox:~$ sudo chage -l nor
Last password change                                 : May 10, 2015
Password expires                                     : Jun 09, 2015
Password inactive                                    : never
Account expires                                      : Dec 31, 2015
Minimum number of days between password change       : 0
Maximum number of days between password change       : 30
Number of days of warning before password expires    :  7  
We'll check this account again after 30 days and use the 'password -u nor' command to re-enable the expired password at that time.