More Postfix Success

I've been delving into why I don't see Postfix bans in my logs.  I think it has to do with the filters that came with my install of Fail2ban...they don't work for Ubuntu.

I looked at my logs and saw a ton of bruteforce attempts against the SMTP service, so I know for a fact that Fail2ban should be blocking these attempts.

I found this page and wanted to test to see if it's filter's regex would work on my server, so created the filter based on what was on that page then edited my jail.local file.  I then restarted Fail2ban but also wanted to see if the filter works, so I ran this:




--------------------------------------------------------

root@linode:/var/log# fail2ban-regex /var/log/mail.log /etc/fail2ban/filter.d/postfix-auth.conf

Running tests
=============

Use   failregex filter file : postfix-auth, basedir: /etc/fail2ban
Use         log file : /var/log/mail.log
Use         encoding : UTF-8


Results
=======

Failregex: 1526 total
|-  #) [# of hits] regular expression
|   1) [1526] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\] )?(?:@vserver_\S+ )?(?:(?:\[\d+\])?:\s+[\[\(]?postfix/smtpd(?:\(\S+\))?[\]\)]?:?|[\[\(]?postfix/smtpd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID \d+ \S+\])?\s*lost connection after .*\[\]$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [10563] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: Year)?
`-

Lines: 10563 lines, 0 ignored, 1526 matched, 9037 missed [processed in 0.88 sec]
Missed line(s): too many to print.  Use --print-all-missed to print all 9037 lines

--------------------------------------------------------

This time I had matches.  The last few days of me trying this with other filters or editing the canned filters netted me nothing.

Then I checked my server's fail2ban logs:

--------------------------------------------------------

 2017-03-29 21:53:56,987 fail2ban.filter         [12346]: INFO    [postfix-auth] Found 156.67.106.244
2017-03-29 21:53:57,037 fail2ban.filter         [12346]: INFO    [postfix-auth] Found 156.67.106.244
2017-03-29 21:53:57,971 fail2ban.actions        [12346]: NOTICE  [postfix-auth] Ban 156.67.106.244
2017-03-29 22:03:56,413 fail2ban.filter         [12346]: INFO    [postfix-auth] Found 105.112.3.167
2017-03-29 22:07:03,231 fail2ban.filter         [12346]: INFO    [postfix-auth] Found 220.178.1.34
2017-03-29 22:12:31,667 fail2ban.filter         [12346]: INFO    [postfix-auth] Found 66.23.212.157

--------------------------------------------------------

So it is working.

Why do I want use Fail2ban to block bad traffic going to my SMTP service?  Well, when I checked one of those IPs above, just to see how noisy it was in the log files, this is what I saw:

root@linode:/var/log# grep 156.67.106.244 mail.log | grep connect | wc -l
1741
root@linode:/var/log# grep 156.67.106.244 mail.log.1 | grep connect | wc -l
1333
root@linode:/var/log# zgrep 156.67.106.244 mail.log.*.gz | grep connect | wc -l
10931

A grand total of 14,005 connection attempts between the 13th and 29th of March.  The filter is configured to block if more than 2 attempts occur in a 5 minute span of time, so it should now block most of these.  I'll watch to see if that filter's parameters work sufficiently, but at least I've a working filter now!

Ubuntu 16.04, Fail2ban and Postfix...Ugh...

So, I've been trying to get Fail2ban working with Postfix.

It has been a bit of a hassle and I'm still not sure if I've got it working properly.

First, when I edit jail.conf to enable the postfix configuration, Fail2ban stops working when I add a ports listing.

Second, I've got it running without errors but can see that Fail2ban isn't blocking incoming bruteforcing attempts on Postfix.  I can see the attacks happening in the mail logs but can't see Fail2ban blocking them.  The Postfix jail is showing when I run "fail2ban-client status".

I've a crapload of studying up to do, as I just found the man pages for fail2ban-client.

I need to configure for FTP and HTTP as well.  SSH is already done.

UPDATE (1/15/2017) - I now have Fail2ban working with more than just SSH.  I'm running it to monitor Apache and Xinetd, as well as MySQL and php-url-fopen attacks.  But I'm stills struggling with getting it to track Postfix brute-forcing attempts.

IPs hammering Postfix

So, I've been lax in checking my logs lately, but decided to check them last night on my Linode server, which is running Ubuntu.

I'm running PSAD but for some reason, it stopped reporting scanning IPs to me. I've fixed that but it won't account for the last few months of PSAD not working.

What I'm seeing in the syslogs are connection attempts to my Postfix mail server. The server appears to have automatically dropped the traffic. I see no indication that Iptables tried to block the traffic, but this is probably due to needing to have Postfix open to the internet.

I can either rely on Postfix to continue to kill these connection attempts or I can add them to the firewall. First, I've to compile a listing of unique IPs (there were many duplicates as well as other connection information that needed to be removed).

The first thing I did was filter any logs not relevant to Postfix and redirect that output to a file. In fact what I did was grep the syslogs using the following filter: 'disconnect from unknown', which showed me every instance of Postfix disconnecting an IP from communication:

root@linode:/var/log# grep 'disconnect from unknown' syslog > /root/postfix_drops

Nov 12 05:44:12 linode postfix/smtpd[15130]: disconnect from unknown[198.206.134.152]
Nov 12 05:46:10 linode postfix/smtpd[15176]: disconnect from unknown[203.157.41.130]
Nov 12 05:57:11 linode postfix/smtpd[15366]: disconnect from unknown[91.205.167.98]
Nov 12 06:03:48 linode postfix/smtpd[15424]: disconnect from unknown[198.206.134.152]
Nov 12 06:05:49 linode postfix/smtpd[15467]: disconnect from unknown[155.208.242.236]
Nov 12 06:17:03 linode postfix/smtpd[15677]: disconnect from unknown[203.122.41.195]
Nov 12 06:17:20 linode postfix/smtpd[15677]: disconnect from unknown[112.175.103.29]
Nov 12 06:21:59 linode postfix/smtpd[15861]: disconnect from unknown[122.154.46.172]
Nov 12 06:22:20 linode postfix/smtpd[15861]: disconnect from unknown[155.208.242.236]
Nov 12 06:23:41 linode postfix/smtpd[15861]: disconnect from unknown[198.206.134.152]
Nov 12 06:35:21 linode postfix/smtpd[16335]: disconnect from unknown[122.154.46.172]
Nov 12 06:48:00 linode postfix/smtpd[16516]: disconnect from unknown[203.157.41.130]

The small snippet above shows that there are repeat offenders.  That'll need to be sorted out but we'll do that later.  Right now, all I need are the IPs.  I've to remove everything else.

I then use the following to remove a good bit of the logs:

root@linode:/var/log# nawk '{print $8}' /root/postfix_drops > /root/postfix_drops_2

The result is:

unknown[91.205.167.98]
unknown[198.206.134.152]
unknown[155.208.242.236]
unknown[203.122.41.195]
unknown[112.175.103.29]
unknown[122.154.46.172]
unknown[155.208.242.236]
unknown[198.206.134.152]
unknown[122.154.46.172]
unknown[203.157.41.130]

Note that everytime I'm making a change to the output, I'm also redirecting the output to a new file.

I then remove all instances of 'unknown' from the file (I had to add spaces between the the pointed brackets and the word "unknown", otherwise neither will render...remove the spaced for the code to work properly):

root@linode:/var/log# sed 's/\< unknown \>//g' /root/postfix_drops_2 > /root/postfix_drops_3

[155.208.242.236]
[203.122.41.195]
[112.175.103.29]
[122.154.46.172]
[155.208.242.236]
[198.206.134.152]
[122.154.46.172]
[203.157.41.130]

All that's left are removing the brackets:

root@linode:/var/log# sed 's/\[//g;s/\]//g' /root/postfix_drops_3 > /root/postfix_drops_4

203.122.41.195
112.175.103.29
122.154.46.172
155.208.242.236
198.206.134.152
122.154.46.172
203.157.41.130

I then sort the file while also only wanting to see unique IPs:

root@linode:/var/log# sort -u /root/postfix_drops_4

94.156.37.145
94.156.37.146
94.156.37.147
94.156.37.148
94.156.37.149
94.156.37.150

I started out with 394 lines and ended up with 94 unique IPs, across two syslog files.  I still have to do the archived syslog files.  I also saw that the 94 IPs show many IPs that are in sequence, which means that the culprits are trying to distribute the attacks over a range of IPs (I've posted similar attacks when analyzing SSH logs).

The resulting log file is here.

So, now, I can either be happy with how Postfix is handling the traffic or add the IPs to my firewall ruleset.  This was more of an exercise in seeing how well I could quickly assess the nature of the traffic in question and I'm happy (it was simple enough, although I'm out of practice in doing such things).

UPDATE:

1. I scrubbed the archived syslog files and the total, including those IPs I already scrubbed, is 959, and when sorted uniquely, there are 301 IPs.

2.  I added the first batch of IPs to /etc/hosts.deny for tcpwrappers to block.  I also found that I couldn't use tcpwrappers since I didn't compile postfix with tcpwrapper support, so I installed xinetd (which can wrap services inside itself with some configuration).  I'll monitor to see if the IPs will be blocked and if it doesn't work, I'll add the IPs to the firewall's ruleset.

PSAD - DoS'd my Linode and my G-mail account

Yes, I DoS'd myself.

How?

I was tuning my firewall so that PSAD wouldn't alert on traffic from my static IP.  This was caused by me using a half-baked firewall policy (ie, the firewall was allowing too much and not blocking what it should've).  So, while making the policy more secure, I ended up blocking some of DNS, which the firewall blocked via the clean-up rule (deny by default policy on all chains).  

I didn't double-check my work, and 24 hours later, I was checking the server via the admin console and saw that the disk I/O was extremely high when looking at the system graphs.  CPU utilization was also wayyy up.  

I initially couldn't figure out what was going on, until I used Webmin to access the server and found that it was taking forever for the server to resolve the domain address.  As well, there were like 30,000 e-mails in the Postfix e-mail queue.  I was basically spamming the hell out of Gmail and DShield.  I'd begun to wipe out all the Gmail notifications, but I soon realized that I wasn't making any headway, so I killed PSAD, cleared ALL the mail in the queue, then restarted PSAD.  It was still generating e-mails, though, so I turned off e-mail notifications, as well as syslog notifications.  I also killed my DShield log feed.  THEN I fixed DNS by just rolling back to a known good policy...then I told PSAD to not log on port 53/UDP (no real need to log that traffic anyways, unless it hits the catch-all rule, but that wouldn't happen now since I fixed DNS within the policy).

It took quite awhile for Gmail to finish processing the e-mails (the ones that I couldn't kill via Postfix).

I just now re-enabled syslogging and the DShield log feed, but may have to reach out to the SANS team to see if they can remove all DNS traffic that was logged by my static IPs.

I think I've everything fixed now.

Postfix Install, OSSIM, Slack 14, Ubuntu, and VPNs

This isn't really a technical post, but I did want to share that I have Postfix running on my server.  I'd never had the need to run my own mail server until I moved my wigglit.com domain.  It was initially hosted at 1and1.com, but I got fed up with their service (or lack thereof).  I had several e-mail accounts set up there and still needed them to stay active, so I was pretty much forced to migrate the accounts as well as the domain.  The domain migration was pretty simple.  The Postfix install was much more difficult, even when using Webmin to set it up.  I used a Ubuntu tutorial (searched on 'webmin', 'ubuntu', 'postfix', and 'configuration') and used it exclusively to set up the server.  I think I have it tuned  pretty well so far, only I found some bounced e-mails going back maybe a month or so...I fixed those today.  Those weren't actually related to Postfix, though.  When I stood up the new server and domain, I forgot to adjust the scripts that kicked off the e-mails (cronjobs).  I'll double-check tomorrow, but I think I've fixed those (was able to test the cronjob successfully...generated a test e-mail).  I've since been editing the main.cf file to make configuration changes (and restarting the mail server afterward).

I've also been trying to use OSSIM, but I think I need a dedicated machine.  I tried to use an install of it within VirtualBox, with very limited success.  It seems it needs considerable resources and doesn't run well on a virtual instance with limited CPU/memory resources.  I ran VirtualBox on my M17xR3...that machine definitely has enough horsepower, but only has 8GB of RAM...it may need a bit more so that I can give OSSIM ample memory.  As well, my RAID 0 drive set may be hindering OSSIM.  I got a taste of it, though, and like it much better than Aanval.  Unfortunately, I don't have a good spare box at the moment, otherwise I'd be running it already.  That was my first time using VirtualBox, also...it's not that much different than VMware...much simpler, though.

So, Slackware v14.0 was released not long ago.  I took the liberty of installing it within VirtualBox.  It runs very nice!  I'm in the process of evaluating it and will soon upgrade my two v12.0 machines.  No, I'm not using Slackware on my public server.  I opted to use Ubuntu (v12.04) instead.  While I love Slack, I needed something less high-maintenance on the public server.  No complaints so far and it's been about a year since I flushed it and gave Ubuntu a try...no complaints whatsoever.  KISS is where it's at.

Lastly, since I've had success with Postfix, I plan to eventually start evaluating security tools again.  I've been out of the loop for awhile and need to push myself to continue to be familiar with Linux and security.  I've never used any of the VPN software before, so I plan to establish a VPN conduit between my LAN and my public server.  We'll see how that goes soon.