PSAD - DoS'd my Linode and my G-mail account

Yes, I DoS'd myself.

How?

I was tuning my firewall so that PSAD wouldn't alert on traffic from my static IP.  This was caused by me using a half-baked firewall policy (ie, the firewall was allowing too much and not blocking what it should've).  So, while making the policy more secure, I ended up blocking some of DNS, which the firewall blocked via the clean-up rule (deny by default policy on all chains).  

I didn't double-check my work, and 24 hours later, I was checking the server via the admin console and saw that the disk I/O was extremely high when looking at the system graphs.  CPU utilization was also wayyy up.  

I initially couldn't figure out what was going on, until I used Webmin to access the server and found that it was taking forever for the server to resolve the domain address.  As well, there were like 30,000 e-mails in the Postfix e-mail queue.  I was basically spamming the hell out of Gmail and DShield.  I'd begun to wipe out all the Gmail notifications, but I soon realized that I wasn't making any headway, so I killed PSAD, cleared ALL the mail in the queue, then restarted PSAD.  It was still generating e-mails, though, so I turned off e-mail notifications, as well as syslog notifications.  I also killed my DShield log feed.  THEN I fixed DNS by just rolling back to a known good policy...then I told PSAD to not log on port 53/UDP (no real need to log that traffic anyways, unless it hits the catch-all rule, but that wouldn't happen now since I fixed DNS within the policy).

It took quite awhile for Gmail to finish processing the e-mails (the ones that I couldn't kill via Postfix).

I just now re-enabled syslogging and the DShield log feed, but may have to reach out to the SANS team to see if they can remove all DNS traffic that was logged by my static IPs.

I think I've everything fixed now.

PSAD

I decided to give PSAD a spin on the Linode since I've never tried it before.  I'm impressed at the features of  it.  I've been running it maybe a bit over a month.  I get alerts whenever PSAD detects a scan or when it logs and drops specific traffic, so I'm aware of what's going on (instead of having to check my firewall logs).  One of the main reasons I decided to give PSAD a spin is because my fwanalog setup stopped working due to a code bug that affects Ubuntu v12.04.

One of the things I've been doing (I used to do this in the past) is I send my dropped logs to Dshield.org (or isc.sans.org).  One of PSAD's features enables me to send the logs, vs. using third-party or Dshield apps.

I noticed when sending my logs that I'm catching bidirectional traffic and my server IP is being flagged as a result.  Why?  I was blocking 118.0.0.0/8 (a large segment of APAC).  I was not only blocking but sending resets, which requires my firewall to send resets when means my IP is talking back, even though it's ending the session.  My firewall logs it as a drop.  To fix this, I just configured the firewall to drop the traffic, although I could've just changed the --log-prefix tag to something other than DROP, which by default PSAD looks for.  I'll monitor the Dshield logs to see how PSAD is now reporting.

Dshield Results From Log Donations

So, every day I submit logs to Dshield, I get a report from them with a breakdown of the submitted logs.

Here's an example:

For 2010-01-15 you submitted 496 packets from 136 sources hitting 2 targets.

Port Summary
============

Port  |  Packets  |  Sources  |  Targets  |      Service       |  Name
------+-----------+-----------+-----------+--------------------+-------------
 445 |        63 |        17 |         2 |       microsoft-ds | Win2k+ Server Message Block
 5900 |        31 |        15 |         2 |                vnc | Virtual Network Computer
 135 |        46 |        14 |         2 |              epmap | DCE endpoint resolution
 1080 |       162 |        13 |         2 |              socks | Proxy Server
  22 |        19 |        12 |         2 |                ssh | SSH Remote Login Protocol
  23 |         9 |         9 |         2 |             telnet |
 1433 |        12 |         9 |         2 |           ms-sql-s | Microsoft-SQL-Server
 3389 |        11 |         7 |         2 |   ms-term-services | MS Terminal Services
 3072 |        12 |         6 |         1 |        csd-monitor | ContinuStor Monitor Port
 4899 |         7 |         5 |         2 |             radmin | Remote Administrator default port
  25 |        20 |         5 |         2 |               smtp | Simple Mail Transfer
 3128 |        13 |         5 |         2 |         squid-http | Proxy Server
 8000 |        10 |         5 |         2 |              irdmi | iRDMI
 8080 |         7 |         4 |         2 |           http-alt | HTTP Alternate (see port 80)
 139 |         7 |         3 |         2 |        netbios-ssn | NETBIOS Session Service
 7212 |         7 |         3 |         2 |                    |
  21 |         5 |         3 |         1 |                ftp | File Transfer [Control]
  80 |         6 |         2 |         1 |                www | World Wide Web HTTP
 2967 |         2 |         2 |         1 |          ssc-agent | Symantec System Center
 1024 |         6 |         2 |         1 |                    |


Port Scanners
=============

   source     | Ports Scanned | Host Name
---------------+---------------+------------
 173.192.192.92|          10   | 173.192.192.92-static.reverse.softlayer.com
 221.192.199.35|           6   |
 78.159.112.84|           5   |
 77.223.143.18|           4   | 77-223-143-18.netdirekt.com.tr
 222.215.230.49|           4   |
 205.209.161.68|           3   |
 67.51.137.218|           2   |
 173.66.248.120|           2   | auth03.cs.net
188.132.196.173|           2   | datacenter-173-196-132-188.sadecehosting.net
 68.237.174.120|           2   | static-68-237-174-120.lsanca.fios.verizon.net
206.217.205.170|           2   | noptr.midphase.com
 66.159.229.149|           2   | netblock-66-159-229-149.dslextreme.com
222.208.183.218|           2   |
  66.160.182.5|           2   | system-5.squaw.com
   64.38.82.20|           2   |
174.129.185.251|           2   | ec2-174-129-185-251.compute-1.amazonaws.com


Source Summary
==============

   source     | hostname  |packets|targets| all pkts | all trgs | first seen
---------------+-----------+-------+-------+----------+----------+-----------
  66.160.182.5|5.squaw.com|    54 |     1 |      143 |       66 | 01-08-2010
  79.125.50.62|azonaws.com|    33 |     1 |    14083 |       94 | 01-11-2010
 79.125.39.245|azonaws.com|    27 |     1 |     5974 |       92 | 01-11-2010
 174.129.93.137|azonaws.com|    27 |     1 |    16623 |      102 | 01-06-2010
174.129.161.206|azonaws.com|    21 |     1 |     8258 |       99 | 01-11-2010
174.129.137.234|azonaws.com|    15 |     1 |     2328 |       90 | 01-14-2010
 221.192.199.35|           |    13 |     1 |    63162 |     2825 | 01-05-2010
  79.125.44.37|azonaws.com|    12 |     1 |     6155 |       88 | 01-12-2010
 173.192.192.92|ftlayer.com|    10 |     1 |   287815 |    25717 | 12-31-2009
 222.215.230.49|           |     9 |     2 |   225112 |     6288 | 05-28-2008
 79.125.32.165|azonaws.com|     9 |     1 |     2346 |       89 | 01-14-2010
 94.59.233.125|           |     7 |     1 |       29 |       13 | 01-15-2010
 77.223.143.18|rekt.com.tr|     7 |     1 |   120123 |    20906 | 12-28-2009
 78.159.112.84|           |     6 |     1 |     7356 |     3206 | 01-15-2010
188.132.196.173|hosting.net|     6 |     1 |     2606 |     1735 | 01-13-2010
118.161.243.145|c.hinet.net|     6 |     1 |       54 |        8 | 01-15-2010
   64.38.82.20|           |     5 |     1 |      894 |      446 | 01-15-2010
 205.209.161.68|           |     5 |     1 |      278 |      245 | 01-15-2010
204.236.194.181|azonaws.com|     5 |     1 |     8563 |       95 | 01-10-2010
204.236.244.234|azonaws.com|     4 |     1 |    10215 |      155 | 01-05-2010

All of this is valuable, and I can sometimes tune the IDS and FW based on the findings of these reports. There are other freeware tools that can do this type of data crunching, but I like the fact that if I'm donating logs, I'm getting a analysis report in return.

Now, the concern is that there's a lot of source IPs that appear to be owned by Amazon (Amazon Web Services). I'm hoping that most of these aren't EC2 hosts. If so, that indicates that Amazon has a security or abuse issue (or a combination of both). I'm hesitant to mention this to ISC since this may well be a trivial concern for them. Regardless of perception, I still believe this is more than likely an issue that should be pursued.

Dshield; Verizon FiOS

I've finally got this running.

I spent a bit of time with it last night and found that the dshield.cnf file had some errors.

I still need to tune it, though, because the script is reporting non-malicious web traffic to Dshield...I'll need to exclude all non-attacks and non-probes.

On another note, I'm at home today since we're getting FiOS installed. This service will replace Direct TV and Comcast. I'm looking forward to a dedicated internet connection. I'll be getting the 25/15 (down/up) internet pipe (YES) and two DVRs (I hope to replace the circa-2003 Tivo soon, with something better).

Worked on...

Reconfigured mnwclient so that I can provide FW logs to MyNetworkWatchman (which is similar to Dshield).

I'd much rather get Dshield working on the Linode but for some reason, I been having difficulties using their supplied clients. I'll continue to work on it, as I had it working prior to the last Linode upgrade.

With that in mind, at some time I'm going to have to upgrade the Linode from v12.0 to v12.2.

Night...