Saturday, September 15, 2007

Log Correlation

I thought I'd talk about the importance of log correlation. For instance, you've found that someone is continually pinging your server. You want to see if your box is responding to the pings. Usually, you'll know right off, since most people know if their firewall was configured to block pings...I'm just using pings as a quick example, though. Log correlation usually consists of checking, for instance, web server logs against firewall logs, or Snort logs against firewall and web server logs. This helps you understand what suspicious activity is actually doing and if your server/workstation responded (and how it responded, if it did).

I run Snort on a server, along with a web server, which is firewalled with IPTables. I have Snort report what it sees to a MySQL database, although it does record captures to a PCAP file locally. I also run Modsecurity, an application firewall that is designed to sniff and possibly block traffic going to/from web servers, mainly Apache. So, I've a ton of logs that I can correlate: Snort, Apache, IPTables, and Modsecurity logs.

We'll pick something easy. In fact, I'll fabricate some logs by generating some false alerts. I'll use 'wget', to visit my website. Keep in mind that what you see below gives you the advantage...you know what you're doing and looking for when we soon check the logs. This won't be the case when some stranger or worm hits your firewall or web server (or any other application server).

-bash-2.05b$ wget wigglit.ath.cx/root.exe
--23:24:05-- http://wigglit.ath.cx/root.exe
=> `root.exe'
Resolving wigglit.ath.cx... 66.160.141.30
Connecting to wigglit.ath.cx|66.160.141.30|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
23:24:06 ERROR 404: Not Found.

-bash-2.05b$
I used root.exe because there is an old vulnerability was was used to exploit the CodeRed worm of old. Now, let's check the web server's logs. I've tail'd my logs:

root@starchild:/var/log/apache# tail -f -n 100 access_log
12.123.12.123 - - [15/Sep/2007:23:34:35 -0400] "GET /root.exe HTTP/1.0" 404 202
12.123.12.123 - - [15/Sep/2007:23:34:35 -0400] "GET /root.exe HTTP/1.0" 404 202 "-" "Wget/1.10.2"
You see that the communcation was rejected (404 code), as the traffic was deemed forbidden by the web server. This is usually a good indication, as the server didn't respond favorably to the attack.

Now, let us check the firewall logs. We already know that the firewall allowed the traffic, since the web server responded with a 404...if the traffic was being blocked, there would be no record of the attack in the logs, because the firewall would have intercepted the traffic before it reached the web server. We're checking the firewall logs just to be sure this guy hasn't done anything else that the web server didn't see:

Sep 12 17:28:34 starchild kernel: Connection attempt (UNPRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=34557 DF PROTO=TCP SPT=46656 DPT=10083 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:28:44 starchild kernel: Connection attempt (PRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=18168 DF PROTO=TCP SPT=40091 DPT=449 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:28:44 starchild kernel: Connection attempt (UNPRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=27053 DF PROTO=TCP SPT=36295 DPT=5060 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:28:54 starchild kernel: Connection attempt (PRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=29291 DF PROTO=TCP SPT=47590 DPT=342 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:28:54 starchild kernel: Connection attempt (UNPRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=11788 DF PROTO=TCP SPT=43604 DPT=1519 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:29:04 starchild kernel: Connection attempt (PRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=17600 DF PROTO=TCP SPT=32783 DPT=577 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:29:04 starchild kernel: Connection attempt (UNPRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=51338 DF PROTO=TCP SPT=47879 DPT=18187 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:29:14 starchild kernel: Connection attempt (PRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=58520 DF PROTO=TCP SPT=41983 DPT=517 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:29:14 starchild kernel: Connection attempt (UNPRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=20255 DF PROTO=TCP SPT=53355 DPT=1986 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:29:24 starchild kernel: Connection attempt (PRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=28213 DF PROTO=TCP SPT=38543 DPT=978 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:29:24 starchild kernel: Connection attempt (UNPRIV): IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=10244 DF PROTO=TCP SPT=45624 DPT=11371 WINDOW=5840 RES=0x00 SYN URGP=0
Sep 12 17:29:37 starchild kernel: ICMP-request: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=28 TOS=0x00 PREC=0x00 TTL=42 ID=53002 PROTO=ICMP TYPE=8 CODE=0 ID=10683 SEQ=16615
Sep 15 17:38:25 starchild sshd[8455]: Accepted publickey for ron from ::ffff:12.123.12.123 port 33557 ssh2
Sep 15 23:21:08 starchild kernel: ICMP-request: IN=eth0 OUT= MAC=fe:fd:40:3e:e7:dc:00:0c:db:f5:90:00:08:00 SRC=12.123.12.123 DST=66.160.141.30 LEN=84 TOS=0x00 PREC=0x00 TTL=55 ID=40988 PROTO=ICMP TYPE=8 CODE=0 ID=614 SEQ=0

Quite a bit of stuff, huh? This looks to be a port scan! This is something that we didn't see in the Apache logs! Good thing we checked! Looks like this IP needs to be blocked with IPTables (which we won't do in this exercise).

Sadly, nothing shows in the Modsecurity logs, but we've enough information already. What about Snort? Because I've PCAP files, the search becomes a bit more involved. I'll spare the intimate details, but this is what we see:

[**] [1:1256:9] WEB-IIS CodeRed v2 root.exe access [**]
[Classification: Web Application Attack] [Priority: 1]
09/15-23:34:35.708546 0:C:DB:F5:90:0 -> FE:FD:40:3E:E7:DC type:0x800 len:0xB0
12.123.12.123:52753 -> 66.160.141.30:80 TCP TTL:56 TOS:0x0 ID:51589 IpLen:20 DgmLen:162 DF
***AP*** Seq: 0xAF32BB68 Ack: 0x3F319F7A Win: 0xFFFF TcpLen: 32
TCP Options (3) => NOP NOP TS: 288652007 1521931210
[Xref => http://www.cert.org/advisories/CA-2001-19.html]

23:34:35.708546 00:0c:db:f5:90:00 > fe:fd:40:3e:e7:dc, ethertype IPv4 (0x0800), length 176: IP (tos 0x0, ttl 56, id 51589, offset 0, flags [DF], length: 162) 12.123.12.123.52753 > 66.160.141.30.80: P [tcp sum ok] 2939337576:2939337686(110) ack 1060216698 win 65535
0x0000: fefd 403e e7dc 000c dbf5 9000 0800 4500 ..@>..........E.
0x0010: 00a2 c985 4000 3806 56c0 47b2 0aa0 42a0 ....@.8.V.G...B.
0x0020: 8d1e ce11 0050 af32 bb68 3f31 9f7a 8018 .....P.2.h?1.z..
0x0030: ffff e9c5 0000 0101 080a 1134 7ae7 5ab6 ...........4z.Z.
0x0040: d3ca 4745 5420 2f72 6f6f 742e 6578 6520 ..GET./root.exe.
0x0050: 4854 5450 2f31 2e30 0d0a 5573 6572 2d41 HTTP/1.0..User-A
0x0060: 6765 6e74 3a20 5767 6574 2f31 2e31 302e gent:.Wget/1.10.
0x0070: 320d 0a41 6363 6570 743a 202a 2f2a 0d0a 2..Accept:.*/*..
0x0080: 486f 7374 3a20 7769 6767 6c69 742e 6174 Host:.wigglit.at
0x0090: 682e 6378 0d0a 436f 6e6e 6563 7469 6f6e h.cx..Connection
0x00a0: 3a20 4b65 6570 2d41 6c69 7665 0d0a 0d0a :.Keep-Alive....

The first is the Snort alert file...this is a fast alert, with minimal detail (no packet capture). The second section is the full alert, including packet capture. It is also garbled (due to the hex code and the fact that this blog has issues with formatting) Note that my logs show no response. Apparently, my Snort install doesn't have a 404 signature. Again, the fact that we can correlate helps me when my Snort install lacks the data that I may have needed. I was able to look at the Apache logs to see the 404 when my Snort logs didn't show the return traffic.

Well, this concludes our chat about correlating existing logs. Note that any log files can be correlated. Correlation can also assist in tracking down network issues or issues with, for instance, a faulty Snort install (ahem). Although this discussion focused more on security, hopefully this helps someone understand their network or software architecture also.

`

Wednesday, September 12, 2007

Assessment of a Hardened Box

I thought I'd do a piece on *nix security. This will be a general guide on how to determine what ports have listening services and how to assess if those ports are available to the world wide web.

I like to do three different things:

1. Run Nmap against localhost (127.0.0.1).
2. Run Nmap against the machine's IP from another machine within the LAN.
3. Run Nmap against the machine from the internet.

Running Nmap against localhost can be deceiving, as the ports that are listening on the machine may not actually available to another machine, on or off the LAN. Note that 127.0.0.1 only pertains to the local machine. This is the loopback address that every machine uses to communicate to itself. I'll compare a localhost scan against a scan that was conducted from another machine on the LAN.

I'll scan localhost first:
-su-2.05b# nmap localhost

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2007-09-12 17:20 EDT
Interesting ports on localhost.home (127.0.0.1):
Not shown: 1674 closed ports
PORT STATE SERVICE
22/tcp open ssh
25/tcp open smtp
3306/tcp open mysql
5801/tcp open vnc-http-1
5901/tcp open vnc-1
6001/tcp open X11:1

Nmap finished: 1 IP address (1 host up) scanned in 10.662 seconds
Now, I'll scan the machine's IP from another machine:

root@slackbox:~# nmap 10.150.1.103

Starting Nmap 4.20 ( http://insecure.org ) at 2007-09-12 17:36 EDT
Interesting ports on delly (10.150.1.103):
Not shown: 1693 filtered ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
3306/tcp open mysql
5900/tcp closed vnc
MAC Address: 00:C0:4F:61:28:1F (Dell Computer)

Nmap finished: 1 IP address (1 host up) scanned in 22.277 seconds
Comparing the two scans, some things are readily apparent: port 25/tcp and 5801/tcp, 5901/tcp, and 6001/tcp are listening on the loopback device, yet they aren't listening on IP that is assigned to the machine. Also, note that certain ports are bound to the IP but not the loopback. The more important aspect of these two scans is the services listening on a non-loopback address, because services use IPs to communicate to other machines, not loopback devices.

So, TCP ports 25, 5801, 5901, and 6001 appear to be open. We can test this. Remember, we're concerned about whether these ports can be seen as open to the internet. We'll test this by using telnet. Below, I'm logged into another machine and attempting to telnet on port 25 to the machine we Nmap'd on localhost:

root@slackbox:~# telnet 10.150.1.103 25
Trying 10.150.1.103...
telnet: connect to address 10.150.1.103: Connection timed out
The connection timed out. It is pretty evident that the machine doesn't have services on 10.150.1.103:25 that just anyone can reach.

We can also test by running 'netstat -an' on the machine. Whle netstat doesn't connect to the port in question, it does indicate what interface services are using:

-su-2.05b# netstat -an | grep -i listen
tcp4 0 0 *.5801 *.* LISTEN
tcp4 0 0 *.5901 *.* LISTEN
tcp4 0 0 *.6001 *.* LISTEN
tcp4 0 0 *.3306 *.* LISTEN
tcp4 0 0 10.150.1.103.8118 *.* LISTEN
tcp4 0 0 10.150.1.103.80 *.* LISTEN
tcp4 0 0 127.0.0.1.25 *.* LISTEN
tcp4 0 0 *.22 *.* LISTEN
tcp6 0 0 *.22 *.* LISTEN
See port 25? What IP address is assigned to the service? Loopback! This means that this service isn't exposed to any other machine on the LAN or internet. The other services are listening (for example, *.22) but the '*' usually means that the service is assigned to an interface (which may have a number of IPs assigned to it). Ports 80 and 8118 have an IP assigned to the service, instead of '*'. This means that only IP 10.150.1.103 is assigned to that interface. Ports 80 and 8118 also do not show up when scanning localhost, but yet I can telnet to port 80 and 8118 when using the telnet command (and also browser), using 'telnet localhost 80' or 'telnet localhost 8118'. Port 80 is a web server. Port 8118 is a Privoxy proxy. The reason is probably because my PF install on that machine filters some loopback traffic. If I scan the IP of the machine, port 80 is then detected. The reason is because PF is configured to allow port 80 traffic to/from 'slackbox' from/to 10.150.1.103.

All of this is very elaborate. Everything has to be taken into consideration: what the firewall does/doesn't allow; what services run on 127.0.0.x; what services run on interfaces; what services run on IP addresses.

The last thing we'll do is scan the LAN from the internet. The results will be determined by what ports the gateway router/firewall are forwarding to machines within the LAN. For example, I know I've port 22/tcp open to the world because I've told my gateway firewall to forward any traffic on 22/tcp to 10.150.1.103. I also don't allow much else and also allow some traffic to be forwarded to different machines within the LAN. So, when a scan is being conducted from the outside, the whole LAN's security posture is taken into account. Now, let us scan from the internet:

Starting nmap V. 3.00 ( www.insecure.org/nmap/ )
Interesting ports on pool-xx-xxx-xx-xx.washdc.fios.verizon.net (xx.xxx.xx.xxx):
(The 1599 ports scanned but not shown below are in state: filtered)
Port State Service
22/tcp open ssh
3306/tcp open mysql
NICE! Of all the scans we've conducted thus far, this is the one that is important. The other scans paint the smaller picture, but this one puts it all together and is THE security assessment that one wants to conduct. So, you can see, ports 22 and 3306 are open to the world. This could be two different machines that need those ports open to the internet, or this could be one machine that has those two ports open to the world wide web. Note that while I can reach those ports, my firewall is configured to allow communication from the machine I scanned from. If a machine, from Google.com or any other IP/domain that is not accounted for within the FW policy, attempted the same thing, those ports wouldn't show as opened, as the firewall would drop the traffic. So, while the two ports appear to be open to the world in the above results, they actually aren't, as the traffic is being filtered.

I hope all of this has helped some in understanding security and how to assess your LAN gateway and hosts within the LAN. We've used Nmap, telnet, and netstat to determine the security posture of a given machine. There are other tools that I didn't mention for brevity, (such as socklist, tcpdump, snort, for example) but Unix and Linux machines offer much in the way of testing security of a machine.

`