Engineering Stories

On the way to work today, I remembered an occasion where a team member who'd left the company had been stockpiling 1U rackmount servers in storage.  He'd reimaged each server with a common image (each had different passwords, though).  I had a listing of passwords for each server, but the listed password for one particular server wasn't working and we needed to get access to that machine.  I couldn't just reimage the machine since, even though it shared a common image, it was prepped for deployment to a certain location and was configured for that specific site.  While I had a copy of the site-specific information, I just did not have the time to reimage the machine and reconfigure it...I saved that as a "last resort" option.

After a bit of research, was able to log in successfully.

I knew the BIOS wasn't locked down, so I went into the BIOS and enabled booting from CDROM.  I had a copy of a Linux CD which I put into the CDROM tray.  I then power-cycled the system.  I was able to use the live-CD to boot up the box.  I mounted the drive within the system and removed the encrypted password within /etc/passwd using 'vipw'.  I then shut the box down, removed the live-CD, then started the system.  I was immediately given a shell.  I then reset the password to what was on the passwords list for that particular system then finished the pre-deployment steps.

This is why I love Linux.  There's always an option.  I could NOT do this with one of the backup Windows servers we had.  That case was similar:  the system was a cold backup and was racked but powered down...it was a new system with a new image but customized for a specific role...it had yet to be used, though.  The password that we had for the device was apparently incorrect.  I even tried to crack the SAM file...that didn't work and I eventually had to reinstall (not reimage) Windows Server (forgot which version) onto the system again.  What made this much worse was that there wasn't an original cloning image to use, as well as the fact that the previous engineer hadn't maintained directions on how he configured the device.  So I had to use the trial-and-error method.  I eventually configured the OS properly and installed and configured the proper software (it was a CA eTrust AV server).  The whole time, the lead client was pestering, badgering, and being overly hostile.

In another case, another contractor had left the company.  He'd been administering a Nessus server that he installed on top of OpenBSD.  This contractor chose OpenBSD and was comfortable with working within a terminal session (as was I).  And really, the box didn't really have an abundance of resources anyways, so it was probably more robust without the GUI enabled.  I understood something of OpenBSD and was aware of how to conduct scans and how to view/store the scan results.  I even had a cron job running that would conduct the scans during maintenance windows.  Everything was working fine.  The same client lead couldn't operate the system because his *nix skills were seriously lacking.  Instead of asking for help/guidance, he directed another contractor to wipe the machine and install Red Hat with the GUI enabled so that he could operate the machine.  Data was not backed up.  The scanning data as well as configuration man-hours were wasted.

Another time, I was working a deployment issue where client remote hands were my remote hands/eyes.  They'd received our Snort sensor that we'd imaged, customized, and configured and had just finished racking and powering it up.  The remote hands did not know anything of how to operate within a terminal session.  I walked him through the process, spelling out the commands he needed to type.  The problem?  We built the machine and while testing it before we shipped, had logged into the machine via SSH.  When the machine was at the remote location, I could not establish an SSH session because the host key had changed.  In order for me to regain access, the remote hands had to remove the existing host key that was tied to the IP of my work machine...the host key resided on the Snort sensor that I was trying to log into.  What made me feel good was that one of the clients was logged into the bridge call and was listening.  After the call, she praised me for my knowledge of guiding the remote hands through the whole process without ever being able to view what was on his screen.  She also commented on how I guided him in what to type.  In this case, I could care less how much they were paying me (which wasn't really all that much)...I was happy that I was able to be of assistance and value.  That was payment enough.  That was one of the few bright days in working with that particular organization.  I soon took a dignified stance and left that contract.  To this day, I will not recommend any person I know to work at that particular location without giving them ample warning.

But the main reason for this post is to share that I love *nix (and why)!

BSD machine fixed!

So, I swapped a known working motherboard into the BSD machine.  It now works.  I also decided to use a quad core AMD AM2+ CPU that I had sitting around.  That's all I changed.

I'd originally thought the problem was related to the hard disk.  So, I decided the night before to disconnect the drives (it has two SATA drives), to determine if it were the real issue.  It still experienced the same symptoms after boot-up attempts, which told me it wasn't a hard disk issue.  I also swapped out the RAM with a known working chip with the same results when trying to boot-up.

So, either the the old CPU (a dual-core AMD...I forget the model) died or something on the motherboard died (or maybe there was a short somewhere?).  I left it running a live instance of Linux Mint, just to see if it stays stable over the next 24 hours.

Next, I need to reinstall FreeBSD (wondering if I should try some others as well, such as OpenBSD or Mint).  I wiped the drive, thinking that there was some corruption issue...shouldn't have done that.

BSD machine still not fixed; Slackware bullet-proof as usual...

So, I've had some time to play with my Slackware install.  I should actually upgrade to the latest, but I think I might try to get that BSD system back up this weekend.  I did upgrade firefox on the Slack machine, though...it was running a VERY old version (v2.x.x, I believe).  I'm running v12.0 now via my regular user account.

I'm tempted to install phpBB3 onto this machine (that's why I want to get into that BSD box...I'd just installed phpBB3 and had a very nice site that contained all my system and sysadmin notes that I've collected over the years...been using that software as a data repository since 2003 or so, on a very old system that runs phpBB2).

I've no real plans this weekend or maybe even the next (no autocross scheduled until next weekend and I'm opting out of that).  That should give me time to delve into the BSD issue as well as wiping the replacement system and installing the latest Slack.

Slackware Reunited!

Well, I'm back to using Slackware.  I don't know if that's actually proper to say, since I still use Slackware as an IDS for my LAN, but that box is pretty much just monitoring the network...nothing else.  I had an issue with my new FreeBSD box (it won't boot properly) and I needed another box, so I powered up an old machine that had Slackware v12 on it.  Yes, I'll upgrade to the latest as soon as I can, since everything seems to be out-of-date, such as my browsers and such.

I was able to get onto irc.freenode.net (was previously logging in via Xchat-aqua using my Macbook), but had a problem with D-bus:

ron@slackbox:~$ xchat
process 7948: D-Bus library appears to be incorrectly set up; failed to read machine uuid: Failed to open "/usr/local/var/lib/dbus/machine-id": No such file or directory
See the manual page for dbus-uuidgen to correct this issue.
  D-Bus not built with -rdynamic so unable to print a backtrace
Aborted

No, I don't use irssi or BitchX (I used to, awhile back).  Found that I could kickstart D-bus with the following command:

dbus-uuidgen --ensure

Dunno why this was an issue, since I didn't have the issue before I powered off the machine...maybe something broke during the power-down cycle of that last shutdown?  Dunno.

I'm glad to be delving in Slackware again...I love tinkering with different environments, but I'm pretty spread thin with Windows 7 (necessary evil to do my hardcore gaming), FreeBSD, Ubuntu (my colo server), and now Slackware...been meaning to reactivate an old box with OpenBSD on it also.  We'll see how I can cope with with all this.  :)

FreeBSD Pains

My 'new' BSD machine (FreeBSD 8.2) crapped out again.  Actually, I think either my wife or one of my kids accidentally shut it down.  Now it tries to boot up and immediately shuts down during the boot-up process.  I can't look at logs because it keeps shutting down.  I tried single mode and it does the same thing.  I've been trying to see the last line of the boot-up process before it shuts down...that's like trying to capture (with your mind) one frame of a film...very hard to do.  Well, it looks like it can't mount the root partition (just from what I've seen the milli-second before it shuts down).  I need to run fsck on it but I can't do that if it's not booting up properly into single mode.

I cheated and tried to boot up live install of Linux Mint, Ubuntu (both of the latter are on USB sticks), and FreeBSD (on DVD)...they must be trying to mount the drive that the FreeBSD install is on, because they all shut down too.  So, I'm going to try a rescue version of FreeBSD (for memory sticks).

Once I fix this, I'm sure there's a rc.conf setting that I'll need to set to force an fsck during boot-up if needed.  This has happened ever since I installed FreeBSD and I'm a bit irritated...this should be enabled by default so that someone doesn't get 'locked' out of their system.  :/

Missing me some Slackware...

I haven't played with Slackware in quite awhile.  I still run a server through Linode.com but I no longer have Slackware installed as an OS (I'm using Ubuntu for ease of use...yes, it is easier to maintain compared to Slackware and I've not run into any 'gotchas' yet).  I run one machine that has Slackware installed (it's sorely in need of an update, though) and it is being used as a NIDS system.  I've another machine with Slack on it that hasn't been turned on in months (it's OS version is even older than the other system).  I'll probably turn on this system and begin to use it again, but it is in very sore need of cleaning (it has 4-5 hard disks with data ALL over the place).

I'm trying to resist the urge to run Slackware in a VM on my Alienware system.  It will require me to probably get more RAM (I'm trying to resist that idea for now).  I do not want to attempt a native install, as I don't feel like experimenting to get Slack to work on that system.  The integrated and dedicated GPUs will probably be an immediate issue, as well as the fact that my system is running two 750GB drives in RAID0.  And, that is also my gaming system.  There's no real need for me to install Slackware natively on my system.  But, I will definitely install Cygwin, since I can leverage it's tools (such as GnuPG) without having to open a shell and have an internet connection.  Cygwin is the less complicated of the aforementioned options.

But I am missing using Slackware, which is why I've been trying to be more active at ##slackware on irc.freenode.net.  The thing is, I also have a fetish for Open- and FreeBSD, so I've been focusing on both of those the past few years.

Power Outages

There have been power outages here that have been taking down my lab equipment. This affected my new BSD machine. The drive became borked due to an unclean shutdown. After a few days, I got it back up again. It was a simple fix but one of the other machines kept me busy until I got to the BSD machine. The old BSD machine had an IP conflict with one of the Verizon set top boxes...I thought I'd set it to a static IP and when I checked, I had, but the damned router gave the set top box the same IP. I had to run around the house at 11PM trying to figure out which box it was (I've five of them). The last one I checked was the one I was looking for...go figure. A quick power-cycle and it got another IP. I wouldn't have figured this out if I hadn't used ARP. I kept pinging the BSD machine's IP but wasn't seeing return traffic...I telnet'd to port 22 and 80 and didn't get a response, either. So, I looked at the ARP results and saw that another machine had the IP...in fact, the set top box had two of them, but the MAC addresses were wrong on one (this was the BSD box entry...the MAC matches that machine). Very weird but hopefully it won't happen again.

I'll be looking to invest in a UPS soon. I need one that will be able to power down 3 *nix machines or at least keep them running for 5 minutes or so. Dunno if I should also ensure that there's room for the router...

Snortreport install

I remember running snortreport awhile back and liked it. I want to try to use it again, but I was having issues installing it in FreeBSD.

It appears that the FreeBSD port of snortreport requires php4. I'm currently using php5 and want to run snortreport with minimal fuss. I do not want to try to run both php5 (for Apache and phpBB3) and php4, as it will break the server. There are several tutorials on how to run both but as I said, I don't want any fuss.

So, I delved a bit into the ports and makefiles. I looked at the makefile for snortreport and decided to remove the php check that stops me from installing the port. It then choked on jpgraph (a dependency)...it appears that jpgraph is actually the port that requires php4. I was going to edit the makefile for jpgraph to allow the install (by commenting out the line that checks for php4), but saw that there is another version of jpgraph called jpgraph2. I looked at that port's makefile and it didn't check for php4 (it did check for php5). I went ahead and installed jpgraph2 instead, then installed snortreport without any warning/error messages.

So, for those of you that want snortreport on FreeBSD and want to leverage the ports system, you can get around the php4 dependency issue by just installing jpgraph2.

Of course, I still have to fully get snortreport up and running before I claim 100% success, right? ;)

Trying to upgrade/revamp my lab

I'm trying to retire some of my older equipment in my lab.  The biggest move will be in migrating my old FreeBSD server to a new one.  Both are currently up and running.

The old:

FreeBSD 6.2-RELEASE #0: Fri Jan 12 11:05:30 UTC 2007
CPU: Pentium II/Pentium II Xeon/Celeron (447.69-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x652 Stepping = 2
Features=0x183fbff
real memory = 268427264 (255 MB)
avail memory = 252989440 (241 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1

The new:

FreeBSD 8.2-RELEASE-p3 #0: Tue Sep 27 18:45:57 UTC 2011
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ (2210.20-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0x60fb2 Family = f Model = 6b Stepping = 2
Features=0x178bfbff
Features2=0x2001
AMD Features=0xea500800
AMD Features2=0x11f
TSC: P-state invariant
real memory = 1073741824 (1024 MB)
avail memory = 1002987520 (956 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1

I've a LOT of data on the old system that I need to somehow offload and retain onto the new one.  I also have to stand up updates services (mysql, ssh, httpd) and apps (phpbb3, BASE).  I already have the new phpbb3 running (it is NICE), but still have to install BASE (although Snort is installed).

I'll keep you all updated on this.

Moving my older domains

So I had wigglit.com hosted at 1and1.com, but ran into issues with them that appear to be recurring.  I previously purchased MobileMe for my mac machines (I can archive data as well as use it's e-mail system and web page authoring), but since Apple is killing MM and migrating to iCloud, some of those capabilities are disappearing.  I decided to host my pages myself, using 1and1.com, but apparently they are idiots.  I sometimes need to shell into the 1and1.com environment to make changes and I've been trying to pipe the data hosted on MM to 1and1.com but they keep locking my account.  I've sent several nastygrams asking them to lessen the lockout threshold on their shell accounts, but they keep blaming the user and not really investigating, sending cookie-cutter responses and such.  So I told them I'm going to discontinue their services as soon as I migrate the data.

So far, I've moved wigglit.com over to my Linode account.  I've moved my SV1000 blog and site to sv1000s.wigglit.com, and my Apple blog was moved to apple.wigglit.com.  I'd never used subdomains before, so that was new to me.  I also had never delved in DNS, as I had to map my subdomains to my Linode account.  Using the Linode tools and a bit of research, I was able to do this seamlessly.  I now have functional subdomains.

I'm going to eventually have everything consolidated on the Linode.  The big one will be migrating my e-mail to my Linode system...I think that's going to be painful.

I will move the rest of the data soon and discontinue using 1and1.com's services within 30 days.

Note that this has nothing to do with Slackware in itself, but I wanted to capture this move in one of my blogs.

Snort and Thresholding Noisy Alerts

I'm trying to stay sharp as a security techie, so I've been trying to contribute to Linux and security forums.  There's a guy who was asking how to use bpf.conf with Snort.  I suggested he use threshold.conf instead.  I actually referenced this (I love TaoSecurity) to help him.  He was being flooded with "SHELLCODE x86 inc ecx NOOP" alerts.  The assistance thread is here, at LinuxQuestions.org.

Connection Tracking and IPTables

Conntrack entries

I'm making a point of trying to read through this Iptables document. The connection tracking function is pretty cool, though. I was aware of the functionality but had never seen the logs at /proc/net/ip_conntrack until this morning.

BASE and Snorby: packet captures

Noticed that someone on the interwebz stated that Snorby captures full payload while BASE doesn't.  I read this as a comment on the Snorby pages.  Unless I'm totally off-base here, that's not the case, unless they're taking about something like netflows or something akin to it.  I believe one of the dev guys stated that only Snorby and Sguil offer full packet capturing.  That does NOT sound right and I believe he should clarify.

I'll dig up the link later, but it should be very apparent on their pages (it was to me, when I was perusing).

So, I pulled up my BASE console and looked at a sample packet.  To look at payload/packets within BASE, you go to a line item then click on the "ID", which would look akin to "2-278900". 

BASE capture view:

Snorby capture view:

Now, I don't see either lacking in that regard.  This is enough for the analyst to determine a false positive vs. a real attack/concern.

Now, if I wanted to further investigate, I can (in BASE), go to a listing, then click the offending IP (or the other IP...doesn't matter).  Then I click "Unique alerts" or "Unique IP links" under "Summary Statistics":

 

Unique alerts

This is basic stuff here.  It shows the history of that particular IP...it shows everything that was ever recorded from that IP, and you can dig down from there.  Source/Destination would show bidirectional traffic between the offending IP and whatever it was communicating with.  I'll get payload every time, IF (BIG IF HERE) the Snort signature is designed to capture payload and if the traffic even has payload.

I don't understand the argument of saying that BASE doesn't capture full payload.  Of course, BASE won't.  It's just a SEM.  Snort would actually do the capturing.  It would also totally depend on who sets up Snort and their requirements.  The admin that configures Snort may not even have all the sigs enabled.  But, BASE will show any payload that Snort does capture.

At this point, Snorby's search and analytical functionality is lacking.  I've said this before and got ridiculed by one of the Snorby developers.  We all know Snorby is relatively new when comparing it to BASE, but until the Snorby dev team enables better query functionality and better ways to quickly track activity, I'm going to stick to my guns.  A pretty (and even simplified) interface is one thing, but when it comes to the meat and potatoes, candy apples doesn't cut it.  As an analyst, I'd not want to lose any type of query features, as this will make a sometimes frustrating job all the more frustrating (been there, done that).

Lastly, I will NOT HAVE A PISSING MATCH over this.  I've been doing such comparisons for YEARS and am fully capable of judging what is acceptable and what is not regarding most security tools (that's why I get paid the big bucks), although I'm always objective in my opinions.  I definitely know what "best of breed" entails.  I'm going to put it out there:  Snorby is NOT best of breed.  I'd love it to be, but right now, it is NOT.  It has to help me sort/organize/filter information that helps me catch malware and such...much more that what it currently offers.  Right now, with Snorby, there's no such thing as digging down or simplifying the search through thousands of potentially bad security events.  "Packet capture options/Customer" isn't going to cut it.  It is good for the small investigation but not for the bigger tasks.  Let's be grown-ups about this topic and offer objective opinions.  If you can't do that, don't even try to leave some nasty comment on this blog.  Comments moderation is enabled.  Yes, I do require clarification on what is considered "full payload analysis", as I feel that's not enough of a description and could actually be relating to something else entirely different that the above (I doubt it, though).

GUIs for Snort

GUIs for Snort --

http://blog.snort.org/2011/01/guis-for-snort.html

Some of these might appeal to you, the network/security administrator, depending on your organization's needs. Note: there is NO best in breed tool...it totally depends on your organization's needs, which will vary when comparing org X to org Y.

HTTP Viewers

I found something that is very similar to Web-sniffer.net (an HTTP viewer/proxy)...it is called "Rex Swain's HTTP Viewer". That's a mouthful, so I'll call it RSHV.

One thing that Web-sniffer can't do is allow for referer configuration. RSHV will let you configure the referer (in fact, this appears to be a recently added feature). Why is this sometimes important? Read here. In comparison to Web-Sniffer.net, RSHV is better documented. A con of RSHV is that it won't do HTTPS.

Why do I call these HTTP viewers proxies? Well, they are. When you utilize those tools to view, for example, pages/headers at wigglit.ath.cx, if you check the web logs at wigglit.ath.cx, you'll see the traffic you generated came from someone else's IP (and not the one that was assigned to your machine when you visited wigglit.ath.cx). That's a protection, in my opinion...this means you can conduct research without having to use a lab system to prevent infection.

Note that the services these two tools provide can be done on pretty much any computer (*nix or win32/64). Just use telnet. Of course, wget can also be used (or fetch or curl), but I consider that to be a more cumbersome solution (although you may be able to create scripts that you can use wget/fetch/curl with).

Utilizing such tools in such a manner is important when conducting security analysis (for instance, validating that a certain website is or isn't compromised and serving malware).

58.221.32.117 hammering my server

I've been seeing 58.221.32.117 in my logs, especially within the last week or so.  So far, I've 5,356 instances of blocking by the firewall for this particular IP.  All traffic is coming from source port 80 of that IP.  Yes, every single instance was blocked.

Has anyone else seen similar activity from this IP?

A whois shows the following:

IP address [?]:	58.221.32.117 [Whois] [Reverse IP]
IP country code:	CN
IP address country:	China
IP address state:	Beijing
IP address city:	Beijing
IP address latitude:	39.9289
IP address longitude:	116.3883
ISP of this IP [?]:	CHINANET jiangsu province network
Organization:	CHINANET jiangsu province network
Local time in China:	2010-10-06 10:29

E-mail Malware Attempt

I've a friend that I got an e-mail from.  It had an empty subject line and one URL in the body.  Twenty others were sent the same e-mail.

I notified the sender that they had an issue.  I then decided to use Web-Sniffer to attempt to visit the link and do a quick investigation.

When visiting via the web proxy, I observed the following:

 The web server was up and running, serving content but threw a code 302.  It also may have attempted to redirect to hxxp://uvuhjomuph.com (I obfuscated the link).  Clicking that URL takes me to an ED page (erectile dysfunction):

Googling that domain, I got at least one good hit:

So, my friend more than likely got phished and her e-mail account is now throwing out spam for penile meds.  :(

Protect your privates!

Protect your privates!

http://isc.sans.edu/diary.html?storyid=9367


In view of all the brute force attacks still being attempted against Secure Shell (SSH), we have long since been extolling the virtues of forgoing passwords and moving to RSA/DSA keys instead.
While key based login indeed nicely addresses the problem of password guessing attacks, it looks like many a Unix admin has been less than diligent in the implementation. In pretty much every Unix security audit recently, we've come across unprotected or badly protected SSH private keys (id_dsa, id_rsa). Some reside plain flat out in the open, in /tmp and such. Others are found in world-readable tar "backup" archives of user and administrator home directories. Some are even built into home-grown Linux RPM and Solaris PKG packages, ready to be plucked off an install server.

Failure of controls...Spanair crash caused by a Trojan

 Failure of controls...Spanair crash caused by a Trojan

Several readers have pointed us to an article about the preliminary report of the Spanair flight that crashed on takeoff in 2008 killing 154. The article suggests that a Trojan infected a Spanair computer and this prevented the detection of a number of technical issues with the airplane. The article speculates that if these issues had been detected the plane would not have been permitted to attempt take off.

NOTE:  Another article is here.  Another is here, and this one supports the error being on the pilots' behalves (bad pre-flight checks).

Splunk?

Been thinking on trying Splunk, a console that can monitor multiple system logs, along with health and security alerts.

I'll post more as I delve into this intriguing tool.