Engineering Stories

On the way to work today, I remembered an occasion where a team member who'd left the company had been stockpiling 1U rackmount servers in storage.  He'd reimaged each server with a common image (each had different passwords, though).  I had a listing of passwords for each server, but the listed password for one particular server wasn't working and we needed to get access to that machine.  I couldn't just reimage the machine since, even though it shared a common image, it was prepped for deployment to a certain location and was configured for that specific site.  While I had a copy of the site-specific information, I just did not have the time to reimage the machine and reconfigure it...I saved that as a "last resort" option.

After a bit of research, was able to log in successfully.

I knew the BIOS wasn't locked down, so I went into the BIOS and enabled booting from CDROM.  I had a copy of a Linux CD which I put into the CDROM tray.  I then power-cycled the system.  I was able to use the live-CD to boot up the box.  I mounted the drive within the system and removed the encrypted password within /etc/passwd using 'vipw'.  I then shut the box down, removed the live-CD, then started the system.  I was immediately given a shell.  I then reset the password to what was on the passwords list for that particular system then finished the pre-deployment steps.

This is why I love Linux.  There's always an option.  I could NOT do this with one of the backup Windows servers we had.  That case was similar:  the system was a cold backup and was racked but powered down...it was a new system with a new image but customized for a specific role...it had yet to be used, though.  The password that we had for the device was apparently incorrect.  I even tried to crack the SAM file...that didn't work and I eventually had to reinstall (not reimage) Windows Server (forgot which version) onto the system again.  What made this much worse was that there wasn't an original cloning image to use, as well as the fact that the previous engineer hadn't maintained directions on how he configured the device.  So I had to use the trial-and-error method.  I eventually configured the OS properly and installed and configured the proper software (it was a CA eTrust AV server).  The whole time, the lead client was pestering, badgering, and being overly hostile.

In another case, another contractor had left the company.  He'd been administering a Nessus server that he installed on top of OpenBSD.  This contractor chose OpenBSD and was comfortable with working within a terminal session (as was I).  And really, the box didn't really have an abundance of resources anyways, so it was probably more robust without the GUI enabled.  I understood something of OpenBSD and was aware of how to conduct scans and how to view/store the scan results.  I even had a cron job running that would conduct the scans during maintenance windows.  Everything was working fine.  The same client lead couldn't operate the system because his *nix skills were seriously lacking.  Instead of asking for help/guidance, he directed another contractor to wipe the machine and install Red Hat with the GUI enabled so that he could operate the machine.  Data was not backed up.  The scanning data as well as configuration man-hours were wasted.

Another time, I was working a deployment issue where client remote hands were my remote hands/eyes.  They'd received our Snort sensor that we'd imaged, customized, and configured and had just finished racking and powering it up.  The remote hands did not know anything of how to operate within a terminal session.  I walked him through the process, spelling out the commands he needed to type.  The problem?  We built the machine and while testing it before we shipped, had logged into the machine via SSH.  When the machine was at the remote location, I could not establish an SSH session because the host key had changed.  In order for me to regain access, the remote hands had to remove the existing host key that was tied to the IP of my work machine...the host key resided on the Snort sensor that I was trying to log into.  What made me feel good was that one of the clients was logged into the bridge call and was listening.  After the call, she praised me for my knowledge of guiding the remote hands through the whole process without ever being able to view what was on his screen.  She also commented on how I guided him in what to type.  In this case, I could care less how much they were paying me (which wasn't really all that much)...I was happy that I was able to be of assistance and value.  That was payment enough.  That was one of the few bright days in working with that particular organization.  I soon took a dignified stance and left that contract.  To this day, I will not recommend any person I know to work at that particular location without giving them ample warning.

But the main reason for this post is to share that I love *nix (and why)!

Missing me some Slackware...

I haven't played with Slackware in quite awhile.  I still run a server through Linode.com but I no longer have Slackware installed as an OS (I'm using Ubuntu for ease of use...yes, it is easier to maintain compared to Slackware and I've not run into any 'gotchas' yet).  I run one machine that has Slackware installed (it's sorely in need of an update, though) and it is being used as a NIDS system.  I've another machine with Slack on it that hasn't been turned on in months (it's OS version is even older than the other system).  I'll probably turn on this system and begin to use it again, but it is in very sore need of cleaning (it has 4-5 hard disks with data ALL over the place).

I'm trying to resist the urge to run Slackware in a VM on my Alienware system.  It will require me to probably get more RAM (I'm trying to resist that idea for now).  I do not want to attempt a native install, as I don't feel like experimenting to get Slack to work on that system.  The integrated and dedicated GPUs will probably be an immediate issue, as well as the fact that my system is running two 750GB drives in RAID0.  And, that is also my gaming system.  There's no real need for me to install Slackware natively on my system.  But, I will definitely install Cygwin, since I can leverage it's tools (such as GnuPG) without having to open a shell and have an internet connection.  Cygwin is the less complicated of the aforementioned options.

But I am missing using Slackware, which is why I've been trying to be more active at ##slackware on irc.freenode.net.  The thing is, I also have a fetish for Open- and FreeBSD, so I've been focusing on both of those the past few years.

I Created some scripts for Snort

I've created (well, modified) a Snort initialization, restart, and shutdown script for Slackware and OpenBSD. They are linked below.

The OpenBSD script works solidly.

The Slackware script works sporadically and I've no idea how to debug it (although I haven't tried 'strace' yet). It appears to work manually every time, but when run as a cron job, it's sometimes, seemingly randomly, doesn't restart. The cron job runs every hours but because it sometimes doesn't start, I now have holes in my website's IDS coverage.

Note that I didn't HAVE to create start/stop scripts for Snort, as I could've started Snort by utilizing the rc.local file, but I'd have still had to manually kill the Snort process whenever I wanted to stop Snort. Having an init script do this is much cleaner.

The fact that I've gotten it working on the OpenBSD machine hints that I've a minor issue with the Slackware script that I have yet to account for, but its frustrating me, so I'll throw it online to see if someone can help with debugging. Yeah, I'd searched for help via Google but didn't see much of Snort init scripts for Slackware (although I may find something if I look at any scripts for other distributions).

I also got Snortalog to process my Snort raw logs into a statistical report, although I had to import 6.2MB of flat files to my FreeBSD box (which Snortalog is installed on), then have Snortalog crunch that data into a HUGE (3.9MB) HTML file! Needless to say, that HTML file takes almost 5 minutes to load into a browser. I've got to filter the logs and only have it crunch certain dates to make the file less bulky.

Snortalog definitely highlights that I could do some tuning, as it shows a very high amount of MS-SQL worm attempts (MS Blaster) hitting my server, amongst other things. This is a good tool that I'd previously used (and had forgotten) at a prior place of employment. It would be nice if I could figure out how to get it to crunch my IPF FW logs.

Another oldie but goddie is SnortSnarf. It is a perl script, as is Snortalog, that parses Snort files (the alert file and the payload files) into readable HTML pages, which is a bit better at searching via command-line. It is not as handy as ACID/BASE is, though, but has lower overhead. Sadly, SnortSnarf's home page is gone, but I've linked Snort.org's archive.

EDIT --

I've found my 'error'. What happened was that I had line 34 commented out and line 35 uncommented. Line 35 is specifically for usage with OpenBSD. Line 34 is specifically for Slackware. I rectified this by uncommenting line 34 and commenting line 35. I'll also put commentary explaining this. Consider this issue solved!

Edited 8/30/2007:

Revised Script that works! *yes, click here*

Shell Scripting | Creation of a Subnet and Securing Wireless Access Points

I've been trying to automate some things on my Linux and BSD boxes, so I've been scripting a bit lately.

For one, I like the mailed stats that FreeBSD and OpenBSD provides the administrator, so I've attempted to do the same for Slackware. I've a version that also runs on BSD, although you have to hack it to get it to work under a BSD machine. I'm currently looking over it to see if I can lessen the hacking of the script when using it on a non-Linux or non-Slackware machine, but for now it does work. It does not mail the admin yet, but it does keep a listing of stats every hour on the hour (via cron).

I've used several web-based resources to create this script:

http://www.linuxcommand.org/
http://www.freeos.com/guides/lsst/

I've also bought a few scripting books:

Unix Shells by Example, 3rd Edition, by Ellie Quigley
Linux Shell Scripting with Bash, by Ken O. Burtch
Learning the bash Shell, 2nd Edition, by Cameron Newham and Bill Rosenblatt

There's a ton of shell scripting books out there, along with a ton of free online tutorials, but the ones I've mentioned gave me the most insight.

I shall post a link to my script when I've finished working it to my liking.

I've also done a few things to my network during the last week.

I bought a Netgear VPN Firewall (FVS114). I want to play with hardware a bit and this unit was cheap. Sometime in the near future, I want to inplement a VPN tunnel from my residence to my linode. Sure, I can implement it via open-source software but I have to start delving with hardware if I want to sell myself as a professional security consultant. Anyways, I was previously utilizing a Linksys WRT54X4 Firewall/Router/WAP as my border router/gateway, which was fine where it was, but in order to utilize the Netgear to its fullest (which I plan to do), it needed to be placed at the border, so I put the Netgear in the Linksys' place. I then put the Linksys inside my LAN, as I needed it's WAP capabilities.

It took me a week to get things the way I wanted them. I wanted the Linksys on its own network, and that required creating a subnet. I opted to let it use its default network segment, 192.168.1.0/24. It pulled an IP from the Netgear (the Netgear is set up to serve IP addresses via DHCP). I was able to run a CAT5 from the Linksys to my Shuttle box and gain access to the administrative browser. Everything was going well, until I found that I couldn't ping other machines on the Linksys network segment. I spent a week trying to figure out why until I got a coworker to come over and take a look at things. He almost immediately got things working. I found that the Sygate firewall that I had installed on the Shuttle was impeding things. I turned it off and the laptop that was associated with the WAP was able to ping the Shuttle. One more problem was apparent: I wasn't able to get out to the internet on any laptop, although I could on the Shuttle. The reason? The Linksys FW was enabled. Once that was turned off, I was able to open up a browser and point it to MSN.com. Those were pretty simple solutions that any competent engineer should have been able to fix. My issue? Well, most engineers run standard tools in their work environment. I'd forgotten about the Sygate firewall, which is only installed on two of my many machines...so, its not so standard within my network environment, so it was easy to forget. That, and I was so wrapped up with getting this setup to work that I didn't check the obvious items.

All that is left now is to add a static route on the Netgear that will allow communcations from the Linksys netrange.

This is a decent setup, as you always want to segregate your WAP from your network. For home users, the basic setup is fine, but I'm not a regular home user. I want to at least TRY to do things the right way.

Now, since we're talking about WAPs, I'll let you know that I'm using WEP, the protocol that's branded 'unsecure'. Why am I using it? Because not all of my wireless devices can use the WPA protocol. Some people say that WEP is so insecure that its better off not using it...that's total B.S. You always want to use security in-depth anyways, which means you need to implement your security in layers to cover all potential weaknesses. Here's what I normally do:

1. Create a good, long password for the administrative GUI.

2. Either limit the DHCP pool to a very small amount (depending on how many wireless devices you have...I've at least 5, so I have a DHCP pool of 7), or turn off DHCP and assign your devices static IPs. This way, if someone gains access to your network, he's smart enough to get his own IP and not have it given to him. The lesson is to not make it easy to bust into your network.

3. Change your WEP key from time to time, maybe once a month. My Linksys will ask for any phrase and create 4 keys based off of that one phrase. Rotate those every once in awhile. Why do all this? The WEP key is supposedly easy to crack. A coworker of mine did attempt to crack a key. He couldn't. Many people think it is easy, but apparently its not as easy as many people say, but in case it IS easy, rotate your keys from time to time.

4. You can use MAC address authentication. Sure, someone could spoof a MAC, but remember that we're layering security...he may spoof a MAC but he won't be able to circumvent the rest of the things you've implemented.

5. Don't broadcast your SSID. Those who know how to hack will find it anyways, but what you want to worry about is the script kiddies out there and the ocassional bandwidth leech. Don't worry about the serious crackers out there, as if they wanted to pop your box, they could easily do it, with or without your basic security layering.

6. On some WAPs, you can dial down the power a bit so the wireless signal doesn't broadcast out to 2-3 blocks. Of course, when you do this, your wireless bandwidth gets throttled. Weigh the cost of this in your own mind and decide what would be best for your network. I don't throttle my signal but I do have my WAP in the basement, which does cut down on the signal. If you've a WAP on a 2nd or 3rd floor home or building, its going to be hard to throttle the signal to the point that others won't see it, since the WAP is physically on higher ground.

I may be missing a few wifi pointers, but I'll let others fill in the blanks. The above are the ways I keep my WAP and network secure. I hope you guys and gals can benefit from those tips.