Ruby, Rails, Gems Redux Part III

I'm starting to get a bit annoyed. I still can't get this working properly. Getting the same error as I got in my last post. I haven't changed anything but I've double- and triple-checked.

Right now, I'm currently posting to the Snorby Goggle group to try to get some assistance, which I usually don't have to do...I hate being dependent upon others, but that's just me.

Anyways, so far, I've been able to rule out MySQL as the culprit, as I'm seeing connections from Ruby to the MySQL server. I'm also able to connect to the server as 'root' and as 'snort'. The web server continues to issue status 500 and the Ruby logs indicate that there's something wrong with the user_session/new.html.erb file (keeps saying 'no credentials provided').

One suggestion I got is to do a 'git pull' to update Snorby from the Snorby directory. That command pulled quite a few changes, but after the pull, I'm still receiving the same error:

root@slackbox:~/RAILS/RAILS/Snorby# git pull
remote: Counting objects: 604, done.
remote: Compressing objects: 100% (522/522), done.
Indexing 542 objects...
remote: Total 542 (delta 393), reused 43 (delta 12)
100% (542/542) done
Resolving 393 deltas...
100% (393/393) done
37 objects were added to complete this thin pack.
* refs/remotes/origin/cache_test: storing branch 'cache_test' of git://github.com/mephux/Snorby
commit: a30cf8e
* refs/remotes/origin/master: fast forward to branch 'master' of git://github.com/mephux/Snorby
old..new: e17ace1..7edf9e9
Updating e17ace1..7edf9e9

Fast forward
app/controllers/application_controller.rb | 2 +-
app/controllers/comments_controller.rb | 57 ++++++++++
app/controllers/events_controller.rb | 4 +-
app/controllers/pages_controller.rb | 25 ++++-
app/controllers/searches_controller.rb | 4 +-
app/controllers/user_sessions_controller.rb | 2 +-
app/helpers/application_helper.rb | 41 +++-----
app/helpers/comments_helper.rb | 2 +
app/models/comment.rb | 5 +
app/models/event.rb | 17 +++
app/models/importance.rb | 3 +-
app/models/report.rb | 2 +-
app/models/search.rb | 4 +-
app/models/user.rb | 17 +++-
app/views/comments/_comment.html.erb | 15 +++
app/views/comments/_form.html.erb | 9 ++
app/views/comments/create.js.rjs | 11 ++
app/views/comments/destroy.js.rjs | 2 +
app/views/comments/edit.html.erb | 3 +
app/views/comments/new.html.erb | 5 +
app/views/events/_comments_for_event.html.erb | 21 ++++
app/views/events/_event.html.erb | 21 +++-
app/views/events/_ip_data.html.erb | 15 ++-
app/views/events/_summary.html.erb | 8 +-
app/views/events/remove_event.js.rjs | 2 +-
app/views/events/send_event.html.erb | 4 +-
app/views/events/show.html.erb | 4 +
app/views/pages/category.html.erb | 13 +++
app/views/pages/category.js.rjs | 1 +
app/views/pages/dashboard.html.erb | 20 ++--
app/views/pages/severity.html.erb | 8 ++
app/views/pages/severity.js.rjs | 1 +
app/views/reports/send_report.html.erb | 2 +-
app/views/searches/send_search.html.erb | 2 +-
app/views/searches/show.html.erb | 4 +-
app/views/settings/index.html.erb | 2 +-
config/email.yml.example | 3 +-
config/routes.rb | 8 +-
db/migrate/20090719222259_create_comments.rb | 16 +++
db/schema.rb | 12 ++-
public/flash/clippy.swf | Bin 5380 -> 0 bytes
public/images/.DS_Store | Bin 12292 -> 12292 bytes
public/images/comment/comment_top.png | Bin 0 -> 4759 bytes
public/images/cross.png | Bin 655 -> 689 bytes
public/images/other/{destroy.png => destroy2.png} | Bin 715 -> 715 bytes
public/images/other/edit.png | Bin 0 -> 497 bytes
public/images/other/is_not_important.png | Bin 648 -> 633 bytes
public/images/other/no_comment.png | Bin 0 -> 604 bytes
public/images/other/slash.png | Bin 714 -> 689 bytes
public/images/other/slash2.png | Bin 0 -> 714 bytes
public/images/other/whois.png | Bin 0 -> 595 bytes
public/stylesheets/snorby.css | 118 ++++++++++++++++++++-
test/fixtures/comments.yml | 11 ++
test/functional/comments_controller_test.rb | 54 ++++++++++
test/unit/comment_test.rb | 7 ++
55 files changed, 504 insertions(+), 83 deletions(-)
create mode 100644 app/controllers/comments_controller.rb
create mode 100644 app/helpers/comments_helper.rb
create mode 100644 app/models/comment.rb
create mode 100644 app/views/comments/_comment.html.erb
create mode 100644 app/views/comments/_form.html.erb
create mode 100644 app/views/comments/create.js.rjs
create mode 100644 app/views/comments/destroy.js.rjs
create mode 100644 app/views/comments/edit.html.erb
create mode 100644 app/views/comments/new.html.erb
create mode 100644 app/views/events/_comments_for_event.html.erb
create mode 100644 app/views/pages/category.html.erb
create mode 100644 app/views/pages/category.js.rjs
create mode 100644 app/views/pages/severity.html.erb
create mode 100644 app/views/pages/severity.js.rjs
create mode 100644 db/migrate/20090719222259_create_comments.rb
delete mode 100644 public/flash/clippy.swf
create mode 100644 public/images/comment/comment_top.png
rename public/images/other/{destroy.png => destroy2.png} (100%)
create mode 100755 public/images/other/edit.png
create mode 100644 public/images/other/no_comment.png
create mode 100644 public/images/other/slash2.png
create mode 100644 public/images/other/whois.png
create mode 100644 test/fixtures/comments.yml
create mode 100644 test/functional/comments_controller_test.rb
create mode 100644 test/unit/comment_test.rb

root@slackbox:~/RAILS/RAILS/Snorby# script/server -e production -b 10.150.1.106 -p 3000
=> Booting WEBrick
=> Rails 2.3.2 application starting on http://10.150.1.106:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2009-07-23 22:18:39] INFO WEBrick 1.3.1
[2009-07-23 22:18:39] INFO ruby 1.8.6 (2007-03-13) [i486-linux]
[2009-07-23 22:18:39] WARN TCPServer Error: Address already in use - bind(2)
[2009-07-23 22:18:39] INFO WEBrick::HTTPServer#start: pid=5752 port=3000


Processing UserSessionsController#new (for ::ffff:10.150.1.106 at 2009-07-23 22:18:40) [GET]
Parameters: {"action"=>"new", "controller"=>"user_sessions"}
Rendering template within layouts/application
Rendering user_sessions/new

ActionView::TemplateError (undefined method `login' for #) on line #8 of app/views/user_sessions/new.html.erb:
5: <% form_for @user_session, :url => user_session_path do |f| %>
6: <%= f.error_messages %>
7: <%= f.label :login %>

8: <%= f.text_field :login %>

9:

10: <%= f.label :password %>

11: <%= f.password_field :password %>


app/views/user_sessions/new.html.erb:8
app/views/user_sessions/new.html.erb:5
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'

Rendering /root/RAILS/RAILS/Snorby/public/500.html (500 Internal Server Error)


Processing ApplicationController#index (for ::ffff:10.150.1.106 at 2009-07-23 22:20:40) [GET]

ActionController::RoutingError (No route matches "/test/" with {:method=>:get}):
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'

Rendering /root/RAILS/RAILS/Snorby/public/404.html (404 Not Found)


Processing UserSessionsController#new (for ::ffff:10.150.1.106 at 2009-07-23 22:20:55) [GET]
Parameters: {"action"=>"new", "controller"=>"user_sessions"}
Rendering template within layouts/application
Rendering user_sessions/new

ActionView::TemplateError (undefined method `login' for #) on line #8 of app/views/user_sessions/new.html.erb:
5: <% form_for @user_session, :url => user_session_path do |f| %>
6: <%= f.error_messages %>
7: <%= f.label :login %>

8: <%= f.text_field :login %>

9:

10: <%= f.label :password %>

11: <%= f.password_field :password %>


app/views/user_sessions/new.html.erb:8
app/views/user_sessions/new.html.erb:5
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'

Rendering /root/RAILS/RAILS/Snorby/public/500.html (500 Internal Server Error)

Grrr. Something isn't quite right here. I've followed the documents properly in installing Snorby, but something was obviously missed. I'm totally reliant upon the developers at this point. While this is normal for some people, it isn't for me. At work, our dev team are the VERY last people I go to, because they tend to either try to make you look stupid or will say some shit like 'it is what it is'. I'm trying to keep in mind that my bad experience with developers is limited to work and not the open-source community. That being said, I've invested quite a bit of time and effort on the Snorby project. While I've learned a few things, I do have an end goal and I'm a goal-oriented person.

I'll stop updating on Snorby until I actually have it working.

Ruby, Rails, Gems Redux Part II

Did a little research on the gem for MySQL and decided to try this:

root@slackbox:~/RAILS/RAILS/Snorby# locate mysql_config
/usr/man/man1/mysql_config.1.gz
/usr/bin/mysql_config
root@slackbox:~/RAILS/RAILS/Snorby# gem install mysql -- --with-mysql-config=/usr/bin/mysql_config
Building native extensions. This could take a while...
Successfully installed mysql-2.7
1 gem installed
Installing ri documentation for mysql-2.7...
Installing RDoc documentation for mysql-2.7...
root@slackbox:~/RAILS/RAILS/Snorby#

Now about my Snort architecture, I'm thinking all I'm gonna have to do is copy my Snort database over to Slackbox and then have my two Snort machines (one internal and one sensor at a datacenter) report to Slackbox....OR, have the Snort sensors report to BOTH the FreeBSD server AND Slackbox! I think the latter will work and it sounds like the better solution.

I'll be updating this post with my successes and failures most of the night, I suspect, or at least until I get good and pissed off. LOL!

=====

Update:

There's nothing like backing up an 83MB database file on old hardware:

Starting: 6:31PM up 23 days, 19:27, 4 users, load averages: 2.89, 2.94, 3.13

Ending: 6:33PM up 23 days, 19:29, 4 users, load averages: 5.88, 3.98, 3.51

While I'm sure that's incomparable to an enterprise database, at one point, I thought the old dell system would lock up.

I also was trying to do this via phpMyAdmin on both machines, but I didn't know the dbase size was that large (4 yrs of sniffing data). phpMyAdmin on the BSD box would say it was finished exporting but I'd check the filesize and it was different each time (did it like 4 times before I decided to go commandline. phpMyAdmin kept giving me a filesize of between 20M and 40M. It must've been choking out. I optimized the dbase, also, so it was more than likely larger than 83MB.

=====

Update:

Had to upgrade MySQL, as my 83MB file wouldn't import into Slackbox's MySQL server. 30 seconds into the import, the import would lock up or die. Apparently, it's a known issue with MySQL's lower versions.

Anyways, after the import and creation of new MySQL users, I had to edit Snorby's config/database.yml file, specifically the development part. The reason:

root@slackbox:~/RAILS/RAILS/Snorby# script/server -p 11001
=> Booting WEBrick
=> Rails 2.3.2 application starting on http://0.0.0.0:11001
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2009-07-17 21:48:14] INFO WEBrick 1.3.1
[2009-07-17 21:48:14] INFO ruby 1.8.6 (2007-03-13) [i486-linux]
[2009-07-17 21:48:14] WARN TCPServer Error: Address already in use - bind(2)
[2009-07-17 21:48:14] INFO WEBrick::HTTPServer#start: pid=3848 port=11001
/!\ FAILSAFE /!\ Fri Jul 17 21:48:17 -0400 2009
Status: 500 Internal Server Error
Can't connect to MySQL server on 'no_not_use' (111)

It's still not clear to me why I had to edit it, but I did because the production portion was populated with the proper credentials but I was still receiving the above error..."Can't connect to MySQL server on 'no_not_use'". When I did it, I stopped getting that error.

*** I found why I was getting the MySQL error. The config/database.yml development entry has 'mysql' for the database entry. It should be 'no_not_use'. I've edited this to what is was originally supposed to be and changed everything back to 'no_not_use'. I no longer get the error when using the production settings. ***

Also, notice that I ran in what I want to call 'debug mode' because I wanted to see what was hanging up the connection.

So, now, after some editing and fiddling, I get the following in 'debug mode':

root@slackbox:~/RAILS/RAILS/Snorby# script/server -e production -b 10.150.1.106 -p 11001
=> Booting WEBrick
=> Rails 2.3.2 application starting on http://10.150.1.106:11001
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2009-07-17 21:55:37] INFO WEBrick 1.3.1
[2009-07-17 21:55:38] INFO ruby 1.8.6 (2007-03-13) [i486-linux]
[2009-07-17 21:55:38] WARN TCPServer Error: Address already in use - bind(2)
[2009-07-17 21:55:38] INFO WEBrick::HTTPServer#start: pid=3915 port=11001


Processing UserSessionsController#new (for ::ffff:10.150.1.106 at 2009-07-17 21:55:40) [GET]
Parameters: {"action"=>"new", "controller"=>"user_sessions"}
Rendering template within layouts/application
Rendering user_sessions/new

ActionView::TemplateError (undefined method `login' for #) on line #8 of app/views/user_sessions/new.html.erb:
5: <% form_for @user_session, :url => user_session_path do |f| %>
6: <%= f.error_messages %>
7: <%= f.label :login %>

8: <%= f.text_field :login %>

9:

10: <%= f.label :password %>

11: <%= f.password_field :password %>


app/views/user_sessions/new.html.erb:8
app/views/user_sessions/new.html.erb:5
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'

Rendering /root/RAILS/RAILS/Snorby/public/500.html (500 Internal Server Error)

The thing is, I see nothing in the web browser, but:

We're sorry, but something went wrong.

We've been notified about this issue and we'll take a look at it shortly.

Something else that is nagging me that I was trying to fiddle with is:

[2009-07-17 21:55:38] WARN TCPServer Error: Address already in use - bind(2)

There is only one Ruby service running and nothing is utilizing that port when I run Ruby. I'm ignoring it for now.

I would love to see what the WEBrick logs show, if there are any.

For now, its time to do some serious Googling and maybe hit up my three Ruby/Rails books.

Ruby, Rails, Gems Redux

I decided to use Slackware this time. I've had better luck.

My install already has Ruby 1.8.6 (the latest stable is 1.8.7, I believe).

Ran into an issue when following these instructions. Was supposed to do 'rake gems:install' but got a 'prawn' error

root@slackbox:~/RAILS/RAILS/Snorby# rake gems:install
(in /root/RAILS/RAILS/Snorby)
rake aborted!
no such file to load -- prawn

Fixed it by using 'gem install prawn'. After running that command, I was able to run the 'rake gems:install' without error.

Now I'm having a similar issue when running 'rake snorby:setup':

root@slackbox:~/RAILS/RAILS/Snorby# rake snorby:setup
(in /root/RAILS/RAILS/Snorby)
Setting Up Snorby Database.
!!! The bundled mysql.rb driver has been removed from Rails 2.2. Please install the mysql gem and try again: gem install mysql.
rake aborted!
no such file to load -- mysql

Running 'gem install mysql' give me a BUNCH of errors:

root@slackbox:~/RAILS/RAILS/Snorby# gem install mysql
Building native extensions. This could take a while...
ERROR: Error installing mysql:
ERROR: Failed to build gem native extension.

/usr/bin/ruby extconf.rb
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/usr/bin/ruby
--with-mysql-config
--without-mysql-config
--with-mysql-dir
--without-mysql-dir
--with-mysql-include
--without-mysql-include=${mysql-dir}/include
--with-mysql-lib
--without-mysql-lib=${mysql-dir}/lib
--with-mysqlclientlib
--without-mysqlclientlib
--with-mlib
--without-mlib
--with-mysqlclientlib
--without-mysqlclientlib
--with-zlib
--without-zlib
--with-mysqlclientlib
--without-mysqlclientlib
--with-socketlib
--without-socketlib
--with-mysqlclientlib
--without-mysqlclientlib
--with-nsllib
--without-nsllib
--with-mysqlclientlib
--without-mysqlclientlib


Gem files will remain installed in /usr/lib/ruby/gems/1.8/gems/mysql-2.7 for inspection.
Results logged to /usr/lib/ruby/gems/1.8/gems/mysql-2.7/gem_make.out

Grrr...!!!

But, I'm a lot closer this time than last time. I'll sort it out either tomorrow night or this weekend.

Oh, and one more thing. Maybe this is more complicated than it has to be, because I've already got BASE running on a server who's internal IP is 10.150.1.103 (FreeBSD on a Dell server/workstation). The MySQL server is also on that box. Maybe I don't actually need the MySQL gem? Can I leverage the actual database on the FreeBSD box and maybe skip some steps? We'll find out, otherwise, I'm going to have to maybe copy the database over to the Slackware machine so I can test. Yeah, I really wanted Snorby on the FreeBSD box but for some reason I'm more comfortable with Slackware.

Rails, Ruby, Gems...PITA

I spent the evening trying to get Snorby installed.

I've sporadically messed with Ruby on Rails before, actually getting it installed and playing a bit with it before moving on to other things.

Now, I've been hindered by an out-of-date Rails install. I tried to install Snorby and every step I have to take two steps backward. I ended up reinstalling to the latest version that FreeBSD (yeah, doing this on a BSD install, as it seems easier to install this way), but that version wasn't high enough.

Now, I'm installing the latest by source. I'm as far as I've ever been tonight, which is good because I'm running out of time. I'm installing the Gems at the moment and the install is agonizingly slow (doing this on a dual proc 450MHz machine). It appears most of this are documentation installs. :/

Maybe I can get this done and still be able to get a bit of sleep before I've to get up for work.

So far, see below .txt file...looks good so far:

http://wigglit.ath.cx/ruby.txt

AAARGH!!

[root@delly /usr/local/www/data/Snorby]# rake snorby:setup
(in /usr/local/www/data/Snorby)
Missing these required gems:
javan-whenever

You're running:
ruby 1.8.5 at /usr/local/bin/ruby18
rubygems 1.3.1 at /root/.gem/ruby/1.8, /usr/local/lib/ruby/gems/1.8

Run `rake gems:install` to install the missing gems.

Almost there but I'm out of time...will continue tomorrow.

Main Slackbox (named slackbox) back online

I finally had time to figure out what was going on with my Slackware tower. It has been down for so long I forget when it actually started having issues. I believe it began having problems since the beginning of 2009.

The culprit? Either the SATA drive that I installed in it awhile back or the SATA controllers on teh board. It's difficult to tell without swapping the SATA drive out for another. I just disabled it (the drive) for now. It works fine without it connected...that tells me its the drive and not the controller.

I began by trying to boot it up by using a Ubuntu LiveCD. It wouldn't boot up and showed many ATA-based errors in the logs. I tried a different Ubuntu LiveCD (v8.10, I believe)...still, same issue.

It was then that I started focusing on the SATA drive. I just unplugged it and tried to reboot...got a reboot and the system has been running for about a week without any issues.

So, I lost a drive. It's not a big deal, as that drive was hosting Windows, I believe. Weird, because that drive is rather young. I believe its a WD (I have seriously bad luck with that brand). I can do without replacing that drive for now, though.

Youch! Freenode bans Mibbit.com connections

New freenode webchat (and why to use it)

OW.

I remember awhile back, the server opers would frown upon banning Mibbit users. I guess they had a change of heart due to abusers using Mibbit to dodge bans. I remember having to place broad bans on Mibbit on a temp basis, but it was always temporary bans that I placed. I even became a Mibbit user. I hadn't joined the server using Mibbit in about a month so something happened recently for Freenode to lock out all Mibbit connections.

Oh well.

For now, Freenode is hosting its own web-IRC client: http://webchat.freenode.net/

Researching and found an old flamefest spark

Reference:

http://mythtv.beirdo.ca/ircLog/channel/1/2008-07-14

Summary: At LQ.org, there was a discussion on the security forums on how vulnerable Linux was to attacks/malware. Someone didn't like what was being discussed because of typical Linux zealotry. What happened on LQ's forums spilled over into ##slackware on IRC. Dagmar, the instigator of a LOT of bad things that used to happen in ##slackware got perm banned by me. Later, documented in the link above, he is his typical self, not even attempting to objectively explain what the whole thing was about, pretty much slandering me about how flawed my thoughts are on the whole thing and is worrying that I'll propagate bad information.

Let me explain some things about myself. I'm an IT security engineer. I don't just mess with routers and I'm not some glorified network engineer. I'm a senior consultant. I not only consult, I'm able to find "needle-in-the-haystack"-type info using packet-level analysis. Most of what I do requires that I be a jack-of-all-trades in network engineering, but my specialty is security. I'm proficient in utilizing many industry-leading security tools, both freeware and commercial software. I work at a very large ISP/telecom within a large managed security services team. I am THE lead of a government security operations center. We manage well over 100 customers' security posture via firewalls, NIDS, HIDS, and IPS appliances, using ArcSight, an aggregation and correlation tool that is fast becoming the standard in security event monitoring.

Every day, we see machines being compromised...this is nothing new. The compromises span every mainstream OS. This includes Linux. Whether it is kernel level or application level is not the argument. The argument is that Linux is not as rock-solid as everyone makes it out to be. Sure, it has more safeguards than Windows-based systems, but it is still susceptible to application-level exploits. Whether this is a coder issue or PEBKAC/user/admin issue is besides the point.

People need to stop thinking that just because they are running Linux, they are safe. That is NOT the case. This is not paranoia speaking. It is from seeing such things happen on a daily basis during security event monitoring. Due to applications such as PHP-Nuke, it is becoming more difficult to secure back end applications. It is much harder to stop SQL injection than it is to stop SSH brute-forcing, for instance. This isn't the only issue, though. The issue is the perception that because Linux code is open and free, the code base is free of vulnerabilities. That is NOT the case. Also, many people think that a majority of the cracker focus is on Win32 because MS has a majority of the market share. That also is NOT the case. That is a big assumption. milw0rm and other such sites document many *nix-based vulnerabilities, along with Bugtraq at Securityfocus track all vulnerabilities. Sometimes, people justify Linux because its security model is better focused than Win32 systems. It is, but that does not mean that Linux is rock-solid. It has its own faults, whether it is the user, the admin, or the software developer (or even kernel developer).

Dagmar has a habit of blocking out people's opinions and sometimes beating people down with his own. Dagmar thinks he knows security more than anyone else when he's just a developer. I see attacks every day on all types of machines. Some of the attacks are successful. I doubt that Dagmar sees those. Dagmar need not worry about me "propagating" untruth, because what I say IS the truth. All you have to do to see the truth is to research and not be blind to other opinions.

Dagmar also stalked. After the IRC discussion, he began to frequent the LQ security forums and respond to every thread I posted to. He was hardly ever in those forums before then. I noticed this immediately (and also checked). I didn't mind this, but when it spilled back over into IRC, I tired of it and wanted it ended...it really had no place in ##slackware and I was fed up with his attitude about the whole thing. I don't suffer drama very well.

Now, Dagmar has been banned several times before for the lack of tact in the way he 'helped' people in ##slackware. He was walking a thin line to begin with. Those with operator status in ##slackware acknowledge that he is knowledgeable, but that is not grounds for him to be dismissed as an abusive ##slackware visitor. Sure enough, he did the same thing with a channel operator (me) and I banned him. I also discussed it with the other operators. The consensus was that he stay banned since his history of being banned was substantial.

That was why he got banned...not because his views went against my own, but because he started regressing back to his former self and became abusive. He did the same in the LQ.org forums, but I was able to filter his posts from my normal views. As an operator at Freenode.net, I can't and shouldn't filter any visitor from my views in ##slackware, so my only option was to ban him, and like I said before, he'd his own infamous nature that was going against him.

As a security consultant, I'm certainly not going to keep my thoughts quiet about what I think is a disservice to my favorite operating system. I certainly know more than someone who is not a security consultant about IT security...its what I get paid to do and its what I've been doing for years. It's the same as a person who has built his own car, vs. someone who works as a senior Mercedes mechanic.

As much as I can, I tell people that there is NO secure OS. It is only as secure as the admin makes it, and even if the admin puts 100% resources into hardening the box, it will never be 100% secure. The LQ security forums is itself proof that Linux systems get compromised more than most people think. 2-3 times a week, someone reports they've been compromised. There's even 4 threads on Linux-based vulnerabilities:

Kernel Vulns
Mozilla Firefox Vulns
The Problem with PHP Application Security
Failed SSH Login Attempts

I can post a ton of other links but why do this when there is Google?

No further issues with gOS so far

It has been a very good experience, so far.

In fact, I've also been leveraging Thunderbird...this is a first time for me (since Netscape Mail back in the 90s). It is very robust!

I've also aliased a few commands that I tend to use alot, mainly ssh commands that I use on remote hosts.

I've also found some decent background images that I've scaled (using Gimp) to 1024x600.

I've also been conducting my typical security audits (BASE and iptables and web server log perusing).

I've not used my Mac in like 3 weeks! I don't know if that's a good or bad thing (probably bad for the battery).

gOS v3.1 installed on Dell Mini 9

I took the plunge and installed gOS v3.1 Gadgets onto the Dell Mini 9.

The install went flawlessly.

The issues I have discovered so far:

1. Wireless would not work. I followed the instructions located at http://gosforums.org/viewtopic.php?f=21&t=48&p=203&hilit=broadcom#p223. I applied this fix (when I was using a cat5 connection) and it worked, so the wireless non-functionality is no longer an issue.

2. The Mini won't suspend when I close the lid. I can manually suspend, though. I'll hunt for a fix and apply it later.

I also just noticed that a swap partition was created and configured for use (automatically, when installing gOS). I already have a gig of physical RAM and I don't want to burn out my SSD card, so I'll disable it for now and consider a workaround if/when I need it.

Overall, this is a pretty solid distro and it is pretty cool to be able to use Google tools (this will save drive space and conserve the limited resources this machine has). The Gadgets can also be used offline, so I won't need to be connected to use them...now, that's cool.

Some screenshots:






EDIT:

Fixed the sound issue by following Step 4 of "Installing Ubuntu 8.10 on the Dell Inspiron Mini 9".

I turned off the swap partition by editing out the swap entry in /etc/fstab. I'll test to see how this impacts my install of gOS before removing the partition.

Power adapters and other news

Wouldn't it be nice if the laptop empire joined forces and standardized laptop power adapter cords? I've several laptop and each requires a dedicated cord. I forgot my cord for my Mini today...fortunately, it has a decent charge, but I now have to watch consumption of power. Good thing my Macbook is in my truck. May have to go get it!

Other news:

I'm still trying to script FW log parsing. I've pretty much nailed it for my BSD machine, but will need to edit what I have, as the script parses and adds IPs to a block list. Sometimes, I just want to parse and find the top 10 offenders. I also still need to do this on my Linux machines. I've a script that parses (for Linux) but it is very rudimentary. And, I still want to port the script to Perl or Python.

tcpdump, Dell Mini, and BASE

So, I'm wondering why tcpdump is missing from the default install of my Dell's Ubuntu...doesn't make sense. I was having issues with getting my wifi card associated with my WAP and wanted to see the packets leaving the wireless interface, so I tried to bring up tcpdump, but it wasn't available. I actually had to hook a cat5 cable to the Mini to get this package, just to troubelshoot. I noticed the same thing with Suse about a year ago.

Apparently, tcpdump was created on the permissive free software license, per Wikipedia. I don't know if this is actually GPL or a derivative of GPL. The manpage doesn't mention what license tcpdump falls under and I'm sometimes wary of Wikipedia, as I like to find the facts on my own to validate (or invalidate) internet claims.

I'll research this and post my findings here.

On another note, I found a very cool bag for my Dell Mini, at Dell's website. I'll try to post pics and a link soon (from my Macbook, as the Mini's keyboard slows me down a bit).

Lastly, I somehow broke access to my MySQL database, so now my snort sensors won't report to it. It's been down for about 2 weeks and I don't have the time to fix it. I'm going on vacation for my birthday and hope to have some personal (ie, QUIET) time to myself to fix this. I'll be visiting my parents for my birthday this weekend and will see about shelling in to fix it remotely.

Worked on...

Reconfigured mnwclient so that I can provide FW logs to MyNetworkWatchman (which is similar to Dshield).

I'd much rather get Dshield working on the Linode but for some reason, I been having difficulties using their supplied clients. I'll continue to work on it, as I had it working prior to the last Linode upgrade.

With that in mind, at some time I'm going to have to upgrade the Linode from v12.0 to v12.2.

Night...

What I've installed on the Mini so far...

So far, I've installed the following on my Dell Mini:

tcpdump (IMO, this is a mandatory package...should've been installed by default)
xchat
gdesklets (I'm going to remove this because desktop space is at a premium)

That's it, so far. Surprisingly, I find myself not needing much more than what apps are already installed. Then again, I've had the Mini for less than a week. :)

EDIT: also installed 'locate' but the install didn't include the updatedb.conf file...working on that now.

My Mini is here!

My Dell Mini has arrived. Went to pick it up at the local Fedex facility last night.

It came in a LITTLE box. Basically, it was just the netbook and the adapter that were in the box, with a few CDs and documents.

I took it out and plugged it up and the battery was almost fully charged already, but I charged it anyways.

The keypad is small and my big hands don't help much but the unit itself is pretty slick.

The tech specs are below:

32gb SSD drive
1.3mb vidcam
1gb RAM
Ubuntu Linux OS
Black in color

I opted for no bluetooth but now I'm thinking maybe I should've went that route (although it would've jacked up the price.

I now need a case. For now, I'm carrying it in my work laptop's bag.

My immediate problem is that it won't connect to any wireless networks, either at home or at work. I did a quick search and this appears to be a known issue. I'll have to delve more into this. When I attempt to have it automatically apply the wireless setting and check those settings, the settings are blank (no IP or DNS). Manually applying them doesn't help.

So, I'll be occupied the next few days... :)

Dell Mini 9 and system upgrades

I just ordered a Dell Mini 9.

It should be here next week. I purchased the 16GB SSD upgrade and webcam. It is also a Windows XP model. Later, I'll more than likely either install Linux on it (or OS X), although this will require a USB optical drive.

The plan is to also upgrade the SSD to 32GB (a $119 purchase at My Digital Discount).

More than likely, I'm going to sell my Macbook (and maybe use the money to get an iMac).

As for my dead motherboard, it looks like I'm looking at this. I've decided to go AMD Phenom. Why? Because, while I want to upgrade to quad core while I can, I don't want to go Core i7 because I'd have to spend butt-loads on the CPU and motherboard in addition to DDR3 RAM...too much upgrading. At this point, I don't even want to go Intel. While Intel has better bang for the buck regarding CPU power and robustness, I'm extremely loyal to AMD. There's nothing wrong with AMD's products if you don't benchmark (and I'm not one to hang his every decision on raw stats alone). I should have the parts within a month...then I can game again. This will leave me with an unused Pentium D 830 (dual core), though. That means I'll more than likely buy a cheaper motherboard just to utilize that CPU in one of my older systems...it is a nice CPU with good power.

EDIT: I upgraded my Mini order because they were offering a $50 discount on Presidents Day. It will have Ubuntu and it will also have a 32GB SSD. More than likely I'm going to send it back, though, since Asus has a better buy, sans the SSD, but also has BT.

About my mainboard and CPU upgrade. I went AMD. I bought a Phenom 940 with a Foxconn mainboard. It is up and running now. It is FAST and I've yet to see all 4 cores maxed out! I also bought another mainboard for the Pentium CPU. That's a project for another day, though.

Foxconn A79A-S AM2+/AM2 AMD 790FX ATX AMD Motherboard

AMD Phenom II X4 940 Deneb 3.0GHz Socket AM2+ 125W Quad-Core Black Edition Processor Model HDZ940XCGIBOX

System dead

Ref: http://slackfiles.blogspot.com/2007_02_18_archive.html

OK. Apparently when the Ultra PS died, it may have taken the motherboard with it. The machine has been blue-screening ever since the PS replacement, just not as frequently as before.

I also noticed that the HSF on the Northbridge isn't working. When searching Google for issues with the Northbridge, I found that this is happening to others and that it actually signifies a motherboard failure. The Northbridge gets HOT and I'm assuming that over time, the heat has killed the motherboard.

This is the time to upgrade, I guess. I'll more than likely buy a new motherboard/CPU combo and it'll more than likely be a quad-core CPU that I'll be getting (Phenom):

(Intel) http://www.newegg.com/Product/ComboDealDetails.aspx?ItemList=Combo.152153

or

(AMD) http://www.newegg.com/Product/ComboDealDetails.aspx?ItemList=Combo.150323

EDIT - I ended up purchasing a

Foxconn A79A-S AM2+/AM2 AMD 790FX ATX AMD Motherboard

and a

Retail AMD Phenom II X4 940 Deneb 3.0GHz 4 x 512KB L2 Cache CPU

and basically rebuilt this system...it is SMOKING. I was gaming and installing game patches for another game AND XP updates, and only two cores were being used while two were idle!! I put 3GB of RAM into the system (I almost put 4GB but then realized that XP 32-bit would only see 3GB of it...time for a 64-bit Windows OS install, I think). I really should install some high-grade RAM, though, as I think I only have 667mhz installed.

I also bought a

BIOSTAR TForce TP43D2A7 LGA 775 Intel P43 ATX Intel Motherboard

for to replace the motherboard in the Pentium-D system...I'll use this motherboard and CPU to rebuild my old Sempron 3000+ system (I'm starting to stack up 'older' motherboards and CPUs).

Recent Happenings

I've added 2 more GB to my gaming system (the Ultra that I bought from TigerDirect 2 yrs ago). The system now has a total of 4GB. It runs Windows XP, so its only seeing 3.25GB of the installed 4GB, as I'm running XP 32-bit. I need to either go to Vista or install XP 64-bit. I may just wait until the new version of Windows desktop is released.

I've also begun to (again) monitor my internal LAN traffic at home. I broke down my security systems this past Fall because I bought a house and had to move. I wanted to reinstall my external IDS but didn't want to run an ethernet tap yet, so I used the same machine to just log internal traffic for now. Later, I'll have this machine start a separate Snort process that will monitor external traffic via the tap.

I need to upgrade my Slackware boxes to v12.2. The current boxes are running v12.0, including my Linode (http://wigglit.ath.cx). I'll try to do this soon, although this will require upgrading from v12.0 to v12.1, then v12.1 to v12.2.

I'll also begin to attempt to convert all (which isn't many) my BASH scripts that manipulate text (FW log parsing scripts and such) to Perl. This is mainly as an exercise to force myself to learn Perl. I could also do Python. This is an educational 2009 goal for me. :)

Distributed SSH Brute Force Attempts, part 3

I wanted to be able to harvest the log data that the brute force attempts are generating, so I've decided to not move the SSH listening port. I'm also logging each event within my firewall logs. The particular machine I'm seeing the attacks on is a FreeBSD box (I should've mentioned that earlier) and I'm using PF as the firewall of choice.

I'd like to show you my listing of blocked IPs. I've been actively gathering them since approximately 11/17/2008. I was unhappy that I hadn't noticed the distributed attempts until November and wanted more trending data, so I reached into my SSH logs and parsed the files present with a very dirty script that added each unique IP to a PF table that is designed to block such activity. The script is below:

[root@delly ~]# cat IPscript
#/bin/bash

cd /var/log
bunzip2 pflog.*

cat /var/log/auth.log | grep sshd | grep -i 'invalid user' > /tmp/auth_IP_list_1
bzcat /var/log/auth.log.[01234567].bz2 | grep sshd| grep -i 'invalid user' >> /tmp/auth_IP_list_1
sed '/Failed keyboard-interactive/d' /tmp/auth_IP_list_1 > /tmp/auth_IP_list_2
awk '{print $10}' /tmp/auth_IP_list_2 > /tmp/auth_IP_list_3

tcpdump -nettttr /var/log/pflog > /tmp/fw_IP_list_1
tcpdump -nettttr /var/log/pflog.0 >> /tmp/fw_IP_list_1
awk '{print $9}' /tmp/fw_IP_list_1 > /tmp/fw_IP_list_2
nawk -F. '{print $1, $2, $3, $4}' /tmp/fw_IP_list_2 > /tmp/fw_IP_list_3
sed 's/ /./g' /tmp/fw_IP_list_3 > /tmp/fw_IP_list_4

cat /tmp/fw_IP_list_4 >> /tmp/auth_IP_list_4
cat /tmp/auth_IP_list_4 | sort -rn | uniq > /tmp/auth_IP_list_5
cat /tmp/auth_IP_list_5 | grep -v '64.62.231.220' > /tmp/auth_IP_list_6
cat /tmp/auth_IP_list_6 | grep -v '66.160.141.30' > /tmp/auth_IP_list_7
cat /tmp/auth_IP_list_7 | grep -v '10.150.1' > /tmp/auth_IP_list_8
cat /tmp/auth_IP_list_8
cat /tmp/auth_IP_list_8 | wc -l

pfctl -t bruteforce -T add -f /tmp/auth_IP_list_8

rm -rf /tmp/fw_IP_list_* /tmp/auth_IP_list_*

I apologize for the lack of comments in the script...as I said, it was an extremely dirty hack that required me to learn a tad of sed and awk. The script outputs the following file: http://wigglit.ath.cx/txt/bruteforce_IPtable

My logs don't actually go back that far. The FW logs go back to 14 Dec. The SSH service logs go back to 26 Nov. I probably have captured IPs reaching back to the beginning of November, though.

My FW policy prevents anything from entering the network unless specifically allowed (default deny policy). The reason I wanted to track the IPs was because my Denyhosts configuration wasn't catching most of these and it was working fine before this new trend occurred. I'm a security consultant and a researcher at heart, so I thought that tracking this would be cool. My home router has ports 22, 443, and 3306 exposed to the wild for the FreeBSD box. The FW only allows certain IPs in on those ports, though, so nothing will get in. The activity that doesn't get immediately blocked is blocked by the bruteforce_IPtable script that I run daily. The script is flawless and I will begin to have it run hourly via a cronjob. I'll also have it send an updated list to my website daily.

The IPs within my block table number 565. The script parsed 115 from the logs tonight, but only added 1 IP. The norm is usually 2-3 daily. The rest of the IPs are from logs over the last month and a half.

Oh yeah, I've another script does a daily copy of the IPs that are added to the table, so I can at least quickly determine (using 'diff') what was added on a certain day. I can create a script that will show me what was added daily for a given timespan (day/week/month).

Distributed SSH Brute Force Attempts, part 2

OK, so I'm still curious about the distributed SSH brute force attempts I've been seeing.

I wanted to check all my logs and not just the most recent logfile. Looking at my /var/log directory:

-su-2.05b# ls | grep auth
auth.log
auth.log.0.bz2
auth.log.1.bz2
auth.log.2.bz2
auth.log.3.bz2
auth.log.4.bz2
auth.log.5.bz2
auth.log.6.bz2
auth.log.7.bz2

I tailed the auth.log file and grabbed a random IP:

Nov 24 21:53:59 delly sshd[75490]: Invalid user bryan from 170.56.255.20
Nov 24 21:54:00 delly sshd[75490]: error: PAM: authentication error for illegal user bryan from 170.56.255.20
Nov 24 21:54:00 delly sshd[75490]: Failed keyboard-interactive/pam for invalid user bryan from 170.56.255.20 port 43229 ssh2

I then wanted to check all the auth.log.* files, but was curious as to how I could check compressed files. I found that there's a command called bzgrep that allows one to grep compressed files, so I used the following command and came up with quite a few hits for the referenced IP over seven (7) log files:

-su-2.05b# bzgrep '170.56.255.20' auth.log.*

The results show 2-3 instances of log entries per login attempt, so I wanted to isolate each instance without having to use arcane sed and sort commands, so I used the following:

-su-2.05b# bzgrep '170.56.255.20' auth.log.* | grep 'Invalid user' | wc -l
19

So, this particular IP generated 19 log entries between 22 and 24 Nov.

That's not particularly good. I'd love to create a script that would break down all of these IPs' unique login attempts (and possibly block them). Sounds like a project, no? :)

I've a bit of time, so I did the following (non-scripted). I cat'd the auth.log file and collected a screens-worth of data:

-su-2.05b# cat auth.log | less
Nov 24 15:00:00 delly newsyslog[74001]: logfile turned over due to size>100K
Nov 24 15:00:49 delly sshd[74014]: Invalid user brand from 218.80.215.198
Nov 24 15:00:50 delly sshd[74014]: error: PAM: authentication error for illegal user brand from 218.80.215.198
Nov 24 15:00:50 delly sshd[74014]: Failed keyboard-interactive/pam for invalid user brand from 218.80.215.198 port 19051 ssh2
Nov 24 15:02:25 delly sshd[74017]: Invalid user brandee from 83.19.224.11
Nov 24 15:02:26 delly sshd[74017]: error: PAM: authentication error for illegal user brandee from dum11.internetdsl.tpnet.pl
Nov 24 15:02:26 delly sshd[74017]: Failed keyboard-interactive/pam for invalid user brandee from 83.19.224.11 port 50163 ssh2
Nov 24 15:03:48 delly sshd[74020]: Invalid user brandee from 194.224.118.61
Nov 24 15:03:48 delly sshd[74020]: error: PAM: authentication error for illegal user brandee from 194.224.118.61
Nov 24 15:03:48 delly sshd[74020]: Failed keyboard-interactive/pam for invalid user brandee from 194.224.118.61 port 6345 ssh2
Nov 24 15:05:21 delly sshd[74026]: Invalid user brandee from 90.176.233.222
Nov 24 15:05:22 delly sshd[74026]: error: PAM: authentication error for illegal user brandee from 222.233.broadband9.iol.cz
Nov 24 15:05:22 delly sshd[74026]: Failed keyboard-interactive/pam for invalid user brandee from 90.176.233.222 port 46108 ssh2
Nov 24 15:06:42 delly sshd[74029]: Invalid user branden from 125.77.106.246
Nov 24 15:06:42 delly sshd[74029]: error: PAM: authentication error for illegal user branden from 125.77.106.246
Nov 24 15:06:42 delly sshd[74029]: Failed keyboard-interactive/pam for invalid user branden from 125.77.106.246 port 46495 ssh2
Nov 24 15:10:59 delly sshd[74035]: Invalid user brandi from 122.224.128.222
Nov 24 15:10:59 delly sshd[74035]: error: PAM: authentication error for illegal user brandi from 122.224.128.222
Nov 24 15:10:59 delly sshd[74035]: Failed keyboard-interactive/pam for invalid user brandi from 122.224.128.222 port 42253 ssh2
Nov 24 15:12:27 delly sshd[74051]: Invalid user brandi from 59.125.200.51
Nov 24 15:12:28 delly sshd[74051]: error: PAM: authentication error for illegal user brandi from 3w.upcc.com.tw
Nov 24 15:12:28 delly sshd[74051]: Failed keyboard-interactive/pam for invalid user brandi from 59.125.200.51 port 14046 ssh2
Nov 24 15:15:23 delly sshd[74057]: Invalid user brandice from 62.112.222.88
Nov 24 15:15:24 delly sshd[74057]: error: PAM: authentication error for illegal user brandice from 3e70de58.adsl.enternet.hu
Nov 24 15:15:24 delly sshd[74057]: Failed keyboard-interactive/pam for invalid user brandice from 62.112.222.88 port 42127 ssh2
Nov 24 15:16:49 delly sshd[74060]: Invalid user brandice from 218.80.215.198
Nov 24 15:16:50 delly sshd[74060]: error: PAM: authentication error for illegal user brandice from 218.80.215.198
Nov 24 15:16:50 delly sshd[74060]: Failed keyboard-interactive/pam for invalid user brandice from 218.80.215.198 port 57929 ssh2
Nov 24 15:18:11 delly sshd[74063]: Invalid user brandice from 65.203.231.41
Nov 24 15:18:11 delly sshd[74063]: error: PAM: authentication error for illegal user brandice from 65.203.231.41
Nov 24 15:18:11 delly sshd[74063]: Failed keyboard-interactive/pam for invalid user brandice from 65.203.231.41 port 38395 ssh2
Nov 24 15:19:43 delly sshd[74066]: Invalid user brandie from 123.14.10.64
Nov 24 15:19:44 delly sshd[74066]: error: PAM: authentication error for illegal user brandie from 123.14.10.64
Nov 24 15:19:44 delly sshd[74066]: Failed keyboard-interactive/pam for invalid user brandie from 123.14.10.64 port 4925 ssh2
Nov 24 15:21:07 delly sshd[74072]: Invalid user brandie from 200.170.141.134
Nov 24 15:21:07 delly sshd[74072]: error: PAM: authentication error for illegal user brandie from 200-170-141-134.static.ctbctelecom.com.br
Nov 24 15:21:07 delly sshd[74072]: Failed keyboard-interactive/pam for invalid user brandie from 200.170.141.134 port 39979 ssh2
Nov 24 15:22:44 delly sshd[74088]: Invalid user brandie from 80.51.31.84
Nov 24 15:22:44 delly sshd[74088]: error: PAM: authentication error for illegal user brandie from 80.51.31.84
Nov 24 15:22:44 delly sshd[74088]: Failed keyboard-interactive/pam for invalid user brandie from 80.51.31.84 port 39453 ssh2
Nov 24 15:24:02 delly sshd[74091]: Invalid user brandon from 200.157.176.13
Nov 24 15:24:03 delly sshd[74091]: error: PAM: authentication error for illegal user brandon from 200.157.176.13
Nov 24 15:24:03 delly sshd[74091]: Failed keyboard-interactive/pam for invalid user brandon from 200.157.176.13 port 54638 ssh2

I then checked every IP for unique log entries within all of my ssh logs:

-su-2.05b# bzgrep '218.80.215.198' auth.log.* | grep 'Invalid user' | wc -l
6

-su-2.05b# bzgrep '218.80.215.198' auth.log.* | grep 'Invalid user'
auth.log.0.bz2:Nov 24 10:55:25 delly sshd[73084]: Invalid user bjorn from 218.80.215.198
auth.log.1.bz2:Nov 23 23:22:28 delly sshd[70023]: Invalid user bahari from 218.80.215.198
auth.log.3.bz2:Nov 23 11:18:54 delly sshd[66908]: Invalid user archibald from 218.80.215.198
auth.log.4.bz2:Nov 23 04:11:52 delly sshd[65051]: Invalid user amy from 218.80.215.198
auth.log.5.bz2:Nov 22 23:20:53 delly sshd[63465]: Invalid user alize from 218.80.215.198
auth.log.7.bz2:Nov 22 01:07:10 delly sshd[57652]: Invalid user claire from 218.80.215.198

-su-2.05b# bzgrep '83.19.224.11' auth.log.* | grep 'Invalid user' | wc -l
8
-su-2.05b# bzgrep '194.224.118.61' auth.log.* | grep 'Invalid user' | wc -l
11
-su-2.05b# bzgrep '90.176.233.222' auth.log.* | grep 'Invalid user' | wc -l
0
-su-2.05b# bzgrep '125.77.106.246' auth.log.* | grep 'Invalid user' | wc -l
9
-su-2.05b# bzgrep '122.224.128.222' auth.log.* | grep 'Invalid user' | wc -l
8
-su-2.05b# bzgrep '59.125.200.51' auth.log.* | grep 'Invalid user' | wc -l
5
-su-2.05b# bzgrep '62.112.222.88' auth.log.* | grep 'Invalid user' | wc -l
10
-su-2.05b# bzgrep '218.80.215.198' auth.log.* | grep 'Invalid user' | wc -l
6
-su-2.05b# bzgrep '65.203.231.41' auth.log.* | grep 'Invalid user' | wc -l
15

-su-2.05b# bzgrep '65.203.231.41' auth.log.* | grep 'Invalid user'
auth.log.0.bz2:Nov 24 14:56:28 delly sshd[73982]: Invalid user bran from 65.203.231.41
auth.log.1.bz2:Nov 23 22:32:01 delly sshd[69793]: Invalid user azra from 65.203.231.41
auth.log.1.bz2:Nov 24 02:04:43 delly sshd[70680]: Invalid user bartholemew from 65.203.231.41
auth.log.1.bz2:Nov 24 04:07:45 delly sshd[71474]: Invalid user beck from 65.203.231.41
auth.log.2.bz2:Nov 23 19:02:03 delly sshd[68866]: Invalid user aurora from 65.203.231.41
auth.log.2.bz2:Nov 23 20:16:18 delly sshd[69213]: Invalid user avi from 65.203.231.41
auth.log.3.bz2:Nov 23 10:08:33 delly sshd[66592]: Invalid user april from 65.203.231.41
auth.log.3.bz2:Nov 23 10:24:43 delly sshd[66657]: Invalid user aquila from 65.203.231.41
auth.log.3.bz2:Nov 23 11:22:12 delly sshd[66933]: Invalid user archie from 65.203.231.41
auth.log.4.bz2:Nov 23 04:22:52 delly sshd[65094]: Invalid user anahid from 65.203.231.41
auth.log.4.bz2:Nov 23 05:32:46 delly sshd[65407]: Invalid user andra from 65.203.231.41
auth.log.5.bz2:Nov 22 23:00:09 delly sshd[63393]: Invalid user alisha from 65.203.231.41
auth.log.6.bz2:Nov 22 12:41:17 delly sshd[60534]: Invalid user abraham from 65.203.231.41
auth.log.6.bz2:Nov 22 16:14:07 delly sshd[61564]: Invalid user africa from 65.203.231.41
auth.log.7.bz2:Nov 22 11:48:33 delly sshd[60289]: Invalid user aaralyn from 65.203.231.41

-su-2.05b# bzgrep '123.14.10.64' auth.log.* | grep 'Invalid user' | wc -l
19
-su-2.05b# bzgrep '200.170.141.134' auth.log.* | grep 'Invalid user' | wc -l
6
-su-2.05b# bzgrep '80.51.31.84' auth.log.* | grep 'Invalid user' | wc -l
3
-su-2.05b# bzgrep '200.157.176.13' auth.log.* | grep 'Invalid user' | wc -l
4

So, someone appears to have a pool of compromised machines and is using each one in a scaled SSH brute force attack, based on the referenced user accounts being bruteforced. I'm seeing more of this than standard, blatant SSH BF attempts. I'll be checking Denyhosts' website to see if they've a resolution on how to track and ban such activity.

Distributed SSH Brute Force Attempts?

I'd read not long ago on the ISC Diary that someone has noticed that a there's a newly discovered way to avoid automated tools such as Denyhosts and Fail2ban. It appears that the attacks are now distributed across an IP pool of compromised machines. Maybe botnet masters are leveraging their botnets to attempt to bruteforce login attempts without risking the attacking hosts.

I think I'm seeing this in my home firewall logs:

Nov 23 15:23:01 delly sshd[67946]: error: PAM: authentication error for illegal user artois from 1-1-4-27a.vhe.sth.bostream.se
Nov 23 15:23:01 delly sshd[67946]: Failed keyboard-interactive/pam for invalid user artois from 82.182.188.187 port 35763 ssh2
Nov 23 15:24:18 delly sshd[67949]: Invalid user arty from 58.26.48.162
Nov 23 15:24:18 delly sshd[67949]: error: PAM: authentication error for illegal user arty from 58.26.48.162
Nov 23 15:24:18 delly sshd[67949]: Failed keyboard-interactive/pam for invalid user arty from 58.26.48.162 port 5785 ssh2
Nov 23 15:25:17 delly sshd[67955]: Invalid user arty from 200.170.141.134
Nov 23 15:25:17 delly sshd[67955]: error: PAM: authentication error for illegal user arty from 200-170-141-134.static.ctbctelecom.com.br
Nov 23 15:25:17 delly sshd[67955]: Failed keyboard-interactive/pam for invalid user arty from 200.170.141.134 port 57360 ssh2
Nov 23 15:26:31 delly sshd[67958]: Invalid user arty from 219.76.222.27
Nov 23 15:26:31 delly sshd[67958]: error: PAM: authentication error for illegal user arty from n219076222027.netvigator.com
Nov 23 15:26:31 delly sshd[67958]: Failed keyboard-interactive/pam for invalid user arty from 219.76.222.27 port 47176 ssh2
Nov 23 15:28:48 delly sshd[67963]: Invalid user arva from 58.196.4.2
Nov 23 15:28:49 delly sshd[67963]: error: PAM: authentication error for illegal user arva from 58.196.4.2
Nov 23 15:28:49 delly sshd[67963]: Failed keyboard-interactive/pam for invalid user arva from 58.196.4.2 port 50637 ssh2
Nov 23 15:33:27 delly sshd[67982]: Invalid user arvid from 125.77.106.246
Nov 23 15:33:27 delly sshd[67982]: error: PAM: authentication error for illegal user arvid from 125.77.106.246
Nov 23 15:33:27 delly sshd[67982]: Failed keyboard-interactive/pam for invalid user arvid from 125.77.106.246 port 51673 ssh2
Nov 23 15:34:40 delly sshd[67985]: Invalid user arvin from 85.39.252.226
Nov 23 15:34:40 delly sshd[67985]: error: PAM: authentication error for illegal user arvin from host226-252-static.39-85-b.business.telecomitalia.it
Nov 23 15:34:40 delly sshd[67985]: Failed keyboard-interactive/pam for invalid user arvin from 85.39.252.226 port 43706 ssh2
Nov 23 15:35:54 delly sshd[67991]: Invalid user arvin from 217.126.90.161
Nov 23 15:35:55 delly sshd[67991]: error: PAM: authentication error for illegal user arvin from 161.red-217-126-90.staticip.rima-tde.net
Nov 23 15:35:55 delly sshd[67991]: Failed keyboard-interactive/pam for invalid user arvin from 217.126.90.161 port 36755 ssh2
Nov 23 15:37:11 delly sshd[67994]: Invalid user arvin from 200.232.181.40
Nov 23 15:37:11 delly sshd[67994]: error: PAM: authentication error for illegal user arvin from 200-232-181-40.dsl.telesp.net.br
Nov 23 15:37:11 delly sshd[67994]: Failed keyboard-interactive/pam for invalid user arvin from 200.232.181.40 port 56318 ssh2
Nov 23 15:39:16 delly sshd[67997]: Invalid user arwan from 200.248.82.130
Nov 23 15:39:17 delly sshd[67997]: error: PAM: authentication error for illegal user arwan from 200.248.82.130
Nov 23 15:39:17 delly sshd[67997]: Failed keyboard-interactive/pam for invalid user arwan from 200.248.82.130 port 53388 ssh2
Nov 23 15:40:24 delly sshd[68003]: Invalid user arwan from 217.126.90.161
Nov 23 15:40:25 delly sshd[68003]: error: PAM: authentication error for illegal user arwan from 161.red-217-126-90.staticip.rima-tde.net
Nov 23 15:40:25 delly sshd[68003]: Failed keyboard-interactive/pam for invalid user arwan from 217.126.90.161 port 43871 ssh2
Nov 23 15:41:33 delly sshd[68006]: Invalid user arwen from 200.209.6.130
Nov 23 15:41:34 delly sshd[68006]: error: PAM: authentication error for illegal user arwen from 200.209.6.130
Nov 23 15:41:34 delly sshd[68006]: Failed keyboard-interactive/pam for invalid user arwen from 200.209.6.130 port 14808 ssh2
Nov 23 15:42:48 delly sshd[68016]: Invalid user arwen from 123.14.10.64
Nov 23 15:42:49 delly sshd[68016]: error: PAM: authentication error for illegal user arwen from 123.14.10.64
Nov 23 15:42:49 delly sshd[68016]: Failed keyboard-interactive/pam for invalid user arwen from 123.14.10.64 port 7600 ssh2
Nov 23 15:43:50 delly sshd[68020]: reverse mapping checking getaddrinfo for techregister.worcesteracademy.org [68.112.227.30] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 23 15:43:50 delly sshd[68020]: Invalid user arwen from 68.112.227.30
Nov 23 15:43:50 delly sshd[68020]: error: PAM: authentication error for illegal user arwen from 68.112.227.30
Nov 23 15:43:50 delly sshd[68020]: Failed keyboard-interactive/pam for invalid user arwen from 68.112.227.30 port 38273 ssh2
Nov 23 15:45:03 delly sshd[68039]: Invalid user arya from 196.28.50.162
Nov 23 15:45:03 delly sshd[68039]: error: PAM: authentication error for illegal user arya from www.cfse.gov.pr
Nov 23 15:45:03 delly sshd[68039]: Failed keyboard-interactive/pam for invalid user arya from 196.28.50.162 port 55647 ssh2
Nov 23 15:46:10 delly sshd[68042]: Invalid user arya from 81.12.221.74
Nov 23 15:46:10 delly sshd[68042]: error: PAM: authentication error for illegal user arya from em.asiban.ro
Nov 23 15:46:10 delly sshd[68042]: Failed keyboard-interactive/pam for invalid user arya from 81.12.221.74 port 16653 ssh2
Nov 23 15:47:23 delly sshd[68050]: Invalid user arya from 190.34.148.178
Nov 23 15:47:23 delly sshd[68050]: error: PAM: authentication error for illegal user arya from 190.34.148.178
Nov 23 15:47:23 delly sshd[68050]: Failed keyboard-interactive/pam for invalid user arya from 190.34.148.178 port 58738 ssh2
Nov 23 15:48:29 delly sshd[68053]: reverse mapping checking getaddrinfo for britannic-iss-medidean-working.e1-4-0-0-57.0.ar2.lon3.gblx.net [64.213.54.106] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 23 15:48:29 delly sshd[68053]: Invalid user asa from 64.213.54.106
Nov 23 15:48:29 delly sshd[68053]: error: PAM: authentication error for illegal user asa from 64.213.54.106
Nov 23 15:48:29 delly sshd[68053]: Failed keyboard-interactive/pam for invalid user asa from 64.213.54.106 port 42991 ssh2
Nov 23 15:49:46 delly sshd[68056]: Invalid user asa from 91.135.200.86
Nov 23 15:49:47 delly sshd[68056]: error: PAM: authentication error for illegal user asa from 91.135.200.86
Nov 23 15:49:47 delly sshd[68056]: Failed keyboard-interactive/pam for invalid user asa from 91.135.200.86 port 10262 ssh2
Nov 23 15:50:52 delly sshd[68062]: Invalid user asa from 200.20.187.222
Nov 23 15:50:53 delly sshd[68062]: error: PAM: authentication error for illegal user asa from 200.20.187.222
Nov 23 15:50:53 delly sshd[68062]: Failed keyboard-interactive/pam for invalid user asa from 200.20.187.222 port 52959 ssh2

Don't focus on the attacking IPs, but look at the referenced users. There are now tools that look like they're scaling attacks on a listing of common logins (or maybe even dictionary attacks) so that there's less risk of detection. There are current tools that look for attacks in a thresholded manner (example: 4 attacks in 5 sec warrants a block of that attacking IP). This new method of attack will not trigger the thresholding blocks.

More than ever, SSH key-based authentication should be used. This will prevent a successful login when under attack via brute forcing methods.

I can already see attack detection tools being adjusted to focus on tracking user accounts being bruteforced and banning all IPs that try to access user accounts based on time (example: 4 attacks on account asa in 5 sec will warrant a ban of all subsequent IPs for the next day or so...and not block if the IP is listed within a whitelist).

Your thoughts?