Archiving Logs

I've logs that I gather and put into a directory named /home/status. There's a script that runs commands and directs the output to files within this directory. The script runs every hour of every day. The output is dumped into a directory akin to "070101/", which means 2007-01-01. Each directory would have 24 files. I periodically archive these (by year) into tar.gz files.

Here's how the directory looks:

root@starchild:/home/status/070101# ls
070101.00.starchild.txt 070101.04.starchild.txt 070101.08.starchild.txt 070101.12.starchild.txt 070101.16.starchild.txt 070101.20.starchild.txt
070101.01.starchild.txt 070101.05.starchild.txt 070101.09.starchild.txt 070101.13.starchild.txt 070101.17.starchild.txt 070101.21.starchild.txt
070101.02.starchild.txt 070101.06.starchild.txt 070101.10.starchild.txt 070101.14.starchild.txt 070101.18.starchild.txt 070101.22.starchild.txt
070101.03.starchild.txt 070101.07.starchild.txt 070101.11.starchild.txt 070101.15.starchild.txt 070101.19.starchild.txt 070101.23.starchild.txt
root@starchild:/home/status/070101#

Here are the archives I currently have:

root@starchild:/home/status# du -h sensorstat_logs-200*
748k sensorstat_logs-2005.tar.bz2
3.5M sensorstat_logs-2006.tar.bz2
7.3M sensorstat_logs-2007.tar.gz
root@starchild:/home/status#

This is a listing of the month of January (unarchived):

root@starchild:/home/status# ls -l | grep 0701
drwxr-xr-x 2 root root 4096 Jan 1 2007 070101/
drwxr-xr-x 2 root root 4096 Jan 2 2007 070102/
drwxr-xr-x 2 root root 4096 Jan 3 2007 070103/
drwxr-xr-x 2 root root 4096 Jan 4 2007 070104/
drwxr-xr-x 2 root root 4096 Jan 5 2007 070105/
drwxr-xr-x 2 root root 4096 Jan 6 2007 070106/
drwxr-xr-x 2 root root 4096 Jan 7 2007 070107/
drwxr-xr-x 2 root root 4096 Jan 8 2007 070108/
drwxr-xr-x 2 root root 4096 Jan 9 2007 070109/
drwxr-xr-x 2 root root 4096 Jan 10 2007 070110/
drwxr-xr-x 2 root root 4096 Jan 11 2007 070111/
drwxr-xr-x 2 root root 4096 Jan 12 2007 070112/
drwxr-xr-x 2 root root 4096 Jan 13 2007 070113/
drwxr-xr-x 2 root root 4096 Jan 14 2007 070114/
drwxr-xr-x 2 root root 4096 Jan 15 2007 070115/
drwxr-xr-x 2 root root 4096 Jan 16 2007 070116/
drwxr-xr-x 2 root root 4096 Jan 17 2007 070117/
drwxr-xr-x 2 root root 4096 Jan 18 2007 070118/
drwxr-xr-x 2 root root 4096 Jan 19 2007 070119/
drwxr-xr-x 2 root root 4096 Jan 20 2007 070120/
drwxr-xr-x 2 root root 4096 Jan 21 2007 070121/
drwxr-xr-x 2 root root 4096 Jan 22 2007 070122/
drwxr-xr-x 2 root root 4096 Jan 23 2007 070123/
drwxr-xr-x 2 root root 4096 Jan 24 2007 070124/
drwxr-xr-x 2 root root 4096 Jan 25 2007 070125/
drwxr-xr-x 2 root root 4096 Jan 26 2007 070126/
drwxr-xr-x 2 root root 4096 Jan 27 2007 070127/
drwxr-xr-x 2 root root 4096 Jan 28 2007 070128/
drwxr-xr-x 2 root root 4096 Jan 29 2007 070129/
drwxr-xr-x 2 root root 4096 Jan 30 2007 070130/
drwxr-xr-x 2 root root 4096 Jan 31 2007 070131/

The command I use to archive the directories within /home/status is:

tar cvjfp sensorstat_logs-2007.tar.gz .

c - creates the archive
v - verbose output once the command is run (this isn't needed, but I like to see what the command is doing.
j - compresses archive
f - uses a file name
p - preserves permissions of files within archive


The "." at the end of the command directs the command to archive everything within the current directory.

Once the files are archived, I can get rid of them using the 'rm' command to free up some space.

I thought all of this would be cool to share...

Did some ##slackware log archiving...

Yeah, I had to do some archiving of the logs, as diskspace usage was at 96%. I didn't just archive the channel logs, but also archived my snort and web logs. About the only thing I haven't archived yet are the modsecurity logs (will do that sometime this weekend). Currently, the host's drive space is currently at 74%. The channel logs are still in place, but I've crunched the logs into monthly tar.bz2 files. This renders the logs unsearchable by google (yeah, this sucks), but I had to compromise...they are still downloadable, just not searchable. So, if you need them, they are there for download. Once you download them, you can grep each tar.bz2 after uncompressing them. Hopefully, Google still has the logs cached so that a person searching for an item can still see the cached files. Maybe I'll purchase more drive space so that I can host the logs in an untarred and uncompressed format in the near future.

Speaking of the channel, there has again been some ruckus about someone being banned 'unduly'. People have to recognize that moderating a channel does come at a price. One of these prices is the fact that people can't visit their frustrations on the channel. An individual visited the channel highly upset that Pat froze Slackware-current relating to issues with both the 2.4 and 2.6 kernel. Instead of following advice to follow up with Pat, he continues to vent on the channel, causing a rather heated flame war over something trivial. He was +q'd (meaning his speech was removed), but he evaded +q. He was then "removed" (meaning he was booted, not kicked, from the channel), but came back in the channel with the same attitude. He was then banned for 30 days. Anyone who evades moderation will automatically get a ban. Why 30 and not 7 days? Because, behind the scenes, in private message, the individual was very argumentive and I didn't feel like dealing with him 2 days later for the same offense. After reading the logs, someone had the gall to mention in the channel that the ban was unwarranted...this person thought that the individual was banned because of his views...WRONG. Read the channel guidelines. It states specifically that any +q/ban evasion will be dealt with in a rather harsh manner. Many people do not realize that the ops will never be able to please every single person's views in the channel. I've been doing this a LONG time (4+ years) and no matter if I just sit there and let the channel run itself or if I step in and boot someone, someone ALWAYS complains. It's a no-brainer for me: moderation is what it is. You can take it or leave it. There aren't too many channels on Freenode that aren't moderated. By nature, moderation pretty much means you can't state everything you feel, especially when it ruins the continuity of the channel chat. Is this an oxymoron, especially since Freenode is inhabited mostly by coders and free-thinkers? Every discussion, whether its in real-life in a conference or in someone's home or online on a forum or in a chat room/channel, will have some type of moderation. So, going forward, I'll not be including comments to the ban messages, as this adds confusion to why the person was banned. Really, the channel doesn't need to know why said person was banned after the fact. The ban messages are for the person being banned and it was designed that way by the people who set up the IRC specifications. If you want to know why someone was banned, speak with them directly or read the logs. I've no time to hold some lengthly dialog with someone who thinks that everyone should join an IRC channel and unload their frustrations. I try to think as objectively as possible on anything that goes on in the channel and to be quite honest, there's been a ton of bitching about the ops lately. When I see the non-ops quit pushing the ops' buttons, I'll take them more seriously and get more active in seeing to their needs...but the bellyaching has to stop first. Seriously, its usually the same people bitching about their rights being violated, and if its not the same people, there's usually some association.