Wednesday, September 05, 2007

Archiving Logs

I've logs that I gather and put into a directory named /home/status. There's a script that runs commands and directs the output to files within this directory. The script runs every hour of every day. The output is dumped into a directory akin to "070101/", which means 2007-01-01. Each directory would have 24 files. I periodically archive these (by year) into tar.gz files.

Here's how the directory looks:

root@starchild:/home/status/070101# ls
070101.00.starchild.txt 070101.04.starchild.txt 070101.08.starchild.txt 070101.12.starchild.txt 070101.16.starchild.txt 070101.20.starchild.txt
070101.01.starchild.txt 070101.05.starchild.txt 070101.09.starchild.txt 070101.13.starchild.txt 070101.17.starchild.txt 070101.21.starchild.txt
070101.02.starchild.txt 070101.06.starchild.txt 070101.10.starchild.txt 070101.14.starchild.txt 070101.18.starchild.txt 070101.22.starchild.txt
070101.03.starchild.txt 070101.07.starchild.txt 070101.11.starchild.txt 070101.15.starchild.txt 070101.19.starchild.txt 070101.23.starchild.txt
root@starchild:/home/status/070101#
Here are the archives I currently have:
root@starchild:/home/status# du -h sensorstat_logs-200*
748k sensorstat_logs-2005.tar.bz2
3.5M sensorstat_logs-2006.tar.bz2
7.3M sensorstat_logs-2007.tar.gz
root@starchild:/home/status#

This is a listing of the month of January (unarchived):

root@starchild:/home/status# ls -l | grep 0701
drwxr-xr-x 2 root root 4096 Jan 1 2007 070101/
drwxr-xr-x 2 root root 4096 Jan 2 2007 070102/
drwxr-xr-x 2 root root 4096 Jan 3 2007 070103/
drwxr-xr-x 2 root root 4096 Jan 4 2007 070104/
drwxr-xr-x 2 root root 4096 Jan 5 2007 070105/
drwxr-xr-x 2 root root 4096 Jan 6 2007 070106/
drwxr-xr-x 2 root root 4096 Jan 7 2007 070107/
drwxr-xr-x 2 root root 4096 Jan 8 2007 070108/
drwxr-xr-x 2 root root 4096 Jan 9 2007 070109/
drwxr-xr-x 2 root root 4096 Jan 10 2007 070110/
drwxr-xr-x 2 root root 4096 Jan 11 2007 070111/
drwxr-xr-x 2 root root 4096 Jan 12 2007 070112/
drwxr-xr-x 2 root root 4096 Jan 13 2007 070113/
drwxr-xr-x 2 root root 4096 Jan 14 2007 070114/
drwxr-xr-x 2 root root 4096 Jan 15 2007 070115/
drwxr-xr-x 2 root root 4096 Jan 16 2007 070116/
drwxr-xr-x 2 root root 4096 Jan 17 2007 070117/
drwxr-xr-x 2 root root 4096 Jan 18 2007 070118/
drwxr-xr-x 2 root root 4096 Jan 19 2007 070119/
drwxr-xr-x 2 root root 4096 Jan 20 2007 070120/
drwxr-xr-x 2 root root 4096 Jan 21 2007 070121/
drwxr-xr-x 2 root root 4096 Jan 22 2007 070122/
drwxr-xr-x 2 root root 4096 Jan 23 2007 070123/
drwxr-xr-x 2 root root 4096 Jan 24 2007 070124/
drwxr-xr-x 2 root root 4096 Jan 25 2007 070125/
drwxr-xr-x 2 root root 4096 Jan 26 2007 070126/
drwxr-xr-x 2 root root 4096 Jan 27 2007 070127/
drwxr-xr-x 2 root root 4096 Jan 28 2007 070128/
drwxr-xr-x 2 root root 4096 Jan 29 2007 070129/
drwxr-xr-x 2 root root 4096 Jan 30 2007 070130/
drwxr-xr-x 2 root root 4096 Jan 31 2007 070131/
The command I use to archive the directories within /home/status is:

tar cvjfp sensorstat_logs-2007.tar.gz .

c - creates the archive
v - verbose output once the command is run (this isn't needed, but I like to see what the command is doing.
j - compresses archive
f - uses a file name
p - preserves permissions of files within archive


The "." at the end of the command directs the command to archive everything within the current directory.

Once the files are archived, I can get rid of them using the 'rm' command to free up some space.

I thought all of this would be cool to share...