Ibuildings Blog: - ibuildings.nl

Boys blog

	ing. Boy Baukema Software Engineer
	Boy is one of our senior software engineers. He has worked on a wide range of projects using many different technologies, JavaScript being one of his favourites. When he blogs, he likes to focus not only on technology, but also on its theoretical background.

Thursday, 24 July 2008

My little cron Bash framework (locking and logging)

Occasionally, I have a maintenance task for one of my programs: something has to be done in the background every so often.
From rsyncing content between servers to importing data received from external parties into my application.

Most of the products we make run on Linux servers, so I use cron to make scripts run every x amount of minutes/hours/days/etc.

With most of these small scripts I want the following:

Locking (so if a script takes particularly long time, it won't be running twice concurrently)

Logging (Never blindly trust the machine, or a human for that matter, to always do the right thing)

Cleaning up aformentioned logs (because over time they can grow very fast)

And while you can do this excellently with PHP's Command Line Interface it doesn't make sense to make a PHP script for simple rsync commands when the native scripting of the shell (Bash scripting) is also available.

So here are some common script snippets that have evolved over the years to perform these simple tasks. Let me know what you think!

Locking
I just use an empty file in /tmp so I can easily see which scripts are running and from what time they are running.

Just replace MY_SCRIPT_NAME by (a short) name of your script.

CODE:

#!/bin/bash

# Lock file in the /tmp directory
LOCKFILE = "/tmp/MY_SCRIPT_NAME.lck"

# Check wether we have a lock file, if so, don't execute
if [ -f $LOCKFILE ]; then
exit 1
fi

# Create the lock file
/bin/touch $LOCKFILE

# Do your thing here

# Done, removing lock file
rm -f $LOCKFILE

Logging
Log to a .log file in a special directory in /var/log/scripts/, always add a start and end time.

Just replace MY_SCRIPT_NAME and make sure the log directory (/var/log/scripts/MY_SCRIPT_NAME/) exists.

CODE:

#!/bin/bash

# Place a log file in it's own directory in /var/log/scripts/
LOGFILE='/var/log/scripts/MY_SCRIPT_NAME/'`date +%Y_%m_%d`'.log'

# Append start date to the log file
date +[START\ TIME:\ %H:%M] >> $LOGFILE

# Do your thing here and log any output by adding '>> $LOGFILE' after the command
# Example:
# ls -lah >> $LOGFILE

# Append end time to the log
date +[END\ TIME:\ %H:%M] >> $LOGFILE

Cleaning up logs
You can do this in the script that logs or you can do it separately if you're really worried about performance with a lot of logs.

Again change MY_SCRIPT_NAME and make sure the log directory is set properly.

CODE:

#!/bin/bash

# How many days old the logs can be, maximum.
MAX_DAYS_OLD ="7"

# Find and delete .log files older than $LOG_MAX_DAYS_OLD days
/usr/bin/find /var/log/scripts/MY_SCRIPT_NAME -type f -name *.log -mtime +$MAX_DAYS_OLD -delete

Everything together
Combined, we get a little 'framework' for Bash cron jobs.
Just change the SCRIPT_NAME variable from MY_SCRIPT_NAME to ... something else and make sure the directory mentioned in the LOGDIR variable exists.

CODE:

#!/bin/bash

## CRON BASH SCRIPT 'FRAMEWORK' ##
## V0.1 Boy Baukema, Ibuildings.com ##
SCRIPT_NAME="MY_SCRIPT_NAME" # Script name, ONLY alphabetical and _ characters

## VARIABLES
LOGDIR           = "/var/log/scripts/$SCRIPT_NAME/" # Directory where the logs are placed
LOGFILE          = $LOGDIR`date +%Y_%m_%d`'.log'    # Log the output in /var/log/scripts/MY_SCRIPT_NAME
LOCKFILE         = "/tmp/$SCRIPT_NAME.lck"        # Lock file in the /tmp directory
LOG_MAX_DAYS_OLD ="7"                               # Keep log files for 7 days

## FUNCTIONALITY

#Check wether we have a lock file, if so, don't execute
if [ -f $LOCKFILE ]; then
exit 1
fi

# Create the lock file
/bin/touch $LOCKFILE

# Append the start time to the log file
date +[START\ TIME:\ %H:%M] >> $LOGFILE

####<MAIN>####

# Do your thing here and log any output by adding '>> $LOGFILE' after the command
# Example:
# ls -lah >> $LOGFILE

####</MAIN>####

# Append the end time to the log file
date +[END\ TIME:\ %H:%M] >> $LOGFILE

# Find and delete .log files older than $LOG_MAX_DAYS_OLD days
/usr/bin/find $LOGDIR -type f -name *.log -mtime +$LOG_MAX_DAYS_OLD -delete

# Done, removing lock file
rm -f $LOCKFILE

If you want to use or modify this, be my guest, but naturally I or Ibuildings cannot accept responsibility for any damages

. Running this is at your own risk.

I do also recommend that you don't just Copy & Paste, but rather take the time to understand what this does and why.
I think it's important that you never blindly run any script on your servers as root without understanding what it does and how it does it. Just like you would not do it with a PHP snippet

Geplaatst door Boy Baukema op 10:21 | Reacties (6)

Tags voor deze bijdrage: bash, framework, linux, locking, logging, maintenance, open source, scripting, shell

Trackbacks

Trackback-URI voor dit artikel

Geen Trackbacks

Reacties

Geeft reacties weer als (Lineair | Samengevoegd)

I found myself migrating a lot of bash scripts to python and php, they where growing in size and complexity and they were becoming very difficult to maintain.

#1 Federico (Homepage) op 24-07-2008 15:51

I generally let my scripts 'evolve'.
The simplest of maintenance tasks usually start out as simple Bash shell script.
Tasks that require more than a screen full of scripting code get split up in a Bash part (for non-logic stuff like logging and locking) and PHP stuff (for the 'meat' of the task), because when you're a PHP shop it's easier to maintain for everybody.

Then some have evolved to require functions for code reuse.

And a select few actually end up entire object oriented applications in and of themselves (though I can only think of one example right now).

I guess the point is that you start as simple as possible.
But I generally like (at least for small to medium tasks) to let the non-logic of a task be handled by the shell, which is better suited for the environment anyway.

#1.1 Boy (Homepage) op 24-07-2008 20:45

What is wrong with using flock?

#2 Shawn op 24-07-2008 18:21

Absolutely nothing.

But if all I want to execute is an rsync, or deleting some cache, or simple maintenance tasks, I generally don't like to make a PHP script for that.

And besides proving to myself that I know a few basics about shell scripting (which I would think is pretty handy when you're writing software which is hosted on Unix platforms) I don't like the overhead of PHP for simple tasks.

I have built some (fairly large) command line tools that are purely PHP, which is great for maintainability.
And I definitely think PHP has a place on the command line.
Though not necessarily in small maintenance scripts where I don't even want to care whether a PHP binary is on the machine.

#2.1 Boy (Homepage) op 24-07-2008 21:05

Using files is racy and invites bugs.
Instead use directories or symlinks
which are guaranteed to be created
atomically.
I.e rather than
if [ ! -f file ]; then touch file; fi

use

if ! mkdir lockdir; then echo already locked; fail;fi

#3 mlh op 26-07-2008 14:26

The script name can be detected like this

# Figure out the name of this script (without the path, etc)
SCRIPTNAME="$(basename $0)"

#4 Shiraz Kanga op 02-08-2008 10:32

De auteur staat geen reacties toe op dit artikel

Ibuildings - Blog - ibuildings.nl

Ibuildings - the PHP Professionals

Blog

Boys blog

Thursday, 24 July 2008

My little cron Bash framework (locking and logging)

Tag cloud

Abonneren op dit weblog