Processing Unix logfiles

Processing Unix logfiles can be tricky because we have to keep track of all the entries already sent to SCOM and because these files usually rotate regularly.

If a logfile entry is unique, either by having a date or by having an ID it is possible to record what was already processed and sent to SCOM, so we only have to report new entries.

I've created a shell script that finds and reports all the new occurrences of a string inside a log file.

In this case I'm only reporting the number the occurrences but the script can easily be changed to report the complete lines.

#!/bin/bash

# -- Parameters:

# $1 - Pathname to logfile

# $2 - String to search: Regular expression for 'grep'

umask 022

# -- Create tmp file name

ftmp=`echo "$1" | tr -dc '[:alnum:]'`"-"`echo "$2" | tr -dc '[:alnum:]'`

> $ftmp.send

# -- If logfile doesn't exist just initialize previous found file

if [ ! -f "$1" ] || [ ! -s "$1" ]; then

   > $ftmp.prev

else

  # -- Search log for string

  grep "$2" "$1" > $ftmp.new

  # -- Process found lines: if not already sent them send them

  if [ ! -f $ftmp.prev ] || [ ! -s $ftmp.prev ]; then

    cp $ftmp.new $ftmp.send

  else

    (while read a

     do

       rm $ftmp.found 2>/dev/null

       (while read b

        do

          if [ "$b" == "$a" ]; then echo "" > $ftmp.found; break; fi

        done) < $ftmp.prev

        if [ ! -f $ftmp.found ]; then echo "$a" >> $ftmp.send; fi

     done) < $ftmp.new

  fi

  mv $ftmp.new $ftmp.prev

fi

# Count the number of lines to send

wc -l $ftmp.send

exit 0