Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Matts (1087)

Matts
  (email not shown publicly)

I work for MessageLabs [messagelabs.com] in Toronto, ON, Canada. I write spam filters, MTA software, high performance network software, string matching algorithms, and other cool stuff mostly in Perl and C.

Journal of Matts (1087)

Thursday March 18, 2004
05:43 PM

Something you might find useful...

[ #17956 ]

I have a problem: very fast moving logs that I want to monitor in realtime for various "tokens". I asked on #perl if anyone knew of a standard tool (like `watch` or `tail`) that could do this but nobody did, so I wrote this:

#!/usr/bin/perl -w
 
use strict;
use Getopt::Long;
use Time::HiRes qw(time);
 
my @Watches;
my $interval = 1;
GetOptions("watch=s" => \@Watches, "interval=i" => \$interval);
 
print "Watching logs every $interval seconds\n";
$|++;
my $secs = 0;
my $total = 0;
my @counts = (0 x @Watches);
my $tzero = time;
my $t0 = time;
while (<>) {
    $total++;
    for my $i (0 .. $#Watches) {
        if (index($_, $Watches[$i]) >= 0) {
            $counts[$i]++;
        }
    }
    my $curtime = time;
    my $diff = $curtime - $t0;
    if ($diff >= $interval) {
        $t0 = $curtime;
        printf "\rLines/s: %0.2f", ($total / $diff);
        $total = 0;
        if (@Watches) {
            for my $i (0 .. $#Watches) {
                printf ", $Watches[$i]/s: %0.2f", ($counts[$i] / $diff);
                $counts[$i] = 0;
            }
        }
        # fixme - don't use fixed num of spaces here
        print "                                  ";
    }
}

Run it without any options to get the number of log lines per second (pipe the logs in with tail -f) or pass in --watch FOO to count the lines containing the string "FOO".

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • Maybe overkill for your purposes, but sec [sourceforge.net] looks interesting. I haven't used it, but heard about it at a PM meeting.
  • The tail -f from the PPT or the File::Tail (some of the very first hits for "tail" from search.cpan.org) don't do what you want?
    • They are totally different. I already have a tail command. If I run the above script with --watch MAIL it tells me how many lines matching /MAIL/ it sees per second.
  • Sounds a lot like what snort does. Perhaps it would be worth sending your logs past some sort of network interface so that snort can see them.