Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

xsawyerx (8978)

xsawyerx
  (email not shown publicly)

Journal of xsawyerx (8978)

Thursday March 05, 2009
03:58 AM

Swiss Knife of the Servers - a short IT story

[ #38595 ]
I thought of "Pirates of the Caribbeans" when I wrote the title

Yesterday we noticed some slowness with HTTP on a machine at $work, and I checked the Apache log. It was - to say the least - larger than expected. tail -f default.access_log showed someone is accessing the machine (through IP, not website) quite often, producing 404 errors and using bad user agents. Stuff like googlebawt and the sort. So, we got an attack on the server through Apache, and they are trying to DDOS.

Fortunately, I'm blessed (and I'm not being cynical about this) with an environment of highly skilled programmers. My boss is an OpenBSD and Gentoo fanatic (who could blame him, really?) to just stress the point.

I started working on blocking the originating attack using mod_rewrite on Apache according to the user agent, when my boss said we should focus on iptables block because perhaps it's not coming from a large range of IPs. I wrote a short oneliner in Perl to put the IPs in a hash with a count of how many times they appear. Apparently we were dealing with roughly 100 or so different IPs generating 400Kbit transfer. We've handled more than 10000 times as much on more serious attacks. This was feather-weight.

We created another chain in the iptables and I wrote a short Perl script (originally oneliner, but I wanted it structured in a file) that blocks with iptables anyone that generates more than $limit 404 errors, which I put in the cron and after 3 seconds, the attack was blocked, almost completely. I left it running all night and the log shows it blocked only an additional 22 IPs over the night.

Here is the script, in case anyone's interested. For some odd reason List::MoreUtils's none didn't work for me. Sometimes I get confused by it and it was just faster to do it without it, I know.

I also created a hash of the IPs and counts according to count (yay for reverse)

#!/usr/bin/perl

use strict;
use warnings;
use File::Slurp;

# these next 3 should be absolute paths
my $log        = 'default.access_log';
my $dropped    = 'dropped';
my $my_logfile = 'drop.log';
my $limit      = 150;
my $real       = 1;
my $read_error = 0;
my %by_ip      = ();
my %by_count   = ();

if ( $ARGV[0] ) {
    $real = 0;
}

foreach my $filename ( $log, $dropped ) {
    if ( ! -r $filename ) {
        print "File '$filename' doesn't exist or is unreadable!\n";
        $read_error++;
    }
}

$read_error && die "We have file read errors\n";

open my $fh, '<', $log or die "can't open file '$log': $!\n";
foreach my $line ( <$fh> ) {
    if ( $line =~ /HTTP\/1\.1\"\s404/ ) {
        ( my $ip, undef ) = split /\s/, $line;
        $ip && $by_ip{$ip}++;
    }
}
close $fh;

# not necessary, but useful to see best targets
# %by_count = reverse %by_ip;

while ( my ( $ip, $count ) = each %by_ip ) {
    if ($count > $limit ) {
        # cron does not have PATH variable set
        my $cmd         = "/sbin/iptables -I ddos -s $ip -j DROP";
        my $found       = 0;
        my @dropped_ips = map { chomp; $_ } read_file($dropped);

        foreach my $dropped_ip (@dropped_ips) {
            if ( $dropped_ip eq $ip ) {
                $found++;
                last;
            }
        }

        if ( !$found ) {
            my $cur_time = scalar localtime time;
            my $msg = "($cur_time) $cmd [number of times: $count]\n";
            if ($real) {
                append_file( $my_logfile, $msg );
                system $cmd;
                append_file( $dropped, "$ip\n" );
            } else {
                print $msg;
            }
        }
    }
}

Notes:

  • I should have used some IPC module for the system command, or an iptables module (I wouldn't be surprises to find one)
  • I should have used List::MoreUtils's none
  • I should have checked if File::Slurp slurps the entire file if I do foreach my $line ( read_file($filename) ) and if not, use it
  • I should have used Getopt::Long
  • I should have documented a bit in the code and provide a usage string
  • I could have written this as a module and add a testing suite for it
  • I should have compiled the regexp with qr// at start and then use the compiled version
  • I should have put it in functions (or methods - if I do it as a module) so I could provide various checking mechanisms besides that specific one
  • If I were to have different checking mechanisms, I could allow using a few, then also use Regexp::Assemble for multiple regexp tests
  • What else am I missing?
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • > I should have compiled the regexp with qr// at start and then use the compiled version

    If you're referring to the "/HTTP\/1\.1\"\s404/" in the loop - perl is smart enough to only compile it once, as it doesn't contain any variables that would change it.

    You can test this be running
    $ perl -Mre=debug -le 'for (1..2) { $_ =~ /a/ }'
    and watching the output

  • There is IPTables::libiptc which is a more modern replacement to IPTables::IPv4 (no longer maintained). The risk with these modules is that the kernel's interface to libiptc changes (as it's not a proper published API).

    It's also a good idea to use some kind of privilege separation when needing to do operations as root. For example you watch the Apache logs as an unprivileged user, then to make a change to Netfilter, send an RPC command to another process using a defined and safe interface. The other proce

  • You might want to check out the fail2ban utility. You can set it to monitor log files for smtp, imap, ssh, apache failures and after a set number ban that IP for a set time. Patterns, durations, actions etc. are all configurable.