use Perl

Yesterday, I was hacking on a script to extract a series of numeric values from a data set. I wanted to understand the data better than just looking at (min | max | average).

So, after I had a list of values, one per line, I followed my first instinct and loaded that file into Excel. (I should mention that I view using spreadsheets a symptom of a larger problem, not a part of a working solution.) After a while, I realized that I had a lot of data to analyze, a lot of tests to run, and this was the fast track to weeks of needless agony and make-work.

Excel did have one benefit -- it helped me understand standard deviation a little better. It's a little annoying though that Excel's =STDEV() function, the most obvious function to use for calculating standard deviation, is actually standard deviation of a sample, not standard deviation of an entire population. Once I refreshed my memory of the concepts involved, it took a while to figure out why the standard formula wasn't agreeing with Excel's results. Sure enough, the =STDEVP() function did match with better than random precision.

I took a quick look on CPAN, but didn't find anything that does standard deviation. I know it's there, but I didn't want to download a huge Math library to calculate a simple function. So I wrote a quick and dirty std-dev instead:

#!/bin/env perl -w

use strict;
use List::Util qw(sum min max);

chomp(my @values = <>);
my $n = @values;
my $avg = sum(@values)/$n;
my $std_dev = sqrt(sum(map {($_ - $avg) ** 2} @_) / $n);

print "total   = $n\n";
print "std_dev = $std_dev\n";
print "avg     = $avg\n";
print "min     = ", min(@values), "\n";
print "max     = ", max(@values), "\n";

The hard part was the single line of code to calculate standard definiton. That was translated verbatim from the definition on the wikipedia page.

This little script, along with nth , reduced a bunch of time consuming Excel drudgery into a nearly autonomic piece of analysis. ;-)