Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

perigrin (3495)

perigrin
  chrisNO@SPAMprather.org
http://chris.prather.org/
AOL IM: marceusx (Add Buddy, Send Message)

After Middle School Chris bounced through various high schools around the state of Florida for a bit. He ended up getting a BA (in English) from the University of Central Florida at about the time others his age were getting their MAs. His (now) wife, their child, and he went to Europe for a bit (there was a programming job in Scotland) and came back unemployed where upon he got offered a job in St. Paul. He moved on from that job after a few years to one where he was paid to do the things he previously did as a hobby. That sounds much more exciting than it really was. Really his life is based on two kids, two cats, and a bunny.

Journal of perigrin (3495)

Wednesday July 04, 2007
10:08 PM

Brad Bird the American Miyazaki

Brad Bird the American Miyazaki

Brad Bird (Ratatouille, The Incredibles) seems perhaps to be one of the best animation writer/directors in America. He is part of the new generation of Animators involved with Pixar, who seem to be viewing Cartoons as a new form of storytelling rather than just as something to pander to children. This view of animation is the ideal Walt Disney had, and the concept found in the Japanese Anime of Miyazaki (My Neighbor Totoro, Spirited Away)

I just finished watching Ratatouille, a story about family, food, france and rats. It tells the tale of Remy, a rat who has a taste for fine cuisine. He ends up moving to paris and finding work (after a fashion) as a Chef in a fine French restaurant. While the movie is appropriate for children it doesn’t treat it’s audience as anything less than intelligent mature people. This is the same feeling I got from watching The Incredibles which felt more like a Graphic Novel than a Super Hero Cartoon.

While I understand this isn't a Perl related post (something I try to maintain here), I felt it deserved a wider audience than my personal blog.

Sunday June 10, 2007
11:58 PM

Spam has worked!

So I run The Open Guide to the Twin Cities. Like all of the guides, I have had issues overcoming spam. I’ve implemented stuff to try to defeat it in the short term, and updated software to give me new tools to work with in the long term. However, what do I do with this mixed blessing. Nearly 5K hits have come in to my guide in the last 7 months via google using the keyword “sexgirl”. That’s 1/6th of my total traffic (5K out of around 30K hits) for a keyword set by a spammer.

Sadly I don’t know enough about the places in the Twin Cities that I could possibly use this keyword recognition for my advantage. Is it wrong of me to want to take advantage of this? Does anybody have any more … intimate knowledge of the cities?

Tuesday May 22, 2007
09:00 AM

Useful SVK One Liner

svk co -l | perl -ne 'm|(/Users/perigrin/.*$)|; print "svk pull $1\n" if $1' | uniq

In my case I've piped this into a script and then run that script. Voila instant updates of all my checkouts. Yes this probably could be improved upon.

Wednesday May 16, 2007
12:16 AM

OpenGuides Work

Since this is “Visit other OpenGuides Week” Kake has been helping out with the TwinCities Guide. After she and Bob sorted out a performance issue for me (which involved not having a proper index), things have been moving nicely. I’d forgotten how much fun it was to work on and with the Guide.

The OpenGuides for those of you who aren’t aware are a series of Perl based Wikis that use Wiki::Toolkit. Unlike most wikis W::T allows for structured Metadata which means you can easily search for “All Restaurants within 500 meters of [here] that are Open at 6pm”. They’ve been slowly growing for a few years starting with London (for locals try the younger Randomness Guide).

All of this is a prelude to saying that the Twin Cities guide is looking for people to come on board as an Admin. If someone in the Cities wants to pitch in and help, please contact me!

Sunday May 13, 2007
07:29 PM

Metawork

Today I’ve been going through my dev folder doing lots of metawork, cleaning out old projects, cleaning and releasing new versions of the tiny ammount of stuff I have on CPAN, checking it all into off site SVN if neccessary.

While I was going though it I discovered my little XSLT style sheet that converts the perlmonger XML into foaf/RDF. For those of you who care you can view the results.

Monday April 30, 2007
07:57 PM

Perl Success Story

I wanted to write down somewhere public my recent lovely experience using Perl.

I’ve been a paid perl developer for a little over 5 years now. I’d used perl for about 2 years before that. So I’m well into my ‘Intermediate’ stage of Perl. I’ve heard the stories of people writing Web apps in a couple of days and putting them online and making millions … most of them involve Ruby on Rails … and extensive lying about how much “millions” really is.

Last week I wrote a web app that saturday night recorded about $25,000 USD in sales. Here’s the tale…

Some months ago my wife asked me if it was possible to write an application to automate the checkout process for our daughter’s school’s Silent Auction event. I thought about it and said “yes that would be possible”. Obviously this meant I volunteered to do it, but it happened long enough ago that it entirely slipped my mind.

Early last month she came back and asked me how long it would take, and being estimation is never my strong suite I said several weeks if everything went well. In a situation like this, nothing goes well … around this time I started formulating a plan as to how I could make the development time as fast as possible. I knew the app would be a basic shopping cart script and I’d been facinated by Chris Laco’s Handel app for months now. I knew it had been ported to DBIx::Class and had heard lots of stories about how quick DBIC was to develop with … it doesn’t hurt that mst is the most vocal user of Bender on the planet.

So I tested out Handel and after several false starts (note only the current dev release of Handel is ported to DBIC, and it required the SVN version of DBIC at the time). I managed to formulate a plan. But I had no data, and worse no schema. The organizers of the auction quickly got me a schema for the Auction items in the form of an Excel spreadsheet. I attempted to convince SQLT to translate this into a SQLite Database but that didn’t work spectacularly well. It choked on the spreadsheet no matter what I tried. Finally I managed to get a two step process where SQLT built me a schema and a hacked version of Spreadsheet::ParseExcel gave me something SQLite could import.

After that the very useful tutorials from DBIC’s pods and Handel got me boot strapping a rudimentary webapp. I used the SAWA framework because I’ve been working with it for 4 years now, and wanted to experiment with it and Moose, for output I used TT2 because it is the most straight forward Template engine I’ve tried (including XSLT, TAL, HTML::Template, and some custom stuff … note Mason isn’t in here) for Perl Object -> HTML/Text generation.

The day of the event I showed up to the event site early to teach the volunteers who were going to do data entry with me how the application works. This is part one of the secret of my success I knew at the beginning I was only going to have 4 - 5 users max, and I could train them personally. There was almost no training really, when I opened up the first page for the order entry workflow I discovered a bug and couldn’t actually train them on it before they had to leave to get ready for the auction!

Happily the bug turned out to be a simple fix, and once I figured out why it was happening I resolved it and my main workflow was restored. I figured out the odds and ends I needed to with the venue’s network to get the computers printing (required a password), and everybody using the website (hosted on my MacBook, with DynDNS providing a subdomain on prather.org for the evening). I fixed a small issue that some items had never made it into the system (A second excel spreadsheet they’d sent two days before but I’d missed in my email). Once the event started, I basically waited around for my wife to appear and spent the first hour or so walking around until they started closing out items.

The rest of the night was a flurry of data entry, only two items hadn’t made it into the database, and a half dozen bidders had IDs but no record in the database (happily some of the hacks I’d done with Handel were able to accommodate this situation though it was not ideal.) I’ve been told that it has gone better than last year’s manual process … we actually finished order entry in real time with the closing of the data, and were able to proceed to checkout fairly quickly.

Checkout was a bit of a nightmare, we only had one printer, and it wasn’t near the checkout process. But after about 30 minutes we reached the point where all of our time was spent dealing with orders that had gotten confused or needed changes (sometimes data was wrong on the sheets, some times there was keying errors by us …we are only human). All in all everything seemed to go well.

Final tally that night was around $25,000 in auction items, and 7K from a “wall of wine” thing that I’m not entire sure I understand … all raised for the PTO which uses the money to pay French interns to come, stay and work at the school.

So I have first hand accounts that you can write a nice perl commerce app in a week and have it perform admirably under the pressure.

Wednesday February 28, 2007
06:49 PM

Perl is Dead. Long Live Perl!

So depending on who you ask, Perl5 is dead. Or is it Perl6? Or wait no any Perl older than 2 years … or wait no Perl1. I’ve spent some time recently exploring all these different camps, and I have to say … Perl is alive and well … all of them.

Perl6 excites a lot of people, and they’re vocal about how when it lands it’ll be the next great thing. And guess what, it’s landed … if you like Haskell and hacking on languages. Nobody is using Perl6 for their pet projects, unless their pet project is perl6 (if I am wrong, write an article, get it published, and tell someone who can promote it). Perl5 is quietly doing what it is supposed to be doing, being damn stable and working for millions of people and billions of dollars worth of code. This isn’t sexy like multi-dispatch-quantum-operators … but it’s sexy like the original Hummer because it get’s the job done in almost any environment. (If you are doing sexy multi-dispatch-quantum-operators in Perl5, write it down, get it published and tell someone who can promote it).

Perl5 is slowly gathering new features as they are either:

  1. back ported from perl6 (//= anyone?)
  2. making things that should be easy, easy (the features pragma that adds switch to the core language?)
  3. make the world as we know it better (dmq’s regexp work?).

Perl1 is being used to test Parrot how sexy is that?! (Someone quick, write an article… never mind you get the idea.) All of these things are good things.

Ultimately Perl5 is growing the way a stable mature system should, Perl6 is growing the way an unstable mutant system should (that’s not derogatory, some of my best friends came from unstable mutants). People need to realize that both are creating exciting things, and the people creating the exciting things need to pause a moment and tell others what they’re doing and why it’s exciting.

Note that this post could bear little to no resemblance to reality as *you* see it, it does however seem to capture he angst-y zeitgeist as *I* see it right now.

Wednesday September 27, 2006
10:12 AM

XML::LibXML Operation in Progress

So I've been playing around with AxKit2 on my laptop. It's a very nice enviroment for playing. Matt and Jorge have done fun things so far. Since I ran into an issue with XML::LibXML twice, before and after the old laptop died, I want to post the solution so I can find it again later.

127.0.0.1:52145 L3 FATAL PLUGIN ERROR: Operation in progressOperation in progressOperation in progressOperation in progressOperation in progressOperation in progress at /opt/local/lib/perl5/site_perl/5.8.8/darwin-2level/XML/LibXML.pm line 530.

Turns out this is libxml trying to lookup external enttiies on the net. Which was failing behind my firewalled/proxy environment. The solution is give libxml net access, in my case by using tsocks and a Socks proxy. Hope this helps.

Thursday September 07, 2006
02:53 PM

Perl 5.8 slowness ...

So at work we are going through and updating our toolset. This included moving from Perl 5.6.1 to 5.8.8 (Yay!). However we discovered a deal breaking issue. Perl 5.8.8 is much slower than 5.6.1. We discovered a slowdown of around 100% between 5.6.1 and 5.8.8 using as close to default installs as we could. (Things that used to take 30 seconds were now taking a minute, and we have some time critical production code.) After doing some inspection I found the largest cause to the slowdown was the change in 5.8.0 to use the system malloc instead of perl’s internal malloc (-Dusemymalloc) by default on Solaris. Fixing this we went back to a roughly 20% slowdown. I never found a clear explanation of why they switched the default malloc other than a reference to corruption and segfaulting when dealing with over 2G of memory (was that RAM or LARGEFILE support?) … does anybody have a pointer?

I’ve been trying to track this down for a couple weeks, and I’ve discovered that this slowdown is true on every platform I’ve tested it on (Solaris, Darwin, Linux). I also found a reference to it on P5P from several years ago that appears to be warnocked. I’m stumped as to what else to try. Below are the benchmarks from my Linux machine along with the benchmark script.

Note that this may look like I’m arguing over peanuts with the tests running in less than 10 seconds. Multiply everything by 4 to roughly get the speeds on our Solaris boxes at work.

perigrin@schroedinger ~/dev/perl_speed/rdsmerge $ perl5.9.4 split_test.pl
do_split:        4 wallclock secs ( 3.59 usr +  0.00 sys =  3.59 CPU) @ 69637.88/s (n=250000)
do_split_loop:   8 wallclock secs ( 7.38 usr +  0.01 sys =  7.39 CPU) @ 33829.50/s (n=250000)
do_split_loop_ia:        7 wallclock secs ( 6.65 usr +  0.00 sys =  6.65 CPU) @  0.15/s (n=1)

perigrin@schroedinger ~/dev/perl_speed/rdsmerge $ perl5.8.8 split_test.pl
do_split:        5 wallclock secs ( 3.99 usr +  0.00 sys =  3.99 CPU) @ 62656.64/s (n=250000)
do_split_loop:   9 wallclock secs ( 8.55 usr +  0.00 sys =  8.55 CPU) @ 29239.77/s (n=250000)
do_split_loop_ia:        7 wallclock secs ( 7.55 usr +  0.01 sys =  7.56 CPU) @  0.13/s (n=1)

# -Dusemymalloc
perigrin@schroedinger ~/dev/perl_speed/rdsmerge $ perl5.8.8+ split_test.pl
do_split:        3 wallclock secs ( 2.82 usr +  0.00 sys =  2.82 CPU) @ 88652.48/s (n=250000)
do_split_loop:   6 wallclock secs ( 5.84 usr +  0.01 sys =  5.85 CPU) @ 42735.04/s (n=250000)
do_split_loop_ia:        6 wallclock secs ( 5.55 usr +  0.01 sys =  5.56 CPU) @  0.18/s (n=1)

# -Dusemymalloc
perigrin@schroedinger /home/projects/perl_speed/rdsmerge $ perl5.8.5  split_test.pl
do_split:        3 wallclock secs ( 2.75 usr +  0.00 sys =  2.75 CPU) @ 90909.09/s (n=250000)
do_split_loop:   5 wallclock secs ( 6.07 usr +  0.00 sys =  6.07 CPU) @ 41186.16/s (n=250000)
do_split_loop_ia:        6 wallclock secs ( 5.52 usr +  0.00 sys =  5.52 CPU) @  0.18/s (n=1)

# -Uusemymalloc aka System Malloc
perigrin@schroedinger /home/projects/perl_speed/rdsmerge $ perl5.8.5- split_test.pl
do_split:        4 wallclock secs ( 3.50 usr +  0.00 sys =  3.50 CPU) @ 71428.57/s (n=250000)
do_split_loop:   8 wallclock secs ( 7.30 usr +  0.01 sys =  7.31 CPU) @ 34199.73/s (n=250000)
do_split_loop_ia:        6 wallclock secs ( 6.28 usr +  0.00 sys =  6.28 CPU) @  0.16/s (n=1)

perigrin@schroedinger ~/dev/perl_speed/rdsmerge $ perl5.6.2 split_test.pl
do_split:        3 wallclock secs ( 2.57 usr +  0.00 sys =  2.57 CPU) @ 97276.26/s (n=250000)
do_split_loop:   6 wallclock secs ( 6.27 usr +  0.00 sys =  6.27 CPU) @ 39872.41/s (n=250000)
do_split_loop_ia:        6 wallclock secs ( 5.63 usr +  0.01 sys =  5.64 CPU) @  0.18/s (n=1)

# -Dusemymalloc
perigrin@schroedinger ~/dev/perl_speed/rdsmerge $ perl5.6.2+ split_test.pl
do_split:        1 wallclock secs ( 2.22 usr +  0.00 sys =  2.22 CPU) @ 112612.61/s (n=250000)
do_split_loop:   6 wallclock secs ( 5.69 usr +  0.00 sys =  5.69 CPU) @ 43936.73/s (n=250000)
do_split_loop_ia:        5 wallclock secs ( 5.20 usr +  0.00 sys =  5.20 CPU) @  0.19/s (n=1)

--- split_test.pl ---

#!/usr/local/bin/perl5

use Benchmark;
use strict;
use warnings;

my $r = timeit( 250000, sub { do_split(); } );
my $r2 = timeit( 250000, sub { do_split_loop(); } );
#my $r2 = timeit( 250000, sub { do_split_noregex(); } );
#my $r3 = timeit( 250000, sub { do_split_bytes(); } );
my $r3 = timeit( 1, sub { do_split_loop_ia(); } );

print "do_split:\t", timestr($r), "\n";
print "do_split_loop:\t", timestr($r2), "\n";
print "do_split_loop_ia:\t", timestr($r3), "\n";

exit;

sub do_split
{
    my $input = "0|hucalc8|hucalc8a0|00:00:04|2|0x00000b0000339fa7|3620316P6|E|P|0.125000|PRICE| BID|3620316P6|GNPL   006278  8.500 07/15/18|100.469000|8.446256|8.446256|5.398844|20060808|20060823|";

    my @fields = split /\|/, $input;
    my $f1 = $fields[1];
    my $f2 = $fields[1];

    return;
}

sub do_split_loop
{
    my $input = "0|hucalc8|hucalc8a0|00:00:04|2|0x00000b0000339fa7|3620316P6|E|P|0.125000|PRICE| BID|3620316P6|GNPL   006278  8.500 07/15/18|100.469000|8.446256|8.446256|5.398844|20060808|20060823|";

    my @fields = split /\|/, $input;
    my $f1 = $fields[1];
    my $f2 = $fields[1];
    my $foo;
    my @out;

    ## loop 15 building array by force
    for ( my $i = 0; $i < 15; $i++ ) {
        if (defined $fields[$i]) {
            $out[$i] = $fields[$i];
        }
    }

    return;
}

sub do_split_loop_ia
{
    my $count = 250000;
    my $input = "0|hucalc8|hucalc8a0|00:00:04|2|0x00000b0000339fa7|3620316P6|E|P|0.125000|PRICE| BID|3620316P6|GNPL   006278  8.500 07/15/18|100.469000|8.446256|8.446256|5.398844|20060808|20060823|";

    my @out;
    my $foo;

    foreach (0..$count) {
        my @fields = split /\|/, $input;

        ## loop 15 building array by force
        for ( my $i = 0; $i < 15; $i++ ) {
            if (defined $fields[$i]) {
                $out[$i] = $fields[$i];
            }
        }

        $foo = join( ',', @out );
    }

    return;
}

Thursday July 21, 2005
10:56 AM

JSAN rocks

Will the sadness never end? Back in the glorious days of 1999, Microsoft in their
infinite wisdom invented DHTM  Behaviors. Which are actually a fairly cool idea, basically you can bind
JavaScript functions to HTML elements via a CSS property.

I know about them because at the last job where the front end guys built a fairly complex
application using them. I wrote the backend and never touched them. They were
Microsoft only too. It never made it very far at the W3C.

Luckily someone has used XBL to port Microsoft's proprietary CSS extension to Firefox. W00t. Sadly though, it works *better* in Firefox than in IE 6. Microsoft in their infinite wisdom seem to only support a single Script object per Behavior file, while Firefox happily parsed and loaded all my script objects and did what I wanted.

Today I discovered the work around for IE. Enter JSAN. (Cue the Heralding Trupets). JSAN imports my javascript and loads it without my having to have several script sections. I simply make sure that JSAN is loaded at the page level then JSAN.use() whatever I need inside the Behaviors. This means that I can build nice clean modular DHTML behaviors that can be applied and controlled the same way CSS is applied and controlled.