use Perl

It sometimes amazes me when I manage to achieve a great deal with a little Perl. Especially so when it leverages (see, I could be management) a module of my own. The latest idea took a while coming. I'm informally part of a group of people coding up a project called Gravel whose task is to build a suite of libraries to parse, manipulate, generate and convert between various vector based animation file formats such as SWF, SVG and later formats like MNG. At the moment, it's mostly aimed at parsing SWF (Flash) files.

Now, the parser is fairly robust, but we wanted a whole range of SWF files to throw at the parser and see what tags we were still missing or whether there would be any crashes (yes, it's written in C, but I stay away from that). One of my more recent contributions to the CPAN has been WWW::Search::Google, which uses the Google Web API. Now, Google allows you to specify the filetype of the results of searches. So all I needed to do was search Google for all the SWFs it could find with, say, "a" in the name and download them to later test them out.

And it was easy. Here's the code I used. Note that you'll need your put your own Google Web API key where it says "XXXX".

#!/usr/bin/perl -w
use strict;
use WWW::Search;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new(timeout => 30, agent => 'SWF-Grabber/1.4');

my $search = WWW::Search->new('Google', key => "XXXX");
$search->native_query("filetype:swf +a");

while (my $result = $search->next_result()) {
    my $url = $result->url;
    my $file = $url;
    $file =~ s:^.+/::;
    next if $file =~ /\?/;
    print "Downloading $url\n";
    next if -f $file;
    $ua->mirror($url, $file);
    sleep 10;
}

951 SWFs and counting...