Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

acme (189)

acme
  (email not shown publicly)
http://www.astray.com/

Leon Brocard (aka acme) is an orange-loving Perl eurohacker with many varied contributions to the Perl community, including the GraphViz module on the CPAN. YAPC::Europe was all his fault. He is still looking for a Perl Monger group he can start which begins with the letter 'D'.

Journal of acme (189)

Monday May 27, 2002
11:00 AM

I love it when Perl comes together

[ #5237 ]
It sometimes amazes me when I manage to achieve a great deal with a little Perl. Especially so when it leverages (see, I could be management) a module of my own. The latest idea took a while coming. I'm informally part of a group of people coding up a project called Gravel whose task is to build a suite of libraries to parse, manipulate, generate and convert between various vector based animation file formats such as SWF, SVG and later formats like MNG. At the moment, it's mostly aimed at parsing SWF (Flash) files.

Now, the parser is fairly robust, but we wanted a whole range of SWF files to throw at the parser and see what tags we were still missing or whether there would be any crashes (yes, it's written in C, but I stay away from that). One of my more recent contributions to the CPAN has been WWW::Search::Google, which uses the Google Web API. Now, Google allows you to specify the filetype of the results of searches. So all I needed to do was search Google for all the SWFs it could find with, say, "a" in the name and download them to later test them out.

And it was easy. Here's the code I used. Note that you'll need your put your own Google Web API key where it says "XXXX".

#!/usr/bin/perl -w
use strict;
use WWW::Search;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new(timeout => 30, agent => 'SWF-Grabber/1.4');

my $search = WWW::Search->new('Google', key => "XXXX");
$search->native_query("filetype:swf +a");

while (my $result = $search->next_result()) {
    my $url = $result->url;
    my $file = $url;
    $file =~ s:^.+/::;
    next if $file =~ /\?/;
    print "Downloading $url\n";
    next if -f $file;
    $ua->mirror($url, $file);
    sleep 10;
}

951 SWFs and counting...

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.