use Perl

Yeah, I visit many websites everyday. what I'm wanting and what I'm always looking for is a facility to automagically keep a record of the URLs and page titles I've just accessed, so that I can analyse the history some time later to find out the focus of my interest in a particular period of time, for example. And it's very likely that I can come up with even more interesting statistical consequences.

The Mozilla browser doubtlessly gives builtin support for accessing history, but unfortunately exporting that history info is not trivial. what I want is not only the URLs, but also the corresponding page titles (if any!) and the visiting time stamp.

Several weeks ago, I happily found that the CPAN module HTTP::Proxy can come to the rescue. What I need to do is just writing several lines of Perl code using that module, running this script at the background as a local HTTP proxy server, and setting my web browser to simply use that. By doing this, my local proxy has a chance to monitor all the HTTP traffic between my browser and the Internet.

It's fun to see that my local proxy server can also use a remote proxy. so my local one then becomes a secondary proxy, no? ;D

The HTTP::Proxy module also supports logging internally, thus my code is even simpler:

use HTTP::Proxy ':log';
my $logfile = ">>$home/myproxy.log";
open my $log, $logfile or
        die "Can't open $logfile for reading: $!";
my $proxy = HTTP::Proxy->new(
        logmask => STATUS,
        logfh => $log,
);

The logmask parameter here controls what kind of things the proxy should record. the STATUS constant indicates only basic URL and response code will be logged. What I get in the log file is something like this:

[Fri Jan 13 17:25:42 2006] (1888) REQUEST: GET http://www.perl.com/
[Fri Jan 13 17:25:53 2006] (1888) RESPONSE: 200 OK
[Fri Jan 13 17:25:53 2006] (1888) REQUEST: HEAD http://www.google.com/mozilla/google.src
[Fri Jan 13 17:25:54 2006] (1888) RESPONSE: 200 OK
[Fri Jan 13 17:25:54 2006] (1888) REQUEST: GET http://www.perl.com/styles/main.css
[Fri Jan 13 17:25:55 2006] (1888) RESPONSE: 304 Not Modified ...

Hmm...very cute! However, HTTP::Proxy's builtin logging mechanism doesn't respect HTML titles. Thus I need to provide a user agent of my own:

package MyUA;
use HTTP::Proxy ':log';
use base 'LWP::UserAgent';
sub send_request {
        my ($self, $request) = @_;
        my $response;
        eval {
                $response = $self->SUPER::send_request( $request );
        };
        if ($@ and not $response) {
                return HTTP::Response->new(500, $@);
        }
        if ($response->is_success) {
                my $type = $response->header('content-type');
                if ($type and $type =~ m[text/html]i) {
                        if ($response->content =~ m[\s*(.*\S)\s*]si) {
                                $proxy->log( STATUS, 'TITLE', $1);
                        }
                }
        }
        return $response;
}

Now we have HTML titles recorded down as well, as witnessed in my log file:

[Tue Jan 17 20:33:47 2006] (2484) REQUEST: GET http://perladvent.org/2004/20th/
[Tue Jan 17 20:33:50 2006] (2484) TITLE: Perl 2004 Advent Calendar: Filesys::Virtual
[Tue Jan 17 20:33:50 2006] (2484) RESPONSE: 200 OK

Then feed the customized user agent to my HTTP::Proxy instance I created earlier:

my $agent = MyUA->new(
        env_proxy => 1,
        timeout => 100,
);
$proxy->agent( $agent );

At last, we enter an infinite loop as every http proxy server:

while (1) {
        eval { $proxy->start(); };
        warn $@ if $@;
}

That's it!

It already works for me, but there're still several pitfalls in this solution:

• Images won't display in MS Internet Explorer (Mozilla works fine, however)
• It seems to me that HTTP::Proxy doesn't support forking by default so it leads to poor performance if I request multiple URLs simultaneously. (BTW, Is there a way to switch to a forking engine? I can't find a word in its POD docs.)
• SSL connection doesn't work on my box.

Have fun!

If you have ever written any Perl OO code, I recommend you to take a look at Luke Palmer's CPAN module Perl6::Attributes, which is extremely simple and handy.

Instead of writing something like

    sub method {
        my $self = shift;
        $self->{attr} = '...';
    }

we can now write

    sub method {
        my $self = shift;
        $.attr = '...';
    }

in Perl5.

Another example is replacing the ugly syntax

    sub method {
        my $self = shift;
        @{$self->{attrs}} = qw(a b c);
    }

with the Perl6-ish syntax

    sub method {
        my $self = shift;
        @.attrs = qw(a b c);
    }

You see, it is pretty cool!

I've already had a try in my recent Perl module developements. Perl6::Attributes makes the source of my Perl OO modules amazingly neat.

The upcoming CPAN module, Test::Masm::Dos, will be a MASM port for the excellent Test::More module on Perl's land. The approach is to give special meanings to MASM comments so that we can write something like

        mov ax, 0ffffh
        ; is ax, -1, "Test a negative number";

directly in the context of 16-bit MASM source code. The implementation for these smart comments is straightforward too -- expand each comment with a piece of assembly code that implements exactly what the comment says. Hence most of the amazing power the Perl QA Group owns can be brought to the poor 16-bit MASM, a legacy MS-DOS assembly language developed by Microsoft.

Here is a complete (thought still trivial) example:

        ; foo.asm
        code segment
                assume cs:code
        start:
                ; tests => 5;
                mov ax, 1011011B
                inc ax
                ; is ax, 91;
                mov cx, 5
                next:
                ; say "# cx = ", cx;
                ; ok cx

        code ends
                  end start

Then we could feed it to my masmt script:

        masmt -o ~foo.asm foo.asm

where ~foo.asm is a temporary program with all the smart comments expanded by corresponding assembly instructions. So now we can assemble this to an executable named ~foo.exe, which is a MASM tester that produces TAP report!

At this point, I've only implemented the "say" comment. However, my classmates and I have already found it very helpful, especially when debugging our assembly programs. You know, it's really a painful experience to do even simple I/O in MASM!

With smart comments in hand, we can easily realize our dreams of applying Perl's debugging strategy to MASM: tweak, run, and tweak, run!

In this journal I'd make one of my secrets public.

Every time I want to use an English expression that I'm not quite familiar with, I always first have a google for it. If there're a lot of hits, I'll then feel pretty confident to write it down. If not, it's still very likely to find the correct form according to the google results. Google is really an extremely rich source of English language materials. It's able to perfectly answer questions like "Do English speakers usually say this way?"

Frankly speaking, all the successful websites, such as Google, eBay, PayPal, and Yahoo, really has changed our ways of life significantly.

I think it'll be a great idea to write this "secret" into my journal. It's doubtless beneficial to share such experience with people from non-English countries all over the world.

Today I did nothing but try Subversion out. Thanks to the free SVN server provided by OSSF. I've hosted two of my CPAN contributions to that server. I also appreciate TortoiseSVN, which is a very handy SVN client on Win32. Centralized version control seems quite sufficient to me. However, I still have an eager to take a look at the famous SVK. Unfortunately I don't seem to have enough time to try SVK tonight.

Geoff Broadwell said in O'Reilly's weblog that, "If you're not already using a version control system, shame on you." Happily I can announce at this moment that I'm already using such a system for my software development. Whee! :=)

I managed to release Makefile::Parser 0.06 and Makefile::GraphViz 0.03 to CPAN tonight. Hopefully I can see them appear on search.cpan tomorrow morning.

The former module is a Makefile parser which parses Makefiles to simple ASTs while the latter is a GraphViz application that generates pretty graphs that depicts the building flow directly from Makefiles. The pictures produced by GraphViz are amazingly good. I don't think I'm able to produce better pictures manually with MS Visio. Furthermore, everything in the output pictures are fully parameterized, leading to good flexibility.

Nadim Khemir told me that he was looking forward to use Makefile::Parser in his PerlBuildSystem, which urges me to improve this stuff actively.

More feedbacks are most welcomed. Thank you in advance.