Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

mpeters (5802)

mpeters
  (email not shown publicly)
http://petersfamily.org/
AOL IM: michael00peters (Add Buddy, Send Message)

Technorati Profile [technorati.com]

Journal of mpeters (5802)

Thursday June 15, 2006
02:04 PM

Mech and hidden fields

I was writing some controller tests for an application that makes heavy use of Javascript and it bit me again that WWW::Mechanize does not allow you to set the values of hidden fields, since HTML::Form considers them read-only.

This has always bugged since if Mech is going to be useful for testing it's almost required that you mess with hidden fields because sometimes your JS will do it, or you want to make sure you've protected yourself against elementary hacking attempts.

Anyway, I always forget how to work around this, so I'm putting it here to always remind myself. If you have a hidden input named 'rm', this is how you would make it settable for Mech:

    my $form = $mech->form_name('my_form');
    $form->find_input('rm')->readonly(0);
    $mech->set_fields(rm => 'foo');

Wednesday May 24, 2006
11:25 AM

another IE WTF

How many of you have heard of "downlevel-revealed conditional comments" in HTML. Of course you haven't. It's something that Microsoft made up. They look like this:

  <![if IE]>stuff<![endif]>;

This means that stuff will only show up in IE browsers < 5.0.

It's bad enough that they have "downlevel-hidden conditional comments", but at least those look like normal comments.

   <!--[if !IE gte 6]>stuff<![endif]-->

This means that stuff will only show up in IE 6+. While this is a great way for MS to increase their market share by encouraging markup that only works in IE, at least it doesn't break anything in other browsers since it actually is a valid comment.

Why does this matter? I'm currently working on a $project where the audience can be using some pretty crufty browsers. We want to provide some eye-candy to those with modern browsers and just prevent those with broken browsers from getting all those annoying JS pop-up boxes with errors. The designer decides to use "downlevel-hidden conditional comments" to hide it from IE 5 and IE(mac). This works ok until we notice that on pages with forms it's broken. That's because we were using HTML::FillInForm (one of my favorite modules) which is based on HTML::Parser. HTML::Parser is smart enough to recognize those funny IE comments as comments, which is helpful. However, it does not treat them any differently than any other comments. So when HTML::FillInForm is creating the final document, it reads in a comment and then outputs a comment. Sounds like the right thing to do, except this breaks those funny IE comments because they now come out looking like normal comments.

The solution? I hacked up a quick sublcass of HTML::FillInForm for the project that will do-the-right thing when it encounters those little suckers. All I had to do was override the comment sub. So in case anyone else needs to use it, here it is:

sub comment {
    my ( $self, $text ) = @_;
    # if it begins with '[if ' and doesn't end with '<![endif]'
    # it's a "downlevel-revealed" conditional comment (stupid IE)
    # or
    # if it ends with '[endif]' then it's the end of a
    # "downlevel-revealed" conditional comment (stupid IE)
    if(
        (
            ( index($text, '[if ') == 0 )
            &&
            ( $text !~ /<!\[endif\]$/ )
        )
        ||
        ( $text eq '[endif]' )
    ) {
        $self->{output} .= '<!' . $text . '>';
    } else {
        $self->{output} .= '<!--' . $text . '-->';
    }
}

Saturday April 01, 2006
09:22 AM

That's it, I'm tired of Perl

So I'm switching to Ruby. Not only does it have cooler videos but when have you ever seen a Perl programmer with a glamour shot in Wired magazine. Perl is a dead-end language (the PHP croud has been saying that for years and their language is awesome!) and the only way I'm going to get noticed is if I switch to Ruby, but not just plain ol' Ruby. My career will be On Rails!
Look out Wired Magazine, I'm way hotter than David Hansson.
Sunday March 26, 2006
05:30 PM

More FC5 breakages

Just to continue the issues that Sam Tregar mentioned about using Fedora Core 5 , I'm adding the following:
  • OpenSSL
    FC5 comes with version 0.9.8 which has actually been available since July of 2005. But Crypt::SSLeay which is used all over the place will not build against 0.9.8. You can, of course patch your version using the info from here.
  • SQLite
    FC5 comes with version 3.3.3 which is isn't even the latest stable version (3.3.4 is). But DBD::SQLite won't build against it. There is a work around, because DBD::SQLite comes with it's own version (3.2.7) of SQLite and uses it if you don't already have one installed. Or you can force it to use it anyway by running Makefile.PL like so

    perl Makefile.PL USE_LOCAL_SQLITE=1

If I notice anything else I'll be sure to let you know.

Thursday March 09, 2006
04:40 PM

HTML Form filling benchmarks

Every web application needs to fill in HTML <form>s at some point. Simple text <input>s and <textarea>s are easy to do. But radio buttons, checkboxes and <select>s are a bit harder.

I love HTML::FillInForm because each form <input> is treated the same. It doesn't matter if the designer wants to change from a <select> box to radio boxes, or replace it with a text <input>. The code that fills it doesn't change.

But HTML::FillInForm has to parse the entire HTML, find the <form> and the inputs and then change the HTML to include all the values and selected="selected" stuff. That's gotta be slow, right?

So, to see how slow I decided to benchmark it (against HTML::Template::Expr). Here's the results (code benchmarked is at the bottom):
With small, fairly simple forms, and using external files so H::T::E could cache -

For 1000 iterations:

      Rate hte fif
hte 1493/s  -- -7%
fif 1613/s  8%  --

      Rate  fif  hte
fif 1075/s   -- -11%
hte 1205/s  12%   --

And it usually went back and forth like that. They were both very close and the winner flipped-flopped.

Now with a very large complicated HTML with lots of other markup taken straight from a client's site:

[mpeters@localhost ~]$ ./bench.pl
      Rate  hte  fif
hte 44.2/s   -- -27%
fif 60.6/s  37%   --
[mpeters@localhost ~]$ ./bench.pl
      Rate  hte  fif
hte 41.8/s   -- -36%
fif 65.4/s  56%   --

FillInForm consistently beat H::T::E on this large form. I was pretty amazed. If you see any problems with my benchmark, please let me know.

Benchmark code

#!/usr/bin/perl
use strict;
use warnings;
use Benchmark qw(:all);
use HTML::FillInForm;
use HTML::Template::Expr;
use HTML::Template;

my %values = (
    prefix      => 'Mr.',
    first_name  => 'Michael',
    last_name   => 'Peters',
    address1    => '1234 Main St.',
    city        => 'Silver Spring',
    state       => 'Maryland',
    zip         => '12345',
    country     => 'US',
);

sub fill_form {
    my $tmpl = HTML::Template->new(
        filename => 'form_text.tmpl',
        cache    => 1,
    );
    my $out = $tmpl->output();

    HTML::FillInForm->new()->fill(
        scalarref => \$out,
        fdat      => \%values,
    );
}

sub fill_tmpl {
    my $tmpl = HTML::Template::Expr->new(
        filename => 'tmpl_text.tmpl',
        cache    => 1,
    );
    $tmpl->param(%values);
    $tmpl->output();
}

cmpthese(
    1000,
    {
        hte => sub { fill_tmpl() },
        fif => sub { fill_form() },
    },
);

Saturday March 04, 2006
02:02 PM

where there's smoke....

In my last big project we had a nice, large test suite and the team was really good about always running it before most commits. But there were always problems with things that passed on one developer's machine and not the others (usually becuase of some files or changes that were not checked in). So I setup an automated smoke test that ran every night and emailed the results to the developers. I even made it nice and pretty using Test::TAP::HTMLMatrix (mentioned here - http://use.perl.org/~mpeters/journal/25612). This worked pretty well, but as always, it could be improved. These are some areas that needed improvement:
  1. It sent emails whether the test suite passed or not. While I like this (it let me know that the automated tests were still running) other developers just wanted to see the report when tests failed
  2. The report could be very, very large (5000+ tests and over 500k of HTML).
  3. There wasn't an easy way for developers to share test reports between themselves without pasting large chunks of test output into emails
  4. It was very tied to that specific project

This is why Smolder was born. It solves the above problems in the following ways

  1. Developers choose when they want to receive emails. Either a) All the time b) just when there are failures, or even c) never.
  2. Developers can also choose what format their email comes in. It can either be the full HTML, or just a summary, or even just a link to the full report. Smolder contains the full report so it can always be viewed at any time.
  3. Test reports can be uploaded by either an autormated test setup, or individually by developers (in either XML or YAML using either Test::TAP::XML or YAML + Test::TAP::Model)
  4. It's a standalone server that's project agnostic. Plus one smolder install can host several projects. The only thing that needs to be adjusted for each project is to add Test::TAP::XML to the test harness.

Screen shots are available if you want to see it (just to make sure it's not too ugly).

Please try it out, give some feedback and request features. It's just at version 0.01 so it's future is very flexible.

Sunday December 04, 2005
09:19 PM

Open source and automobiles

Since most of my friends and family are not associated with software (except for my dad) and are almost completely ignorant of OSS, I feel compelled to explain my views to them in a way that doesn't completely bore them. For some reason, most of my explanations and analogies involve cars. It's not because I'm particularly interested in cars; but when I tried to find an example of other people who like to tinker, sometimes as a hobby, sometimes as a profession, the first thing that came to mind was a mechanic.

So I thought I'd share these examples here and maybe others will find them useful.
  • Red Cars and Firefox:
    It has been reported that red is the most popular color for stolen cars. Are red cars any more insecure than white cars? When comparing Internet Explorer and Firefox, MS fans defend IE by saying it is more targeted by viruses and spyware because it's more popular. Even if Firefox isn't any more secure than Internet Explorer, isn't choosing a white car over a red one a good idea just because it won't be as big of a target? I think so.
  • Tweak out your engine and I'll sue you

    Why should I care if I don't get the code when I buy software? I'll never look at it or change it?

    It's true that most people buy their cars and have no interest in ever even opening the hood. And most of those that do will only change their oil, or replace belts, etc. Very few will ever replace their alternators or rebuild their engine. But some will.

    While most people don't care to get their hands that dirty, how would people react if Ford suddenly sold all of their cars with the hoods locked such that only an authorized dealer could open it? (And in a comparison to some companies, threaten legal action if you figure out a way to open it yourself). If they decided that you cannot get any repairs (patches or upgrades) from anyone except the dealership? I'm sure they'd love the extra revenue, but the business they would lose would greatly offset that. The public outrage caused by such a business decision would be quite significant.

    Why shouldn't you be allowed to tinker under the hood of the software you buy? Even if you don't want to, shouldn't you be allowed to take it to another professional who would be experienced enough to do it for you?

Monday September 26, 2005
01:54 PM

Flowcharts... Evil, evil, evil!

[ #26892 ]
I'm in the final phases of a project that has been going on for some time. We have extensive documentation in all modules and scripts and several POD files filled with details and overviews of various sections and how the pieces fit together (141 pages of HTML with indeces, and nicely linked together).

Now we are getting ready to hand the code over to the $client's internal developers and we have asked them to peruse the docs to let us know if there is anything lacking. They (actually just one person) gave us a list of several things (most of which were there so it makes me think they didn't even look at it) that they want, most notably flow charts that show every control module (we're using MVC), every run mode (using CGI::Application), every template and every hyperlink (or form submission) between them.

I have a problem with this. Flowcharts are nice when used to gather requirements but quickly become stale if you do any kind of iterative development and I hate maintaining documentation that is separate from code. So I've been pushing back on this.

As it just happens I've been reading "The Mythical Man-Month" and just came across a great section on flowcharts. (written by Frederick P. Brooks, JR. in 1975)

The detailed blow-by-blow flow chart, however, is an obsolete nuisance, suitable only for initiating beginners into algorithmic thinking.

Speaking of the fact that know one keeps good up-to-date flowcharts, he said:

I think this universal experience is not an embarrassing and deplorable departure from good practice, to be ackknowledged only with a nervous laugh. Instead it is the application of good judgement, and it it teaches us something about the utility of flow charts.
...
A basic principle of data processing teaches the folly of trying to maintain independent files in synchronism...Yet our practice in programming documentation violates our own teaching.

Saturday July 09, 2005
07:32 PM

Perty Testing Harness

So I'm working on this web project that has a custom test harness and nightly automated tests. Right now it runs the tests using Test::Harness, captures the output and then emails it to everyone on the mailing list for the project. It's worked really well so far, but the output is kinda plain and because there are 96 test files, and 3000+ individual tests the test output is a little hard to skim quickly in the morning to make sure everything worked...

Enter Test::TAP::HTMLMatrix.

This module will run the desired tests, record the output from any TAP compatible tests (eg, anything using Test or Test::More, etc) and then format them into nice, pretty and colorful HTML results.

So I decided to try and capture this out and send out a multipart email using MIME::Lite. And the code looked something like this:

  # create the TAP model to run and capture the
  # TAP test output
  my $model = Test::TAP::Model::Visual->new();
  $model->run_tests(@testfiles);

  # now get the HTML output
  my $matrix = Test::TAP::HTMLMatrix->new($model);
  my $output = $matrix->html();

  # now send it in an email
  my $msg = MIME::Lite->new(
    From    => 'testing@plusthree.com',
    To      => 'project_list@plusthree.com,
    Subject => 'Automated Test - ' . scalar(localtime),
    Type    => 'multipart/alternative',
  );
  $msg->attach(
    Type => 'text/html',
    Data => $email_output,
  );
  $msg->send()

Pretty simple huh? But darn cool.