Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

tsee (4409)

tsee
  {smueller} {at} {cpan.org}
http://steffen-mueller.net/

You can find most of my Open Source Perl software in my CPAN directory [cpan.org].

Journal of tsee (4409)

Sunday January 25, 2009
05:01 PM

PAR evilness to make remote auto-upgrading work

When I think about telling people about PAR internals, a reply from a colleague readily comes to mind, when he was asked about an icky detail of his analysis:

You don't want to know how sausages are made!

But then I can't resist grossing out people with some details anyway...

Two years ago, I wrote PAR::Repository::Client as an interface for loading PARs and thus arbitrary modules from a remote server. If the client is installed, all you need to do to auto-load missing modules from the server is:

  use PAR { repository => 'https://foo.com/myapp' };
  use Foo; # will be loaded from remote if necessary

But since this may become expensive, and caching the binaries only removes part of that, the "install" option was part of the interface almost from the start:

  use PAR { repository => 'https://foo.com/myapp', install => 1 };
  use Foo; # will be loaded AND INSTALLED if necessary

Back then, I also added most of the code necessary for an "upgrade" option.

  use PAR { repository => 'https://foo.com/myapp', upgrade => 1 };
  use Foo; # will be loaded AND INSTALLED OR UPGRADED if necessary

Unfortunately, it was missing a few critical details until today. The repository client is normally only invoked when all other sources fail. But that's a problem if you're trying to check for upgrades. Thus, repositories in upgrade-mode are now checked early in the module-loading process.

The real bummer was that in order to check for upgrades, the locally installed version has to be determined. Since this is hard to do reliably without loading the module, that's what PAR has to do. But that means require()ing module X from within an early @INC hook that ran due to a "require X;". There's so many things wrong with that idea, it's not even funny. It seems that creating an infinite recursion in an @INC hook segfaults perl 5.8.9. Regardless, it can be (and was) made to work:

  • Before running the client's upgrade_module method, dynamically override the set of active (via PAR.pm) PAR::Repositories to be empty.
  • Run the current repository client's upgrade_module method which will attempt to require the module for checking its version.
  • Afterwards, check whether the module was loaded using %INC.
  • If not, continue normally, probably ending up failing to load the module from anywhere or loading the freshly installed copy.
  • If the module was loaded, prevent an additional loading with an evil trick in the @INC hook:

  my $line = 1;
  return \*I_AM_NOT_HERE, sub { $line ? ($_="1;",$line=0,return(1)) : ($_="",return(0)) };

Even disregarding the slight obfuscation, can you figure out how this works?

One obscure feature of @INC and the module loading is the return value(s) of a subroutine @INC hook. It normally simply returns a file handle that the module code is then read from. But if it returns a code ref as its second return value, that code ref is called repeatedly until it returns false. After each invocation, $_ is assumed to contain the next line of the module code. If the first argument was a file handle nonetheless, $_ is initialized to a new line from the file handle before calling the subroutine.

The motivation here is mostly that we want to set the file contents to "1;". Unfortunately, passing undef as the file handle resulted in the subroutine not being called. This smells like a bug in perl to me, but I'll have to check that more closely with blead. Furthermore, it's not wise to load any unnecessary modules in PAR.pm as they would have to be included verbatim in an uncompressed part of PAR::Packer created executables. Therefore, instead of simply passing a IO::Handle->new(), I'm supplying an arbitrary GLOB ref.

Finally, the subroutine itself simply sets $_ to "1;" in the first invocation and returns zero on the second to stop the evaluation, thus essentially short-circuiting require()'s loop through @INC.

After going through this considerable pain, I got the auto-upgrading feature of PAR::Repository::Client to work. There's probably still bugs and testing it as part of the test suite is no fun (but still feasible).

Stay tuned for a new release of the involved modules.

Cheers,
Steffen

Tuesday December 23, 2008
02:14 PM

Padre 0.22 highlights and checks Perl6 code

Padre version 0.22 has just been uploaded to the PAUSE. That means it will propagate to the CPAN mirrors without a few hours. Like the previous release, the list of changes is quite long, but one particular achievement is support for highlighting Perl6 code and checking its syntax if Padre::Plugin::Perl6 is installed and enabled. Christmas is close.

Once the distribution has reached your CPAN mirror, you will be able to access the full Change Log here.

The next Padre release is very preliminarily scheduled for December 28th or 29th and we're still looking for a new release engineer.

Best regards,

Steffen

Thursday December 18, 2008
12:09 PM

On a similarity between C++ and Perl popularity

Today, I read an interesting interview with Bjarne Stroustrup, the father of C++, on DevX from August of this year. It's a good read, so if you're a C++ user, you should have a look. But even if you never touched any C++ code, there's a very interesting bit of information:

On page 6, the interviewer, Danny Kalev, asks Stroustrup:

Is C++ usage really declining, as some biased analysts and journalists have been trying to convince us for years (often not without ulterior motives), or is this complete nonsense?

Does that ring a bell? The "Perl is dead" crap that's been splashing down the gutters of the interweb waste disposal system, anyone? I urge you to read the full answer. Much of it applies to Perl as well. Please note that Stroustrup doesn't simply dismiss the issue. Here's an excerpt from his reply:

[...] C++ use appears to be declining in some areas and appears to be on an upswing in other areas. [...] Most of the popular measures basically measures noise and ought to report their findings in decibel rather than "popularity." Many of the major uses are in infrastructure (telecommunications, banking, embedded systems, etc.) where programmers don't go to conferences or describe their code in public. Many of the most interesting and important C++ applications are not noticed, they are not for sale to the public as programming products, and their implementation language is never mentioned. [...]

It's a really big world "out there" and the increase in the number of users of one language does not imply the decrease in the numbers of another. [...]

One simple thing that confuses many discussions of language use/popularity is the distinction between relative and absolute measures. For example, I say that C++ use is growing when I see user population grow by 200,000 programmers from 3.1M to 3.3M. However, somebody else may claim that "C++ is dying" because it's "popularity" has dropped from 16 percent to 10 percent of the total number of users. Both claims could be simultaneously true as the number of programmers continues to grow and especially as what is considered to be programming continues to change. [...]

Most of the popularity measures seem to measure buzz/noise, which is basically counting mentions on the web. That's potentially very misleading. Ten people learning a scripting language will make much more noise than a thousand full time programmers using C++, especially if the thousand C++ programmers are working on a project crucial to a company—such programmers typically don't post and are often not allowed to. My worry is that such measures may actually measure the number of novices and thus be an indication of a worsening shortage of C++ programmers. Worse, managers and academics may incautiously take such figures seriously (as a measure of quality) and become part of a vicious circle.

I know first hand about large C++ systems that don't produce the slightest bit of publicity for the language they're implemented in. It's what I deal with every day. Dito for large Perl code bases. Stroustrup hits the nail on the head about this issue (and C++). It's exactly what I think about Perl in the same context. There may be an issue with not generating as much noise as others (not enough new blood), but it's by no means an indication of stagnation or decline. People simply use it do what they always did. People also use it to do new stuff. But they don't blather about it all day. They earn their salary and at the end of the day, they go home to their families and spend their spare time on more interesting things than blogging about their favourite new toy language*. You have to realize: This applies to easily more than 95% of all professional programmers.

* That reminds me of something... got to go.

Sunday December 14, 2008
02:54 PM

Padre 0.21 - More improvements than can fit in this margin!

I'm quite proud to announce the release of Padre 0.21.

It features the biggest list of changes of any Padre release so far. The menu code has received a major overhaul, the editor has become multi-threaded, and we see more "advanced" Perl-specific features like reliably finding the location of a variable declaration or the experimental feature of replacing a lexical variable. The full list of changes can be found in the Changes file of the distribution.

The list of developers has grown to thirteen, but that list is probably not even complete. The program has been translated to nine languages at this point.

With this release, we are starting to rotate the release duty so the weekly or bi-weekly releases don't block on the availability of Gabor at those rare points in time when no major refactoring or feature implementation is going on.

You can find more information and the Padre mailing list, irc information, bug and issue tracking, etc. on the Padre site as usual.

Cheers,
Steffen

Sunday November 30, 2008
07:07 AM

Class::XSAccessor with constructors = Object::Tiny::XS

Since Class::XSAccessor now supports constructors (see previous journal entry), there's everything in place to implement Object::Tiny in XS. Hence you can find Object::Tiny::XS 1.01 on CPAN very soon. It's been uploaded to PAUSE.

The following benchmarks can be found in the distribution. They're comparing O::Tiny::XS to O::Tiny and Class::Accessor::Fast.

Benchmarking constructor plus accessors...
             Rate accessor     tiny  tiny_xs
accessor 107325/s       --     -40%     -57%
tiny     177837/s      66%       --     -29%
tiny_xs  248931/s     132%      40%       --
 
Benchmarking constructor alone...
             Rate accessor     tiny  tiny_xs
accessor 168659/s       --     -49%     -57%
tiny     330831/s      96%       --     -17%
tiny_xs  396844/s     135%      20%       --
 
Benchmarking accessors alone...
          Rate accessor     tiny  tiny_xs
accessor 331/s       --     -16%     -54%
tiny     395/s      19%       --     -45%
tiny_xs  715/s     116%      81%       --

Saturday November 29, 2008
03:56 PM

Class::XSAccessor - now doing XS constructors

So far, when using Class::XSAccessor or Class::XSAccessor::Array to generate fast accessors implemented in XS for you, you'd still have to write a simple new:


    sub new {
        my $class = shift;
        my $self = bless { @_ } => $class;
        return $self;
    }

So I just added an XS implementation of the above constructor which you can optionally import into your class. The gains are modest compared to the large speedup of the accessors. You gain only something like 40% (without arguments). However, I haven't compared this with any of the heavier method generators out there.

Now, if you think that this kind of micro-optimization is pointless: You're right. Besides being an exercise in XS, this module means that you don't have to spend your time micro-optimizing your simple methods. Everybody knows it's bad, but almost all succumb to the inner voice that makes them do it regardless eventually. With this being done, you can't get much faster, so you don't even have to try!

Cheers,
Steffen

       

Monday November 03, 2008
04:22 AM

Marcel's accessor generator benchmarking

So Marcel has been benchmarking accessor generators. He used Class::Accessor::Fast::XS which is a verbatim copy of Class::XSAccessor with the Class::Accessor interfacing code added. But he missed out on the fastest ones bar XS accessor hand-rolling. It's a combination of using Object::Tiny's constructor and Class::XSAccessor's accessor generation. (Let's call it Object::Tiny::XS!) Here's the adapted code and then the results:

Benchmarking attribute access only

#!/usr/bin/env perl

use warnings;
use strict;
use Benchmark qw(cmpthese timethese :hireswallclock);

package WithMoose;
use Moose;
has myattr => ( is => 'rw' );

package WithMooseImmutable;
use Moose;
has myattr => ( is => 'rw' );
__PACKAGE__->meta->make_immutable;

package WithMouse;
use Mouse;
has myattr => ( is => 'rw' );

package WithClassAccessor;
use base qw(Class::Accessor);
__PACKAGE__->mk_accessors(qw/myattr/);

package WithClassAccessorFast;
use base qw(Class::Accessor::Fast);
__PACKAGE__->mk_accessors(qw/myattr/);

package WithClassAccessorFastXS;
use base qw(Class::Accessor::Fast::XS);
__PACKAGE__->mk_accessors(qw/myattr/);

packag e WithClassAccessorComplex;
use base qw(Class::Accessor::Complex);
__PACKAGE__->mk_new->mk_scalar_accessors(qw/myatt r/);

package WithClassAccessorConstructor;
use base qw(Class::Accessor::Constructor Class::Accessor::Complex);
__PACKAGE__->mk_constructor->mk_scalar_accessors(qw/ myattr/);

package WithMojo;
use base qw(Mojo::Base);
__PACKAGE__->attr('myattr');

package WithClassMethodMaker;
use Class::MethodMaker
    [ scalar => [ qw/myattr/ ],
      new    => [ qw/-hash new/ ],
    ];

package WithAccessors;
use accessors qw(myattr);

sub new { bless {}, shift }

package WithObjectTiny;
use Object::Tiny qw/myattr/;
sub set_myattr { $_[0]->{myattr} = $_[1] }

package WithSpiffy;
use Spiffy -base;
field 'myattr';

package WithClassSpiffy;
use Class::Spiffy -base;
field 'myattr';

package WithClassXSAccessor;
use Class::XSAccessor accessors => { myattr => 'myattr' };
sub new {my $class = shift; bless {@_} => $class}

package WithClassXSAccessorArray;
use Class::XSAccessor::Array accessors=> { myattr => 0 };
sub new {my $class = shift; my %args = @_; bless [$args{myattr}] => $class}

package WithObjectTinyXS;
use Object::Tiny qw/myattr/;
use Class::XSAccessor accessors => { myattr => 'myattr' }, replace => 1;

package main;

my $moose                      = WithMoose->new;
my $moose_immutable            = WithMooseImmutable->new;
my $mouse                      = WithMouse->new;
my $class_accessor             = WithClassAccessor->new;
my $class_accessor_fast        = WithClassAccessorFast->new;
my $class_accessor_fast_xs     = WithClassAccessorFastXS->new;
my $class_accessor_complex     = WithClassAccessorComplex->new;
my $class_accessor_constructor = WithClassAccessorConstructor->new;
my $mojo                       = WithMojo->new;
my $class_methodmaker          = WithClassMethodMaker->new;
my $accessors                  = WithAccessors->new;
my $object_tiny                = WithObjectTiny->new;
my $spiffy                     = WithSpiffy->new;
my $class_spiffy               = WithClassSpiffy->new;
my $direct_hash                = {};
my $class_xsaccessor           = WithClassXSAccessor->new;
my $class_xsaccessor_array     = WithClassXSAccessorArray->new;
my $object_tiny_xs             = WithObjectTinyXS->new;

cmpthese(timethese(-5,{
    moose => sub {
        $moose->myattr(27);
        my $x = $moose->myattr;
    },
    moose_immutable => sub {
        $moose_immutable->myattr(27);
        my $x = $moose_immutable->myattr;
    },
    mouse => sub {
        $mouse->myattr(27);
        my $x = $mouse->myattr;
    },
    class_accessor => sub {
        $class_accessor->myattr(27);
        my $x = $class_accessor->myattr;
    },
    class_accessor_fast => sub {
        $class_accessor_fast->myattr(27);
        my $x = $class_accessor_fast->myattr;
    },
    class_accessor_fast_xs => sub {
        $class_accessor_fast_xs->myattr(27);
        my $x = $class_accessor_fast_xs->myattr;
    },
    class_accessor_complex => sub {
        $class_accessor_complex->myattr(27);
        my $x = $class_accessor_complex->myattr;
    },
    class_accessor_constructor => sub {
        $class_accessor_constructor->myattr(27);
        my $x = $class_accessor_constructor->myattr;
    },
    mojo => sub {
        $mojo->myattr(27);
        my $x = $mojo->myattr;
    },
    class_methodmaker => sub {
        $class_methodmaker->myattr(27);
        my $x = $class_methodmaker->myattr;
    },
    accessors => sub {
        $accessors->myattr(27);
        my $x = $accessors->myattr;
    },
    object_tiny => sub {
        $object_tiny->set_myattr(27);
        my $x = $object_tiny->myattr;
    },
    spiffy => sub {
        $spiffy->myattr(27);
        my $x = $spiffy->myattr;
    },
    class_spiffy => sub {
        $class_spiffy->myattr(27);
        my $x = $class_spiffy->myattr;
    },
    direct_hash => sub {
        $direct_hash->{myattr} = 27;
        my $x = $direct_hash->{myattr};
    },
    object_tiny_xs => sub {
        $object_tiny_xs->myattr(27);
        my $x = $object_tiny_xs->myattr;
    },
    class_xsaccessor => sub {
        $class_xsaccessor->myattr(27);
        my $x = $class_xsaccessor->myattr;
    },
    class_xsaccessor_array => sub {
        $class_xsaccessor_array->myattr(27);
        my $x = $class_xsaccessor_array->myattr;
    },
}));

Results:

class_accessor_constructor   2454/s
moose                        4989/s
mouse                       11187/s
class_methodmaker           35946/s
mojo                        52058/s
class_accessor_complex      54325/s
moose_immutable             60575/s
class_accessor              75348/s
class_spiffy                79642/s
class_accessor_fast         89620/s
class_accessor_fast_xs     107349/s
spiffy                     134988/s
class_xsacessor_array      155608/s
class_xsacessor            173799/s
object_tiny                234721/s
object_tiny_xs             287949/s
direct_hash                449616/s

Benchmarking object creation and attribute access

#!/usr/bin/env perl

use warnings;
use strict;
use Benchmark qw(cmpthese timethese :hireswallclock);

package WithMoose;
use Moose;
has myattr => ( is => 'rw' );

package WithMooseImmutable;
use Moose;
has myattr => ( is => 'rw' );
__PACKAGE__->meta->make_immutable;

package WithMouse;
use Mouse;
has myattr => ( is => 'rw' );

package WithClassAccessor;
use base qw(Class::Accessor);
__PACKAGE__->mk_accessors(qw/myattr/);

package WithClassAccessorFast;
use base qw(Class::Accessor::Fast);
__PACKAGE__->mk_accessors(qw/myattr/);

package WithClassAccessorFastXS;
use base qw(Class::Accessor::Fast::XS);
__PACKAGE__->mk_accessors(qw/myattr/);

packag e WithClassAccessorComplex;
use base qw(Class::Accessor::Complex);
__PACKAGE__->mk_new->mk_scalar_accessors(qw/myatt r/);

package WithClassAccessorConstructor;
use base qw(Class::Accessor::Constructor Class::Accessor::Complex);
__PACKAGE__->mk_constructor->mk_scalar_accessors(qw/ myattr/);

package WithMojo;
use base qw(Mojo::Base);
__PACKAGE__->attr('myattr');

package WithClassMethodMaker;
use Class::MethodMaker
    [ scalar => [ qw/myattr/ ],
      new    => [ qw/-hash new/ ],
    ];

package WithAccessors;
use accessors qw(myattr);

sub new { bless {}, shift }

package WithObjectTiny;
use Object::Tiny qw/myattr/;
sub set_myattr { $_[0]->{myattr} = $_[1] }

package WithSpiffy;
use Spiffy -base;
field 'myattr';

package WithClassSpiffy;
use Class::Spiffy -base;
field 'myattr';

package WithClassXSAccessor;
use Class::XSAccessor accessors => { myattr => 'myattr' };
sub new {my $class = shift; bless {@_} => $class}

package WithClassXSAccessorArray;
use Class::XSAccessor::Array accessors=> { myattr => 0 };
sub new {my $class = shift; my %args = @_; bless [$args{myattr}] => $class}

package WithObjectTinyXS;
use Object::Tiny qw/myattr/;
use Class::XSAccessor accessors => { myattr => 'myattr' }, replace => 1;

package main;

cmpthese(timethese(-5,{
    moose => sub {
        my $obj = WithMoose->new(myattr => 27);
        my $x = $obj->myattr;
    },
    moose_immutable => sub {
        my $obj = WithMooseImmutable->new(myattr => 27);
        my $x = $obj->myattr;
    },
    mouse => sub {
        my $obj = WithMouse->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_accessor => sub {
        my $obj = WithClassAccessor->new({ myattr => 27 });
        my $x = $obj->myattr;
    },
    class_accessor_fast => sub {
        my $obj = WithClassAccessorFast->new({ myattr => 27 });
        my $x = $obj->myattr;
    },
    class_accessor_fast_xs => sub {
        my $obj = WithClassAccessorFastXS->new({ myattr => 27 });
        my $x = $obj->myattr;
    },
    class_accessor_complex => sub {
        my $obj = WithClassAccessorComplex->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_accessor_constructor => sub {
        my $obj = WithClassAccessorConstructor->new(myattr => 27);
        my $x = $obj->myattr;
    },
    mojo => sub {
        my $obj = WithMojo->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_methodmaker => sub {
        my $obj = WithClassMethodMaker->new(myattr => 27);
        my $x = $obj->myattr;
    },
    object_tiny => sub {
        my $obj = WithObjectTiny->new(myattr => 27);
        my $x = $obj->myattr;
    },
    spiffy => sub {
        my $obj = WithSpiffy->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_spiffy => sub {
        my $obj = WithClassSpiffy->new(myattr => 27);
        my $x = $obj->myattr;
    },
    direct_hash => sub {
        my $h = {};
        $h->{myattr} = 27;
        my $x = $h->{myattr};
    },
    object_tiny_xs => sub {
        my $obj = WithObjectTinyXS->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_xsacessor => sub {
        my $obj = WithClassXSAccessor->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_xsacessor_array => sub {
        my $obj = WithClassXSAccessorArray->new(myattr => 27);
        my $x = $obj->myattr;
    },
}));

Results:

class_accessor              159366/s
mojo                        240146/s
class_spiffy                250628/s
mouse                       270458/s
spiffy                      277170/s
moose                       288516/s
moose_immutable             293403/s
class_accessor_fast         295518/s
class_accessor_constructor  302447/s
accessors                   318795/s
class_accessor_complex      352492/s
class_methodmaker           389809/s
object_tiny                 435876/s
class_xsaccessor            537101/s
class_accessor_fast_xs      572424/s
class_xsaccessor_array      620285/s
object_tiny_xs              638328/s
direct_hash                1183721/s

I'll leave the analysis of the results to the reader.

Tuesday October 28, 2008
11:28 AM

Gabor's post on Perl application distribution

Since Gabor has disabled comments in his blog, I'll have to write in my own.

This is about Gabor Szabo's post about Perl application distribution.

There is a whole lot of things that need taking care of with a scheme like the one that Gabor outlines. The whole notion of managing dependencies gives me headache. In my experience, there are very few distribution schemes which work well (e.g. dpkg, CPAN) and many which don't work quite as well (various CPAN-alikes for other languages, PPM). Sometimes, it's a conceptual issue, sometimes just that such a beast carries a large technical and maintenance burden.

For example, I once created a PAR::Repository with (almost) the whole of CPAN in it. Took a horribly long time to generate... and was slow as molasses because of some technical issues. (In this case, it was a combination of large DBM::Deep databases and the dynamic dependency resolution of PAR::Repository::Client. It would be somewhat faster today.)

But Gabor suggests reducing this whole nightmare to a relatively "flat" dependency tree. It shifts parts of the problem to the developer of the app (need to include all dependencies which are not part of the platform) and eliminates some.

I could go on for ages about this and Gabor has had to suffer through that already, but what I really wanted to point out is that a tiny part is solved already: Soon, I'll release a tool that can install an arbitrary .par into an arbitrary perl installation on the system. That may sound simple enough, but I dare you to try install into a non-running perl using CPAN(PLUS). This means that you can just ship a single, binary executable that can install stuff if you just select the perl executable and the URI of the package to install. I hate bootstrapping.

Monday October 20, 2008
02:25 PM

Tired of ignorant users

Warning. A bit of a frustrated rant ahead.

I have many, many distributions on the CPAN and have been struggling very hard to support them as well as I possibly can. Unsurprisingly, providing user support and fixing bugs is not the only thing I do and certainly not the only thing I like to do. In order to be a good CPAN citizen, I have adopted various unmaintained modules and applied tiny changes or made large overhauls resulting in a total of 488 .tar.gz files in my BACKPAN directory. Sorry if I'm coming across as a narcissistic, self-righteous bastard, but I'm trying to get across that the time it takes to maintain all that is not negligible.

One of those distributions is PAR, the Perl Archive Toolkit, and many are related. If you happen not to know what that is, it's a set of tools for packaging, managing, and distributing binary packages of Perl code and related resources and was written by Audrey Tang. It's a complex and very system dependent piece of software and I'm very, very glad that unlike many other packages in my CPAN directory, I'm not the only one who works on it. There is an active mailing list with quite a few incredibly nice, competent, and patient subscribers. In particular, there are various people who have a better overview over the code and its peculiarities on some platforms than I do. (I don't do Windows nor MacOS, for example.)

Now, each and every manual page of a PAR-related module states explicitly that support requests shall go to the PAR mailing list and bugs to the request tracker (which also sends a copy to the list, thanks to Jesse Vincent's tireless work). The PAR homepage prominently displays that same information and nowhere does it as much as mention me.

Regardless, I regularly receive support requests for the PAR modules from users to my personal e-mail address. I'm glad to hear they're using PAR, but they're usually very badly written and do not even give enough information for me to help them.
So far, I think I have answered all of these mails except maybe a handful that were either extremely impertinent or just unlucky because I was much busier than usual.
I started replying with an answer to their problem if I had one. Eventually, I grew tired of that and gave them their answer after noting that they should follow up on the mailing list.
This, however, has turned out to be highly ineffective. Those who got their answer usually didn't end up participating in the mailing list but just disappeared again.

How do you solve this dilemma? I still want to try to help wherever I can, but should I really spread my spare time thin between those who read the documentation, do some testing, and write a well-phrased mail to the mailing list and those who just send me a two-line mail saying that my shit doesn't install on their computer with an unspecified operating system? I'm exaggerating here. There's lots of shades in between.
Replying simply RTFM or Send it to the mailing list and include your system details and logs feels really rude. Forwarding the terrible thing to the list seems much worse: I'd be rude to those people who spend their spare time supporting the software. I'm actually thinking about not answering those mails at all unless they seem particularly interesting.

I can't be the only one in this situation. How do you handle this?

Friday September 05, 2008
09:19 AM

PAUSE on CPAN, indexing $stuff

Most likely, everybody who reads this directly or indirectly depends on the operation of the PAUSE indexer (aka mldistwatch). The PAUSE indexer scans new distributions on the CPAN (really on PAUSE at that point) for the packages/namespaces and associated versions they contain, sends the uploader a friendly message with the results, and adds the information to the metadata that's used by our toolchain when people install modules from the CPAN.

The PAUSE code was written and is still maintained by Andreas König. It's a rather large and unquestionably a rather complex piece of software.

I don't think I'm giving anybody a big surprise if I say that being able to run this same indexer on a given tarball or zip offline may be useful for some toolchain modules. One example would be generating the META.yml provides section.

At the social event of YAPC::EU 2008, Andreas and a posse of other PAUSE admins, including me, sat down to talk about the directions our tools are heading as well as policies. I don't think anybody disagreed to that it'd be great to have components of PAUSE available individually from CPAN. But that's one ambitious goal!

A long time ago, I had spent a significant amount of time on porting the PAUSE indexer code in order to be able to index PAR distributions for injection into a PAR::Repository. I could do all sorts of simplifications for that purpose. For example, .par files are always in ZIP format, no tarballs, etc.
Last night, I decided to give it a shot at making the PAUSE indexer it's own CPAN module.

But I failed.

It turned out to be very, very tightly woven into the whole PAUSE code. I'm really not sure how I got the PAR file scanner to work on the basis of the PAUSE indexer. So I switched to a less ambitious goal: split the PAR indexer out of the PAR::Repository code into the PAR::Indexer module (and distribution) for general consumption.

That's where it stands today. For the future, I figure adding some code back into the mix and making a more generic indexer distribution would get it up to producing 98% of the same results as the real PAUSE indexer. I can do this, but:
Now I'd like to know, would you consider this useful?
And a challenge for all the testing gurus: How would you try to exhaustively test this thing agains the PAUSE indexer?

Cheers,
Steffen