Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

Ovid
  (email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Monday September 11, 2006
03:49 AM

Testing All of CPAN at Once

[ #30959 ]

Last week, while at the same Test Automation Conference Acme attended, I had an inspiration. Given that Yahoo! is using my TAPx::Parser, Torsten Shoenfeld has written a GTK front end for it (I need to include his name directly in the POD), Shlomi Fish is integrating TAPx::Parser with Test::Run, people are starting to send me patches and there's a general increase in interest for it, it's time for me to get this thing out of alpha.

How better to do that than to run TAPx::Parser against all of CPAN? Since I use CPAN::Mini, I have most of the CPAN on my hard-drive. As it turns out, testing everything isn't practical, but I can test most of it. My initial plan is to discard all Apache and Win32 specific modules, anything without a Build.PL or Makefile.PL file or a t/ directory. I might be tossing out too much, but I still have thousands of packages I can test on my iBook.

I get through about 200 of them before my code can no longer fork (something about "resource not available") and I can't even run ls in a directory until I kill my code. This has turned out to be very difficult, but I've learned a lot about my code and have already found some strange edge cases it can't handle.

In other news, Acme wants to install your tests with your code so you can test that your installation works. What a brilliant idea!

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • Is that a common problem, installation failures? Instead of installing the tests somewhere can't it just do some kind of -I to get @INC set?

    I don't know of any traditional location for tests in production and I'd rather there not be a pile of clutter under my @INC directories while a dozen different people all decide to use some different scheme for storing their tests somewhere.
    • I want you to install your (Test::Class-based) tests as normal modules so I can reuse it when I reuse your modules.

      • That is just the mental shove I needed to see why I'd possibly ever care about Test::Class. Thanks!
        • There are plenty of other reasons to use Test::Class. Do you have thirty t/*.t programs? You've just loaded the perl runtime at least 30 times (probably more). You've also probably reloaded the tested modules many times and possibly duplicated a lot of test code. With Test::Class, you only load the Perl interpreter once and the associated modules once. Large test suites can run much faster and are easier to refactor.

          And inheriting tests to ensure that derived classes don't break their parents makes m

  • I should note that I blogged about it here [perl.org], with some more information.

  • Even if a distribution lacks a ./t directory, it may have a test.pl script. Hence, you may want to adjust your criteria for discarding untestable distributions.
    • I had considered that, but a number of test.pl modules don't output TAP (largely because they're older code). As a result, my parser can't reliably parse their output. My goal here, in reality, is not to test all of the CPAN, but to have a massive stress test of TAPx::Parser [cpan.org]. I still have several thousand test suites left which I can use and I've found a few bugs I need to address.

      That being said, I doubt all tests in t/*.t are based on Perl's regular testing tools, but I have to start somewhere.

      • Yes, I had considered that you had considered that.

        In which case, it would be handy to know what test.pl scripts aren't TAP-compliant. I know early versions of Compress::Bzip2 (before it was taken over by the current maintainer) featured a test script that produced nothing useful.

        It would be nice to have an exhaustive list. Then we could go about fixing them, if the distributions in question are useful enough to be worth the attention.