Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Alias (5735)

Alias
  (email not shown publicly)
http://ali.as/

Journal of Alias (5735)

Monday October 13, 2008
07:41 PM

What is the longest Perl test suite you've ever seen?

[ #37656 ]


ok( Perl::Dist::Strawberry->default_machine->run );

With this release of Strawberry, I've managed to find a reasonably safe way to write tests that generate distributions without wiping out your current installation.

One of these tests the default Perl::Dist::Machine for Strawberry.

A "distribution machine" takes the options for building a distribution and generates several variants of the same thing. In this case, the machine is handling the creation of the three 5.8.8, 5.10.0 and Portable variations of Strawberry.

For completeness, this process now has a test script of it's own.

The test takes around 4-5 hours to complete (each distribution takes about an hour and a half to build) (on top of the 4-5 hours for the three tests that do the single builds on their own).

It's so long that it's now pretty much pointless to run under $ENV{AUTOMATED_TESTING} because it's going to hammer the CPAN Testers hosts (it also generates around half a gig or so of temporary data).

Currently these distribution building tests only run under RELEASE_TESTING.

So my question to you...

Am I right in thinking this is the longest test on the CPAN?

And secondly, what's the longest (Perl) test suite you've ever seen?

For me, it's my current test suite at work, which is starting to head past an hour now.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • The longest suite I've worked with took about an hour and a half to complete. Regrettably, it was written with a bespoke (and broken) test harness.

    Why does your current test suite take so long to run? There are numerous strategies which can generally be applied to mitigate long test suite run times (or are you not bothered by this?).

    • In Perl::Dist(::Strawberry) I build Perl from scratch, about 7 times.

      Across all of them, I also install about 200+ CPAN modules...

      And unfortunately, none of it knows when to test in parallel on it's own, so much of my the power of my quad-core machine is wasted.

      At work, much of the extra work is necessary, because the application is ENORMOUS (250k source lines of code) with lots of lots of potential variation and state.

      Before every single test script, we kill and rebuilt users from scratch. Some test direct

      • For the long setup time, have you considered creating a set of general test data that can be used for many of the tests?

        If you can do that, then maybe you can cache the setup of it and avoid long processing time just to get to the correct state.

        So the app code is used to move the app to the correct state only once per test suite run, or even outside of the test run. Once that is done, make a snapshot of it that can be restored quicker for each test.

        This is something we haven't done yet, but I'm thinking of

  • Smolder tells me that the test suite for our biggest project at $work took 1:28:55 the last time it ran. Granted our smoke box isn't as powerful as our production or even our development machines. But it's still in active development with lots of new features planned, so I expect that to get worse...

  • I had the impression that CPAN::Reporter is usually the slowest distribution, and really, if I run the following one-liner in a directory with my saved reports:
    grep -r 'wallclock secs' .|perl -nle '/^(\S+).*=\s(\d+\.\d+)\s+CPU/ and push @d, [$1,$2]; END { print join("\n", map { "$_->[1]\t$_->[0]" }sort { $b->[1]$a->[1] } @d) }'
    then this shows up as the top 5, sorted by CPU seconds:
    1132.65 ctr1/done/pass.CPAN-Reporter-1.1702.i386-freebsd.6.1-release-p23.1223500100.952. rpt:Files=36,
    932.16