Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Eric Wilhelm (6389)

Eric Wilhelm
  (email not shown publicly)
http://scratchcomputing.com/

Journal of Eric Wilhelm (6389)

Tuesday September 11, 2007
04:03 AM

Parallel Testing

[ #34414 ]

So I decided to finally (after all of this compatibility work) take TAP::Harness out for a spin and make TAP::Harness::Parallel. That comes with some pain about the output formatting API (and I'll push it to CPAN as soon as we stabilize that -- right now you have to ride along and get TAP::Harness from its svn too (and possibly r491 because we're still working-through the single-process assumptions in the formatter.)) But, the preliminary results are between 40-60% reduced waiting. If your test suite takes more than 10s to run, it is probably worth it.

runtests -r -j 9

Note: I symlink bin/prove to runtests -- the Test::Harness transition isn't quite complete.

Expect about 40% time reduction in typical tests suites. The perl 5.8.8 core tests dropped to 200s from 500s. The parrot tests seem to be playing along as well. (Both of these require the directory-based partitioning.)

The biggest problem is resource conflicts. Have you ever run your tests simultaneously before? Probably not -- it is likely that they reuse tempfile names, server ports, etc. Time to start tacking $$ onto your tempfiles or using File::Temp. Of course, the directory-based partitioning (run each directory as a process) tends to help drop-in compatibility, but can block performance if things are unbalanced.

Next steps probably involve more switches and various forms of partitioning.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • Eric, this is a huge win for us. People used to ask "why are we rewriting Test::Harness?" It's because of things like this that they've stopped asking this question.

  • I'd hacked evil equivalents several times in the past but never had the time and brains (or the new T::H :-) to make something sanely generic.

    You are most exellent!
  • ...about the fact that it's my refactoring of our display code that's causing him a load of pain. Great work in the face of considerable adversity Eric :)
  • with Parallel...make sure to use unique tmp file names. :-)

    Lesson #2: make sure no single test depends on a certain run order for lesson #1.

    Is there a -shuffle or some other method in the new stuff to randomize the order tests are run in. I was addicted to that feature in Apache::Test. It keeps you honest.
  • Is there, or will there be a way that test scripts can alert harness that they can't, or shouldn't be run in parallel due to some external resource issues?
    • That’s what partitioning is for. Within a partition, tests run sequentially, not in parallel.

    • Yep. Most simply, partitioning based on dirname($test) will work for most people. The trick is how to add options to prove when '--jobs N' is triggering a run-time load of the optional module. Perhaps it requires its own frontend, but that's actually quite easy due to prove being abstracted into App::Prove (well, except for the lack of composite-ing support in Getopt::Long.)
  • I have had to hack something like this into Test::Harness before, and it was annoying. I haven't looked at TAP::Harness yet, so hopefully it is better.

    One thing I remember being particularly annoying... The harness shouldn't assume that it is running the test directly. The test may actually run on another host, and the stdout/stderr saved to a file on that host and then copied back and fed to the harness. I'm not saying ::Parallel needs to provide all that, but it should be fairly easy to do with a su