Stories
Slash Boxes
Comments

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Eric Wilhelm (6389)

Eric Wilhelm
  (email not shown publicly)
http://scratchcomputing.com/

Journal of Eric Wilhelm (6389)

Tuesday September 11, 2007
05:03 AM

Parallel Testing

[ #34414 ]

So I decided to finally (after all of this compatibility work) take TAP::Harness out for a spin and make TAP::Harness::Parallel. That comes with some pain about the output formatting API (and I'll push it to CPAN as soon as we stabilize that -- right now you have to ride along and get TAP::Harness from its svn too (and possibly r491 because we're still working-through the single-process assumptions in the formatter.)) But, the preliminary results are between 40-60% reduced waiting. If your test suite takes more than 10s to run, it is probably worth it.

runtests -r -j 9

Note: I symlink bin/prove to runtests -- the Test::Harness transition isn't quite complete.

Expect about 40% time reduction in typical tests suites. The perl 5.8.8 core tests dropped to 200s from 500s. The parrot tests seem to be playing along as well. (Both of these require the directory-based partitioning.)

The biggest problem is resource conflicts. Have you ever run your tests simultaneously before? Probably not -- it is likely that they reuse tempfile names, server ports, etc. Time to start tacking $$ onto your tempfiles or using File::Temp. Of course, the directory-based partitioning (run each directory as a process) tends to help drop-in compatibility, but can block performance if things are unbalanced.

Next steps probably involve more switches and various forms of partitioning.


List all Journal entries
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Parallel Testing  Comments More | Login | Reply /

 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.

  • Eric, this is a huge win for us. People used to ask "why are we rewriting Test::Harness?" It's because of things like this that they've stopped asking this question.

    --
    Silence is Evil [easystreet.com]
  • I'd hacked evil equivalents several times in the past but never had the time and brains (or the new T::H :-) to make something sanely generic.

    You are most exellent!
  • ...about the fact that it's my refactoring of our display code that's causing him a load of pain. Great work in the face of considerable adversity Eric :)
  • with Parallel...make sure to use unique tmp file names. :-)

    Lesson #2: make sure no single test depends on a certain run order for lesson #1.

    Is there a -shuffle or some other method in the new stuff to randomize the order tests are run in. I was addicted to
  • Is there, or will there be a way that test scripts can alert harness that they can't, or shouldn't be run in parallel due to some external resource issues?

    • That’s what partitioning is for. Within a partition, tests run sequentially, not in parallel.

    • Yep. Most simply, partitioning based on dirname($test) will work for most people. The trick is how to add options to prove when '--jobs N' is triggering a run-time load of the optional module. Perhaps it requires its own frontend, but that's actually qu
  • I have had to hack something like this into Test::Harness before, and it was annoying. I haven't looked at TAP::Harness yet, so hopefully it is better.

    One thing I remember being particularly annoying... The harness shouldn't assume that it is running t

Stories, comments, journals, and other submissions on use Perl; are Copyright 1998-2006, their respective owners.