Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

  (email not shown publicly)
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Tuesday January 15, 2008
05:52 AM

Where to add brakes to a test suite

[ #35391 ]

A while ago, there was a debate on Perl-QA as to whether or not one must have test programs run in separate processes. What a few people ignored is:

There are pros and cons to each approach.

Run all tests in separate processes and you can know that your tests are focusing on exactly what you want to test. Run them in the same process and you can catch strange state issues. Sometimes those are in the tests, but sometimes they are in the code. Both approaches offer benefits and drawbacks.

Today's navel-gazing exercise involves being able to have a test stop when a failure occurs or optionally bailing out of the test suite when a failure occurs. For example, right now I have one test program whose 4,039 tests take over six minutes to run (yeah, it sucks, but hey, it happens). Since I'm using Test::Most, I just add a 'die' to the import list and the test program halts at the first failure. This failure happens about 38 seconds into the run and then I can see what causes the failure and rerun it with a subset of its data and recreate the error faster. This makes debugging much easier than being hunched over the keyboard waiting to hit ctrl-C if I see a failure. Or maybe I should just pipe everything through less and manually page through those results, eh? (Because this is programming and we want to increase manual involvement?)

Or I could just tell the damned tests to stop when a failure is encountered. Simple, eh?

The debate, however, has ungracefully broken down into all sorts of positions, the two most prominent which seem to be:

  1. The harness runs the tests, so the harness should be the only thing which stops the tests (this should require a patch to Test::Builder to remove BAIL_OUT functionality).
  2. The builder produces the test output, so the builder should be responsible for halting the test output.

You know, while I advocated the second position, I know that each approach has strengths or weaknesses. Unfortunately, somehow the "harness should stop the tests" camp has become entrenched in their position and are arguing strongly for it, even going so far as to hint that their's is the "one true way." While I think their position has a lot of merit, I'm distressed at the dogmatism surrounding the topic.

Meanwhile, Test::Most does what it says on the tin.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.
  • Last I checked, Test::Builder was pretty much at the center of testing, and as far as I'm concerned, pretty much all functionality possible should be there.

    Just to get really wild, I don't know a whole lot about how all the various parts of testing interact, so I'm looking at it with a clean (and probably ignorant slate), and I wonder if there should even be a harness.....

    J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
    • Well, the problem with that is that TAP is the Test Anything Protocol, which is to say you don’t know if the tests on the other end of the harness use Test::Builder, or if they are written in Perl in the first place, or even if they run on something that has a microprocessor at all – for all you know, you’re parsing the self-test output of an embedded device received over a serial port. So the “harness should be responsible” camp has a point.

      On the other hand (which is the ar

  • It's about separation of concerns. Test::Builder produces TAP. TAP::Parser parses TAP. The harness is the mediator between the two.