Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

Ovid
  (email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Thursday September 14, 2006
03:42 PM

Massive TAPx::Parser regression tests

[ #30999 ]

After Schwern pointed me to the sample tests for Test::Harness, I've started adding them to TAPx::Parser. I got to 613 tests without a single bug.

Then I added a regression tests for programs with no test output. Just that simple requirement added 22 tests. That's when I finally hit my first bug. My parser died a horrible death. That bug is now fixed, but that uncovered another bug, also fixed. I don't know how soon I can upload, though, as I still have 21 more sample test programs to add. I think that's the point where I can bump the parser up to 0.30.

This pushes me up to 1096 tests altogether.

Question: does anyone ever use the exit status of a test program? I haven't provided this, but it strikes me that some folks might use it.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • Test::Harness uses the exit status of a test script.

    —Theory
    • Yes, but it's apparently to represent the number of failed tests and that caps out at 255, so you can't even report the number of failed tests correctly unless it's less than that magic number. VMS doesn't even get that right, so the exit status doesn't seem terribly useful. On the Perl-QA list, chromatic suggested that I just ignore it. However, if someone is actually using it, I'll need to know how and why.

      Plus, this is part of a project to replace Test::Harness and its limitations. This appears to

      • I don't see why you don't validate that the exit status is zero. Your programs never segfault or die without ever printing useful information?

        Or maybe the thing runs all the way through and segfaults in an END block. Maybe the official part of the test is already over but you can notice that the test still died a bad death.
        • Yes, that's what I meant, though that's certainly not what I wrote to perl-qa. An exit status of zero or not-zero is useful information. Anything else seems somewhat useless.

          • Absolutely. It's not like we're dealing with sendmail, where it does need to deal with the different varieties of code in sysexits.h.

            -Dom

  • The exit status is crucial information for automated installers, to know whether the test (program) succeeded or failed.

    See this bug in Test::Harness [cpan.org] for what can go wrong.

    The precise exit number is not quite relevant, but zero vs non-zero is.