Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

Ovid
  (email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Saturday January 06, 2007
08:21 AM

TAPx::Harness and Test::Harness results

[ #32091 ]

The new TAPx::Parser is not on the CPAN yet, but if you remove the t/sample-tests/bailout from the sample test directory in the TAPx::Parser distribution and run prove across those tests, you get the following summary report:

Failed Test                    Stat Wstat Total Fail  List of Failed
------------------------------------------------------------------------ -------
t/sample-tests/bignum                         2   ??  ??
t/sample-tests/bignum_many                    2 1999  3-100000
t/sample-tests/combined                      10    2  3 10
t/sample-tests/die                1   256    ??   ??  ??
t/sample-tests/die_head_end       1   256    ??   ??  ??
t/sample-tests/die_last_minute    1   256     4    0  ??
t/sample-tests/duplicates                    10   ??  ??
t/sample-tests/head_fail                      4    1  2
t/sample-tests/inc_taint          1   256     1    1  1
t/sample-tests/no_nums                        5    1  3
t/sample-tests/no_output                     ??   ??  ??
t/sample-tests/segfault           0    11    ??   ??  ??
t/sample-tests/simple_fail                    5    2  2 5
t/sample-tests/switches                       1    1  1
t/sample-tests/todo_misparse                  1    1  1
t/sample-tests/too_many           4  1024     3    4  4-7
t/sample-tests/vms_nit                        2    1  1
(2 subtests UNEXPECTEDLY SUCCEEDED), 2 tests and 3 subtests skipped.
Failed 17/31 test scripts. -10/101 subtests failed.
Files=31, Tests=101,  5 wallclock secs ( 0.34 cusr +  0.23 csys =  0.57 CPU)
Failed 17/31 test programs. -10/101 subtests failed.

Note that there are lots of question marks. You also don't know which tests skipped or which tests unexpectedly succeeded. The number of tests which failed is also a bit weird. Here's the output from runtests:

Test Summary Report
-------------------
t/sample-tests/bignum           (Wstat: 0 Tests: 4 Failed: 2)
  Failed tests:  136211425-136211426
  Errors encountered while parsing tap
t/sample-tests/bignum_many      (Wstat: 0 Tests: 11 Failed: 9)
  Failed tests:  99997-100005
  Errors encountered while parsing tap
t/sample-tests/combined         (Wstat: 0 Tests: 10 Failed: 2)
  Failed tests:  3, 10
  TODO passed:   9
  Tests skipped: 7
t/sample-tests/die              (Wstat: 256 Tests: 0 Failed: 0)
  Errors encountered while parsing tap
t/sample-tests/die_head_end     (Wstat: 256 Tests: 4 Failed: 0)
  Errors encountered while parsing tap
t/sample-tests/duplicates       (Wstat: 0 Tests: 11 Failed: 0)
  Errors encountered while parsing tap
t/sample-tests/head_fail        (Wstat: 0 Tests: 4 Failed: 1)
  Failed tests:  2
t/sample-tests/inc_taint        (Wstat: 256 Tests: 1 Failed: 1)
  Failed tests:  1
t/sample-tests/no_nums          (Wstat: 0 Tests: 5 Failed: 1)
  Failed tests:  3
t/sample-tests/no_output        (Wstat: 0 Tests: 0 Failed: 0)
  Errors encountered while parsing tap
t/sample-tests/out_of_order     (Wstat: 0 Tests: 15 Failed: 0)
  Errors encountered while parsing tap
t/sample-tests/simple_fail      (Wstat: 0 Tests: 5 Failed: 2)
  Failed tests:  2, 5
t/sample-tests/skip             (Wstat: 0 Tests: 5 Failed: 0)
  Tests skipped: 2
t/sample-tests/skip_nomsg       (Wstat: 0 Tests: 1 Failed: 0)
  Tests skipped: 1
t/sample-tests/todo_inline      (Wstat: 0 Tests: 3 Failed: 0)
  TODO passed:   2
t/sample-tests/todo_misparse    (Wstat: 0 Tests: 1 Failed: 1)
  Failed tests:  1
t/sample-tests/too_many         (Wstat: 1024 Tests: 7 Failed: 4)
  Failed tests:  4-7
  Errors encountered while parsing tap
t/sample-tests/vms_nit          (Wstat: 0 Tests: 2 Failed: 1)
  Failed tests:  1
Files=31, Tests=122,  2 wallclock secs ( 1.07 cusr +  0.39 csys =  1.46 CPU)

It reports more tests as having been run because it can count them more accurately. It also has no problem figuring out which tests have been skipped, which have unexpectedly passed (these are passing 'TODO' tests) and if you want to see those parse errors in detail, you just pass it the '-p' switch.

It's still not perfect, but it's getting one heck of a lot closer. Mine's also more verbose, but I'm reporting a heck of a lot more information. If you don't like the report format, all you'll have to do is override the &TAPx::Harness::summary. Lots of other things can also be overridden, if need be.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.