Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

Ovid
  (email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Thursday April 10, 2008
04:37 AM

Test::TAP

[ #36120 ]

One of the things which we faced at the Oslo Perl QA Hackathon is that there are precious few tests for TAP itself. TAP consumers will have a hard time verifying that they are doing the right thing if we don't make it easy to test. As a result, along with the spec, YAML descriptive tests have been started. Two gentleman (Oystein and Andreas? Last names? Spelling?) started on the draft TAP specification and Adrian Howard was writing the Perl code which tested their YAML. A sample test looks like this:

---
name: parse error, no plan
TAP: |
    TAP version 13
    TAP done
Result:
    Error:
        Type: Parser
        Desc: No plan found before TAP done.
...

With this, anyone writing a TAP consumer in any language can just write code to parse the YAML and verify that their consumer is correct.

However, there's another problem we stumbled on: nested TAP. We'd like something like this:

TAP version 13
1..5
ok 1 some name
begin 2 - fribble
    TAP version 13
    1..3
    ok 1
    ok 2
    not ok 3
    TAP done
not ok 2 - sucess or failure of above block
ok 3 - a separate test
... more tests

That turns out to be completely backwards compatible since lines with leading whitespace are ignored. However, that was shot down because this means that a TAP producer also needs a TAP consumer to parse the nested TAP to determine if it succeeded or not. Producing TAP is easy. Parsing it is much harder than people would suspect and embedding a TAP parser into a TAP consumer raises the bar to what we felt was an unacceptable level. Thus, this non-backwards-compatible suggestion was adopted:

TAP version 13
1..5
ok 1 some name
begin 2 - fribble
    TAP version 13
    1..3
    ok 1
    ok 2
    not ok 3
    TAP done
ok 3 - a separate test
... more tests

Note that we switch from 'ok 1' to 'ok 3'. 'not ok 2' is implied by the failure of the nested TAP. This is not backwards compatible, but it was felt that this was OK because the consumer would specifically have to ask for nested TAP.

That sucks. It sucks hard. Just because the consumer didn't ask for nested TAP doesn't mean that the producer won't produce it. Producing TAP that is not backwards compatible is very disturbing to me if we can figure out a way around it. However, since it was agreed that interpreting the nested TAP required a parser, we would have to leave off the 'not ok 2' line.

Last night I figured a way out. Interpreting the nested TAP does not require a parser, it only requires that we check three things:

  1. One and only one plan leading or trailing plan.
  2. The plan is correct.
  3. No 'not ok' tests unless they are TODO.

That's all we need. Things like 'tests out of sequence' are parse failures, not test failures. and the parser is required to catch that. Thus, I think we can get backwards-compatible nested TAP. To check this, I am now writing this:

#!/usr/bin/env perl

use strict;
use warnings;

use Test::Most 'no_plan';
use Test::TAP;

is_failing_tap 'ok 1', 'Invalid tap if we do not have a plan';
is_passing_tap <<'END', '... but it should be valid if we do have a plan';
1..1
ok 1 - some message
END

is_passing_tap <<'END', '... even if it is a trailing plan';
ok 1 - some message
1..1
END

is_failing_tap <<'END', '... but not if it is an embedded plan';
ok 1 - some message
1..2
ok 2
END

is_passing_tap <<'END', '... but it is ok if the plan is in nested TAP';
ok 1 - some message
begin 2 - fribble with nests
    TAP version 16
    1..2
    ok 1
    ok 2
    TAP done
ok 2 - refers to preceding block
1..2
END

Backwards compatibility matters.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.