Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

Ovid
  (email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Thursday April 03, 2008
11:03 AM

Oslo Perl QA Hackathon

[ #36044 ]

Tomorrow I'll be arriving in Oslo as part of the Oslo QA Hackathon. I'll be presenting my "Turbo Charged Test Suites" talk and generally looking forward to meeting people heavily involved in Perl QA. I'm particularly interested in the Test::Builder 2.0 discussion and TAP improvements.

I want the new Test::Builder because I keep hitting the limitations of the current one. Its plan handling is awful, it's difficult to alter the output and diagnostics are a mess. Few people hit these problems, but when they do, they're hard to overcome.

The TAP improvements are possibly more important. People are constantly talking about how to generate test results as XML. When I was at a Google Test Automation conference, one topic which came up over and over again is how they needed a common XML test result serialization format. Somehow, the assumption that it must be XML was made over and over again. TAP, however, isn't an answer because we simply don't capture the needed information. It is, however, stable and proven.

If we feel TAP is a good tool for others, we need to make sure that it supports how they do things. For example, here's a jUnit test:

public void testEquals() {
    Money m12CHF= new Money(12, "CHF");
    Money m14CHF= new Money(14, "CHF");

    Assert.assertTrue(!m12CHF.equals(null));
    Assert.assertEquals(m12CHF, m12CHF);
    Assert.assertEquals(m12CHF, new Money(12, "CHF")); // (1)
    Assert.assertTrue(!m12CHF.equals(m14CHF));
}

That's one test, not four. One suggestion for handling that in TAP:

ok 1 - testEquals
    ok 1 - True(!m12CHF.equals(null))
    ok 2 - Equals(m12CHF, m12CHF)
    ok 3 - Equals(m12CHF, new Money(12, "CHF"))
    ok 4 - True(!m12CHF.equals(m14CHF)

With nested TAP, the top level test would fail if any "subtests" fail. Also, it's completely backwards compatible with current TAP because the regex for a TAP line is basically /\A(?:not )?ok/, thus ensuring that indented TAP doesn't get recognized.

Nested TAP has its own problems, but Schwern's been here for a couple of days and we've discussed some of the issues involved. I really hope we get some serious traction on this.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • hey, re this -- I don't know if you saw this on Planet Perl, but I recently hacked up a conversion script for the other direction:

    http://jmason.org/software/scripts/tap-to-junit-xml.txt [jmason.org]

    head1 NAME

    tap-to-junit-xml - convert perl-style TAP test output to JUnit-style XML

    =head1 SYNOPSIS

    tap-to-junit-xml "test suite name" [ outputprefix ] ) format, and produce XML output in a similar format to that produced by the ant task. This is useful for consumption by continuous-integration systems like Hudson (C).

    thanks f
    • damn, guess who forgot to preview. ;)

      here's the missing text:

      =head1 SYNOPSIS

      tap-to-junit-xml "test suite name" [ outputprefix ] < tap_output.log

      =head1 DESCRIPTION

      Parse test suite output in TAP (Test Anything Protocol, http://testanything.org/ [testanything.org] ) format, and produce XML output in a similar format to that produced by the junit ant task. This is useful for consumption by continuous-integration systems like Hudson ( https://hudson.dev.java.net/ [java.net] ).
  • "When I was at a Google Test Automation conference, one topic which came up over and over again is how they needed a common XML test result serialization format".

    Do you remember any conclusions from that discussion, i.e., whether any XML test result formats were proposed or demonstrated?

    John.
    --