Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

Ovid
  (email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Friday April 18, 2008
07:34 AM

Time to Rename TAP

[ #36191 ]

On the TAP discussion list, there's a robust debate over the future of TAP diagnostics. To a large extent, it's been pretty much settled that TAP diagnostics will be in YAML format. Currently, there are no diagnostics. You might think there are, but what you see is unstructured information dumped to STDERR. By a happy accident, you can deterministically read the file and line number, but that's it. Any other diagnostic information is effectively lost and a human has to eyeball it. People have tried a number of times to work around this, but to no avail. Thus, we need structured diagnostics. Most of you won't, but just because you are happy with plain text, someone who requires RTF isn't going to be happy with Notepad.

For the most part, we aim to ensure that the casual user sees no difference. Got that? The casual user should see no difference! Under the hood, however, you should be able to tweak that engine to your heart's content. People like me need to get under the hood. People like Yahoo! need to get under the hood (and they've already shoehorned some of their needs on top of TAP). If we really want the 'A' in TAP to stand for 'Anything', then we need to provide for this. Thus, diagnostics need to allow user-supplied keys.

That's where the disagreement starts. Schwern argues that we should have them completely free-form. We can't predict what the user needs and therefore we shouldn't try.

The counter-argument from chromatic is that if two TAP streams supply a key named 'Time' and one means 'start time' and the other means 'duration', how can a client possibly disambiguate them? Requiring identifying prefixes on the keys eliminates the ambiguity.

While I am very sympathetic to Schwern's point of view, the problem is the same problem with blessed hashes and large applications. Key collision can be very painful and difficult to debug. People with serious test suite needs aren't going to be terribly impressed when you tell them to build rich clients on unreliable keys.

The problem with chromatic's point of view is that at the end of the day, we're actually trying to attach semantic meaning to these keys and the only unambiguous way to do that is to define authorities for any prefix and this goes way beyond the scope of TAP.

I am, however sympathetic to chromatic's point of view and while I don't think he would argue for providing canonical authorities for prefixes, I think he would agree with requiring users to supply arbitrary but meaningful prefixes (in fact, I think this is what chromatic meant) for their keys or to group them under said prefix. This is admittedly ambiguous, but less so than just an unstructured mass of data. Clients can then selectively choose to use official TAP keys and known prefixes.

There was a claim that since we don't know what people will do with these, we're arguing from ignorance. I don't buy this because there is plenty of prior art in this area. Just look at TestNG, ant XML test output or any other well-used test-information structure. We can see plenty of what people need in the real world. Much of it doesn't always apply to different problem domains and hence the necessity of allowing systems to disambiguate their keys from other's keys.

Failing to provide even basic namespace safety leads to the problems you see in COBOL or PHP where you have ridiculously long function names which embed a faux namespace. It's a nightmare to work with (I speak from plenty of experience here).

If we go down the typical Perl route of being Captain of the USS Make Shit Up, how on earth can we convince those with serious testing needs that TAP is a serious protocol? It's a stated goal of ours to bring TAP to a wider audience. If we don't do this, we need to rename TAP to TAPDANCE: Test Anything Protocol Doesn't Address Needed Commercial Exigencies.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • For the most part, we aim to ensure that the casual user sees no difference. Got that? The casual user should see no difference!

    I'm a little confused here. Who is the target audience for TAP? And are you confusing that audience with the "casual user" that installs a module from CPAN and sees lots of junk scroll by on the terminal?

    The fact that CPAN module installation shows a lot of diagnostics to "casual" users is a historical accident and one that distribution packagers, whether PPM or .deb or .rp

    • use.perl seems buggy if I'm previewing several times. The reply below was for you.

  • I can define multiple target audiences and potentially different needs they have:

    Installers like CPANPLUS
    Should ignore POD failures.
    Smoke Testers
    Should potentially alert authors to POD failures, but not fail the module because of this.
    Developers (module authors)
    All tests must pass.
    Power Users
    Test timings, code coverage over time, tagged tests, aggregating multiple TAP streams from multiple projects, critical failure response and so on.

    All of the above issues have cropped up mult

    • My point was that you said "casual users" shouldn't see a difference. None of the examples you gave are casual users.

      I think worrying about two different companies with internal "Customer" modules is over-engineering a solution that can be solved with culture instead of code. All you need to do is document that using a generic module name increases the odds of a namespace conflict. Companies that care can name their internal test modules Yoyodyne::Internal::Customer or whatever suits their fancy.

      I think
      • Casual users: How would you define 'casual user' then? Someone who never runs tests? Then that's someone who will never see an impact anyway. I thought that the installers should be the 'casual users' because most of use use CPAN or CPANPLUS or make to install a new module. Those are the ones who shouldn't see a difference and those are the ones I view as casual users.

        Namespaces: I'm rather unhappy with the 'lower-case reserved' suggestion as I think the embedded dot is a reasonable compromise, bu

  • Can't we just steal the standard namespace trick of leeching off IANA?

    ---
    line: 5
    column: 6
    tap.yahoo.com:
            mykey: whatever
    expected: foo
    got: bar
    • I think this can work. Eric Wilhelm suggested the idea excluding dots from standard keys are recommending them (I would say "require") for user keys. Users would then be recommended to define a namespace for their keys (but ".time" would still be allowed). It's not perfect because clashes are still possible, but it does minimize the problem considerably. As we see more examples of what people are doing with TAP, we can build on their work. Simply allowing an "anything goes as long as it's not lower cas