All the Perl that's Practical to Extract and Report
use Perl Log In
Ovid (2709)
(email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)
Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.
Narrative Testing
One of the lies of agile development: tests are documentation. I keep wanting to create Acme::Tests::Are::Documentation and have the tests spit out a long POD rant about how tests usually (note that word) are not documentation.
Tests can be part of the API documentation, if done well, but they're often not. Consider the following verbose test output:
ok 50 - HTML::TokeParser::Simple::Token::Tag::Start->can('set_attr')
ok 51 - set_attr() works
ok 52 - HTML::TokeParser::Simple::Token::Tag::Start->can('get_attrseq')
ok 53 - works
ok 54 - elements
ok 55 - element order
That's awful, but that's commonly what I see in test suites. All the time, I see that in test suites. Now take a look at the actual test output I produce:
ok 50 - HTML::TokeParser::Simple::Token::Tag::Start->can('set_attr')
ok 51 - set_attr() should accept what get_attr() returns
ok 52 - HTML::TokeParser::Simple::Token::Tag::Start->can('get_attrseq')
ok 53 -... and it should return an array reference
ok 54 -... with the correct number of elements
ok 55 -... in the correct order
I don't always have test output this clear, but I try. You can read my test output in a narrative style and hey, we're starting to get closer to that mythical "tests are documentation" grail.
What I want my test output to do is give the developer the ability to read the output and have a decent idea of what's going on. Sometimes test names that developers assign are so worthless that they may as well leave them off and leave us with just the numbers.
ok 50
ok 51
ok 52
ok 53
ok 54
ok 55
You remember how that used to be testing? Argh! Admittedly, at least using the function name as a test name is better than nothing, but not by much. Give the poor maintenance programmer a clue and write your tests in narrative style so they can run the tests and have an overview of what's going on. At that point, they might at least know where to look for an issue and can start reading the test code itself with a greater degree of confidence.
usually is. :-)
--Larry Wall in <1991Jul31.174523.9447@netlabs.com>
"tests are documentation" (Score:1)
That's an interesting take on "tests are documentation" - you implicitly read it as "test output is documentation", whereas I read it as "test files are documentation" - meaning the contents of the .t files - they provide an example of usage, and expected results.
I try to avoid naming tests completely - it's bad enough trying to keep the *real* documentation in sync with the code, never mind a 2nd copy of the documentation, spread through hundreds of test files.
Re: (Score:2)
Note: it's been agreed at work that reproducing BBC code here is acceptable as the public pays for this code and we really don't have "business secrets" in the sense that other businesses do.
I understand what you're saying, but there's a critical difference here. Test names are embedded in the tests and tend not to get misplaced. They're also close enough to their code that they're easier to keep in synch.
Many times I find that the tests are opaque to the point of incomprehensibility and I'm forced to re
agreed (Score:1)
I agree, and I like the proposed text output format. Thanks for sharing it!
But they don't really work in isolation (Score:1)
Re: (Score:2)
That's a good point. I hadn't thought about from the CPAN testers standpoint. For me, the vast majority of my testing, though, is at work either using my code or the code of a colleague. As a result, this style of testing is perfectly suited to that environment. For the CPAN, though, you are correct that you'd have to rerun the test to understand the context. However, I suspect this is mitigated by the fact that it's your module you're trying to understand and, as a result, it will be easier.
Now that I
Re:But they don’t really work in isolation (Score:1)
I think for CPAN you want to tag your test names with a short subject, somewhat like this:
Even then, narrative test names won’t always contain all necessary hints to know which one failed, but they’ll still allow you to know at a glance what aspect of the test code is affected by the breakage.
Re: (Score:1)
to produce output like
. Test::Group looks similar, but it runs all the tests in a group as a single Test::More test.
Tags (Score:2)
In that case, what you're really looking for are tagged tests. This is something we hope to properly support with Test::Builder 2.0.