Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

Ovid
  (email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Friday June 20, 2008
07:52 AM

The Test::Most From HELL!

[ #36738 ]

So far Test::Most has garnered me nothing but kudos. People love the fine-grained control over their test suites and now it makes plans almost irrelevant. However, here's what it exports into your namespace:

TODO
BAIL_OUT
all
all_done
any
array
array_each
arrayelementsonly
arrayl ength
arraylengthonly
bag
bail_on_fail
blessed
bool
can_ok
cmp_bag
cmp_d eeply
cmp_methods
cmp_ok
cmp_set
code
diag
die_on_fail
dies_ok
eq_array
eq_deeply
eq_hash
eq_or_diff
eq_or_diff
eq_or_diff_data
eq_or_diff_data
e q_or_diff_text
eq_or_diff_text
eq_set
explain
fail
hash
hash_each
hashkey s
hashkeysonly
ignore
is
is_deeply
isa
isa_ok
isnt
like
listmethods
li ves_and
lives_ok
methods
noclass
num
ok
pass
plan
re
reftype
regexpmat ches
regexponly
regexpref
regexprefonly
require_ok
restore_fail
scalarrefo nly
scalref
set
shallow
skip
str
subbagof
subhashof
subsetof
superbagof
superhashof
supersetof
throws_ok
todo
todo_skip
unlike
use_ok
useclass
warning_is
warning_like
warnings_are
warnings_like

Today I tracked down a nasty bug where someone was importing these (but not using them) into a module which inherits from Class::Accessor. This wound up inappropriately overriding the &set method and it was no end of trouble trying to find this.

I'm trying to consider how to deal with this. One way is to check to see if caller->can($function), but that doesn't work if the offending function is created after Test::Most is used. I could check this in a CHECK block, but I'm unsure if this is the best way to manage this. There need to be other strategies here, but my primary goal is to keep this simple.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • Have you consider using

          die_on_fail {
                # your tests go here.
          }

    ?

    --
    life is short
    • Actually, I've been thinking about putting a failure handler there. I think it's a better solution:

      use Test::Most
          tests   => 3,
          on_fail => sub {
              # whatever you want
          };

      That gives end users maximum flexibility.

      Right now, though, I've discovered that the latest release, which incorporates Test::Deep, doesn't play well with Test::Class [cpan.org].

      • But that only gives you a global handler, with little knowledge of the test that failed.

        The other suggestion is more local. And even lexical.

        --
        life is short
        • Actually, I'd be passing in the same information the current die handler gets, so you would have complete knowledge of the test which failed (well, as much as Test::Harness has). And it would be file scoped, not global.

  • ...and now it makes plans almost irrelevant.
    This is one of those bits of yak shaving that Perl programmers seem to think is necessary. It always baffled me, even when I was a full fledged Perl guy, that I had to specify my plan up front. Just run the tests I give you!
    • Plans haven't saved me a lot, but when they have, they have (particularly on ridiculously obscure occasions when tests exit prematurely but with a 0 status code. There's no reason to know anything is wrong without a plan).

    • So how do you know you didn’t run fewer tests than you meant to?

      Yes, OK, that was easy. Now tell me how you know you didn’t run more tests than you meant to.

      • Just how are you supposed to know how many tests you planned?

        Anyway... how about an end marker, "yes I'm done", a sub you call at the end of your test file. All it has to do is set a flag in the test module.

        done;
        If the tests are interrupted, it won't get called, and then you'll know, as the test module will check the status of that flag in its END block.

        Something like that.

        • As noted, Test::Most does this.

        • Just how are you supposed to know how many tests you planned?

          What am I supposed to say to that?

          If the tests are interrupted, it won’t get called, and then you’ll know

          Right, that was the easy part I talked about.

          • What am I supposed to say to that?

            What I was aiming as was that with the old school test modules, you have to state how many tests you plan on executing. But how can you know? Counting them manually? Running the tests, toping they all get executed, and check how many were done in the error message?

            That is not ideal. I'd even prefer a dry run, using ok/nok to just count the tests without actually doing them, to this. Maybe do the real test in a live run immediately after that.

            • But how can you know?

              How much confidence do you have in a test if you don't know exactly what you expect it to do?

            • When you read this, the number one thing to remember is to take it with a grain of salt! Whether or not you choose to adopt a plan depends on whether or not you feel it brings enough added value to offset your dislike of it. If it doesn't, don't use a plan. I'm not dogmatic :)

              What I was aiming as was that with the old school test modules, you have to state how many tests you plan on executing. But how can you know? Counting them manually? Running the tests, toping they all get executed, and check how many were done in the error message?

              I've been trying to understand what you were meaning by this because I think that there's a miscommunication here. I could be wrong.

              If I'm presented with a 300 line test program, you're absolutely right that I can't just know

              • Running the tests, toping they all get executed, and check how many were done in the error message?

                I've been trying to understand what you were meaning by this because I think that there's a miscommunication here. I could be wrong.

                There was a typo that I noticed after I pressed "submit". What I wanted to say was tis:

                >Running the tests, hoping they all get executed, and check how many were done in the error message?

                There error message is, of course, the info of Test::* complaining about "It looks like you planned to run 20 tests, but you actually ran 23" so you can fix the plan number to 23.

                Or I can always assert a plan and never have to worry about it.

                My problem is that when I'm writing tests for a module, I may be adding a test every few minutes and I always have to update the plan. And t

                • You can get around that too.

                  use Test::More;
                  plan tests => my $num_tests;

                  BEGIN { $num_tests+=2 }
                  is( 1, 1, 'identity' );
                  isnt( 1, 2, 'identity' );

                  BEGIN { $num_tests++ }
                  is( 1 + 1, 2, 'addition' );

                  # etc
                  • That's clever! And it is possible because plan is a runtime function call, instead of the usual parameter to the use statement.

                    I must say, that plan is well hidden in the docs [cpan.org]. It doesn't even get its own entry in the module's list of functions.

  • See Test::Deep in rt.cpan.org. Incompatible isa() and blessed() functions lead to very hard to catch bugs.
  • Instead of exporting all those subs you could export an AUTOLOAD into main and have that import them on demand ;)