Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • by Alias (5735) on 2008.12.11 3:48 (#66406) Homepage Journal

    Pop quiz: is the following a failure? Why or why not? Is the existing behavior wrong?

            1..3
            ok 1 - Booting
            ok 2 - Got dem boots!
            ok 3 - We have foobar # TODO Waiting on foobar shipment

    Asking if this is a fail is similar to asking if you run a test.

    In the install context, this is a merely a curiosity. It's a difference in expected behaviour, and such for the author it's a fail(ure), but it's not a FAIL.

    If you recall, the working definition of failure we used twice now for AUTOMATED_TESTING and RELEASE_TESTING is "Is the failure significant enough to forbid the user from installing".

    An unused feature, documented as broken, that unexpectedly passes does not qualify as something that should prevent installation.

    • That's pretty much spot on, though I think the existing behavior might be wrong for authors. If you're an author, I would say that the "unexpectedly succeeded" should be a fail. Otherwise, it should not. We don't want heisenfails. I want to know if code I am developing is susceptible to this, but it shouldn't cause pain for anyone else. However, just like running tests in xt/, this is something the tester must explicitly request.