Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Alias (5735)

  (email not shown publicly)

Journal of Alias (5735)

Thursday July 26, 2007
10:41 PM

Testing doesn't matter as much as you might think

[ #33893 ]

The Perl/CPAN community places a lot of emphasis on testing. Testing early, testing often, and testing thoroughly.

But unfortunately, sometimes we go a bit over the top.

Yes, it is extremely important that the tests are written and run by the author.

Yes, it is very important that the tests are run by a collection of people other than the author, especially on other platforms and locales. This catches lots of common errors, such as file path names in tests, forgetting to specify dependencies in the Makefile.PL, locale assumptions, and so on.

HOWEVER, in the context of an end-user installation, things are different and it is actually less important that the tests are actually run.

Test failures in both the author and CPAN Testers contexts are iterative events. The author is getting feedback and using that feedback to improve the codebase. And nothing matters except for the information such failures provide to the author.

However, when an end-user is installing a module, test failures mean installation failure for the end-user. There are no steps they can reasonably and safely take to resolve the problem immediately.

So it is important to consider whether you REALLY want to any specific subset of tests to run or not.

Because by running a test, you are effectively saying that you would rather the end user not use a module at all, rather than continue.

For core tests of functionality, this is obviously the case. If it's breaking, you DON'T want them using it.

But for things like POD tests or coverage of Perl::Critic, or optional features, or expanded tests, and so on and so forth, this is not always going to be the case.

As a module author, you can often make a fairly reasonable judgment call that if the core tests and platform tests all pass, and the optional feature testing works for both the author of the package and for CPAN Testers folks, the nature of that optional feature is such that it is highly unlikely to break for end users.

And so it's perfectly fine to skip those tests and have them not run.

The same thing is the case for PREREQs (module dependencies).

By listing something as a dependency, you are saying you would rather not have the end user install your module AT ALL, if that module is not available.

I mention this mostly because of this journal entry.

The author in this case is so concerned about not running tests, that he has contorted the dependencies in a circular.

This really shouldn't be a problem.

You just avoid running those optional tests for the end-user, or consider moving them into the optional package.

Because that optional module isn't a dependency if it depends on the original package. It's a plug-in type of package.

The feature used should probably be removed altogether from the main package, so that anything needing the additional feature should have to install the plug-in separately, and module authors should have a separate dependency for it.

And if an interactive question and answer "which features do you want" type thing is REALLY needed, than we can follow the example of Task::Catalyst, and wrap an interactive Task module over the top of the lot.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.
  • I didn't install a module recently because the modules it wanted me to install were specifically for the author's testing and useless for me as the end user. Test::Pod, yeah I really need to know that on my end. I agree 100%. I guess sometimes we are thinking so much of testing we aren't thinking what context those tests need to be run in.
  • In the extreme case you could have functionality that exists in a core module that is not tested because it depends on an optional module, and then you could have another module that lists both modules (the core and the optional) as dependencies, tests the optional functionality, and pretends to the user to be the module you have to install if you want that functionality. It could be a test-only distribution, although it might not make that obvious to the end-user.

    J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
  • "The author in this case is so concerned about not running tests, that he has contorted the dependencies in a circular."

    That post has nothing to do with running tests. I'm merely pointing out that a 'feature' is NOT the same a prereq, yet the toolchain treats them exactly the same way. The 'feature' has it's own tests, and they would be run regardless of whether the second dist is install AFTER the primary dist, or inline with the primary dist.

    The reason I have a circular, is I simply wanted to add a 'featu
    • And yes, I mentioned tests, but only because I'm sick and tired of getting testers reports from broken smokers/toolsets that can't tackle circular deps, dep install orders+@INC, and sending failed reports to ME when one of the prereqs fails to get installed by the smoker. Using the 'feature' part of M::I is a great way to let people know that can install additional dists to get more functionality. Sometimes those extra dists are totall unrelated, soetimes they more like 'expansion packs' that are totall us
      • There's two problems with this though.

        1. A feature IS a set of a dependencies, even if you don't think of it that way.

        2. You shouldn't be using the feature() command to "let people know that can install additional dists to get more functionality.".

        The second one is more important, because you should never assume that the end user themselves understands what your module does (they can be using something several levels up the dependency tree).

        Your feature question is going to do nothing but confuse them.
        • "2. You shouldn't be using the feature() command to "let people know that can install additional dists to get more functionality."."

          Huh? That's EXACTLY what it's for [or how everyone uses it]... "Install X to make Y work"....why everyone does feature 'XYZ Support' in their Makefiles.

          From Task::Catalyst:

          feature 'Log4perl Support',
              -default                  => 1,
              'Params::Validate'        => 0,

          • > Why should I have to make a separate Task::Handel just so I don't have a circular when CPANPLUS can get it right, but other tools can't?

            Reduction of confusion.

            Installation without requiring user intervention.

            Not limiting RedHat users all to the same choices for the answers.

            And finally...

            Just because one particular tool compensates for some particular author mistake, doesn't mean all of them necessarily can or will.
  • I totally agree. Thanks for the post!
  • So it is important to consider whether you REALLY want to any specific subset of tests to run or not.

    This is one place where Kwalitee diverges from Quality.