Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Alias (5735)

Alias
  (email not shown publicly)
http://ali.as/

Journal of Alias (5735)

Thursday January 14, 2010
10:47 PM

Request for Comments - Module::Install support for xt

[ #40102 ]

With the increasing usage of the xt directory, and with a more robust understanding of the appropriate mechanics behind release tests, I feel the time has come to build support for xt into Module::Install by default.

But before I do so, I'd like to lay out the rules I'll be applying to the implementation for public review and commentary.

-------------------------------------------------------------------

1. The xt directory, if it exists, will automatically be added to the list of tests to be executed if the Makefile.PL is running with either..

1.1. Author mode enabled

1.2. With RELEASE_TESTING true.

1.3. With AUTOMATED_TESTING true.

2. Recursion on the xt directory will be triggered with the same tests_recursive command that triggers recursion on the t directory.

3. The behaviour of the individual test scripts in xt is assumed to be...

3.1. Always run when RELEASE_TESTING is enabled.

3.2. Run during AUTOMATED_TESTING only if the support modules are installed and at a new enough version.

3.3. Run during author mode only if the support modules are installed and at a new enough version. If the test script is unable to M:I's author mode status, the test should be skipped to prevent accidental executing during end user install.

I believe this combination of rules will give us the following scenarios.

Module Authors

Having checked out a module the xt tests will be visible during make test and will run if they can, but if not they will skip without error while telling the author what extra dependencies they need to install.

When building a release tarball, the xt tests will forcefully always run and fail if they are missing dependencies. This will ensure releases are always subject to full Quality Assurance.

CPAN Testers

The xt tests will be included in the list of tests to run, but will only be run when the system already has the appropriate dependencies in place to prevent excessive dependency recursion (which might fail and prevent any testing of the package at all).

End User

During installation, the end user will never see the xt tests displayed on their screen, even if they would otherwise always skip.

-------------------------------------------------------------------

If you have any comments, questions, or suggestions, please reply now.

Assuming no obvious issues arise, my plan is to implement this in the next M:I release some time in the next month.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • I generally think of CPAN Testers as trying to reflect the "end user experience" -- and some people just happen to report in bulk using automated smokers. I would rather just have xt tests skipped if AUTOMATED_TESTING is true, since an xt FAIL is meaningless to an end user.

    -- David

  • Is there a reason for assuming all xt tests are "release tests"?

    I'm currently looking at using "xt" for all "extra tests", with "xt/release" for release tests, and "xt/slow" for slow tests.

    I'd want to run the release tests for both automated and release testing (as you've described), but I'd only ever want to run the slow tests if I'd explicitly asked for them.

    Now I could put them in yet another top-level tests directory, but that's getting messy, I'd rather keep all the "extra tests" in the "extra tests" d