Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

schwern (1528)

schwern
  (email not shown publicly)
http://schwern.net/
AOL IM: MichaelSchwern (Add Buddy, Send Message)
Jabber: schwern@gmail.com

Schwern can destroy CPAN at his whim.

Journal of schwern (1528)

Monday December 28, 2009
08:43 PM

Numbered test file abuse

[ #40054 ]

I hate numbered test files. Not that it isn't useful to force the ordering of some tests, but because its not then necessary to force the ordering of EVERY test. At the worst case you're back to BASIC. Observe the test suite from SQL::Statement.

00error.t
01prepare.t
02executeDirect.t
03executeDBD.t
04names.t
05create.t
06group.t
07case.t
08join.t
09ops.t
10limit.t
11functions.t
12eval.t
13 call.t
14allcols.t
15naturaljoins.t
16morejoins.t
17quoting.t
18bigjoin.t
    19idents.t
20pod.t
21pod_coverage.t

Now, how much of that is really trying to order the tests and how much of it is just the order it happened to be written? Does the limit test really have to go after the functions test? Why is there a quoting test in the middle of three join tests? I see three that make any kind of sense, 00error.t (since most of the tests use RaiseError), 20pod.t and 21pod_coverage.t, which maybe go last, though its honestly not important that they do.

What's the harm? It cripples command line completion. And you've got the old BASIC problem of renumbering. I want to add a new test, where does it go? Do I have to puzzle out the implied dependencies? Do I stick it at the end? But then the POD tests aren't last any more. Do I renumber everything? Do I use a duplicate number? Do I say the hell with it and cram it into an existing test file?

Not worth it.

Realistically most test dependencies really want to express two things: Run this first and run this last. For that you have 00foo.t and zz-bar.t. 00compile.t, 00setup.t, zz-teardown.t, zz-pod.t, etc... Anything else, just write it without the number. Or if you really do have a fixed order use Test::Manifest.

If you do have tests that would do better running in order, then instead of smashing them together into one arbitrary numbering system, group them. That is, stick them into a common subdirectory and then apply the first/last numbering again. A clear candidate in SQL-Statement would be t/join to contain naturaljoins.t, morejoins.t and bigjoin.t. This has the advantage of easily letting you run all one group's tests in one shot. "prove -lr t/join".

I don't mean to pick on SQL::Statement, its just what I'm patching right now. Lots of distributions obsessively number their test files to no real advantage.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • What I usually recommend people to do, particularly if the test suite gets large, is to name tests after their packages. For SQL::Statement [cpan.org], you would get something like this:

    t/sql/dialects/ansi.t
    t/sql/dialects/anydata.t
    t/sql/dialects/csv.t
    t/sql/eva l.t
    t/sql/parser.t
    t/sql/statement.t
    t/sql/statement/function.t
    t/sql/statem ent/functions.t
    t/sql/statement/getinfo.t
    t/sql/statement/operation.t
    t/sql/s tatement/placeholder.t
    t/sql/statement/ram.t
    t/sql/statement/term.t
    t/sql/sta tement/termfactory.t
    t/sql/stateme

  • Realistically most test dependencies really want to express two things: Run this first and run this last. For that you have 00foo.t and 99bar.t. 00compile.t, 00setup.t, 99teardown.t, 99pod.t, etc... Anything else, just write it without the number.

    Wouldn't that make a test file starting with an alphabetic character run after the 99-prefixed files?

  • I don't necessarily use consecutive numbers, instead, I use three-digit numbers, with the first digit indicating a "category".

    This is from one distribution of mine.

    001_compile.t
    100_machine.t
    102_misc.t
    103_wix_component.t
    104_wix_fragment.t
    111_environment.t
    112_files_entry.t
    113_files_component.t
    114_files_directoryref.t
    115_startmenu.t
    116_registry.t
    117_directorytree.t
    118_icons.t
    119_createfolder.t
    120_feature.t
    121_files.t
    122_removefolder.t
    500_new.t
    800_perlcritic.t
    801_pod.t
    802_pod_coverage.t
    8

    --
    The new Strawberry Perl for Windows has been released! Check http://strawberryperl.com for it.