Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

ajt (2546)

ajt
  (email not shown publicly)
http://www.iredale.net/

UK based. Perl, XML/HTTP, SAP, Debian hacker.

  • CPAN: ATRICKETT [cpan.org]
  • PerlMonks: ajt [perlmonks.org]
  • Local LUG: AdamTrickett [lug.org.uk]
  • Debian Administration: ajt [debian-adm...ration.org]
  • LinkedIn: drajt [linkedin.com]

Journal of ajt (2546)

Saturday May 17, 2003
12:21 PM

Testing

[ #12267 ]

I've decided that tests are one of the most useful thing I've learnt about Perl in the last 12 months. In the past I've seen tests whizz past like a blur when installing anything from CPAN, and they are noticeable by their absence in PPM files, but now I grok them.

They are a great help to the module developer. I know I'm preaching to the converted (well I hope I am), but I'm constantly amazed by the defects that turn up during testing that I didn't know were there. You know the story you add a new feature, add a few tests, and then think you'd better add a new test to an old method, and then you discover a new unhandled exception, so you fix your module, and add yet more tests.

My current task has moved XML::RSS::Tools up to 94 tests. The latest tests require a live Internet connection, to test the HTTP clients. It's a bit clunky as the user has to select a reachable URI, but that allows the test to work behind firewalls, or over SSH tunnels where things are odd! I've made it so that the user can select to abort the test during the build Makefile and during the testing stage, but it's messy requiring user input....

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • As for the absense of tests in PPM files, I heard a rumor that ActiveState only allows modules to be turned into PPMs if they pass all tests. If that's true, that means some of the largest, most useful modules may never see the light of day because one test out of two thousand might fail and small, broken modules that only have a use_ok() test might get included. I wonder if anyone can (or will) confirm this?

    The latest tests require a live Internet connection, to test the HTTP clients. It's a bit clunky

    • I've often wondered about ActiveState's PPM policy. Most pure Perl modules don't need PPMing, CPAN works perfectly well, and I don't think it's that hard to use on Windows. The modules that require a compiler, something Windows doesn't usually have, are more troublesome, and it's nice of them to provide the files all nice and packaged up. BUT, it's a lot of work for them to keep up with all of CPAN, and it's clear that there is a lot of popular stuff they just haven't done, as it's only available elsewhere

      --
      -- "It's not magic, it's work..."
    • Yes, I can (and will) confirm that only CPAN packages that successfully pass their regression tests are included in the ActiveState PPM repositories.

      The processes that build the repository are still being tweaked, but ultimately, a module will have to build and test. Otherwise it is broken on that platform.

      It is the package authors responsability to make sure that tests work in an environment that for example don't have a test database set up or where STDIN is not available to Makefile.PL.

      We are still w

      • So ActiveState does realize that this means that the overall quality of modules made available will probably be lower? While I will agree that it's the author's responsibility to ensure that there modules pass all tests, it can be very difficult to test complicated modules on platforms that the authors don't have access to. For example, the page you direct us to lists the following platforms:

        • windows
        • linux
        • solaris
        • hpux-pa-risc
        • hpux-ia64
        • hpux-ia64-lp64

        Will you create a PPM for a module for

        • If the tests fail on a platform, it means that either the module or the tests don't work on that platform. I totally disagree that it is likely that the module will work and it is just the tests that are failing. If tests are known to fail on some platform, then they should automatically skip, and not just fail (if the failure is acceptable). Known failing tests are worse than not having tests at all, because they are bogus tests.

          For the 5.6 repositories, it is unfortunately necessary that all platforms

  • Just use a SKIP block that skips over any group of tests that is likely to fail if a first ping probe fails.
    SKIP: {
      skip "skipping online tests... not connected", 17
        if not_connected();
      ... 17 tests here ...
    }
    It's up to you to write probes that make sense for the group of tests you're skipping.
    --
    • Randal L. Schwartz
    • Stonehenge
  • You really shouldn't be prompting from your Makefile. It makes it very awkward to automate. Sure, it's just one module, you only install it once. But it's really annoying when I'm building about 50 modules off of CPAN in one go (say for a new site installation) and some of them are interactive. I come back from lunch and find that it's stopped after the 3rd one.

    Of course I want the foo option, dammit, it's the default!

    Normally, most of these modules provide some way to avoid interacting, but every single one is different! It's a most unsatisfactory state of affairs.

    You might also wish to review some [perl.org] of [perl.org] acme's [perl.org] old [perl.org] journals [perl.org] for more details on the subject.

    -Dom

    • I couldn't agree more that returing from lunch to discover that something stoped 5 minues after you left it is a big pain. Then again, it's also annoying when a tests fails just because you're behind a firewall. I'd love to make it work like magic, testing if it can, skipping if it can't!

      I've had some good ideas suggested, even some via Perlmonks from this jorunal thread, so things are looking up.

      --
      -- "It's not magic, it's work..."
      • Look at Makefile.PL [cpan.org] from Net-DNS! It shows you how to detect if a direct connection to the internet is available.

        It also shows how you use the prompt() function of MakeMaker. That one will automatically choose the default value if STDIN is not a tty, or if the PERL_MM_USE_DEFAULT environment variable has been set. This makes it much more likely that your module will build (and test) in an automated environment.

        Oh, and a hint from looking at another PPM build failure: Make sure that you always accept th

        • Jand,

          Actually it wasn't a discussion per se, rather Corion [perlmonks.org] suggested that you would include HTTP::Demon in the test suite as WWW::Mechanise::Shell does. According to the credits in the file, Merlyn made the original suggestion.

          The Perlmonks node that started this whole thing was: LWP::Parallel vs. HTTP::GHTTP vs. IO::Socket [perlmonks.org] but there have been several emails, and P2P messages that are missing from my use.Perl journal.

          --
          -- "It's not magic, it's work..."