Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • I still have in my mind some sort of graphical test tool for Perl, but I don't know what I quite want it to do and what new features, techniques, and consequences would come out of it.

    Test::Aggregate certainly comes out of the consequences of mainstream Perl testing and the framework that most everyone uses. It's what works right now: "Do what you can, with what you have, where you are".

    Without any goal or intention of actually implementing something new, if you started all over with whatever test harness y

    • Off the top of my head, there are a few things I would want. First, I'd want tests installed with modules. That would allow us to test our current installation, not just the module we intend to install.

      Second, I'd want a way to track tests over time. That would be difficult and I'm not sure how we would do it, but basically, if test some-cpan-module/t/foo.t fails in a particular way that we can live with and it fails in the same way after we upgrade other modules, we can at least have a controlled environment. Right now, even if we could install the tests, we have no way of knowing if particular failures are behavioral changes or if they were always there. Yes, all tests should pass, but right now, we don't even know if those tests would fail, much less pass. If we knew they would fail, we could devise better strategies for dealing with them (that's why we have a "patches" directory at work which comes first in our @INC, thereby ensuring that we can at least have somewhat safer local changes).

      As for cherry picking tests, that's one of the reasons we want to add 'tags' to the next version of TAP. You could tag tests as 'database' and run only the database tests, if desired. Test::Class is very close to being able to do this now, but it's not quite there yet.

      For those who think that testing is simple, imagine trying to run and maintain a test suite for your entire installation. That significantly increases the difficulties involved, particularly after installing huge packages which might pull in hundreds of dependencies. Some of them (Jifty, I think) ignore dependency test failures on the theory that if their tests pass, that's good enough. The potentially leaves people with broken modules having been installed and no clear way of dealing with it.

      I might add that lately I've been making major changes to Test::Harness' App::Prove::State module. I'm giving it a proper API so that I later I can subclass it and capture test run information in a database. This should provide interesting ways of dealing with some of these issues.