Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Ovid (2709)

Ovid
  (email not shown publicly)
http://publius-ovidius.livejournal.com/
AOL IM: ovidperl (Add Buddy, Send Message)

Stuff with the Perl Foundation. A couple of patches in the Perl core. A few CPAN modules. That about sums it up.

Journal of Ovid (2709)

Monday December 15, 2008
06:34 AM

Idealism Versus Reality: Reality Wins

[ #38089 ]

What do you do when people keep doing the wrong thing? I see three general types of responses:

  1. Tell them to do the right thing
  2. Make it easy to do the right thing
  3. Make the wrong thing less wrong

Those are in the order in which they should be applied. The problem is that item 3 is considered heresy by some people. It's such an awful heresy that sometimes when I bring it up publicly, I get soundly chastised for this. Sometimes I deserve it and it forces me to rethink my position. Sometimes I ignore the criticism and the people who use my solutions are happier for it. Data::XML::Variant is a case in point. I hope you never have to use that steaming pile of ones and zeroes, but if you are sending, say, IDIF feeds to Yahoo! in their pseudo-XML format, you're not going to get them to fix things and you know it, so option #3 is on the table.

Now I have a problem with our test suite at work. If we run the "fast" version of our tests, they take about 7 to 8 minutes. The regular version of our tests takes 35 to 45 minutes. Lately it seems that developers are running the fast version of our tests before merging branches to trunk and not running the slow tests. As you might expect, our trunk has broken a few times.

We use Hudson for an integration server, so we find out about this, so it's not the end of the world, but I'm trying to figure out how to solve this.

The "fast tests" have saved us so much time and agony that they're "making the wrong thing (not running all the tests) less wrong" and we get more work done, but how to get the developers to run the full test suite before merging to trunk? Once again I need to go back to step 1 (do the right thing) and then to step 2 (make it easy to do the right thing) before moving to step 3.

Welcome to the real world: sometimes it's not nirvana.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • It sounds like your version control system could help you here. It should be easy to use Devel::Cover to generate a source file -> relevant test mapping. From there, you can see which files are affected by a merge. Then, you can compute a minimum set of relevant "slow" tests to run. This should prevent the merger from having to run all the tests, and save him some time.

    Obviously this could go wrong in a number of ways, but it might be helpful anyway.

    • Devel::Cover is a wonderful tool, but we have a lot of trouble using it. Our test suite is slow enough, but with Devel::Cover, it can only be run once, and that's an overnight run (7 to 10 hours).

      What this means is that we can only run it on trunk, but we need it run on our branches. Since they change, add and (sometimes) delete tests, we've no real guarantee of which tests from a branch should be run, thus putting us back at square run.

      And for extra credit, create a multi-user development environment wh

    • Devel::CoverX::Covered [cpan.org] does the mapping (which turned out to be far from "easy").

      But it doesn't quite solve the problem.

      Which isn't really a technical problem anyway. People just needs to run the tests.

  • Our work setup is that everything runs thru a QA branch (trunk -> QA -> branch ). This is a patch for us not having a proper path to live so we just have a branch for stage. But this affords us a single path to trunk. So it would be feasible to merge to QA, run the fast tests on QA, if I feel good enough about my changes and I'm lazy then you can have a process that runs at night that:
    • locks the QA repo (no more commits)
    • runs the long tests
    • if all passes then merge to trunk
    • else email all the devel
    --
    benh~