Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • I wonder if your post today was inspired by what I wrote in reply [perl.org] to your last post or we're just on the same wavelength with this. :)

    At YAPC::NA this year I did a lightning talk on a web-testing framework I built for a client*. The main topic was just that the tests were all data-driven, and because of that I gained a pile of great functionality, not least of which was the ability to turn my test suite into a benchmarking suite seamlessly.

    By running via prove or directly, each script was a set of unit
    • I think Alias addressed most of my would-be response, so I'll limit my comments to:

      All of the efforts I've personally seen to turn unit tests into benchmarks ended up being more of a benchmark of perl's compiler than anything else.  Unit tests tend to focus on tiny portions of the overall code, and test just them; they tend to not do unnecessary looping.  As such, the naive 'turn them into benchmarks' puts the loop around them, and unfortunately either re-evals them, or re-forks and evals them.

      One would be better off turning ones functional tests into benchmarks - on average (at least for me) they test a larger portion of the code per script, but tend to not be as complex, proportionately speaking.  Some of my functional tests have even been distilled down to the point of "Directory A contains a bunch of input files, directory B contains files with the same names which are their corresponding output files.  See if they generate the right output."  When that's running against a persistent process, it can be looped without too much worry.

      Of course, the real benchmark is probably more like, "Directory A contains a bunch of input files; each one corresponds to a real session with its data munged to eliminate personal details.  Each session file resets its state at the end.  Directory B contains the expected output for each of these sessions.  Run them all at the same time, repeatedly."

      • Indeed, in this case the test harness *is* testing on a more functional/integration level. As we refactor the code that runs the client's site, we want to ensure that we neither break any part of the site nor slow it down.

        Still, using this type of technique has made me wonder if there is a good way of encouraging benchmarks of perl modules by piggy-backing on the tests.

        An example of this can be found in the test suite for Sort::Maker. http://search.cpan.org/dist/Sort-Maker/ [cpan.org] (It was actually Uri's advice to

      • One of the other problems we have is that because our system is enormous, with many links to external systems and pre-compiled caches, it takes two to three orders of magnitude more time to set up for a functional test than it does to run the actual tests.

        I think any sufficiently robust solution is going to need a way to factor that out, and this is a problem for the test script overloading approach.

        The test scripts would need a way to isolate the code you are interested in from the code you are running to