Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • The longest suite I've worked with took about an hour and a half to complete. Regrettably, it was written with a bespoke (and broken) test harness.

    Why does your current test suite take so long to run? There are numerous strategies which can generally be applied to mitigate long test suite run times (or are you not bothered by this?).

    • In Perl::Dist(::Strawberry) I build Perl from scratch, about 7 times.

      Across all of them, I also install about 200+ CPAN modules...

      And unfortunately, none of it knows when to test in parallel on it's own, so much of my the power of my quad-core machine is wasted.

      At work, much of the extra work is necessary, because the application is ENORMOUS (250k source lines of code) with lots of lots of potential variation and state.

      Before every single test script, we kill and rebuilt users from scratch. Some test directories, this also means uploading PDF files, generating all sorts of stuff, running back-office sweeps etc.

      If we aggregated absolutely everything like you do, we'd probably only save 5 minutes out of that hour, because much of the work is in the web client to server round tripping, and in the database queries, and in the massive amount of setup tasks which we need to redo each test script.

      It gets worse next year, when I start to borg in some external nightly and weekly maintenance code into the test suite as well.

      Updating the external Endecca instance (specialised search engine vendor product) with a new baseline update is a process which takes almost an hour on it's own, much of that due to dumping out search data from the 100 gig Oracle instance.

      A full resync from our upstream ERP takes another three quarters of an hour, potentially.

      I think everyone here recognises that there's absolutely no way that we can ever be in a position to have developers running the whole test suite regularly once the level of coverage starts to get up to somewhere we are happy with. Or even run it all pre/post commit.

      And so that takes us over into the range of "nightly test run" scenarios, and running the test suite when building the installation RPMs to feed into the UAT and deployment workflows.

      And so as long as we can contain the entire test run within about 5-10 hours max, we can continue to get nightly runs quite comfortably.

      At the moment, we're a lot more concerned about issues like rollback strategies, and horrors like load testing, which is insanely hard to do (right) when you hit this complexity. Reproducing the caching behaviours of production load alone is bloody hard.

      • For the long setup time, have you considered creating a set of general test data that can be used for many of the tests?

        If you can do that, then maybe you can cache the setup of it and avoid long processing time just to get to the correct state.

        So the app code is used to move the app to the correct state only once per test suite run, or even outside of the test run. Once that is done, make a snapshot of it that can be restored quicker for each test.

        This is something we haven't done yet, but I'm thinking of