Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.
  • At Socialtext, we had a similar problem with long test suite runs. The solution was to set up a system exclusively for running the test suite repeatedly. It'd check out various branches (including trunk), run the tests, and update Smolder. Smolder would email people watching the branch if tests failed.

    Since we did all our unstable dev on branches (back then), this meant that we saw test failures well before they got merged to trunk, and usually we saw them within an hour or two of the actual checkin, making it relatively easy to track down the problem.

    We'd also run tests manually as part of our dev, but I for one rarely ran the whole test suite that way.

    • This is the point I was trying to make in []. Maybe it's a management thing that Ovid can't control, but there's no reason to force people to repeatedly run automated tests. Computers do a great job at boring repetitive tasks.

      • It's not a management thing at all. I understand that smoke testing is great for this, but I still want to be sure that I can run a good set of tests repeatedly while developing and not wait "an hour or two" to find out if there's a problem. I especially want to do this prior to a check in. Perhaps it's a difference in style, but I want comprehensive feedback immediately.

        • I still want to be sure that I can run a good set of tests repeatedly while developing and not wait "an hour or two" to find out if there's a problem. I especially want to do this prior to a check in.

          I completely agree with this. But why does that "good set of tests" have to be a predetermined list? Why can't it just be the tests that excercise the feature you're working on? This means it will be different for every developer and changing pretty much every day.

          • Why can't it just be the tests that excercise the feature you're working on?

            If that gives you enough confidence that you haven't caused regressions elsewhere, great! That's not always the case.

            A comprehensive test suite is incredibly valuable, and (sometimes) end-to-end tests are the best way to achieve that. I have my doubts about the utility of continuous integration servers however, and I firmly agree with James Shore on ten-minute builds [].

            • You write unit test, then the unit, then unit test the unit which uses that unit. Do you test the kernel first or just the functionality you're running on top of the kernel (and libc (and perl))?

              So, end-to-end testing for real-world inputs and outputs gives you all of the coverage you actually need, but of course unit tests have the nice property of isolating the functionality under test so that you can see what you're doing when you're working on that given chunk.

              The question is whether the test is a deve

              • If you have good unit test coverage, and if you have well-coupled and well-factored units, you can get away with only a few comprehensive end-to-end tests. The trouble comes when your test suite is so slow that it's impractical to run it before every commit. Then you face the temptation to shove your tests off into a continuous integration server, and you risk checking in broken code and losing your momentum when you interrupt your current task to switch back to the previous task you didn't actually finis

                • For me, waiting more than 30 seconds for the tests to run is too long and I've already lost momentum. If a well-covered change passes all of its unit tests and perhaps one level of units up from that, plus the bug test, your probability of failing any other test due to that change is low enough that you win overall by simply handing the rest of the smoke+checkin off to a bot (that could even run on your machine.)

                  In the 5% of commits where the bot comes back and yells at you, you're still at break-even in t

                  • I suppose everything depends on how often you run the full tests. For me, it's on average every 20 - 60 minutes. I can invest five minutes in that confidence before checking in a change.

                    If I ran the full test suite between every change I made to the source code even if I'm not ready to check in, that's a different story.

    • Running the whole test suite before committing the merge to trunk is a given. That's not the problem, the trade-off between time and stability is very easy.

      The problem we have is when we are too many people working on the same thing, in the same branch (generally one branch per feature).

      So if you check in, breaking something, it's not so bad if it only affects you. Even if you don't know about it until an hour later, you still know what you were doing you know it's up to you to fix it.

      The problem is when so

      • Thanks for clarifying that. I should have pointed that out.

      • To clarify. At Socialtext, when we did this sort of thing (this was 2 years back and I hear things have changed), we had many, many branches. Most branches belonged to very few devs (often 1), and were for one feature or one bug fix. Some branches lasted only a commit or two.

        That limited the scope of whow as affected by breakage that was caught by the smoke tester.

        The problem is, even getting a full test run down to 7 minutes (from 30, say) is still way,way too long to run all that often. When something tak