Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Mark Leighton Fisher (4252)

Mark Leighton Fisher
  (email not shown publicly)

I am a Systems Engineer at Regenstrief Institute []. I also own Fisher's Creek Consulting [].
Friday July 21, 2006
11:46 AM

Hidden Assumptions of Testing

[ #30368 ]

We software testers take a lot for granted. Our overall assumption is that the only thing that will fail is the software under test, but underlying that assumption are a number of other assumptions about the testing environment assumptions about the CPU, the disk drives or other mass-storage devices, the operating system, the libraries, our testing framework... The list goes on and on.

Here is a non-exhaustive list of infrastructure pieces that could fail while you are testing your software:

Modern CPUs are a miracle of design and manufacturing, but they are not error-free. Usually, the OS and compiler obscure most CPU defects. This is not always the case, however remember Intel's troubles of years back as immortalized in the phrase:

We are Pentium of Borg. Division is futile. You will be approximated.

Unless you are writing software to test CPUs, you can put worrying about whether your CPU has an error near the bottom of your list.

Hard Drives (Mass Storage)
Mass storage has been dominated by hard drives for (I think) 50+ years now, and is likely to continue under that domination for some time yet, although modern semiconductor memories are getting large enough to take over some of the smaller mass storage applications (USB keychain "drives", anyone?) Hard drives are a particular problem, because their moving parts make them much more vulnerable to failure. Hard drives don't always fail catastrophically, either occasionally-bad sectors can make using a hard drive problematic at best.
Networking Hardware
Intermittent network hardware failures can be devilish to solve, as they just add to the basic problem of networks not all of a network is likely to up at once, and the larger the network, the more likely this is (the Internet can be seen as the Reductio ad absurdum of this line of reasoning, as there is never a time when all the servers on the Internet are available).
Operating System
All though much current attention is directed towards OS security defects, other OS defects can interfere with your testing just as well. For example, in Microsoft Windows, tests of window message handling can fail because another, errant program is sending messages to your program's window by mistake a design defect (IMHO) that I suspect is left over from Windows 1.0.
Each new field in technology requires taming before it can be put to wide use, and compilers are no exception. When Dave Cutler was starting on the architecture of VAX/VMS, compilers were not up to his exacting standards, so the bulk of VAX/VMS (as I understand it) was written in assembly language (yes, despite the previous Unix kernel experience of writing most the OS in a high-level language). Tracking compiler bugs was practically a hobby among some PC programmers during the 1980s, as compilers were still developed mainly by hand, rather than using automated lexers and parser-generators, which introduces more chances for defects to creep in during compiler development. I really can't remember the last time I tracked down an actual compiler bug, but it was probably sometime during the 1990's.
Libraries now dwarf the size of their associated compilers and interpreters, and the ever-expanding size of these library codebases increases the chance there will be one or more defects in the libraries (see the recent news about how most binary search implementations are likely to be broken, as they run into 32-bit size limitations). One of the beauties of CPAN is that you get the library sources, so you can fix problems you might encounter.
Testing Framework
Just because you use your testing framework to test your other code doesn't mean your testing framework is defect-free. Perl's Test::Simple and Test::More come with complete sets of automated tests. Recently, over a few lunch hours at work, I wrote VB6Unit, a VB6-based testing framework using the Visual Studio Test System API and plug-in output modules with a reasonably complete automated test suite. Although it is likely that I will find defects in VB6Unit, the automated test suite helps to ensure that the remaining defects are in the edge cases but not the core testing code (Nothing vs. Null vs. Empty, anyone?)
Security Permissions
Security permissions are a good example of external factors that can affect your test environment. Unless you use mock objects for all your external interfaces, you have likely had one or more tests not go right because you had a file, database, or network permission set wrong. Servers seem especially vulnerable to security permissions problems, as permissions that are fine for one application (maintained by SysAdmin Foo) are not quite right for another application (maintained by SysAdmin Quux), which leads to problems when Foo and Quux don't think to talk about all their changes in their shared application objects.

That was eight of the myriad ways for your tests to fail even when you have constructed the tests correctly. Do not despair, dear reader computers and their software are now reliable enough that errors in your tests can first be assumed as defects in your reasoning, not defects in your testing environment. My little guide is just a "heads-up" that once in a while, occasionally, (and usually without warning), your perfectly good testing code will fail you because of a testing infrastructure dependency fault.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.