use Perl

Recently I was doing debugging-via-email of some graphics code and it turned out that the problem became perfectly obvious with a single screenshot of the demo scene. I realized that I have a habit of designing test scenes for my graphics code that make rendering errors pop out to the human eye -- most any problem becomes obvious in a second or two. This is most definitely a Good Thing.

Of course, many readers will point out that I should have a strong automated test library to catch these sorts of issues; the user should just be able to run make test and get a nice test failure describing the problem. That's great advice, and I'd love to give the users a nice automated test suite . . . but how?

The problem here is that OpenGL is guaranteed to produce consistent output from run to run, but only for the same hardware, drivers, resolution, color depth, and so on. Change any of these, and the output is allowed to change quite drastically. Yes, there are some bounds to how far afield the output can stray -- but these bounds are, to put it lightly, pretty loose.

So how do you automate the testing, when OpenGL drivers are often buggy and sometimes vary widely from vendor to vendor and system to system? You can't keep canonical images in the testing library to compare against because the output from the developer's system is unlikely to match pixel-by-pixel on basically anyone else's system. You can't even save the output from the first test run, and compare against it in succeeding builds; if the user happens to upgrade their video drivers, or alter the desktop resolution or color depth, or make some other completely reasonable change to their system, the pixels are quite likely to be different.

I could spend a great deal of effort writing a fuzzy image matcher that determined whether two images matched within the available OpenGL guarantees -- and submit the result as a dissertation on computer vision. It's certainly an interesting subject for research, and I wouldn't be surprised to hear that SGI did exactly this to create their conformance testing library, though I've never seen it -- but it has little to do with the graphics work in front of me. I'd rather spend my time actually producing working rendering code.

This brings me back to my original method -- design test scenes that can take advantage of the human visual system by making real rendering errors pop out of the noise of absolute pixel value differences. An incorrect pixel here or there would be difficult for the eye to detect (unless the color was WAY off), but I'm looking more for "Well, clearly something's wrong with texturing -- the big wooden box is rendered as a plain black cube."

Yes, a human needs to get involved. But with proper test design any problems should be obvious -- so obvious that the user may react even before being able to verbalize the issue, because reacting quickly to seeing something wrong is a survival trait. In many cases, this process can actually be faster than running a full automated test suite for a complex system. Granted, it won't help autobuilders with automated test runs on the latest source revision, but it does have its place.

None of this is specific to graphics code, of course; that's just the area I've played the most with. These testing issues will come up anywhere you need to test interactions with the real world, where APIs often don't guarantee perfect reproduction on all systems, in all environments. How do you know, for instance, that the user can actually hear the music you are trying to play?

In the last few years I've come across quite a few books and articles espousing automated testing, and going into great detail about how to write automated tests and test harnesses in every development environment and programming language under the sun. What seems to be missing is a decent treatise on producing good user-oriented tests. Maybe I'm just looking in the wrong places . . . .

Wayne Keenan is back working on the SDL_perl port for Win32, and I am hosting the binary builds for him. This release brings the Win32 port up to 1.20.3 (matching most of the free *nixen), which means all of the examples from my Building a 3D Engine in Perl articles should now work perfectly.

Full PIGGE support and other fixes beyond 1.20.3 are in progress.

Following the smashing success of yesterday's show, PIGGE has now been held over for a second straight day, now in a spanking new version: 0.1.2! Faster, better, spiffier, and now with a changelog!

Changes copied below for your viewing pleasure:

• Improve and refactor font and HUD support
• Improve and refactor font test routine (the default HUD)
• Autogenerate toggle_show_foo commands for all render.show.foo config items
• Add ported toggleable HUD to challsall-blackhole port
• Add ported HUD to chalsall-cube port
• Skip rendering for invisible WorldObjects
• Implement toggle_show_axes for chalsall-blackhole port
• Mark toggle_lighting command as working instead of broken for chalsall-cube port
• Fix some incorrect and/or incomplete statistics in the trends doc
• Fix a previously untriggered braino bug in Callgraph.pm resulting in bad code output for the refactored draw_hud routine

Testing, testing . . .

Check, check, check.

Is this thing on? Ah, good. Hello there! Good to see you all out there. We've got a marvelous show for you tonight and . . . What? What do you mean they cancelled? Bloody hell. Well what else have we got? Is that it? OK, we're running with it.

Right, OK folks, we've got a new show for you! It involves farm animals and explosions and . . . What was that again? . . . and dancing boxes! You'll laugh, you'll cry, you'll think I'm insane to have this on my stage! And you'll be right.

One and all, I present to you: PIGGE version 0.1.1!

That's an awful low version number . . . first time on stage? You handed me an act that's never been on stage before?!? Oh man, I have got to get out of this business . . . I'll be ruined for sure, I can tell already . . . .