In doing some more work with Test::Aggregate, I discovered that many of our tests rely on $0 (script name) being set. As it turns out, you can assign to $0. The current API for Test::Aggregate now looks something like this:
my $tests = Test::Aggregate->new(
dirs => 'aggtests/',
dump => 'dump.t', # optional
shuffle => 1, # optional
matching => qr/customer/, # optional
set_filenames => 1, # optional (sets $0)
I also tried to clean up my Test::Builder hack by properly subclassing it. I gave up after about half an hour since it's not designed to be subclassed. My original hack works just fine.
You can now randomly order the tests (very useful for shaking out bogus state assumptions), run tests from multiple directories, and only run tests matching a given regex. Hopefully this will be on the CPAN soon.