About a year and a half ago, I wrote CPAN::Reporter, to bring CPAN Testers reporting to good-old CPAN.pm. Since then, some intrepid individuals have built automated smoke testers upon CPAN::Reporter, all writing their own programs to do so.
CPAN.pm itself added an experimental smoke command to test recent uploads, but it requires XML::LibXML and doesn't test distributions in isolation -- all tested distributions and their dependencies keep getting added to PERL5LIB until things explode.
Yesterday, I released the first, alpha version of CPAN::Reporter::Smoker to provide a better-behaved, more turn-key approach to smoke testing with CPAN::Reporter.
(... configure CPAN and CPAN::Reporter as usual
$ perl -MCPAN::Reporter::Smoker -e start
This initial version still has some limitations (e.g. no developer versions smoked), but I'm ready for anyone interested to try it out and start sending feedback -- compliments, suggestions, or complaints all welcome.