Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Alias (5735)

Alias
  (email not shown publicly)
http://ali.as/

Journal of Alias (5735)

Tuesday October 07, 2008
08:43 AM

This is the sort of crap that "patch" test reports cause...

[ #37614 ]

One of the modules that we rely on at work is PDF::API2, and I'm deeply grateful for the fact it exists. But we're also highly sensitive to it, so if it ever goes rotten we have some hard thinking to do. But lets ignore that part for now.

If you were considering writing a PDF application, and happened to be searching around for something to use, and you stumbled across the PDF::API2 module on CPAN you would see this.

Now, looking at that page, I can see that it's failed a third of all CPAN Testers reports, and it hasn't had a release of note in a while. To me, that looks like a fairly rotten not very well maintained module that I should avoid.

So lets look at it's CPAN Testers report page.

And if you want to see what it looks like with my settings, which only shows production releases of the module that have occurred on production versions of Perl (that is, in the real world) then you see something like this.

And what you'd see, if you only care about the real world, is that in the real world PDF::API2 has no less than 100% PASS rate (ignoring the 2 UNKNOWNs).

Not a single damned FAIL.

And yet looking at the graph on the CPAN testers page, you could be forgiven for thinking that the module is decending into rot.

The real situation is that the module works perfectly, unless you try to build it one of two dozen patched versions of Perl that nobody other than the P5P team will ever see, and which are probably already fixed.

It's all well and good that we run tests of patched versions and developer versions, and that we collect results from patched versions and developer versions.

But the poison is spreading and as a result we're staining the reputation of reputable modules in public forums.

EVERY feed of information coming out of the CPAN Testers database needs to be limited to production releases of modules on production versions of Perl BY DEFAULT, unless someone has a specific need to request otherwise.

This bullshit needs to end.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • I like that people test my modules on any version of Perl, but I'd like to see the Testers figure out how to have two sets of results: stable version results and everything else. The stable results are for the public. The everything else gets sent to developers and isn't in the testers database.

    I'd like to see things such as CPAN Search just use the stable results. That might be as easy as slicing the data for public consumption without making any changes to the people doing the testing.

    • That's why the CPAN Testers Matrix (e.g. the PDF::API2 results [radzeit.de]) exists. You can easily see if failures are OS or perl dependent and feel free to ignore it.
      • The CPAN Testers matrix isn't something see on CPAN Search though, which is the point. This isn't the normal techie problem of thinking that the data is there and if people would just look at it everything would be fine. People see the summary of PASS and FAIL on a module page first, and then maybe (maybe) the matrix.

        I want that first impression to be better tuned to what their experience will be, which is using the module with a stable version of Perl.

        • If you're refering to the number of PASSes and FAILs on the summary page --- these numbers are nearly meaningless, just like the cpanrating stars or the number of open RT tickets.
          • Those numbers are quite meaningful to large set of people unable to reliable judge the code themselves. They use results like 100% PASS as a guide of it they can trust the code to work in some random place.

            • People really choose a CPAN distribution based on a single number?
              • Pretend you're not an expert Perl contributor for ten minutes. Imagine that someone told you to find an XML module for Perl from the CPAN. You shuffle through several dozen pages of search results and read the distribution pages. Some of them have a big red "CPAN TESTERS SAYS THIS CODE GETS A FAIL!!!" notice.

                How many non-experts are going to dig into matrices of platform combinations and bleadperl patchlevels to realize that, oh, you can't rearrange the struct members between a PV and a PVMG and retain s

          • Why publish meaningless data?

            • It's just a teaser. If it's 100% PASS, then I don't need to look at the results. If not, then I can look at the matrix and see if there are patterns. Or dive deeper into the results.
  • Yes I get it. You don't like it. You've bitched about it all over the place. I get the message. Instead trying to beat the hell out of me, perhaps you can help fix it.

    Do I really deserve this public thrashing?

    Stop and think for a moment. Would you be so ready to fix problems within your code when someone is persistently telling you how crap you are for not fixing their specific problem?

    • You wouldn't believe the amount of crap I get about some of the stuff Strawberry does that people want changed.

      And yes, I'm trying to fix it.

      Also, I deeply grateful for the stuff you in particular have done to fix the problem on the places you control. You've probably done more to fix this problem than anyone.

  • Your argument may or may not have merit, but PDF::API2 is a terrible example to make your point.

    Have you looked in the t/ directory of PDF::API2? Here, I'll save you the time:

    use Test;
    BEGIN { plan tests => 1 }
    use PDF::API2; ok(1);

    With no unit tests, a PASS is meaningless. Only a FAIL has any meaning at all.

    That said, I think PDF::API2 is good code and Fredo has been remarkably responsive in my experience, but my own not-really-competing CAM::PDF [cpan.org] library has unit tests with code coverage over 50% --

    • True that, but the sheer percentage of FAILs all of which are on non-prod environments provides an excellent visual example.

    • Also, alas, we're doing PDF creation, and we've implemented pretty much an entire graphic design environment, with box geometry model and a ton of other crazy stuff.

      PDF::API2 is the only thing we've found with enough features and control over the generation of the PDF content.

      The old system this replaced used to use PDFLib, but it's not good enough any longer.

      We'll probably end up contributing patches and improvements to PDF::API2, rather than switching.

  • I could agree about the reports for 5.11.0. These can be neglected for some time. I cannot agree about the reports for 5.10.x. So if you look at the filtered reports for 5.10.0 [radzeit.de], then it's clear that PDF-API2's test suite is broken with every perl-5.10.1-tobe. And what did you do about it? Where's the report in the PDF-API2 RT queue? Where's the report on perl5-porters? If nobody takes action, then 5.10.1 will come out, PDF-API2 will be broken and you have to switch to another PDF module.

    I think you could