Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

chromatic (983)

chromatic
  (email not shown publicly)
http://wgz.org/chromatic/
Blog Information [technorati.com] Profile for chr0matic [technorati.com]

Journal of chromatic (983)

Monday June 09, 2008
01:16 AM

Kwalitee Without a Point

[ #36627 ]

Gabor Szabo requested comments on several new CPANTS metrics. Short comment: these are the worst metrics yet.

Medium comment: Yeah, yeah, Kwalitee isn't a measure of Quality. I get that, but it's no excuse for piling on even more lousy metrics that not only don't know Quality, but have never even bumped into Quality accidentally at the grocery store.

Longer comment:

  • distributed_by_debian -- I'm sure all users of Win32-specific modules, for example, will be happy to learn that their Kwalitee could be higher if they sweet-talked a Debian Developer into polluting their repository with code useless to Debian.
  • latest_version_distributed_by_debian -- maintainer on vacation? Sorry, no Kwalitee for you!
  • has_no_bugs_reported_in_debian -- finally, the Win32 developers get a break. Again, I'm not sure what this really means for them, but hey, it's a free Kwalitee point. Life is good.
  • has_no_patches_in_debian -- I've only been doing this for a decade, but I'm still under the impression that most of the OSI-certified licenses seen in CPAN distributions allow downstream modification and redistribution. In that decade, I've also seen how often downstream never notifies upstream of changes. (Though Brendan O'Dea of Debian is one of the most responsive downstream developers from any distribution for Perl 5.)
  • uses_test_nowarnings -- because the POD testing metrics work oh-so-well and because testing for custom lexical warnings is evil and bad and wrong, or at least stuffing a do-nothing string "use Test::NoWarnings" in your t/ directory somewhere is a certain sign of Kwalitee.

80% of these new metrics only measure Kwalitee of a distribution to Debian users. That may be useful for Debian users, but it's highly inappropriate (and worse, irrelevant) for everyone else. (I believe Ubuntu has a bug somewhere that not all of the world uses a Debian-based distribution.)

Constructive criticism: How about a way to opt into only the useful CPANTS metrics? I'd hate for users of my code to get confused by meaningless Kwalitee Theater.


List all Journal entries
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Kwalitee Without a Point 25 Comments More | Login | Reply /

 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.

  • Wow. I have to confess that I can't see much point to those. I would much prefer to focus on serious issues which clearly point to Kwalitee. Not many people debate strictures, but Test::NoWarnings? I often have code to specifically disable that module because it's lovely, but problematic.

    Kwalitee shouldn't be focusing on specific modules, but I can see where he's going with the Debian stuff. But yes, there should be greater control. I'd prefer "core" Kwalitee metrics with the other issues being "opt

    • Well, with metrics like these, CPAN module developers can get a small link into the Debian package review process, can't they?

      "Given enough eyeballs, all bugs are shallow", IIRC. Putting a module into a larger context (more users => more developers => more eyeballs => more feedback) SHOULD increase the quality of the software - at least over time.

      Wether or not to make a "competition" of this metric is another issue, and possibly solved by splitting up the CPANTS games into smaller ones, where the "
      • All of this kwalitee hoohah assumes that authors give a crap about their point score. I wonder what percentage of CPAN authors indeed give a crap. I'm guessing, what, maybe 1%, max? Let's call it 5% to be generous. I'd suggest that the better time would be spent on increasing the author_gives_crap score than adding more arcane little tweaks that can only damage the current author_gives_crap rank.
        --

        --
        xoa

        • Remember that author_gives_crap just is a product of all the other metrics. ;-D
          • My very point. I can't imagine that does_debian_whatever is going to help the author_gives_crap rating.

            My concern with CPANTS all along has been that the metrics are being written for the CPANTS developers, not the CPAN developers.

            --

            --
            xoa

            • Sure it helps. It helps me as a user of some module to see if it's author gives_crap about Debian. If he doesn't, then I'll at least know in advance that the module may need some Debian love, or that I may have to look for another one that can fit my needs.

              Of course, this assumes that the author _wants_ feedback, but I think that's a pretty safe assumption (and those who don't want feedback, probably trickle towards the bottom end of the CPANTS ranking anyway.)

              My point is that metrics like has_no_bugs_repor
              • My point is that there are so many authors out there that are probably entirely unaware of CPANTS, and more still that know about CPANTS and think it's silly. I think it would serve CPAN better if more people were to use the good stuff in CPANTS more than tweaking what's already there.
                --

                --
                xoa

                • What you describe here is a different type of problem, requiring another solution - e.g. a CPANTS rating score + link to more info, located somewhere on the main package description page found on CPAN.

              • It helps me as a user of some module to see if it's author gives_crap about Debian.

                That's exactly backwards.

                These metrics only hint if Debian cares about the distro. That may be useful to you if you use the Debian-packaged version, but it's useless to everyone else. If the distro has a Debian package, that means that some Debian developer went through the work to package the distro for Debian. If the distro doesn't have a Debian package, that means some Debian developer didn't go through that work.

                • These metrics only hint if Debian cares about the distro. That may be useful to you if you use the Debian-packaged version, but it's useless to everyone else. If the distro has a Debian package, that means that some Debian developer went through the work to package the distro for Debian. If the distro doesn't have a Debian package, that means some Debian developer didn't go through that work. That's all.

                  And conversely, if the distro HAS a Debian package, it means that this Debian developer trusts the distro enough to put it into Debian. It's not the work that is important, it's the question of trust that is.

                  Similarly, the presence or absence of open bugs in the Debian queue is a useless indicator. Those bugs may never come to the distro author's attention. CPAN already has a request tracker. Any bugs which aren't Debian specific should go there -- and again, Debian-specific information is only useful to people using the Debian-packaged version.

                  Open bugs in Debian could mean one of several things (and if you use your imagination, I'm sure you'll find a few more):

                  • The Debian dev doesn't trust the distro completely, only just enough to let it into Debian with some modifications.
                  • The distro has platform-specific issues, perhaps including a
        • Your response makes the common mistake of assuming that we can only do one or the other, which is of course not true.

          If there's isn't something for authors to care ABOUT, there's zero point in making them aware of it.

          It's quite reasonable to work on making the kwalitee metrics more accurate, more relevant, and less vulnerable to cargo cults.

          That way when authors DO give a crap, at least they have good information to work from.

          And the better the information, the more likely they are to agree with them, once
          • Exactly. I look at http://cpants.perl.org/author/PETDANCE [perl.org] and it tells me that many of my distributions fail certain checkboxes. So what?

            Here it says WWW::Mechanize has no README. So what? Why do I as an author care?

            It says that Mech's META.yml doesn't conform to a known spec (At least that's what I think the arcane code in the hover box tells me). So what? Why do I as an author care?

            My META.yml doesn't have a license in it. So what? Why do I as an author care?

            Perl::Critic::Bangs fai

            --

            --
            xoa

            • This is actually a good point. There should be more documentation available about _why_ a Kwalitee metric is a good thing.
              • Not just why Kwalitee is good, but why each individual data point is good. Let the author make the decision about whether it's to his advantage or not to have a META.yml that conforms to a spec, or a more likely example, whether it's worth putting out a release of a module that he wasn't going to work on for a while for the sake of putting out a META.yml that conforms to a spec.
                --

                --
                xoa

                • I don't use META.yml in any of my dists. I have no reason to. This apparently means I am bad not just on that score, but all the others that have to do with META.yml. ONOZ.

            • I just got curious and looked at my modules. They all got docked for not having a human readable LICENSE. So I looked. Under AUTHOR AND COPYRIGHT they say, This may be modified and distributed on the same terms as Perl. Somehow I think that humans would disagree with CPANTS on how human readable that is. :-)

              All of my modules are docked for my deliberate policy of not introducing unnecessary dependencies. That includes warnings, Test::Pod and Test::Pod::Coverage.

              Sometime I should get around to removing
        • This is an important point, I think. But I'm not sure what you're talking about.can you give us an example?

          This is test washback. We want a test that just doesn't tell us whether the author gives a crap, but in fact MAKES, or at least encourages, the author to give a crap. Or at least tells the author how to give a crap.

          As it stands, each checkbox is just a rap over the knuckles for not having your shoelaces tied.

          How do we learn to dress properly?

          Perhaps we have to be working at a higher level here, rather
  • The main reason I wholeheartedly agree with Gabor's Kwalitee tests is because I am involved - at the other end. Yes, I am a CPAN author, although a very modest one, but my main work in this regard is in the Debian pkg-perl group. We are currently responsable for about half of the Perl packages in Debian, and we are doing our best effort to get as close as possible to working with you (what we call the upstream authors). Take a look at our QA report page [debian.org] - In our packages, when latest_version_distributed_by_

    • Measuring the quality of Debian-specific packaging makes a lot of sense for Debian developers and users, but it's irrelevant for everyone else. As you mention, pkg-perl already tracks these statistics. What possible value is there for CPANTS do to the same?

      (Win32::OLE [cpan.org] is no less useful because there's no Debian package for it and no more useful because its Debian bug queue is empty.)

      • If CPANTS can centralise tracking for all the main downstream channels, I for one would find it useful.

        Also, I wouldn't write off Win32::OLE in Debian just yet.

        I know of at least one person that has been installing Win32 Perl modules on Debian on top of wine.

        As for Win32 modules getting a free "no bugs in debian" point, that at least won't cause any harm.

        • As for Win32 modules getting a free "no bugs in debian" point, that at least won't cause any harm.

          No one has reported any bugs for my distros on ReactOS, Multics, MVS, Windows 7, VxWorks, or IOS. Some of those are in wide use. I struggle to decide why anyone who doesn't use my code on those platforms should care, and why someone should care to inform them on my behalf.

          • Because it shows that you pay attention to all of the downstream channels, no matter how weird, and make sure none of them accumulate unfixed bugs?
        • If SomeThingElse can centralise tracking for all the main downstream channels, I for one would find it useful.
          --

          @JAPH = qw(Hacker Perl Another Just);
          print reverse @JAPH;
  • Yes, the new metrics are silly, perhaps even more so than the old ones. Luckily, CPANTS is mostly invisible (i.e. not found on search.cpan.org), so I can go on ignoring it.
Stories, comments, journals, and other submissions on use Perl; are Copyright 1998-2006, their respective owners.