My personal overall opinion is that CPAN Ratings has been a mild success. Key to its usefulness is that all these reviews, noise or signal, are easily reachable directly from the CPAN search itself. This close association makes it better equipped to be relevant than any other venue we currently have.
Of course, most reviews are just about useless and don’t manage to tell you more than that the author thinks “this module is great!” This is no surprise; reviewing well requires empathy, which is a talent few people have; particularly relating to an audience. Reviews which show that the reviewer has put the module through its paces or at least taken an in-depth glance at it, however short, are relatively few and far between. Still, their absolute number is high enough for the system to be useful, in my opinion. I’ve spent quite a bit of my own time trying to write reviews I hope are useful, in hopes of contributing positively to the system.
What really annoys me are the cases of which this one or that are only the latest examples: people use the rating system to write bug reports or to respond to a review. These things are not reviews! They don’t belong where they are posted.
With bug reports, the case is clear: they belong on RT, not on CPAN Ratings. One might argue that grave bugs might warrant a rating (and thus review) all by themselves, but then how many people are there who will actually go back once the bug has been fixed and amend or retract such a “bug review?” Better to keep these things were they belong. Let’s not forget that someone who checks the ratings can easily take a look at the bug tracker as well.
With commentary on reviews, the case gets murkier. CPAN Ratings does not provide a way to contact the author of a review nor does it offer a way to comment on a review. What’s worse is that the lack of a discussion platform is compounded by the abuse as a bug reporting system. I do not have a link handy, but I’ve seen authors respond to bug reports on the rating system by… writing a post on the rating system. Frankly, I have no good ideas on how to solve this problem. The most practical approach (note: not solution) might be to allow posting “reviews” which carry no rating so that they can be used for responses and comments. Still, the rating system is not a discussion forum (and in my opinion, rightly so), so the format will always be ill-suited to discussion.
And of course, the question which remains is: how do we get people to direct comments (even if we knew where those should go) and bug reports to their respective, correct venues?