Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

brian_d_foy (44)

brian_d_foy
  brian.d.foy@gmail.com
http://www.pair.com/~comdog/

Journal of brian_d_foy (44)

Thursday May 25, 2006
07:02 PM

Benchmarks suck, but so do people who believe them

[ #29713 ]

Chris alerted me to a wacko going around Wikipedia misinterpreting a single slide from my Uniforum benchmarking talk (PDF). This is the same wacko that wanted everyone on Perlmonks to boycott O'Reilly, apparently. When enough people in Perlmonks told him what he could do to himself, he moved over to trashing Perl (and here too after being slapped around in the Perl article) on Wikipedia. Chris keeps cleaning up the mess. I think this guy has infected a lot of other places, but Chris says he keeps cleaning it up, so I don't go looking for it.

In one slide from the talk, I say "Benchmark.pm comes with Perl..." and in the next, I say "..and it sux". I used the deliberate mispelling and a smaller font-size analogy to kwalitee to point out that I don't mean that the module is bad, but that a lot of people mis-interpret benchmarks because they trust the numbers too much and they let the computers do the thinking for them. Later I show an example from an earlier Stonehenge class where we constructed a benchmark that didn't show the Schwarzian Transform as good as it really is. Simply quoting me saying something "sux" because I said it as a throw away joke in a talk doesn't really say anything about anything, and certainly is far from quoteworthy in an encyclopedia.

I had written about bad benchmarks earlier in "Wasting time thinking about wasted time" on Perlmonks and also in the chapter-in-progress in Mastering Perl . Those explanations aren't as sexy as using "sux" though, and the reasonableness of the much longer discussions aren't as useful for misrepresentation and shock value. It's much harder to characterize me as anti-Perl when you read the whole thing, especially when I'm pointing out the flaws in things I've done myself.

So, to perfectly illustrate my point that you can't trust people with statistics, this wacko is using my comment that people mis-interpret and mis-represent benchmarks to support some vague notion that Perl is just bad. It doesn't have any relevance to anything to do with Perl as a language, really, and certainly doesn't support the anti-Perl stuff.

The curious thing, however, is that Wikipedia even tolerates this. Several other people keep reverting this guy's edits, and he keeps putting them back. The "play nice" attitude of Wikipedia ignores thousands of years of history that show we can't do that. Just because something is on the internet doesn't mean people are going to change. The LART and the clue stick were invented for a reason.

It's all part and parcel of attaching my name to something though. I get to be the target of militants and fanatics who have nothing better to do then think about something they hate.

This doesn't mean I'm going to stop doing anything. Once wacko is hardly worth losing sleep over.


List all Journal entries
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Benchmarks suck, but so do people who believe them 20 Comments More | Login | Reply /

 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.

  • So, I guess you think it's wacky for me to be against a publisher who post-copyrights technical books, which people tend to want to be up-to-date, and a publisher of a book that "Rather than simply describing the vulnerabilities and their exploits theoretically or showing you how to use pre-existing tools to exploit the vulnerabilities...provides the nuts & bolts you need to learn how to program your own exploit code." At least you linked to my post, but I'm wondering exactly what you disagree with abou

    • I guess you think it's wacky for me to be against a publisher who post-copyrights technical books...

      As explained before, this is what book publishers do with the permission of the Library of Congress so as to receive copyright the entire length of the copyright period, and not just part of the last year.

      (Of course, O'Reilly books enter the public domain long before the copyright period expires.)

      If the copyright year of 2006 for a book that goes to the printer in December 2005 really causes you that

      • There was a greater difference between the date I bought the book and the printed copyright date than that. Lets try to not be misleading with our examples. I don't know when the book first went on sale, but I bought the book in November and the copyright was listed as the next year. See my original post about it here [perlmonks.org], where I also quote copyright law. If there's some aspect of the law that makes printing a later copyright legal, and if it's done on all major books when applicable, then I'd withdraw my com
        • It is common practice for publishers of technical books to postdate the copyright date. For example, the Hibernate Quickly [amazon.com] sitting on my desk, has a copyright of 2006, but was released August 2005. The date is simply the date the copyright is registered, which you are not required to do. Copyright protection actually begins when you start writing your book, article, program, etc.
          • It's not just technical books.

            But August seems a bit early by the standards I got used to when I was in the used book trade part time.

            Rule of thumb in publishing used to be a book released in the last 3 months of the year carried next year as copyright date (he says working from fuzzy memory). This is not unlike Detroit model years.

             
            --
            Bill
            # I had a sig when sigs were cool
            use Sig;
        • ... at this point I'm not interested in looking into it more deeply than I have.

          There's a word for people who refuse to change their opinions and continue to complain anyway after being corrected.

    • You shouldn’t have said Benchmark.pm “sux” if it doesn’t,

      Apparently you don’t understand the concept of irony.

    • "It's hard to be sure whether other slides are meant to support the sux statement or define it, so I just quoted the sux statement and some other slide without assuming anything."

      Ha!

    • You shouldn't have said Benchmark.pm "sux" if it doesn't...

      It's obvious (to me, anyway) from the slide that he's implying that saying "Benchmark sux" means that it actually sucks in the same way that saying CPANTS "measures kwalitee" means that it actually measures quality. So just saying that "Benchmark sux" is taking the slide out of context.

      • I saw that kwalitee analogy in the slide, but...well, if I could find a link to get to the top post I would look at the slide again, but this forum software sucks. I though the kwalitee thing was just explaining that quality is to kwalitee as sucks is to sux.

    • Mmmm, I missed that:

      “Rather than simply describing the vulnerabilities and their exploits theoretically or showing you how to use pre-existing tools to exploit the vulnerabilities…provides the nuts & bolts you need to learn how to program your own exploit code.”

      I have explained to a number of people how to conduct SQL injection attacks. Will you boycott me now?

      • It's not as bad if you're selective about who your teaching, and it depends on why you're teaching them, but yes, I'll boycott you too. Not sure what you sell though.
        • I would also like to be boycotted. I have written an article, in The Perl Journal or maybe The Perl Review, one of the two, about how to do SQL injection attacks AND hack poorly written CGI scripts, on the way to explaining why taint checking is a good thing. What do I need to do to be boycotted? Is my bringing it to your attention here sufficient? Is there some form I need to fill out? Please reply at your earliest convenience.
          --

          --
          xoa

    • It's hard to be sure whether other slides are meant to support the sux statement or define it, so I just quoted the sux statement and some other slide without assuming anything.

      Why in the world would you quote something you clearly don't understand?

      "I saw it on the internet so it must be true!!"

        - ask
      --

      -- ask bjoern hansen [askbjoernhansen.com], !try; do();

      • Because it was written by a notable author and I was quoting it, not interpreting it in a way I was unsure of. It was written as though "sux" was the bottom line, and it would have been ok to quote it even if I quoted nothing else. I provided a link to the source. Other Wikipedians could have given the module the benefit of the doubt and added more details if they thought it would clarify things. Quoting "sux" was appropriate whether there's total suckiness or not, if that's what a notable author and edito

        • Speaking of suckiness, there's ... you.

          Your opinion is ignored, since I'm the one writing the code, and I hate you.
Stories, comments, journals, and other submissions on use Perl; are Copyright 1998-2006, their respective owners.