Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

perrin (4270)

perrin
  (email not shown publicly)

Perrin is a contributor to various Perl-related projects like mod_perl, Template Toolkit, and Class::DBI. He is a frequent speaker at OSCON, YAPC, and ApacheCon, and a contributor to several perl-related books.

Journal of perrin (4270)

Thursday September 09, 2004
01:38 PM

how NOT to benchmark

[ #20799 ]
So, Tim Bray wrote this hairy regex which he ran in Perl and in Java, and it ran faster in Java. He then posted this result to his blog. There is no sample data to try it yourself, it's a single test involving unusual unicode stuff, and he says that "they don’t produce quite the same results, with occasional variation around international characters", but this has not stopped Java fans from declaring that Java's regex engine is faster than Perl's.

Then he goes on to say that perl 5.8.3 gives a different result from perl 5.6.1 and that this is somehow a strike against Perl's suitability for "enterprise" work. This sounds like a result of the many bugfixes in unicode and regex stuff that happened between those versions to me, but he appears to be saying that the inability to fix bugs in Java is a positive thing, since it means you can get the wrong answer consistently. He also doesn't say whether or not he was using the same versions of Java in each test.

The problem with blog postings like this isn't so much that they are exist as that many people who read them will not have any context for evaluating the validity of the benchmark, and will just start spouting nonsense about Java's regex speed. Very frustrating.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • It doesn't help when you have people like Simon Cozens [simon-cozens.org] agreeing with him. :)

    Actually, I reckon he would have been better off using common lisp and cl-ppcre [weitz.de]

    -Dom