Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

domm (4030)

domm
  (email not shown publicly)
http://domm.plix.at/

Just in case you like to know, I'm currently full-time father of 2 kids, half-time Perl hacker, sort-of DJ, bicyclist, no longer dreadlocked and 33 years old

I'm also head of Vienna.pm [pm.org], maintainer of the CPANTS [perl.org] project, member of the TPF Grants Commitee [perlfoundation.org] and the YAPC Europe Foundation [yapceurope.org].

I've got stuff on CPAN [cpan.org], held various talks [domm.plix.at] and organised YAPC::Europe 2007 in Vienna [yapceurope.org].

Journal of domm (4030)

Thursday July 02, 2009
03:49 PM

Need for Speed Part I: DBIx::Class vs DBI

[ #39213 ]

Yesterday I had to write some code that goes through ~700.000 datasets (seperated into 6 tables) and denormalise them (see the yet unwritten Part II). As we're using DBIx::Class, I first used it. Even though I avoided some in/deflators and used columns to only get the stuff I needed, the process took ages (~50 items per second or aprox 4 hours for the whole job). Well, 4 hours might be bearable, but this was only the Swiss dataset. The German one, which we have to tackle soon, is at least 10 times as big, and 40 hours is just a way too long runtime.

So I rewrote the core of the programm using raw DBI calls (I had several flashbacks to the 90's :-) The results were quite astounding, as I was now getting 500 items a second. 10 times faster!

And that was before koki told me to set work_mem to a bigger value. I choose 30MB, and got another speedup to ~900 items per second. Sweet!

Of course, it's completely unfair to compare DBI with DBIx::Class performance wise, because DBIx::Class is so much nicer to work with. But if speed is an issue, dropping back to raw DBI is the way to go.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • If you write raw SQL, don’t use DBI itself, use DBIx::Simple [p3rl.org]. It is chock full of sensibly named methods for fetching results in various forms, making client code much nicer to read.