All the Perl that's Practical to Extract and Report
use Perl Log In
samtregar (2699)
Putting MIME::Lite on a diet
There's nothing quite like some justified optimization to get the heart pumping. I've been working on a system which needs to produce multipart MIME messages very quickly. It's using MIME::Lite, which, it turns out, isn't actually all that lite. After solving the other bottlenecks in my code I was left with around 30% of my processing time locked-up in MIME::Lite's code.
My first stop was MIME::Fast. It certainly is fast, but unfortunately it also has some large memory leaks. Running at full speed it was losing around 5MB per second! I spent a few hours confirming that the leaks are somewhere in the huge XS codebase of MIME::Fast and not in the underlying gmime library. I gave up and filed a bug with the author.
That left me back with MIME::Lite, so I decided to see if I could speed it up. After a few hours of work I've come up with a patch which offers a 50% speedup for my use-case (creating two-part messages from parts in memory). For the curious, here's my work-log:
- Starting work => MIME::Lite is building messages at 750/s
- Use direct hash access for Attrs => 900/s
- Change how Attrs are stored, moved sub-attrs into a separate structure. This avoids the {''} access for the vastly more common case of attribs without sub-attribs. => 980/s
- Did some work optimizing fields() which is a hot method. => 1046/s
- Tightened-up fields_as_string a bit. Might be worth making the pretty-printing optional if the spec doesn't require it. => 1057/s
- Started testing with more realistic part sizes, new base => 910/s
- inlined the popular routine known_types() => 930/s
- Played with speeding up IO_* to little effect. Removing the wrap() indirection entirely would no-doubt help but it's a big project.
I sent the patch to the MIME::Lite maintainer, so hopefully this code will be available to all someday soon.
-sam
Or is it a BUG?"
"Let's call it an accidental feature. :-)"
--Larry Wall in <6909@jpl-devvax.JPL.NASA.GOV>
I'd be interested in that patch (Score:2)
I'd be interested in that patch as at my previous job we had a system that sent thousands of big multipart MIME emails at a time and that could improve things greatly.
I'd also possibly use it in my new job too.
@JAPH = qw(Hacker Perl Another Just);
print reverse @JAPH;
Re:I'd be interested in that patch (Score:2)
-sam
Maintaining MIME::Lite (Score:2)
First of all, as I'd like to duplicate your benchmarks, I wonder how exactly you measured things? What exactly did you test?
Second, I have my doubts on how
$sub_attrs->{'content-disposition'}{'filename'}would ever be faster than$sub_attrs->{'cRe:Maintaining MIME::Lite (Score:2)
2) The code needs to be able to loop through all the sub-attrs for a given attr in fields(). That would be rather complicated with the scheme you suggest, although it's possible it would be faster (and there's only one way to find out!). However, this isn't a change I made: the old code looked up every single attribute like $attr->{$attr}{$sub} where $sub was '' for the "main" attribute. My change r
Re:Maintaining MIME::Lite (Score:2)