Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Alias (5735)

Alias
  (email not shown publicly)
http://ali.as/

Journal of Alias (5735)

Monday May 05, 2008
12:21 AM

Major CPAN::Mini::Extract upgrade

[ #36326 ]

One of the things I noticed at the Oslo Hackfest was the surprising number of people using CPAN::Mini::Extract to selectively suck out bits of the CPAN (All the Changes files, all the Makefile.PL, all the tests, etc etc) and scan them for stuff.

Of course given that it was a meeting of most of the CPAN QA people, it was pretty much the entire userbase of CPAN::Mini::Extract as well :)

And with offline support now finally working properly in CPAN::Mini I can now make offline support for CPAN::Mini::Extract work as well, which (unlike offline support in CPAN::Mini) is actually useful.

Since I find that most of my random inspirational hacking happens when I'm offline, this will be hugely useful for me, if nothing else.

At the same time I took another look at the speed of the extraction, since I noticed memory usage was all over the place when extracting. From what I can tell, the extraction in IO::Zlib happens half a meg at a time, but continuously allocates and frees 50meg of memory for each block it expands. Weird.

By shifting expansion to a one-shot extraction to a temp file, and then opening tarballs once from the temp file, I managed to get a two to three times speed up for file extraction.

Combined with CPAN::Mini pipelining, this makes CPAN::Mini::Extract massively faster (a 200%-300% overall speed up).

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.