use Perl

For Strawberry Perl, and Padre, I use a custom entry in the Start Menu, which technically is a shortcut (*.LNK file). For example, for Padre the command line in the shortcut file was:

C:\WINDOWS\system32\cmd.exe /c PATH=c:\strawberry\perl\bin;c:\strawberry\c\bin;%PATH% && padre

Likewise, the command line for my Strawberry shell was:

C:\WINDOWS\system32\cmd.exe /k PATH=c:\strawberry\perl\bin;c:\strawberry\c\bin;%PATH%

Overnight, these both stopped working.

After a bit of puzzling, I figured out that a new program had been installed by Windows Update, and this had added a new directory to PATH. As a result, after the environment variable was substituted with its real value, the length of the expanded command line was now longer than 256 bytes, and now PATH got truncated.

Remember, folks:

The length of the command line for a shortcut, after expansion, should never be longer than 256 bytes.

If you put the code for modification of PATH in a *.BAT file, no such restriction applies.

So now, my shortcut to start the command shell is:

C:\WINDOWS\system32\cmd.exe /k c:\strawberry\strawberry_path.bat

where c:\strawberry\strawberry_path.bat contains the lines

@echo off
PATH=C:\strawberry\c\bin;C:\strawberry\perl\bin;%PATH%

which makes it easier for me to add more Perl based tools that depend on these entries in PATH: I now have a central location for the definition, if I ever need to add or modify a directory.

Two days ago, after successfully installing Alien::wxWidgets, Wx and Wx::Demo on Strawberry Perl, with a bit of trouble and a lot oit f time, I was curious to see how Wx::Demo works on ActivePerl, and if it shows the same screwed up result in the wxComboCtrl demo. (For the impatient: it does.)

ActivePerl has PPM, right? So this should be a piece of cake. Let me see... Uh, nothing. Ooh yes, I forgot to add repositories. I'll add my favorites: TCool, Bribes, Trouchelle, UWinnipeg.

Good, now both Alien::wxWidgets and Wx appear. Smooth, in less than a few minutes, both are installed. Now to check, run

perl -MWx -le "print Wx->VERSION"

Uh oh... I get a Windows dialog box telling me some DLL (I forgot its exact name, something with "custom" in its name) can't be found, and the above command line just produces a syntax error, saying the module can't be loaded.

What next? Well, ActivePerl now supports the MinGW compiler, and I've got one installed as it came with the Strawberry distro... just add "c:\strawberry\c\bin" back to PATH, and I'm good to go.

So, uninstall Wx again, and try to install it with CPAN. Wait a minute, now it says I don't have a (usable) Alien::wxWidgets? But I installed 0.45 with PPM? Aargh! So if you install Alien::wxWidgets with PPM, you can't use it to install Wx with CPAN, and that's the main purpose of that module! Right, uninstall that too, and install it with CPAN, too.

I was not expecting a smooth ride with CPAN, and that's exactly what I've got. Installing in one go didn't work, obviously, so I broke it down into smaller tasks installing troublesome dependencies first. That did go smoothly, apart from one module that failed tests: Module::Build. I don't get why a module that is seen by many as the future of CPAN, can be so much trouble. I assumed it would work well enough, so I force installed it.

Installing Alien::wxWidgets took... forever. I gave up waiting and went doing something else, even forgetting all about it. I was surprised to see, accidently stumbling back to this console window, that it was still actively chugging along. Anyway, it tested smoothly, and installed with no problems.

Installing Wx next, went a lot faster. In a matter of minutes it was installed. Well, sortof...

I got some trouble trying to open Wx. Even just "look Wx" tried to unpack the package, ending in the error message

gzip: stdout: broken pipe

CPAN barfing and refusing to open a shell.

I assume this implies that pipe between the external gzip and tar program are probably treated in text mode by mistake; maybe that my ports of these programs are broken. But replacing the UnxUtils programs with those from GnuWin32 offers no improvement whatsoever.

Strawberry Perl didn't have that problem. Let me see what's in its "o conf" settings... a space?? In ActivePerl's CPAN, both are empty. How can you do that...

o conf gzip ' '
o conf tar ' '

That works. I don't know what it means, I assume it now will try to use Perl modules instead of external programs, but at least, it no longer produces a broken pipe. That's what matters.

And now: no more broken pipe!

There was a problem with ExtUtils::XSpp, which refused to be installed as a dependency, it looks like it didn't even try (??), so I had to install that manually, and retry.

Argh, CPAN, I hate you! If in the CPAN shell, some step in the build process fails, and you retry, it'll happily assume that step worked and go to the next step, and then croak. Exiting CPAN and relaunching it is often enough to really make it start over, but sometimes you may have to manually delete the built files.

Anyway, installing Wx finally worked. So did installing Wx::Demo. Running the demos shows that it all worked.

Conclusion: yes, you can install Wx on ActivePerl and the MinGW C compiler, but it'll take a lot of time, and some kicking of the (CPAN) engine in strategic spots.

Oh, and yes, the combo control shows the exact same weird behavior as it did in Strawberry Perl, here's a screenshot to show what I mean). I'm curious to hear if it works better on other platforms than Windows it does. I'm trying to decide if it's a bug in Wx, or in Wx::Demo, and if it's tied to the platform.

p.s. Just earlier today I found there's a PPM repository on WxPerl's own site. Aargh! Well, it was an interesting experience anyway. And now I've got Wx version 0.93, which is a bit more recent than the version on that repository (0.89)... :)

Yesterday evening I felt like it was time for something new, and I decided to install Strawberry Perl and Padre. So I grabbed the MSI installer file from Strawberry Perl's website, and ran it. Soon enough it finished, and I felt simply lost. Was that it?

I glanced in the Start Menu, and I found a few links to docs, 2 to CPAN, and one to the help channel on IRC. I've never used IRC in my life, I don't even have IRC software, so that didn't feel welcoming. No welcome message, no introduction message "What now?", nothing. I'm an experienced Perl user, and as I felt lost, I can't imagine what kind of hell it must feel like to people who are new to Perl.

So I opened a command line window, and I typed "perl -v". Wow! At least that was something: apparently Strawberry Perl had put itself in my PATH, and in front of my default Perl install. (The reason for that is because my default perl is in the user environment variable PATH, and Strawberry put itself in the system environment variable PATH, and the latter comes in front of the former. It doesn't feel right to me, but at least, it's a annoyance caused by Windows.)

But I had, half and half, expected that Padre would have been there. It wasn't. So I went to the win32 Wiki (following the link in the Start Menu and initially ending up in the wrong place), and tried to find a "What now?" page. Still nothing. There's a (very incomplete) page on editors usable for Perl (I can name, off the top of my head, 3 free Windows text editors that aren't listed). I needed an editor, this is a fresh Windows install and there's no editor there yet apart from Notepad.

I decided to grab Padre anyway. So I dropped down to the command line window, and typed "CPAN", followed by "install Padre". That should go swiftly, shouldn't it? It didn't. I got all sorts of weird errors, most of them relating to "missing prerequisite", at least most due to a module that failed to install, due to a "missing file" error message. Say what??

So I decided to figure out which module failed to install, and installed, by hand, the first one that produced such an error: IPC::Run. Hmm, that installed without glitch. But "install Padre" still doesn't work. So, digging down, I decided to try "install Wx" first. Big mistake. After a very long compilation time, Wx appears to exist out of trillions of little C files, it still failed to install successfully. Again, it appears the reason is because of prerequisites that failed to install.

So I dug further down, installing module after module by hand: starting up with the huge ones like PPI, and ending up in really tiny ones, like Class::Accessor. Eventually, they all installed.

So what was the culprit? Nothing. They all installed. But I get the distinct impression CPAN throws away part of the build before it even finishes, and therefore, installing huge dependency trees fails. You have to split it up in smaller chunks and install them one at a time.

Seriously, I can't expect somebody new to perl and CPAN to get this far. From installing Strawberry Perl to finishing installing Padre through CPAN took over 2 hours.

So after everything installed fine, I sighed a sigh of relief, and typed "padre" at the command line. A spinning cursor for 1 second, and the command line prompt was back. That was it. No error, no error message. Nothing.

After a pause of more than an hour, I decided to tackle the problem, and started by the most likely culprit: Wx, which I had more or less forced to install. I dropped back to the CPAN command line and entered "test Wx". Again, some mysterious error message about a missing file (something like "can't copy file ... to ...: file does not exist). "look Wx", in order to unpack the distro, died with the same error. After trying a few times, it finally worked, as it seems CPAN had decided to start afresh. This reinforces the idea that CPAN is the culprit, apparently cleaning up before it even completes building. Eventually, after going through "perl Makefile.PL" and "dmake" manually, "dmake test"... succeeded? So, back in the CPAN shell, I installed it again. I dropped out of CPAN, hopefully typed "padre" and... still nothing.

So I tried one more module: Wx::Demo. (Seriously, guys, the docs of that demo module are seriously lacking. You run the demo through the script it installs: wxperl_demo.pl, but I really had to browse the repository to figure that out.) Anyway: the demos all worked. (One demo looks like shit: "WxComboCtrl", but that's a problem for another day.)

Back to "padre": still nothing. Okay, debugging time. "perl -x -S padre.bat". Huh, I get an editor window?? "padre"... It works??

It no longer fails. It all works.

Okay, the nightmare is over. It took me nearly 3 hours to get this far. This is not something you want to make everybody go through. And the fault is, most likely, in the module CPAN.

p.s. This post was written using Markdown using Showdown and posted after conversion to HTML.

When I downloaded the headers of the usenet groups I follow, this morning, I saw no new headers in comp.lang.perl.misc. None at all??

So I doublechecked. In the last 24 hours, there have been 3 (no, 4) automated FAQ postings, and 2 spam messages. That is all.

If I were a festival promotor (but I'm not) these are the bands I'd like to see, because they did impressive things (more than 1 song) in the last year:

• School of Seven Bells
• Friendly Fires
• Fever Ray
• Juana Molina

and for the dance tent:

• Pogo
• Deadmau5

However, it doesn't look very likely to see them all (or even,, half of them) in a single event.

(Note: a first draft of this post has been lying on my shelf for over a year. Now, I finally got around to finishing it.)

Now and then in Perl forums, the discussion comes up about how bad it is to use grep in boolean context. And now it's even been poured into a Perl::Critic rule, based on Damian Conway's PBP book, and the argument they bring up is always the same (see the POD for the module):

Using grep in boolean context is a common idiom for checking if any elements in a list match a condition. This works because boolean context is a subset of scalar context, and grep returns the number of matches in scalar context. A non-zero number of matches means a match.

But consider the case of a long array where the first element is a match. Boolean grep still checks all of the rest of the elements needlessly. Instead, a better solution is to use the any function from List::MoreUtils, which short-circuits after the first successful match to save time.

Now, why would you expect the item you're looking for to be the first one in the array? I see no reason for such an assumption, at all. On average, you still have to look through about half of the array items, if the item is even in the array. If it isn't, you still have to look through all items, anyway.

So, depending on the likelihood that an item is in the array, you might save between 0% and 50% of execution time by leaving the loop early. Personally, I don't find that overly impressive... As O(n) is still O(n).

The implied assumption is that the overhead of grep and any is ignorable, or at least, that it is the same for both, somebody that nobody actually bothered to verify. Well, I bothered. I decided to benchmark grep vs. any.

The kind of code that I benchmarked is the simple common problem of testing for the presence of a string in an array, just like IN in (some dialects of) SQL, and in_array of PHP. Here are the prerequisites and the functions I tested:

my @letters = 'A' .. 'Z';
my %letter; $letter{$_} = 1 foreach @letters;

# List::MoreUtils' any
any => sub { my $x = any { $_ eq $search } @letters },
# grep with expression
grepE => sub { my $x = grep $_ eq $search, @letters },
# grep with block
grepB => sub { my $x = grep { $_ eq $search } @letters },
# explicitly written code with foreach and last
foreach => sub { my $x=0; $_ eq $search and $x=1, last foreach @letters; },
# the expected overall winner: prefilled hash
hash => sub { my $x = exists $letter{$search} },
# hash on the fly, rebuilt on every test
temphash => sub { my %h; @h{@letters}=(); my $x = exists $h{$search} },

I searched for 'Z' (last item), 'M' (center item), 'C' (pretty up front), 'A' (first item), and 'banana' (not in the list). And here are the results (ActivePerl 5.8.8, Windows XP, 2.4GHz):

Search for Z
               Rate     Time   any temphash foreach grepB grepE hash
any         50172/s  19.93µs    --     -60%    -65%  -69%  -71% -98%
temphash   125555/s  7.965µs  150%       --    -12%  -22%  -27% -96%
foreach    142037/s   7.04µs  183%      13%      --  -11%  -18% -95%
grepB      160313/s  6.238µs  220%      28%     13%    --   -7% -94%
grepE      172645/s  5.792µs  244%      38%     22%    8%    -- -94%
hash      2828893/s 0.3535µs 5538%    2153%   1892% 1665% 1539%   --

Search for M
               Rate     Time   any temphash grepB grepE foreach hash
any         65938/s  15.17µs    --     -47%  -59%  -62%    -74% -98%
temphash   124203/s  8.051µs   88%       --  -22%  -29%    -51% -95%
grepB      160049/s  6.248µs  143%      29%    --   -8%    -36% -94%
grepE      174201/s   5.74µs  164%      40%    9%    --    -31% -94%
foreach    251362/s  3.978µs  281%     102%   57%   44%      -- -91%
hash      2733577/s 0.3658µs 4046%    2101% 1608% 1469%    988%   --

Search for C
               Rate     Time   any temphash grepB grepE foreach hash
any         87975/s  11.37µs    --     -29%  -45%  -50%    -85% -97%
temphash   124136/s  8.056µs   41%       --  -22%  -29%    -78% -96%
grepB      158854/s  6.295µs   81%      28%    --   -9%    -72% -95%
grepE      175068/s  5.712µs   99%      41%   10%    --    -69% -94%
foreach    571092/s  1.751µs  549%     360%  260%  226%      -- -80%
hash      2926393/s 0.3417µs 3226%    2257% 1742% 1572%    412%   --

Search for A
               Rate     Time   any temphash grepB grepE foreach hash
any         94486/s  10.58µs    --     -23%  -40%  -47%    -90% -96%
temphash   123096/s  8.124µs   30%       --  -22%  -30%    -87% -95%
grepB      158407/s  6.313µs   68%      29%    --  -10%    -84% -94%
grepE      176637/s  5.661µs   87%      43%   12%    --    -82% -93%
foreach    978762/s  1.022µs  936%     695%  518%  454%      -- -64%
hash      2687559/s 0.3721µs 2744%    2083% 1597% 1422%    175%   --

Search for banana
               Rate     Time   any temphash foreach grepB grepE hash
any         51867/s  19.28µs    --     -58%    -62%  -69%  -72% -98%
temphash   123717/s  8.083µs  139%       --     -9%  -25%  -34% -95%
foreach    136015/s  7.352µs  162%      10%      --  -18%  -28% -95%
grepB      165077/s  6.058µs  218%      33%     21%    --  -12% -94%
grepE      187691/s  5.328µs  262%      52%     38%   14%    -- -93%
hash      2670255/s 0.3745µs 5048%    2058%   1863% 1518% 1323%   --

Well, that is looking bad for any: it is the worst performer in all cases. In the average case ('M'), it is almost 3 times slower than grep (grep with expression is, as I expected, a bit better than grep with a block, as the latter has an overhead of entering/exiting a lexical block). Even in its best case, any is still almost twice as slow as grep. So much for saving. The manually written loop is, in the average case, about a third faster than grep. But, if the item isn't found, it is actually slower.

No surprise that, if you really want a high speed test, and you need to test against the same array often, it is best to prepare a hash and simply test if the item is in the hash.

What I found rather surprising, is that populating a hash and using it once ('temphash'), isn't such a bad performer.

For completeness' sake, here's the same benchmark on Strawberry Perl 5.10.

Search for Z
               Rate     Time   any temphash foreach grepE grepB hash
any         46995/s  21.28µs    --     -60%    -62%  -65%  -67% -98%
temphash   117471/s  8.513µs  150%       --     -6%  -13%  -19% -96%
foreach    125313/s   7.98µs  167%       7%      --   -7%  -13% -95%
grepE      135294/s  7.391µs  188%      15%      8%    --   -6% -95%
grepB      144589/s  6.916µs  208%      23%     15%    7%    -- -95%
hash      2740628/s 0.3649µs 5732%    2233%   2087% 1926% 1795%   --

Search for M
               Rate     Time   any temphash grepE grepB foreach hash
any         63959/s  15.64µs    --     -46%  -54%  -55%    -71% -97%
temphash   118040/s  8.472µs   85%       --  -15%  -18%    -46% -95%
grepE      138646/s  7.213µs  117%      17%    --   -3%    -36% -94%
grepB      143388/s  6.974µs  124%      21%    3%    --    -34% -94%
foreach    218040/s  4.586µs  241%      85%   57%   52%      -- -91%
hash      2516377/s 0.3974µs 3834%    2032% 1715% 1655%   1054%   --

Search for C
               Rate     Time   any temphash grepE grepB foreach hash
any         86710/s  11.53µs    --     -25%  -36%  -39%    -85% -96%
temphash   115756/s  8.639µs   33%       --  -15%  -19%    -79% -95%
grepE      135977/s  7.354µs   57%      17%    --   -5%    -76% -94%
grepB      142657/s   7.01µs   65%      23%    5%    --    -75% -94%
foreach    559639/s  1.787µs  545%     383%  312%  292%      -- -76%
hash      2315599/s 0.4319µs 2571%    1900% 1603% 1523%    314%   --

Search for A
               Rate     Time   any temphash grepE grepB foreach hash
any         85902/s  11.64µs    --     -26%  -36%  -41%    -90% -96%
temphash   115752/s  8.639µs   35%       --  -14%  -21%    -87% -95%
grepE      134479/s  7.436µs   57%      16%    --   -8%    -85% -94%
grepB      146193/s   6.84µs   70%      26%    9%    --    -84% -94%
foreach    902933/s  1.108µs  951%     680%  571%  518%      -- -62%
hash      2405894/s 0.4156µs 2701%    1978% 1689% 1546%    166%   --

Search for banana
               Rate     Time   any temphash foreach grepE grepB hash
any         51088/s  19.57µs    --     -56%    -61%  -69%  -71% -98%
temphash   116528/s  8.582µs  128%       --    -12%  -29%  -33% -96%
foreach    132626/s   7.54µs  160%      14%      --  -19%  -24% -95%
grepE      164236/s  6.089µs  221%      41%     24%    --   -6% -94%
grepB      175040/s  5.713µs  243%      50%     32%    7%    -- -93%
hash      2655952/s 0.3765µs 5099%    2179%   1903% 1517% 1417%   --

As an aside: I've got the distinct impression that overall, Strawberry Perl 5.10 is slower than ActivePerl 5.10, by about 5-10%. The biggest surprise to me, however, is that grep BLOCK is no longer faster than grep EXPRESSION. So weird.

Now, the thing you would test for must not always be that simple. If it is very slow or otherwise expensive, you might still, and rightfully so, want to leave the testing loop early. But, although elegant as a solution, you should not blindly assume that any is the best solution for every such problem... Though I wish that grep would be made smarter, and that it either knows to leave the test loop early, or that you could manually do so, for example, with last.

p.s. The code to convert Rate to Time was kindly supplied by GrandFather on Perlmonks.

p.p.s. I'm sorry about the formatting problem, that "µ" should look like "µ", but even though it is a "µ" character in the text I posted, the journaling system shows it as "µ". It's a bug, I'm sure.

Running the CPAN shell on ActivePerl 5.8.x on Windows XP, I can see in Process Explorer that the bare CPAN shell uses a whopping 195MB of RAM (size of working set: 120MB). I mean... wauw!!

That does not even include the programs, including perl, that are invoked from make, that actually test or install stuff. No, just the bare shell.

WTF is going on? Does the CPAN shell perhaps try to keep the whole module database in RAM? And if so: why? It's not as if looking up a distro by module name is so system critical, that it requires a sub-millisecond response time. So a slightly slower system, that greps through the data on file, would work just as well.

Fact is: memory consumption is 3 times lower before the metadata is first loaded. And it doesn't ever go down again... (As if it could.)

CPAN is now using several times more RAM than the average computer had, when CPAN.pm first came out. I doubt it used that much RAM, in those days.

p.s. For some bizarre reason, Strawberry Perl 5.10.x uses "only" 120MB. I have no idea why.

The verbose and arcane syntax of POD always distracts me from what I want to write, whenever I write POD directly. I prefer markdown, which doesn't get in the way.

And with going from markdown through html2pod, I get a reasonable headstart. It works pretty well.

The one thing I commonly need in POD that markdown is lacking, is itemized bulleted lists, that in HTML you'd write with DL/DT/DD lists, and in POD you write as

=over

=item one

This is item one.

=item two

This is item two.

=back

(Gah! POD is ghastly!)

In a reference manual, I need them a lot.

But, starting from a plain bulleted list, you can get the basic POD list syntax, and by just tweaking the generated POD a little (replacing the bullets with the item text), I get where I want.

In the last few weeks I've taken up the habit of writing my posts and replies on use.perl.org as plain text, using markdown markup. I then run the markdown script from the CPAN module redirecting the output to an HTML file, preview in a browser, and finally copy/paste the HTML into the use.perl.org textarea, and submit.

You may know markdown as the markup system utilized by websites such as Reddit and StackOverflow.

The difference in effort in marking up prose text in markdown rather than in any other system use.perl.org allows (such as the allowed subset of plain HTML) would appear to be slight, yet it is quite relevant to me. Even my rather clumsy workflow is still preferable to me, than using the site's standard HTML markup.

It's a really nice thought that markdown originated in Perl, although it has been ported to Javascript, complete with a live preview, which is still primitive compared to the system used on StackOverflow.

And maybe it's an idea to modernize the comment system in use.perl.org a little, as more and more people apparently utter their dislike for the current system.

But likely they think there's still a lot more wrong with it, than just the markup system. :)

(title inspired by a blog post)

CPAN.pm used to ask a bunch of questions the first time it is run. One of the questions is what CPAN mirrors to use.

Now it doesn't any more: it comes preconfigured. But that comes at a price: a lot of distros simply assume use of http://cpan.perl.org/ or of http://www.cpan.org/ (while several perl ports use their own private CPAN mirror by default, such as Strawberry Perl for Windows, and, I thought, ActiveState's ActivePerl.

Is the idea of using CPAN mirrors simply outdated? Or, should the CPAN client be smarter, and figure out for itself which mirrors to use? The latter feels like overkill to me. It presumes inclusion of a geolocator module and database, like Geo::IP (the free version of that database is far more than sufficient for this purpose, so the license price is no objection). But having that module and database on every Perl installation, just to get a list of mirrors once, or maybe a few times, in the lifetime of a perl installation, really is far too much.

I can remember how http://www.perl.com/CPAN, thanks to Tom Christiansen IIRC, used to have a built in redirector, where it figured out where in the world you are, and hence, which (single) mirror to use. But if that one mirror was offline, you were out of luck. It didn't check the status of the mirror, it just redirected you there.

If we still wish to use mirrors, why not drag CPAN into the age of webservices? (actually we're already late for that, as the age of webservices seems to have passed, already... :)) Set up a main page on a site, for example on www.cpan.org, where CPAN.pm can simply ask "Can you suggest me what mirrors to use?" (pun intended). Then only the central site needs to have this geolocation database, to check what part of the world the request comes from, and compose a list of preferable mirrors. The output could be as simple as a text/plain page with one URL of a mirror per line, returning maybe 5 or 10 URLs in total. Easy to generate, and dead easy to parse.

(Note: the order of mirrors that are close to each other in level of preference could be randomly shuffled for each request, to avoid that all users in one area all hammer the same mirror.)

CPAN.pm can still be made a bit smarter, and for example, use ping to test responsiveness of the mirror, or, simpler still, time the fetch time of a page from the currently chosen mirror, and check if it's fast enough (depending on your internet connection; it should keep track of responsiveness of the mirrors, so it can compare them); and switch the order of mirrors, if that may, likely, seriously improve matters.