use Perl

http://perlmonks.org/?node=Corion asked about measuring memory allocation in IRC but I was reminded of something I've poked at recently which is measuring how much memory is copy-on-write shared between forked mod_perl processes. Thus far, when on Linux the only answer I know of is to use the exmap kernel module. The main page is http://www.berthels.co.uk/exmap/ but Dave Olszewski wrote some bug fixes for it at http://github.com/cxreg/exmap. exmap uses a kernel module to add a new /proc/exmap file. To read physical page stats, write the PID to this file, then read the results. The exmap distribution comes with a C++ and perl GTK program to interpret the kernel data. Below is what I know of the format for the kernel data.

To use:

$ echo $pid > /proc/exmap
$ cat /proc/exmap
VMA 400000 87
1 0 1c6e7
1 0 1c6e8
1 0 1d328
...

$ grep 400000 /proc/$pid/maps
00400000-00457000 r-xp 00000000 08:01 722755 /usr/bin/screen

The sections provided by /proc/exmap correspond to each of the chunks in /proc/$pid/maps. Each line then details a page, whether it is swapped, and whether it is writable.

(
    VMA $address $page_count
    ( $resident $writable $page_id )+
)+

Anyway, just thought I'd share. If you know a better trick, I'd love to hear of it. When I next get around to looking at the mod_perl in question I'll likely actually try to use this but for now this is just a tool I think I plan to use but haven't done serious work with yet.

I want to migrate from use.perl to someplace nicer. Where should I go? Is there a blog migration protocol so RSS gets some kind of automatic redirect? People seem to find my blog through use.perl - how do I get into the appropriate aggregators?

I'm tired of writing a long stanza to declare "yes, I'm going to write an ordinary perl program."

use strict;
use warnings;
use feature ':5.10';

I'm going to go use chromatic's Modern::Perl and see how it feels.

use Modern::Perl;

I saw the screen shots of Padre 0.46 in http://ahmadzawawi.blogspot.com/2009/09/whats-new-in-padre-046.html and thought I'd try it myself. I've got a non-threaded perl but potentially http://search.cpan.org/dist/forks will let me run without it. I've just spent the last hour trying to get http://search.cpan.org/Wx and its dependencies installed. Mostly its just oddness like a default in wxWidgets to demand OpenGL. Now I'm just hanging out in Wx's build:

/opt/perl-5.10.0/bin/perl /opt/perl-5.10.0/lib/5.10.0/ExtUtils/xsubpp  -noprototypes -nolinenumbers  -typemap /opt/perl-5.10.0/lib/5.10.0/ExtUtils/typemap -typemap ../../typemap -typemap typemap  RichText.xs > RichText.xsc && mv RichText.xsc RichText.c
Cannot open 'xspp -t typemap.xsp -t ../../typemap.xsp XS/RichTextCtrl.xsp |': No such file or directory in RichText.xs, line 65
make[2]: *** [RichText.c] Error 1
make[2]: Leaving directory `/home/josh/.cpan/build/Wx-0.92-psm_3c/ext/richtext'
make[1]: *** [subdirs] Error 2
make[1]: Leaving directory `/home/josh/.cpan/build/Wx-0.92-psm_3c/ext'
make: *** [subdirs] Error 2

Anyway, kind of annoying. Wx is prettier than Tk but Tk is easier to install.

Using ruby-1.9, I just got the helpful exception "bin/pp-pretty:60: undefined (?...) sequence: / (RegexpError)". I don't know what it's complaining about since it's not telling me where inside the 70 line regexp it started failing. :-(

When I write Perl, I get useful context like:

Reference to nonexistent named group in regex; marked by <-- HERE in m/
  (?<object>
    \#<
    (?&class_nm)
    : [ \t\n]*
    (?:
      0x[0-9a-f]+ [ \t\n]* (?=\@) (?&object_data <-- HERE )
      | [^>]*
    )?
    >
  )/

At work, I've got a problem on the back burner which is kind of interesting. We've got some mod_perl processes with big data sets. The processes fork and then serve requests. I've heard from Operations that they're not using Linux's Copy-on-Write feature to the extent desired so I'm trying to understand just what's being shared and not shared.

To that end, I wanted to map out where perl put its data. I made a picture (http://diotalevi.isa-geek.net/~josh/090909/memory-0.png, a strip, showing the visible linear memory layout from 0x3042e0 to 0x8b2990. The left edge shows where arenas are. The really clustered lines to the middle show the pointers from the arenas to the SV heads. The really splayed lines from the middle to the right show the SvANY() pointer from the SV heads to the SV bodies.

I kind of now suspect that maybe the CoW unshared pages containing SV heads because of reference counting are maybe compact or sparse. They sure seem to be highly clustered so maybe it's a-ok to go get a bunch of values between two forked processes and not worry about reference counts. Sure, the SV head pages are going to be unshared but maybe those pages are just full of other SV heads and it's not a big deal. If SV heads weren't clustered then reference count changes could have affected lots of other pages.

Anyway, there's a nice little set of pics at http://diotalevi.isa-geek.net/~josh/090909/. I started truncating precision by powers of two to get things to visually chunk up more. So when you look at memory-0.png, there's no chunking but when you look at memory-4.png, the bottom 4 bits were zeroed out.

There's a github repo of this at http://github.com/jbenjore/Internals-GraphArenas/tree/master for the interested.

Today at work the toy problem of extracting some data from $more_json in a line like "$json|$more_json". The obvious thing seemed to use Text::Balanced to skip the first $json, snip off the '|' and then use a real JSON parser on the next json blob.

So uh... there's no Text::Balanced in Ruby. Not that I can find. We solved our work problem by just declaring that we'd only use delimiters which didn't occur in our input so we wouldn't need to find a Ruby Text::Balanced. I'm thinking maybe the easiest thing to do if we do turn out to need this in Ruby will be to write a gem that embeds a Perl interpreter. Then Ruby's Text::Balanced is just a short adapter over the Perl version.

It's just a thought though.

I've got a preforking Apache server at work that's not sharing enough memory among the many children. This snippet helped identify post-fork module loading. I put an @INC hook at the end of my httpd.conf and it writes a warning to my log whenever the forked child loads something.

# httpd.conf
...
<Perl>
  # This fragment goes at the end, when I think I'm ready to let Apache start forking
  $main::PARENT_PID = $$;
  unshift @INC, sub {
    print STDERR "LATE LOAD: $_[1]\n" if $$ != $main::PARENT_PID;
    return undef;
  };
</Perl>

I just found 233 modules I was loading post-fork. Oops! It's now trivial to make sure my httpd.conf has the proper lines to load the stuff I was forgetting to:

# httpd.conf
LoadModule XML::LibXML
LoadModule LWP::UserAgent
...

Today, I'm using Ruby for a project at work and Oh! My! God! Every week I get deeper with Ruby I become more impressed with how much I like writing ordinary things in it and how much I loath having to solve hard problems in it. This language, or the implementation seems to go miles out of its way to make it nigh impossible to debug.

Today's frustration:

• Without changing anything I'm aware of my trepan gem (http://github.com/jbenjore/trepan/tree/master) has started throwing Bus error
• I've found that the regexp engine is matching "something" wrong
• To debug a regex in 1.8.6, I recompiled ruby with CFLAGS=-DDEBUG_REGEX
• -DDEBUG_REGEX produces only the inscrutable output "regex loop(5): matching 0x01" which doesn't seem to map to anything like characters or offsets or anything. I can't use it
• Ruby 1.9 uses Oniguruma but it also fails to match the same regexp
• Installing the latest Oniguruma from http://www.geocities.jp/kosako3/oniguruma/ yields libonig, not something I can use in ruby. Oops
• Building 2.x per the page tries to patch my Ruby 1.8 source code but fails
• There's apparently a /third/ version of Oniguruma at http://rubyforge.org/projects/oniguruma. Maybe's it's just old? Maybe it's the version that works. Dunno

  I just want a language that doesn't make me spend immense amounts of energy! Is that so much to ask? I throw tons of energy at perl but I get cool things for it. I'm throwing tons at ruby now and not even getting things I have for FREE in perl.

  FWIW, after spending several hours struggling with Ruby, I threw in the towel. Read a book and went to bed early, exhausted.

Hi there $world and PAUSE administrators,

Commit bits to my CPAN modules are given freely. If a PAUSE administrator or some other core hackers can vouch for you, I don't mind if upload and indexing permission bits are handed out and I only find out about it later.

This is prompted by an IRC conversation. I'd gone away for the weekend and left people hanging about whether some permissions were available. The answer in general is "yes, and be nice with them."

-
Josh