rhesa's Friends' Journals

The mod_perl debugger you've always wanted

jjore — 2010-07-22T17:53:07+00:00

"The mod_perl debugger you've always wanted" is on my new blog at: http://bit.ly/agzWSU

Unicode in Perl, FTW (Я очень ра

jjore — 2010-07-07T06:01:11+00:00

Hi,
This blog post is actually at http://blogs.perl.org/users/josh_ben_jore/2010/07/unicode-in-perl-ftw.html.

Under the covers of perldoc

jjore — 2010-07-07T05:44:28+00:00

Hi, I posted to my new blog at blogs.perl.org:

http://blogs.perl.org/users/josh_ben_jore/2010/07/under-the-covers-of-perldoc.ht ml

JVM's -XX:-DontCompileHugeMethods & dynamic languages

jjore — 2010-05-26T22:02:34+00:00

In http://nerds-central.blogspot.com/2009/09/tuning-jvm-for-unusual-uses-have-some. html which I just picked out of reddit.com's front page I notice there's an ordinary 8K bytecode size limit for JVM heuristics.

Allegedly adding -XX:-DontCompileHugeMethods to the command line eliminates this limit.

In "current" versions of JVM which don't have the very new invokedynamic bytecode operation and languages implemented on top of the JVM like JRuby, I assume their implementation does all the dynamism as ordinary code. I assume then this means the bytecode sizes are artificially inflated and then possibly just missing out on this nice optimization.

Allegedly, this is one of those heuristics that moved Java from the "slow" category to "faster than C++".

Last year I found that MRI Ruby-1.9 and JRuby were equivalently fast as Perl 5 but now I wonder if the JRuby implementation I was using was missing out and possibly could go even *faster*.

FWIW, MRI Ruby-1.8 is 1/10 the speed of Perl 5.

Stop the debugger when a test fails

jjore — 2010-05-09T17:43:40+00:00

I was debugging some test failures earlier this morning but they were in the middle of a long test script. I thought it'd be nice if I could step through the test with the debugger and just stop whenever a test failed.

Here's the result. A test failure immediately followed by being in the debugger:

not ok 236 14: return $result; DB<1>

and then I could ask for the stack trace which is nice:

DB<1> T . = main::ok(undef) called from file `t/02_methods.t' line 796 . = main::check_tar_extract(ref(Archive::Tar), ref(ARRAY)) called from file `t/02_methods.t' line 507

I wrote the below snippet to automatically stop the debugger whenever something like ok() failed. Currently you can just drop this right after your "use Test::More" statement. I think possibly this kind of idea should be packaged in a no-muss CPAN module or perhaps directly right into Test::More.

# Wrap Test::More functions with debugger breakpoints BEGIN { use B; # Enumerate over all symbols in the main:: symbol table. SYMBOL: for my $symbol ( sort keys %main:: ) { # Choose only things that are functions. { no strict 'refs'; next SYMBOL if ! defined &$symbol; } # Get a reference to the function. my $code = do { no strict 'refs'; \&$symbol; }; # Get a B::CV object so I can get at meta-data about the # function. my $cv = B::svref_2object( $code ); # Get the path to the compilation file. Will often be a path like # '/usr/share/perl/5.10/Test/More.pm'. # # To visually inspect other meta-data possibly available about this function: # # use Devel::Peek; # Dump( $code ); # my $src_file = $cv->FILE; # Select only functions coming from files named like # .../Test/More.pm. next SYMBOL if $src_file !~ m{/Test/More\.pm$}; # Propagate the old function's prototype to the new function. my $prototype = prototype $code; my $new_prototype = defined $prototype ? "($prototype)" : ''; # Generate the new function and replace the old function. The new function # has access to the original via the closed-over $old variable. my $src = qq{ no warnings 'redefine'; sub $symbol $new_prototype { # Call the original function and get the test pass/fail. my \$result = \$code->( \@_ ); # Debugger single-stepping mode if the test failed. if ( ! \$result ) { \$DB::single = 1; } # Return the test/failure. return \$result; } # Compilation of the new function succeeded? 1; }; eval $src or die $@; } }

Bob Jacobsen interview on FLOSS Weekly

merlyn — 2010-04-23T03:55:11+00:00

Last week, I interviewed Bob Jacobsen for FLOSS Weekly. Bob used Perl's Artistic 1.0 license on some Java code to manage model trains. The code was later patented by an Oregon-based company(!) and then Bob got sued(!!) for Bob distributing the other company's patented code(!!!). The good part of the story is that this is the first test at the US Federal Appeals Court level for an open source license to be enforceable even if no money exchanges hands, and... we won!

Bob spent a lot of time and money on the case though. Listen to the podcast and contribute to his legal defense if you care about open source.

Devel::Symdump + Compress::Zlib crash

ChrisDolan — 2010-02-15T02:59:26+00:00

Lazyweb,

The following is a known bug that crashes Perl 5.10.0 (solved for 5.10.1 via Perl RT#52740)

perl -MCompress::Zlib -MDevel::Symdump -e'Devel::Symdump->rnew'

Does anyone know of a workaround? I want my software to work on 5.10.0 because that's what Apple ships with Snow Leopard. I don't want to use the 5.8.9 that ships with SL because I want 64-bit.

Attaching to a process and calling a kernel function

jjore — 2010-01-09T01:30:33+00:00

I figured out last night that I could use allocate memory within another process for the purposes of using more complicated APIs. In the below example, I attached to one of my shells and queried the groups out of it.

In this snippet, the target shell tells me its process id is 40728.

Last login: Fri Jan 8 16:51:00 on ttys000 w3m211a:~ jjore$ echo $$ 40728

I'm calling the (int)getgroups( int, int* ) kernel function. I start by getting the size of an integer because I'm going to allocate some multiple of this.

w3m211a:~ jjore$ gdb -p 40728 (gdb) print sizeof(int) $1 = 4

Call (int)getgroups(int,int*) once to get the number of groups I'm a member of.

(gdb) print (int)getgroups(0,0) $2 = 16

Allocate 64 bytes of memory to hold my groups[16] array.

(gdb) print (void*)malloc(64) $3 = (void *) 0x10010fca0

Call (int)getgroups(int,int*) again but this time the kernel will write to my memory with the group ids.

(gdb) print (int)getgroups($2,$3) $4 = 16

Print the 16 decimal numbers out of memory

(gdb) x/16d $3 0x10010fca0: 2042662593 401 1612991420 1178352327 0x10010fcb0: 102 800006232 204 100 0x10010fcc0: 98 81 80 79 0x10010fcd0: 62 12 155049148 2133303823

Clean up

(gdb) print (void)free($3) $5 = void

Unicode URLs, wtf?

jjore — 2010-01-07T05:43:29+00:00

Hey internet, ⠸⠙⠱ ⠝⠉⠁⠈ ⠅⠝⠁⠕⠕⠉⠃ ⠝⠆⠏⠍⠞?

A year or more ago I was fixing work's web site to handle Unicode as entered by users into fields. We don't use CGI.pm because....? Well ok, we just don't. It also doesn't handle Unicode properly either. Or at least almost no version. Huh.

If a user types "Coatıcook" you'll probably get the dotless "i" character as either %C4%B1 or %u131 but CGI.pm as supplied by perl almost most of the time won't do something reasonable.

not ok 5.11.3 CGI-3.48
not ok 5.10.1 CGI-3.43
ok 5.10.0 CGI-3.29
not ok 5.8.9 CGI-3.42
not ok 5.6.2 CGI-2.752

Wut?

for v in 5.11.3 5.10.1 5.10.0 5.8.9 5.6.2;do /opt/perl-$v-64-thr-dbg/bin/perl\ -le ' use CGI; my $input = "a=%u2021"; my $expect = "\x{2021}"; my $got = CGI->new( $input )->param( "a" ); print $expect eq $got ? "ok $] $CGI::VERSION" : "not ok $] $CGI::VERSION" '; done

Decoding multiple encoded utf-8 in perl or ruby

jjore — 2010-01-07T03:57:25+00:00

I'd recently encountered some data that had been re-encoded five times. Ugh. The key was to guess that if a character 0xC0-0xFF is followed by 0x80-xBF, it's likely that the bytes are actually utf-8. What follows is a function which guessed a reasonable way to deal with the data and turn it into the right utf-8.

use Test::More tests => 1; use Encode (); my $bad = "\xc3\x83\xc2\x83\xc3\x82\xc2\x83\xc3\x83\xc2\x82\xc3\x82\xc2\x83\xc3\x83\xc2\x8 3\xc3\x82\xc2\x82\xc3\x83\xc2\x82\xc3\x82\xc2\xa9"; my $good = "\xe9"; is( multiple_decode( $bad ), $good ); sub multiple_decode { my ( $str ) = @_; Encode::_utf8_on( $str ); while ( $str =~ /[\xc0-\xff][\x80-\xbf]/ ) { utf8::downgrade( $str ); Encode::_utf8_on( $str ); } return $str; }

I tried doing this in Ruby because I actually needed this for an EventMachine (http://rubyeventmachine.com/) server but never quite got it working. Iconv seemed to want to be strict about rejecting the originally ostensibly invalid input.

# Iconv::IllegalSequence: "\303\203\302\203\303\202\302\203\303\203\302\202\303\202\302\203"... require 'test/unit' require 'iconv' class MDecode < Test::Unit::TestCase def test_multiple_decode conv = Iconv.new( 'UTF-8', 'ASCII' ) bad = "\xc3\x83\xc2\x83\xc3\x82\xc2\x83\xc3\x83\xc2\x82\xc3\x82\xc2\x83\xc3\x83\xc2\x8 3\xc3\x82\xc2\x82\xc3\x83\xc2\x82\xc3\x82\xc2\xa9" good = "\xe9" assert_equal( good, conv.multiple_decode( bad ) ) end end class Iconv def multiple_decode( str ) while str =~ /[\xc0-\xff][\x80-\xbf]/ str = iconv( str ) end return str end end

Building a lot of perl today...

jjore — 2009-12-29T21:49:08+00:00

#!/bin/bash set -x set -e perl="$HOME/src/perl" function build-it () { bversion=$1 shift 1 dir=/opt/$bversion tar=/opt/$bversion.tar.bz2 if [[ ! -e $tar ]]; then echo "Clobbering $dir" rm -rf $dir mkdir $dir cd $HOME/src/perl echo "Configure "$(date) > $dir/stamp.log ./Configure -des -Dcc='ccache gcc' -Dprefix=$dir $* 2>&1 | tee $dir/config.log echo "make "$(date) >> $dir/stamp.log make 2>&1 | tee $dir/make.log echo "make test "$(date) >> $dir/stamp.log make test 2>&1 | tee $dir/test.log echo "make install "$(date) >> $dir/stamp.log make install 2>&1 | tee $dir/install.log echo "End "$(date) >> $dir/stamp.log chown -R jbenjore $dir/ cd / tar cjf $tar $dir cd $HOME/src/perl git clean -xdf git reset --hard fi } git clean -xdf git reset --hard for tag in\ v5.11.3\ perl-5.10.1 perl-5.10.1\ perl-5.8.9 perl-5.8.8 perl-5.8.7 perl-5.8.6 perl-5.8.5 perl-5.8.4 perl-5.8.3 perl-5.8.2 perl-5.8.2 perl-5.8.1 perl-5.8.0 do git checkout $tag build-it $tag-64-dbg -DDEBUGGING -Duse64bitint build-it $tag-thr-dbg -DDEBUGGING -Dusethreads build-it $tag-64-thr-dbg -DDEBUGGING -Duse64bitint -Dusethreads build-it $tag-dbg -DDEBUGGING build-it $tag -DDEBUGGING=-g done for tag in\ perl-5.6.2 perl-5.005_04 do git checkout $tag build-it $tag-64-dbg -DDEBUGGING -Duse64bitint build-it $tag-thr-dbg -DDEBUGGING -Dusethreads build-it $tag-64-thr-dbg -DDEBUGGING -Duse64bitint -Dusethreads build-it $tag-dbg -DDEBUGGING build-it $tag -g done

Bugging myself

jjore — 2009-12-06T23:11:05+00:00

I've been seriously neglecting my CPAN code. It's been pretty overwhelming having too many modules up there and no time to really get into anything. I figured it was worth at least an hour and a half's time to read the CPAN tester reports.

I just filed 34 bugs based purely on CPAN testers output. Bleagh. :-(

Acme-Anything

FIXED: #52481: Makefile.PL isn't valid perl 5.005 https://rt.cpan.org/Ticket/Display.html?id=52481

Acme-Nothing

FIXED: #52482: Makefile.PL doesn't declare 5.008 as mininum perl version https://rt.cpan.org/Ticket/Display.html?id=52482

Alien-Judy

FIXED: #52483: Build fails when prefix isn't absolute https://rt.cpan.org/Ticket/Display.html?id=52483
#52484: Mysterious hanging build https://rt.cpan.org/Ticket/Display.html?id=52484
FIXED: #52485: Requires sh implicitly https://rt.cpan.org/Ticket/Display.html?id=52485

App-Perldoc-Search

#52486: Basic search failing for some users https://rt.cpan.org/Ticket/Display.html?id=52486

B-Deobfuscate

#52487: Redundant nextstate https://rt.cpan.org/Ticket/Display.html?id=52487
#52488: Illegal seek https://rt.cpan.org/Ticket/Display.html?id=52488
#52489: B::Concise: Can't use an undefined value as a SCALAR reference https://rt.cpan.org/Ticket/Display.html?id=52489

B-Generate

#52490: XS_B__SVOP_sv: invalid lvalue in assignment https://rt.cpan.org/Ticket/Display.html?id=52490

B-Lint

#52492: Makefile.PL doesn't declare minimum version https://rt.cpan.org/Ticket/Display.html?id=52492
#52493: Can't detect $& usage https://rt.cpan.org/Ticket/Display.html?id=52493
#52495: Too-old version of B? https://rt.cpan.org/Ticket/Display.html?id=52495
#52496: Tests finding flaws in test harness https://rt.cpan.org/Ticket/Display.html?id=52496
#52498: Can't call method "name" on an undefined value https://rt.cpan.org/Ticket/Display.html?id=52498

B-Utils

FIXED: #52491: Depends on cc https://rt.cpan.org/Ticket/Display.html?id=52491

Devel-Dt

FIXED: #52499: Makefile.PL doesn't declare minimum version https://rt.cpan.org/Ticket/Display.html?id=52499

Devel-OptreeDiff

#52500: Tests too static https://rt.cpan.org/Ticket/Display.html?id=52500

Devel-StackBlech

#52501: Mininum version is 5.9.x, not declared https://rt.cpan.org/Ticket/Display.html?id=52501
#52502: structure has no member named `next_op' https://rt.cpan.org/Ticket/Display.html?id=52502

ESPPlus-Storage

#52504: Makefile.PL doesn't declare minimum version https://rt.cpan.org/Ticket/Display.html?id=52504
#52505: Can't use an undefined value as filehandle reference https://rt.cpan.org/Ticket/Display.html?id=52505

Enbugger

#52503: Can't get terminal size https://rt.cpan.org/Ticket/Display.html?id=52503

Geo-TigerLine-Abbreviations

#52506: Missing Cache::Cache dependency https://rt.cpan.org/Ticket/Display.html?id=52506

Judy

#52507: error: Judy.h: No such file or directory https://rt.cpan.org/Ticket/Display.html?id=52507
#52508: undefined symbol: JudyLPrevEmpty https://rt.cpan.org/Ticket/Display.html?id=52508
#52509: Symbol not found: _Judy1ByCount https://rt.cpan.org/Ticket/Display.html?id=52509
#52510: Fetch Word_t test fails https://rt.cpan.org/Ticket/Display.html?id=52510

Regexp-NamedCaptures

#52513: Makefile.PL fails to prevent perl-5.9+ from using this https://rt.cpan.org/Ticket/Display.html?id=52513

Runops-Trace

#52514: t/03even fails "sometimes" https://rt.cpan.org/Ticket/Display.html?id=52514

Term-HiliteDiff

#52515: Tests completely broken https://rt.cpan.org/Ticket/Display.html?id=52515

UNIVERSAL-ref

#52516: Makefile.PL doesn't declare minimum version https://rt.cpan.org/Ticket/Display.html?id=52516

overload-eval

#52511: Test fails https://rt.cpan.org/Ticket/Display.html?id=52511

perl-lint-mode

#52512: Signature is invalid https://rt.cpan.org/Ticket/Display.html?id=52512

Measuring copy-on-write on Linux

jjore — 2009-10-06T18:06:05+00:00

http://perlmonks.org/?node=Corion asked about measuring memory allocation in IRC but I was reminded of something I've poked at recently which is measuring how much memory is copy-on-write shared between forked mod_perl processes. Thus far, when on Linux the only answer I know of is to use the exmap kernel module. The main page is http://www.berthels.co.uk/exmap/ but Dave Olszewski wrote some bug fixes for it at http://github.com/cxreg/exmap. exmap uses a kernel module to add a new /proc/exmap file. To read physical page stats, write the PID to this file, then read the results. The exmap distribution comes with a C++ and perl GTK program to interpret the kernel data. Below is what I know of the format for the kernel data.

To use:

$ echo $pid > /proc/exmap $ cat /proc/exmap VMA 400000 87 1 0 1c6e7 1 0 1c6e8 1 0 1d328 ...

$ grep 400000 /proc/$pid/maps 00400000-00457000 r-xp 00000000 08:01 722755 /usr/bin/screen

The sections provided by /proc/exmap correspond to each of the chunks in /proc/$pid/maps. Each line then details a page, whether it is swapped, and whether it is writable.

( VMA $address $page_count ( $resident $writable $page_id )+ )+

Anyway, just thought I'd share. If you know a better trick, I'd love to hear of it. When I next get around to looking at the mod_perl in question I'll likely actually try to use this but for now this is just a tool I think I plan to use but haven't done serious work with yet.

Migrate from use.perl to ...?

jjore — 2009-10-02T16:40:45+00:00

I want to migrate from use.perl to someplace nicer. Where should I go? Is there a blog migration protocol so RSS gets some kind of automatic redirect? People seem to find my blog through use.perl - how do I get into the appropriate aggregators?

use Modern::Perl?

jjore — 2009-09-30T17:50:53+00:00

I'm tired of writing a long stanza to declare "yes, I'm going to write an ordinary perl program."

use strict; use warnings; use feature ':5.10';

I'm going to go use chromatic's Modern::Perl and see how it feels.

use Modern::Perl;

Grumble, mutter, Padre's dependencies, Grr, Mutter

jjore — 2009-09-18T01:33:49+00:00

I saw the screen shots of Padre 0.46 in http://ahmadzawawi.blogspot.com/2009/09/whats-new-in-padre-046.html and thought I'd try it myself. I've got a non-threaded perl but potentially http://search.cpan.org/dist/forks will let me run without it. I've just spent the last hour trying to get http://search.cpan.org/Wx and its dependencies installed. Mostly its just oddness like a default in wxWidgets to demand OpenGL. Now I'm just hanging out in Wx's build:

/opt/perl-5.10.0/bin/perl /opt/perl-5.10.0/lib/5.10.0/ExtUtils/xsubpp -noprototypes -nolinenumbers -typemap /opt/perl-5.10.0/lib/5.10.0/ExtUtils/typemap -typemap ../../typemap -typemap typemap RichText.xs > RichText.xsc && mv RichText.xsc RichText.c Cannot open 'xspp -t typemap.xsp -t ../../typemap.xsp XS/RichTextCtrl.xsp |': No such file or directory in RichText.xs, line 65 make[2]: *** [RichText.c] Error 1 make[2]: Leaving directory `/home/josh/.cpan/build/Wx-0.92-psm_3c/ext/richtext' make[1]: *** [subdirs] Error 2 make[1]: Leaving directory `/home/josh/.cpan/build/Wx-0.92-psm_3c/ext' make: *** [subdirs] Error 2

Anyway, kind of annoying. Wx is prettier than Tk but Tk is easier to install.

Missing Perl's useful regexp errors

jjore — 2009-09-16T18:10:05+00:00

Using ruby-1.9, I just got the helpful exception "bin/pp-pretty:60: undefined (?...) sequence: / (RegexpError)". I don't know what it's complaining about since it's not telling me where inside the 70 line regexp it started failing. :-(

When I write Perl, I get useful context like:

Reference to nonexistent named group in regex; marked by <-- HERE in m/ (?<object> \#< (?&class_nm) : [ \t\n]* (?: 0x[0-9a-f]+ [ \t\n]* (?=\@) (?&object_data <-- HERE ) | [^>]* )? > )/

Starting to chart a map of where perl locates data

jjore — 2009-09-09T23:32:46+00:00

At work, I've got a problem on the back burner which is kind of interesting. We've got some mod_perl processes with big data sets. The processes fork and then serve requests. I've heard from Operations that they're not using Linux's Copy-on-Write feature to the extent desired so I'm trying to understand just what's being shared and not shared.

To that end, I wanted to map out where perl put its data. I made a picture (http://diotalevi.isa-geek.net/~josh/090909/memory-0.png, a strip, showing the visible linear memory layout from 0x3042e0 to 0x8b2990. The left edge shows where arenas are. The really clustered lines to the middle show the pointers from the arenas to the SV heads. The really splayed lines from the middle to the right show the SvANY() pointer from the SV heads to the SV bodies.

I kind of now suspect that maybe the CoW unshared pages containing SV heads because of reference counting are maybe compact or sparse. They sure seem to be highly clustered so maybe it's a-ok to go get a bunch of values between two forked processes and not worry about reference counts. Sure, the SV head pages are going to be unshared but maybe those pages are just full of other SV heads and it's not a big deal. If SV heads weren't clustered then reference count changes could have affected lots of other pages.

Anyway, there's a nice little set of pics at http://diotalevi.isa-geek.net/~josh/090909/. I started truncating precision by powers of two to get things to visually chunk up more. So when you look at memory-0.png, there's no chunking but when you look at memory-4.png, the bottom 4 bits were zeroed out.

There's a github repo of this at http://github.com/jbenjore/Internals-GraphArenas/tree/master for the interested.

Whither Ruby's Text::Balanced?

jjore — 2009-08-31T19:07:40+00:00

Today at work the toy problem of extracting some data from $more_json in a line like "$json|$more_json". The obvious thing seemed to use Text::Balanced to skip the first $json, snip off the '|' and then use a real JSON parser on the next json blob.

So uh... there's no Text::Balanced in Ruby. Not that I can find. We solved our work problem by just declaring that we'd only use delimiters which didn't occur in our input so we wouldn't need to find a Ruby Text::Balanced. I'm thinking maybe the easiest thing to do if we do turn out to need this in Ruby will be to write a gem that embeds a Perl interpreter. Then Ruby's Text::Balanced is just a short adapter over the Perl version.

It's just a thought though.

Simple trick to get more copy-on-write goodness

jjore — 2009-08-23T23:45:26+00:00

I've got a preforking Apache server at work that's not sharing enough memory among the many children. This snippet helped identify post-fork module loading. I put an @INC hook at the end of my httpd.conf and it writes a warning to my log whenever the forked child loads something.

# httpd.conf ... <Perl> # This fragment goes at the end, when I think I'm ready to let Apache start forking $main::PARENT_PID = $$; unshift @INC, sub { print STDERR "LATE LOAD: $_[1]\n" if $$ != $main::PARENT_PID; return undef; }; </Perl>

I just found 233 modules I was loading post-fork. Oops! It's now trivial to make sure my httpd.conf has the proper lines to load the stuff I was forgetting to:

# httpd.conf LoadModule XML::LibXML LoadModule LWP::UserAgent ...

A Test::Class idiom

jplindstrom — 2009-08-21T14:26:53+00:00

Test::Class is very nice, but I realize I've started to use it in a peculiar way that I think is an indication of a missing feature.

Let's say there's a class with quite a few different tests and I'm currently TDD-ing this single method.

sub ion_favourites : Test(no_plan) { # ... }

It's quite annoying to run the whole test class, because all of the test methods are run, not just the one I'm working on. The output of my test is easily lost in the sea of output, and running all of the tests increase the turnaround time.

To only run this test, I temporarily rename it and put an exit at the end:

sub a__ion_favourites : Test(no_plan) { # ... exit(1); }

The naming ordering is actually mentioned in the manual, but seems to be more geared to running sanity checks first.

The full test suite fails becuase of the exit status, but I can easily see whether the stuff I'm working on passes and that's what I care about during TDD.

Smells a bit hacky, but it works. So, does anyone else do this?

Ruby's regexp engine sucks to debug

jjore — 2009-08-19T15:27:42+00:00

Today, I'm using Ruby for a project at work and Oh! My! God! Every week I get deeper with Ruby I become more impressed with how much I like writing ordinary things in it and how much I loath having to solve hard problems in it. This language, or the implementation seems to go miles out of its way to make it nigh impossible to debug.

Today's frustration:

Without changing anything I'm aware of my trepan gem (http://github.com/jbenjore/trepan/tree/master) has started throwing Bus error
I've found that the regexp engine is matching "something" wrong
To debug a regex in 1.8.6, I recompiled ruby with CFLAGS=-DDEBUG_REGEX
-DDEBUG_REGEX produces only the inscrutable output "regex loop(5): matching 0x01" which doesn't seem to map to anything like characters or offsets or anything. I can't use it
Ruby 1.9 uses Oniguruma but it also fails to match the same regexp
Installing the latest Oniguruma from http://www.geocities.jp/kosako3/oniguruma/ yields libonig, not something I can use in ruby. Oops
Building 2.x per the page tries to patch my Ruby 1.8 source code but fails
There's apparently a /third/ version of Oniguruma at http://rubyforge.org/projects/oniguruma. Maybe's it's just old? Maybe it's the version that works. Dunno
I just want a language that doesn't make me spend immense amounts of energy! Is that so much to ask? I throw tons of energy at perl but I get cool things for it. I'm throwing tons at ruby now and not even getting things I have for FREE in perl.
FWIW, after spending several hours struggling with Ruby, I threw in the towel. Read a book and went to bed early, exhausted.

CPAN commit bits, given freely

jjore — 2009-08-17T17:41:22+00:00

Hi there $world and PAUSE administrators,

Commit bits to my CPAN modules are given freely. If a PAUSE administrator or some other core hackers can vouch for you, I don't mind if upload and indexing permission bits are handed out and I only find out about it later.

This is prompted by an IRC conversation. I'd gone away for the weekend and left people hanging about whether some permissions were available. The answer in general is "yes, and be nice with them."

-
Josh

[non-perl] Hacking iPhoto via SQLite

ChrisDolan — 2009-08-14T02:33:23+00:00

(a very off-topic SQL exercise follows...)

Apple's iPhoto 2009 has a very cool face recognition feature where it will show you faces and prompt you to enter names. When it finds resemblances, it will suggest names for you to confirm or reject.

When it's wrong, it makes for funny little screenshots you can send to friends: "Haha, cute girl! You look like an older heavier man!"

But once you reject the match, you can't undo or get it back in any other way. Luckily, iPhoto stores ALL of its metadata in a collection of SQLite databases. So, in the particular case that I linked above I was able to undo by telling the database not to ignore the marked mismatches from IMG_0095.JPG:

% osascript -e 'tell application "iPhoto" to quit' % cd "Pictures/iPhoto Library" % sqlite3 iPhotoMain.db sqlite> select photoKey,relativePath from SqFileImage,SqFileInfo where sqFileInfo=SqFileInfo.primaryKey and relativePath like '%IMG_0095.JPG'; 110|Data/2003/Roll 15/IMG_0095.jpg 110|Originals/2003/Roll 15/IMG_0095.JPG 3666|Data/2006/Apr 25, 2006/IMG_0095.jpg 3666|Originals/2006/Apr 25, 2006/IMG_0095.JPG sqlite> .exit % sqlite3 face.db sqlite> update similar_faces set ignore=0 where image_key=110; sqlite> .exit % open -a iPhoto.app

Life... draining... out..., ruby-1.8 to blame

jjore — 2009-08-05T00:17:23+00:00

Losing...

the will to live...

must...

have...

useful stack traces...

I wish Ruby were as useful as Perl. I wish the following had the arguments like I'd get from any perl program using Carp. I also wish this code had just used perror(3) so it'd report the ordinary message "Address already in use" instead of making up its own message "no acceptor."

>> Thin web server (v1.2.2 codename I Find Your Lack of Sauce Disturbing) >> Maximum connections set to 1024 >> Listening on 127.0.0.1:8859, CTRL+C to stop Failed to start the service RuntimeError no acceptor /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:56 5:in `start_tcp_server' /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmac hine.rb:565:in `start_server' /usr/lib/ruby/gems/1.8/gems/thin-1.2.2/lib/thin/backends/tcp_ser ver.rb:16:in `connect' /usr/lib/ruby/gems/1.8/gems/thin-1.2.2/lib/thin/backends/base.rb:49:i n `start' /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242 :in `call' /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242: in `run_machine' /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine. rb:242:in `run' /usr/lib/ruby/gems/1.8/gems/thin-1.2.2/lib/thin/backends/base.rb:57:in `start' /usr/lib/ruby/gems/1.8/gems/thin-1.2.2/lib/thin/server.rb:156:in `start' /usr/lib/ruby/gems/1.8/gems/thin-1.2.2/lib/thin/controllers/controller. rb:80:in `start' ./basic.rb:68 ./basic.rb:51:in `initialize' ./basic.rb:51:in `new' ./basic.rb:51

My phone knows perl

jjore — 2009-08-03T04:37:19+00:00

It's cool to run perl on my phone. I tried compiling bleadperl on it just now but it didn't generate a Makefile. I guess I'm going to learn how the configuration system works now. The last few weeks have been endless yak shaving. I tracked down some debuild problem through diff to FSF's diffutils and all their various versions, learned how to override parts of Makefiles for ExtUtils::MakeMaker, installed and set up Gnus, have been getting into the internals of Git in a fairly serious way.

Blech. It seems it's so difficult to get to /actual/ work.

http://www.webos-internals.org/wiki/Application:Terminal fwiw

[non-perl] mdfind vs. locate

ChrisDolan — 2009-07-26T19:08:54+00:00

I've used the "locate" command for many, many years on various POSIX systems. It's a very fast search for filenames containing the specified string on your machine.

Here's a Mac command that simulates "locate" using the Spotlight database:

#!/bin/sh mdfind "kMDItemFSName == '*$1*'"

It's not as fast as locate, but it is always up to date (as opposed to locate, which relies on cron to reindex the disk periodically). As anyone can guess from the syntax, it's also very flexible. To find all files that originated as email attachments from my friend Glenn, for example, I can do:

mdfind "kMDItemWhereFroms == '*medianinja.com*'"

Or to find all files bigger than about a gigabyte:

mdfind "kMDItemFSSize > 1000000000"

(the WhereFroms search is fast and the size search is dramatically slower, maybe because all files have sizes but not many have kMDItemWhereFroms metadata)

To get a list of the supported keywords, type "mdimport -A"

Update: I found the Spotlight query syntax to be useful

Your future Unicode overlords

jjore — 2009-07-26T09:20:25+00:00

Recently while testing perl-5.10.1-RC0 at $work I found that it worked seamlessly for everything except our custom debianized build. It turns out http://cpansearch.perl.org/src/ADAMK/Parse-CPAN-Meta-1.39/t/data/utf_16_le_bom.y ml a UTF-16LE encoded file with a BOM (byte order mark) breaks my build of a .deb including this.

debuild makes both the binary and source packages. It's nice for presenting the whole picture to someone trying to follow along later. debuild uses dpkg-source to generate a diff between the vendor's source package and the source tree used in the actual debian build. dpkg-source uses your diff which probably comes from diffutils-2.8.1 which was the last release in 2002 (http://ftp.gnu.org/pub/gnu/diffutils/).

It looks like dev continued on til 2008 including the dev releases (ftp://alpha.gnu.org/pub/gnu/diffutils/, http://git.savannah.gnu.org/cgit/diffutils.git). Somewhere along the way it got support for multibyte characters. It still doesn't work for that UTF-16LE text file with a BOM.

Oh well. Anyway, consider this a tale of basic tools just not working nicely when your source code is in Unicode.

You'll miss composable functions

jjore — 2009-07-24T20:34:11+00:00

In perl or almost any other functional language (I guess) you can combine functions together and the innards of one thing aren't going to affect the flow control of the other. Here's a simple bit of perl:

method trap (CodeRef $block) { $block->() } uffda(); # Called in Perl, not called in Ruby } method moo ( ... ) { trap( sub { # Early return because ... return ... if ...; ... } ); }

Writing the equivalent flow control in Ruby and I found out today that the return() calls go all the way through and past the wrapping "trap" function.

WTF? I don't think I understand how Ruby programmers put up with this. Maybe 1.9 is better about this.

Gems driving me to drink

jjore — 2009-07-24T00:09:13+00:00

I don't know my blood pressure but I'm considering getting a desk-side monitor just to empirically know when I'm being driven out of my mind.

I wish:

that Ruby's RubyForge had something like CPAN's browseable directories like http://cpan.org/modules/by-module/Test/ for finding related things all grouped together
that Ruby's gems were named categorically instead of "Bob's Unit Testing Framework"
that there was a search.rubyforge.org which worked at least as nice as http://search.cpan.org/
that I could find where the test reports of gem's tests are aggregated like the CPAN Tester's http://matrix.cpantesters.org/?dist=Test-Simple+0.92

Work day's done now. I feel ill.