cogurov's Friends' Journals

DBD::SQLite 1.31 releasing next week and may break your code

Alias — 2010-09-09T02:11:54+00:00

After 6 or 7 months (mainly waiting around for the next "recommended upgrade instruction" from the SQLite project) the latest DBD::SQLite release should occur next week.

You can get the 1.30_06 release candidate from the CPAN, or from the following URL if your mirror hasn't synced yet.

http://svn.ali.as/cpan/releases/DBD-SQLite-1.30_06.tar.gz

Apart from the normal batch of SQLite upgrades (from 3.6.22 to 3.7.2), bug fixes, and minor enhancements, this release has two changes that may break your code.

These changes have been in the dev releases for some time, but you may want to take the opportunity to test more intensively if you use either of the following features.

1. BLOB columns with UTF8 content

- Resolved #54271: Inserting a string with utf-8 flag on corrupts BLOB data; now BLOB data is always stored as bytes (without the utf-8 flag) even if it has the flag set (ISHIGAKI)

2. FTS3 queries

- Added support for FTS3 tokenizers written in Perl. Added tests and documentation on how to use FTS3. Changed compilation flag to use the recommanded -DSQLITE_ENABLE_FTS3_PARENTHESIS *** MAY POSSIBLY BREAK OLD APPLICATIONS THAT ALREADY USED FTS3 *** (DAMI)

If you are currently using FTS3, please see DBD::SQLite::FTS3Transitional which contains a helper function for automatically upgrading old FTS3 queries to the new syntax.

Should Module::Install move to explicit plugin declaration?

Alias — 2010-09-06T02:26:06+00:00

Module::Install has been through a long period of gradual stability over the last year, without any really dramatic improvements to the grammar or APIs.

With the more urgent "it doesn't work with blah" stuff mostly solved now, one of the big remaining issues is around error clarity and excessive magic.

For example, some random author that is trying to checkout a Catalyst project needs:

1. To have Module::Install installed.
2. To have Module::Install::Catalyst installed.

In the case of the former, you get the semi-cryptic but at least standard "Can't find inc/Module/Install.pm in @INC" message, so the error is resolvable.

But in the latter case, you're likely to get something like "Unknown command 'catalyst_ignore'", with no real obvious resolution mechanism.

I think this idea of automatic plugin discovery is starting to hit it's limits in terms of clarity.

And so I'd like to do something counter to my natural instincts here, and make M:I more verbose.

I'm thinking of something like the following for explicitly declaring the use of a non-core Module::Install extension.

use inc::Module::Install qw{ Catalyst XSUtil };

This would both allow M:I to error with a much more meaningful error when you don't have a plugin, and also prevent the loading of unused plugins which should prevent accidental plugin collisions (some of which I've seen occurring in the CPAN Testers machines).

Thoughts?

DC Perl Mongers Rakudo Star Pizza Party Meeting

awwaiid — 2010-09-04T00:50:48+00:00

We're starting our usual monthly DC Perl Mongers meeting a bit early this Tuesday (September 7th) to have a little pizza and celebrate Rakudo-Star! Arrive at 6:30pm at the Starbucks at 18th and K Street NW (call me, Brock, if you miss us and need to be let in, number on the website) if you want food. But feel free to wander in any time thereafter, we usually stay as late as 10:00pm. We'll swoop down and look for people at the normal 7:30pm time too :)

Other activities:

Bring your laptop!
Installing Rakudo-Star
Giving hands-on tutorials
Beginner and Advanced welcome!
Never been to DC.pm before? No problem, come have some free pizza!

Please put your name on http://dc.pm.org/Rakudo-Star_Party (or email the mailing list, which can be found at http://dc.pm.org/) so we know how much food to get.

If you have any questions, email the list or me directly -- awwaiid@gmail.com will do :)

About Rakudo Star: "Rakudo Star" is a useful and usable distribution of Perl 6. Current releases are aimed at "early adopters" of Perl 6. We can tell you ALL about it when you arrive :)

Pittsburgh Perl Workshop CFP ends Monday Sept 6

rblackwe — 2010-09-02T15:22:01+00:00

The Pittsburgh Perl Workshop 2010 will be Saturday October 9 and Sunday 10 at the Gates Center at CMU.

Don't miss your chance to speak at this years Workshop. The Call For Papers ends Monday Sept 6.

Submit your talk.

Speaking at Microsoft TechEd - Any issues you want raised?

Alias — 2010-08-20T03:20:46+00:00

Next week I will at Microsoft's TechEd Australia event, courtesy of Microsoft Australia and Microsoft Open Source Labs.

More specifically, I'll be attending the Open Source mini-conf and discussion day on Tuesday, and presenting in the Community Presentations to Microsoft session on the current state of Perl and Windows on Wednesday.

Likely topics will include a review of the first year of the CPAN Testing Lab and a second-generation based on their Cloud Services, free code signing certificates for open source developers, and what issues are slowing us down or blocking progress.

So consider this your opportunity to raise any outstanding issues you have with Microsoft and Perl. What problems are you still seeing, what would like fixed or changed, and what is on your want-to-have list?

I'll try to address as many of your issues as possible in the time I have available with them (which is actually pretty substantial).

New WWW::Salesforce release details

Phred — 2010-08-19T17:24:05+00:00

I've taken over the maintainership role for WWW::Salesforce and have pushed out a maintenance release that resolves some long standing issues.

http://search.cpan.org/dist/WWW-Salesforce/

0.12 Tue Aug 17 19:34:00 2010 PST
- New maintainer PHRED
- Thanks to Mark Stosberg for several patches for this version
- Die with an error string instead of carping and returning
- Skip tests in automated testing mode
- Skip tests unless user, pass, and sectoken environment vars set
- Fix failing test - base64binary => base64Binary namespace change
- Perltidy file contents and remove unnecessary package scope braces
- Handle undefined return values from SOAP client
- Fix Type => type doc error in create()
- Add describeSObjects method [tom@eborcom.com]

Perlmongers Dinner

Phred — 2010-08-17T20:49:00+00:00

We'll be having a group dinner for the August meeting, and have
a few drinks after for those interested. This will mostly
be a planning meeting for future meetings, but all are welcome
for Perl discussion and agreat food.

"Naan-N-Curry" at 336 O'Farrell Street, between Mason and Taylor.

http://maps.google.com/maps?q=336+OFarrell+St,+San+Francisco,+CA+94102,+USA

This place has moved around a few times, and has many satellite
locations now, so look at that address carefully. This is across the
street from the Hilton, and next to the entrance to a large parking
garage.

From the Powell Street Bart station: walk two blocks north along Powell,
and 1.5 blocks west. Don't try to walk up Mason or Taylor, unless
you're in an adventurous mood.

The food is inexpensive, high quality Indian food. They have a buffet
these days, which makes things simpler. Free chai. The dining room
is double-sized, with large tables: there's no need to worry too much
about RSVPs.

http://naancurry.com/branches.php?brn=5

This place used to be 24 hours, but I guess they've scaled back to
11:00 AM to 4:00 AM. But I don't think we'll need to rush out of
there.

Announcement posted via App::PM::Announce

RSVP at Meetup - http://www.meetup.com/San-Francisco-Perl-Mongers/calendar/14453668/

TPF News website redesign

jj — 2010-08-16T19:57:22+00:00

I've just uploaded a new design for http://news.perlfoundation.org. Based on the perldoc.perl.org style, it implements Phil Smith's idea for the "Explore Perl" toolbar. I'll blog in more detail about this later, but essentially we've now got a single list of links that appear on multiple sites but are maintained in a single place.

Many thanks to Ask, Squeaky, Phil Smith, and Karen for their help and ideas.

P.S. As with any redesign there may be the odd browser issue or bug, so if anything looks amiss please email me at jj@jonallen.info and I'll look into it.

Class::XSAccessor now even faster'er

Alias — 2010-08-16T03:04:05+00:00

The new 1.07 release of Class::XSAccessor mentions the use a new somewhat-evil technique for making the code even faster than it was previously.

But how much faster is it?

The following are being run on a fairly typical corporate Windows XP machine, with Strawberry Perl 5.10.1 and thread support.

First, some benchmarks using the previous 1.05 release (two runs)

Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true... accessor_get: 1 wallclock secs ( 2.51 usr + 0.00 sys = 2.51 CPU) @ 3976143.14/s (n=10000000) accessor_set: 2 wallclock secs ( 3.09 usr + 0.00 sys = 3.09 CPU) @ 3233107.02/s (n=10000000) constructor: 16 wallclock secs (15.67 usr + 0.00 sys = 15.67 CPU) @ 638080.65/s (n=10000000) false: 2 wallclock secs ( 1.91 usr + 0.00 sys = 1.91 CPU) @ 5243838.49/s (n=10000000) getter: 1 wallclock secs ( 2.34 usr + 0.00 sys = 2.34 CPU) @ 4266211.60/s (n=10000000) predicate: 1 wallclock secs ( 2.38 usr + 0.00 sys = 2.38 CPU) @ 4210526.32/s (n=10000000) setter: 2 wallclock secs ( 3.27 usr + 0.00 sys = 3.27 CPU) @ 3061849.36/s (n=10000000) true: 1 wallclock secs ( 1.80 usr + 0.00 sys = 1.80 CPU) @ 5564830.27/s (n=10000000) Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true... accessor_get: 3 wallclock secs ( 2.51 usr + 0.00 sys = 2.51 CPU) @ 3976143.14/s (n=10000000) accessor_set: 3 wallclock secs ( 3.14 usr + 0.00 sys = 3.14 CPU) @ 3183699.46/s (n=10000000) constructor: 15 wallclock secs (15.73 usr + 0.00 sys = 15.73 CPU) @ 635566.29/s (n=10000000) false: 2 wallclock secs ( 1.86 usr + 0.00 sys = 1.86 CPU) @ 5379236.15/s (n=10000000) getter: 3 wallclock secs ( 2.50 usr + 0.00 sys = 2.50 CPU) @ 4000000.00/s (n=10000000) predicate: 3 wallclock secs ( 2.47 usr + 0.00 sys = 2.47 CPU) @ 4050222.76/s (n=10000000) setter: 4 wallclock secs ( 3.13 usr + 0.00 sys = 3.13 CPU) @ 3200000.00/s (n=10000000) true: 2 wallclock secs ( 1.98 usr + 0.00 sys = 1.98 CPU) @ 5037783.38/s (n=10000000)

And now again with the new 1.07 release.

Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true... accessor_get: 2 wallclock secs ( 1.75 usr + 0.00 sys = 1.75 CPU) @ 5711022.27/s (n=10000000) accessor_set: 1 wallclock secs ( 2.69 usr + 0.00 sys = 2.69 CPU) @ 3721622.63/s (n=10000000) constructor: 15 wallclock secs (15.62 usr + 0.00 sys = 15.62 CPU) @ 640000.00/s (n=10000000) false: 1 wallclock secs ( 1.28 usr + 0.00 sys = 1.28 CPU) @ 7806401.25/s (n=10000000) getter: 1 wallclock secs ( 1.56 usr + 0.00 sys = 1.56 CPU) @ 6397952.66/s (n=10000000) predicate: 2 wallclock secs ( 1.92 usr + 0.00 sys = 1.92 CPU) @ 5205622.07/s (n=10000000) setter: 3 wallclock secs ( 2.50 usr + 0.00 sys = 2.50 CPU) @ 4000000.00/s (n=10000000) true: 2 wallclock secs ( 1.55 usr + 0.00 sys = 1.55 CPU) @ 6464124.11/s (n=10000000) Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true... accessor_get: 2 wallclock secs ( 1.78 usr + 0.00 sys = 1.78 CPU) @ 5614823.13/s (n=10000000) accessor_set: 3 wallclock secs ( 2.63 usr + 0.00 sys = 2.63 CPU) @ 3809523.81/s (n=10000000) constructor: 16 wallclock secs (15.69 usr + 0.00 sys = 15.69 CPU) @ 637429.88/s (n=10000000) false: 2 wallclock secs ( 1.22 usr + 0.00 sys = 1.22 CPU) @ 8203445.45/s (n=10000000) getter: 2 wallclock secs ( 1.53 usr + 0.00 sys = 1.53 CPU) @ 6535947.71/s (n=10000000) predicate: 2 wallclock secs ( 1.78 usr + 0.00 sys = 1.78 CPU) @ 5614823.13/s (n=10000000) setter: 2 wallclock secs ( 2.56 usr + 0.00 sys = 2.56 CPU) @ 3903200.62/s (n=10000000) true: 2 wallclock secs ( 1.48 usr + 0.00 sys = 1.48 CPU) @ 6738544.47/s (n=10000000)

The numbers are pretty impressive.

The 'accessor', 'setter', 'predicate' and 'true' methods are about 25% faster, while 'getter' is a whopping 60% faster and (curiously) 'false' is about 50% faster as well.

Constructors are really the only thing that hasn't changed.

Impressive work, even if the code is a bit risky.

Why does Object::Tiny only support getters

Alias — 2010-08-15T01:39:50+00:00

http://perlalchemy.blogspot.com/2010/08/objecttinyrw-and-moosexnonmoose.html

Zbigniew Lukasiak tries out Object::Tiny and wonders why it is that I didn't allow for the creation of setters when it is only a one line change.

Like most ::Tiny modules the reason is a bit complex and the result of compromises.

Object::Tiny began as an attempt to create a lighter, faster, version of Class::Accessor. A way to bulk-generate the accessor code I had to type over and over again.

However, where I differ is a strong preference for light and elegant API design.

And so I decided to implement mine with as little implementation code as possible, and as little API code as possible.

Once you have decided to go down the simplicity path, there's a couple of standard techniques you often end up using.

The first and most important is state reduction.

In their introduction to Erlang, the founders of that language describe state as one of the main sources of failures in programs. And so anything that removes state, at the very least unnecessary state, is a positive. Especially if the state reduction also results in code reduction, and a reduction in computation.

So take the following example, where we create an object with some attributes and then run some code that will use those object attributes..

my $object = Class->new; $object->foo(1); $object->bar(2); $object->do_something;

This is a use case that we see fairly often, but it's really quite horrible code. It is really only the object-oriented equivalent of something like the following.

our $Object::foo = 1; our $Object::bar = 2; do_something('Object');

It is especially bad code if the following code would throw an exception.

my $object = Class->new; $object->do_something;

If this blows up, then you are REALLY doing something wrong, because you have allowed the creation of completely invalid objects. Now anybody taking one of these objects as a parameters needs to do with following.

sub foo { my $object = shift; unless ( $object->isa('Class') and defined $object->foo and $object->foo > 0 and defined $object->bar and $object->bar > 2 ) { die "Invalid object"; } }

If you are going to create an object for something, you HAVE to be sure that the objects are trustworthy.

And so you should never allow objects to exist that are invalid. EVERY object should be a valid object.

At the absolute minimum objects should be able to default every attribute to something reasonable and unlikely to cause problems.

But this still results in excess and wasteful work, because the object has to transition through two or more states.

You start with an object with parameters and defaults, and you validate them. And then you change on of the attributes immediately, validating it AGAIN. In the mean time, your object exists in a state that it will never actually be used in.

And so everywhere you possibly can, you should be setting attributes in the constructor rather than afterwards.

my $object = Class->new( foo => 1, bar => 2, ); $object->do_something;

Less state, less complexity, less CPU, and less bugs.

If we accept this model of pushing all the configuration into the object up front to reduce state, then why change the object arbitrarily?

In fact, anything that you ARE going to change should be done under very controlled conditions.

It should require a dedicated method to apply the change, it should require validation, and work. It shouldn't be trivial, and it shouldn't be automatic.

If I had my way, Moose would set is => 'ro' by default, to make people think before they go about simply allowing stuff to change.

It also happens to let you shrink down the API markedly.

There are three potential use cases available when implementing accessors. Everything readonly, everything readwrite, or mixed.

With Object::Tiny, I was aiming for the smallest possible code.

Implementing either all-readonly or all-readwrite can be done with the following.

use Class qw{ foo bar };

By contrast, if we want to allow mixed readonly and readwrite, we would need some way of distinguishing. Something like the following.

use Class { readonly => [ 'foo' ], readwrite => [ 'bar' ], }; use Class [ qw{ foo } ], [ { bar } ]; use Class { foo => 'ro', bar => 'rw', };

No matter how you try, there's always an inherent additional element of complexity that results from the split between them.

And so the decision to go with all-readonly in Object::Tiny is a combination of these two issues.

If went with all-readwrite, I'm practically encouraging bad behaviour and more bugs. If I went with mixed accessors, the API would remain relative complex.

In the end, the best way to achieve both API simplicity and code safety is to only provide read-only accessors, and anything more complex should require both though and effort.

CPAN Testers 2.0: The death of via email is wrong.

jk2addict — 2010-08-10T02:49:45+00:00

I finally got around to installing perlbrew, 4 flavors of perl and updating some modules for recent perls. I've been sending a lot of email test reports lately.

Last night I got the email from bitbucket about CPAN Testers 2.0 and the death of sending reports via email with wiki links/instructions on how to setup .cpanreports/config.ini for a metabase id/transport via HTTP.

This is all well and good, but it feels wrong to me. I use CPAN::Mini. The point of which is partly to have CPAN when you're not online. The same reason I use git. With SMTP emails I could just queue up test reports in the local postifx and they would get delivered later when I'm online.

Now, that's is no more. I'm forced to be online to send HTTP reports. Seems like a bad idea. NOw, the only thing I can do is toggle off reporting when I'm offline, which really means I'll hate doing that and just turn off reporting all together.

Why did reports via email have to go away?

Trailer Theory - Reinvented for Ignite Sydney as Economics

Alias — 2010-08-04T05:40:53+00:00

Back in 2005 in only my fifth use.perl post ever, I outlined an idea I had been developing for a couple of years that I called "Trailer Theory".

A few years ago on my Portable Perl world hack'a'tour, I took with me a lightning talk version of the concept. It was a pretty crude talk but was received, it seemed, fairly well by the development community.

Since that trip, and inspired by the unexpected conversion of my "Perl is unparsable" claims in the PPI docs into a formal mathematical proof (complete with "Kennedy's Lemma") I've been wondering if this "Trailer Theory" idea could really be developed as a proper scientific proof, and if so what would that look like.

A couple of months ago I presented a new version of the talk at Ignite Sydney, speaking to an mixed audience of Twitterati, social media, advertising and journalist types.

I've rebuilt the talk from scratch and tried to outline the same idea, but in the form of a kind of layman's Economics Proof.

I hope you enjoy the result.

Adam Kennedy - Using Economics to make movies suck less

Notes for other speakers:

1. Ignite advances slides every 15 seconds, no clickers allowed. This turns out to take a shitload of practice to get right.

2. When someone says to you "Here's your mark, you need to stay on this line to be in the fixed spot" it helps to pay attention. FAIL :)

SF.pm grant funding proposal to TPF up for public review

Phred — 2010-08-03T20:43:17+00:00

A grant proposal I wrote for SF.pm is up for public review. All views welcome - http://news.perlfoundation.org/2010/08/2010q3-grant-proposal-sfpm-fun.html

Protesting the Moose with sugary alternatives

Alias — 2010-08-03T05:09:02+00:00

At work, we've been experimenting with Moose because in our enormous and complex codebase we think we can probably benefit a lot from the extra rigour that it brings.

Before I continue, let me note that ours is a typical mod_perl enterprise setup with 6gig of memory per machine, so any memory consumed before the Apache fork is essentially free.

So none of the issues people (including me) have with startup code and memory consumption apply in this case, and I won't be addressing performance issues in this post.

The consensus of the half-dozen people at work is how Moose tries to look like a declarative extension, but doesn't actually act like it.

The following is what Moose seems to have been aiming for.

package Foo; use Moose; extends 'Bar'; with 'Role1'; with 'Role2'; has this => ( is => 'ro', isa => 'Str', ); has that => ( is => 'ro', isa => 'Str', ); 1;

Unfortunately, this is what we've had to do instead.

package Foo; use Moose; BEGIN { # When we use Catalyst extends 'Bar'; } has this => ( is => 'ro', isa => 'Str', ); has that => ( is => 'ro', isa => 'Str', ); with 'Role1'; with 'Role2'; no Moose; __PACKAGE->meta->make_immutable;

This "real Moose" code totally spoils the dream of what we felt like we were going to get when we started to play with it.

Most of our current options for fixing this amount to either.

a) Add this extra dependency that will unscrew one of the other of the problems (namespace::autoclean)

b) Use this SECOND heavy sugar layer on top of the FIRST sugar layer, on top of Class::MOP.

Is fixing the syntax or writing light weight sugar really so hard?

As a kind of protest, I tried it for myself and managed to create MooseX::Atom.

This still has some flaws, but the current equivalent of the above would just be this.

package Foo; use MooseX::Atom [ extends => 'Bar', with => 'Role1', with => 'Role2', has => [ this => ( is => 'ro', isa => 'Str', ), ], has => [ that => ( is => 'ro', isa => 'Str', ), ] ]; 1;

You can do the same thing for roles with MooseX::Role::Atom.

Now clearly, this might have some issues. It's the work of an hour and not a whole lot of thought.

But it's still light and clean, with all the class spec in one place up the top where people are used to seeing the declarative stuff in Perl modules.

Perhaps something like this might be a little better...

package Foo; use MooseX::Hash { extends => 'Bar', with => [ 'Role1', 'Role2' ], default => { is => 'ro' }, has => { this => { isa => 'Str' }, that => { isa => 'Str' }, }, ); 1;

Help?

Alias — 2010-07-29T13:23:44+00:00

(Before I begin, I should clarify I did not write this code, I'm just trying to maintain it)

The following error is the first thing spat out by make test for the Padre sync server, located at http://svn.perlide.org/padre/trunk/Madre-Sync.

Wasn't the move of Catalyst to Moose going to make things easier?

Can someone explain how you debug this?

I get the basics, I can see the "Can't locate Madre/Sync/Schema.pm". But that file should be, I think, automatically generated. And I don't really get how to dig down the 75 caller levels from the start to the end to work out where the actual functionality is failing...

not ok 1 - use Catalyst::Test; # Failed test 'use Catalyst::Test;' # at t\01app.t line 7. # Tried to use 'Catalyst::Test'. # Error: Couldn't load class (Madre::Sync) because: Couldn't instantiate component "Madre::Sync::Model::padreDB", "Can't locate Madre/Sync/Schema.pm i n @INC (@INC contains: blib\lib blib\arch C:/strawberry/perl/lib C:/strawberry/perl/site/lib C:\strawberry\perl\vendor\lib .). at C:\strawberry\perl\vendor \lib/Class/MOP.pm line 132 # Class::MOP::load_first_existing_class('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137 # Class::MOP::load_class('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Catalyst/Model/DBIC/Schema/Types.pm line 21 # Catalyst::Model::DBIC::Schema::Types::__ANON__[C:\strawberry\perl\vendor\lib/Ca talyst/Model/DBIC/Schema/Types.pm:21]('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Moose/Meta/TypeCoercion.pm line 63 # Moose::Meta::TypeCoercion::__ANON__[C:\strawberry\perl\vendor\lib/Moose/Meta/Ty peCoercion.pm:67]('Madre::Sync::Schema') called at C:\strawberry\per l\vendor\lib/Moose/Meta/TypeCoercion.pm line 97 # Moose::Meta::TypeCoercion::coerce('Moose::Meta::TypeCoercion=HASH(0x4a2b444)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Moose /Meta/TypeConstraint.pm line 90 # Moose::Meta::TypeConstraint::coerce('Moose::Meta::TypeConstraint=HASH(0x4a29854 )', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/M ooseX/Types/TypeDecorator.pm line 206 # eval {...} called at C:\strawberry\perl\vendor\lib/MooseX/Types/TypeDecorator.pm line 205 # MooseX::Types::TypeDecorator::AUTOLOAD('MooseX::Types::TypeDecorator=HASH(0x4a3 1424)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\l ib/Moose/Meta/Attribute.pm line 743 # Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x4b467 fc)', 'Madre::Sync::Schema', 'Madre::Sync::Model::padreDB=HASH(0x4db 7c64)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Attribute.pm line 398 # Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(0 x4b467fc)', 'Moose::Meta::Instance=HASH(0x4db7ed4)', 'Madre::Sync::M odel::padreDB=HASH(0x4db7c64)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Class.pm line 567 # Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/C lass.pm line 540 # Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Class.pm line 256 # Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Object.pm lin e 25 # Moose::Object::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at generated method (unknown origin) line 4 # Catalyst::Model::DBIC::Schema::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at C:\strawberry\perl\vendor\lib/MooseX/ Traits/Pluggable.pm line 131 # MooseX::Traits::Pluggable::new_with_traits('Madre::Sync::Model::padreDB', 'Madre::Sync') called at C:\strawberry\perl\vendor\lib/CatalystX/Componen t/Traits.pm line 146 # CatalystX::Component::Traits::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Cl ass/MOP/Method/Wrapped.pm line 48 # Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:49]('Madre::Sync::Model::padreDB', 'Madre::Sync', ' HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 89 # Catalyst::Model::DBIC::Schema::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/C atalyst.pm line 2502 # eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2502 # Catalyst::setup_component('Madre::Sync', 'Madre::Sync::Model::padreDB') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2416 # Catalyst::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 54 # Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:64]('Madre::Sync') called at C:\strawberry\perl\ven dor\lib/Class/MOP/Method/Wrapped.pm line 89 # Madre::Sync::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 1142 # Catalyst::setup('Madre::Sync') called at blib\lib/Madre/Sync.pm line 62 # require Madre/Sync.pm called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 114 # Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:118]() called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 74 # eval {...} called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 67 # Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125 # Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137 # Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24 # Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493 # Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424 # Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S ub/Exporter.pm line 742 # Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per l\vendor\lib/Catalyst/Test.pm line 112 # Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2 # main::BEGIN() called at blib\lib/Madre/Sync.pm line 0 # eval {...} called at blib\lib/Madre/Sync.pm line 0 # eval 'package main; # use Catalyst::Test @{$args[0]}; # 1; # # ;' called at C:/strawberry/perl/lib/Test/More.pm line 858 # Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p m line 833 # Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7 # at C:\strawberry\perl\vendor\lib/MooseX/Types/TypeDecorator.pm line 208 # MooseX::Types::TypeDecorator::AUTOLOAD('MooseX::Types::TypeDecorator=HASH(0x4a3 1424)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\l ib/Moose/Meta/Attribute.pm line 743 # Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x4b467 fc)', 'Madre::Sync::Schema', 'Madre::Sync::Model::padreDB=HASH(0x4db 7c64)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Attribute.pm line 398 # Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(0 x4b467fc)', 'Moose::Meta::Instance=HASH(0x4db7ed4)', 'Madre::Sync::M odel::padreDB=HASH(0x4db7c64)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Class.pm line 567 # Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/C lass.pm line 540 # Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Class.pm line 256 # Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Object.pm lin e 25 # Moose::Object::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at generated method (unknown origin) line 4 # Catalyst::Model::DBIC::Schema::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at C:\strawberry\perl\vendor\lib/MooseX/ Traits/Pluggable.pm line 131 # MooseX::Traits::Pluggable::new_with_traits('Madre::Sync::Model::padreDB', 'Madre::Sync') called at C:\strawberry\perl\vendor\lib/CatalystX/Componen t/Traits.pm line 146 # CatalystX::Component::Traits::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Cl ass/MOP/Method/Wrapped.pm line 48 # Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:49]('Madre::Sync::Model::padreDB', 'Madre::Sync', ' HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 89 # Catalyst::Model::DBIC::Schema::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/C atalyst.pm line 2502 # eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2502 # Catalyst::setup_component('Madre::Sync', 'Madre::Sync::Model::padreDB') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2416 # Catalyst::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 54 # Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:64]('Madre::Sync') called at C:\strawberry\perl\ven dor\lib/Class/MOP/Method/Wrapped.pm line 89 # Madre::Sync::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 1142 # Catalyst::setup('Madre::Sync') called at blib\lib/Madre/Sync.pm line 62 # require Madre/Sync.pm called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 114 # Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:118]() called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 74 # eval {...} called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 67 # Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125 # Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137 # Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24 # Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493 # Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424 # Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S ub/Exporter.pm line 742 # Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per l\vendor\lib/Catalyst/Test.pm line 112 # Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2 # main::BEGIN() called at blib\lib/Madre/Sync.pm line 0 # eval {...} called at blib\lib/Madre/Sync.pm line 0 # eval 'package main; # use Catalyst::Test @{$args[0]}; # 1; # # ;' called at C:/strawberry/perl/lib/Test/More.pm line 858 # Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p m line 833 # Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7"Compilation failed in require at C:\strawberry\perl\vendor\lib/Class /MOP.pm line 114. # at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 121 # Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:125]('Couldn\'t instantiate component "Madre::Sync::Model::padreDB"...') called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 98 # Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125 # Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137 # Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24 # Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493 # Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424 # Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S ub/Exporter.pm line 742 # Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per l\vendor\lib/Catalyst/Test.pm line 112 # Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2 # main::BEGIN() called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 2 # eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 2 # eval 'package main; # use Catalyst::Test @{$args[0]}; # 1; # # ;' called at C:/strawberry/perl/lib/Test/More.pm line 858 # Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p m line 833 # Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7 # BEGIN failed--compilation aborted at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2.

Rakudo Star Install Party

Phred — 2010-07-28T18:44:20+00:00

Hi Mongers,

I'd like to offer my place in Bernal Heights Tuesday next week from 7:30pm until whenever for a Raduko Star installation & Perl 6 lightning talks. As some of you know a significant milestone in Perl 6's release history is coming up this Thursday, http://rakudo.org/node/73 and what better way to get through an install than with local PM'ers!

For those that haven't been chez moi we have a basement, bar, projector, wifi, yard, BBQ, etc so we can eat, drink & give presentations. There's space for at least a dozen seated inside, and more outside (for those that can withstand the Day Star).

Who's interested in something like this? How about giving a talk?
Doesn't need powerpoint but we have a screen & sound if you do.
Anything at all raduko/perl6 related seems cool. Ping me off-list and I'll collate.

Summary:
What: Rakudo Star Install Party
When: Tuesday 3rd August 2010, 19:30 'til (e.g.) 22:00
Where: Paul's place, SF, 94110 (address on RSVP)
What to bring: computer, snacks & drinks (although I'll seed unless someone wants to sponsor it?!). We could even BBQ!

Bonus feature: there's a ton of interesting links in bite-sized chunks at http://twitter.com/rakudoperl

Paul

Announcement posted via App::PM::Announce

RSVP at Meetup - http://www.meetup.com/San-Francisco-Perl-Mongers/calendar/14243128/

For people running Perl conferences

Alias — 2010-07-25T12:55:04+00:00

Leo Lapworth comments in "For speakers at Perl conferences" that you should record your own talks at conferences, because organisers cannot be trusted to release the videos they take of you.

I wholeheartedly agree. I've spoken at numerous conferences, including several large ones, and only in about 10-20% of cases have the videos of me EVER appeared.

I'm amazed that conference organisers can put in so much effort into recording (dozens of tapes, multiple cameras, multiple operators) and then never produce anything as a result.

The only time I'm aware of that a full talk of mine has appeared online was at a Linux.conf.au, who had a dedicated video team of eight people.

Where is all this footage, and why does it never get processed? Why even bother recording it? If you don't have time, send me the raw tape of my own talk and I will do it myself if needed.

YAPC::NA? You listening?

A reminder - Padre Second Birthday Hackathon this weekend

Alias — 2010-07-23T03:53:28+00:00

This is just a quick reminder that this weekend is the Padre's second birthday party and hackathon.

If you are a Padre user, please drop in and say hello to the team. We'd love to hear how you are using Padre and where your main needs are for the next year.

If you are interested in trying out Padre for the first time, or trying your hand at improving it for the first time, it's a great time to get started because we'll have plenty of people around to provide guidance and advice.

Some of the plans for this weekend are to bring all the plugins up to date with the latest versions of the plugin API, to start the merge of the ConfigSync branch, and if we can, to start on the Madre server that will serve as the ConfigSync and Telemetry server for Padre.

I look forward to seeing you there!

Profiling your website while live on a production cluster

Alias — 2010-07-22T02:27:33+00:00

As I mentioned in my last post, by day I wrangle a large web application that occasionally verges on being too complex to for mere humans to understand.

Curiously, because it is private and the income rate is high (we probably average something like $5 in gross turnover per page view) we don't have to deal with a lot of servers to deliver it, by internet standards anyway.

But the application is still way too big to profile easily by normal methods, and certainly on production it's way too heavy, even if we applied targeted profiling using something like Aspect::Library::NYTProf.

Between the web servers, transaction server, database, search engine and cache server, we are probably only dealing with 12 servers and 30 CPUs. Of course, these servers are all horribly expensive, because they are all server-virtualised, network-virtualised, doubly redundant (high-availability + disaster-recovery) and heavily monitored with high end support contracts.

One of our most sensitive issues is database load.

We have a ton of database tables (about 200) and lots of medium sized queries running across them. One of our main failure modes is that some deep change to code boosts the quantity of some significant query, which stresses the database enough to cause contention and lock-storms, leading to site failure.

Complicating things, big parts of some pages are embedded in other pages. So attributing load and lag to one part of the website, or to Oracle, is tricky and hard to benchmark in advance (although we do load test the main load paths to catch the worst cases).

For a long time, we've had a mechanism for zoned profiling the production site, so we can allocate wallclock costs to different general areas of the site.

But it is fragile and unreliable, requiring perfect maintenance and every developer to remember to write this kind of thing everywhere.

# Embed a foo widget in the current page $perf->push_timing('foo'); foo($bar); $perf->pop_timing;

Since you don't know this profiling system exists unless you've seen it somewhere else in the code before, and it's hard to care about something that is orthogonal to the problem you are actuall solving, this mechanism has degraded over time. While we still get some pretty cacti graphs showing load breakdown, they are highly unreliable and you can never be entirely sure if the load attribution is correct.

This kind of performance monitoring as a "cross-cutting concern" is a textbook use case for Aspect-Oriented Programming, and so in our Christmas 2009 code freeze window I set about trying to rewrite the push/pop_timing profiler using Aspect.pm.

Unfortunately, Aspect.pm turned out to be a bit too slow and naive for my needs. But after a 6 month delay to do a 90% rewrite of Aspect.pm, I now finally have something sophisticated enough (and fast enough) to meet my needs.

So today I'm releasing the shiny new Aspect::Library::ZoneTimer, which will serve as the main plank of our new production profiling system.

The idea behind ZoneTimer is to define each performance zone as a pointcut. The aspect will track the movement of your code across each zone boundaries and build a running total of the exclusive time spent in each performance zone.

When your program exits the top-most performance zone, the totals will be sent to a handler function.

A typical use of the ZoneTimer aspect looks something like this

use Aspect; aspect ZoneTimer => ( zones => { main => call 'MyProgram::main' & highest, search => call 'MyProgram::search', widgets => call qr/^MyProgram::widget_.*/, templates => call 'MyProgram::render', dbi => call qr/^DB[DI]::.*?\b(?:prepare|execute|fetch.*)$/, system => ( call qr/^IPC::System::Simple::(?:run|runx|capture)/ | call 'IPC::Run3::run3' | call qr/^Capture::Tiny::(?:capture|tee).*/ ) }, handler => sub { my $top = shift; # "main" my $start = shift; # [ 1279763446, 796875 ] my $stop = shift; # [ 1279763473, 163153 ] my $exclusive = shift; # { main => 23123412, dbi => 3231231 } print "Profiling from zone $top\n"; print "Started recording at " . scalar(localtime $start) . "\n"; print "Stopped recording at " . scalar(localtime $stop) . "\n"; foreach my $zone ( sort keys %$exclusive ) { print "Spent $exclusive->{$zone} microseconds in zone $zone\n"; } }, );

This example breaks out the cost of a typical small web application into a general zone, a special zone for the search page, and then splits the costs of generating widgets, rendering the HTML template, waiting for the database, and making calls to the underlying system.

Start and stop times are returned as a two element array exactly as returned by C, and the exclusive zone totals are returned as integer microseconds (all math is done in these integer microseconds to prevent floating point corruption).

The use of Aspect allows us to easily mix special cases via the pointcuts, such as the use of "highest" which makes sure that "main" only matches the first time it is seen, and any widget which that does a re-entry into main still has that cost attributed to the widget. We've also hooked into multiple system call modules to measure system call cost, because we know different modules our program consumes will use different methods for interacting with the system.

While the handler I've shown here will just print out a summary of the call, in our environment at work the profile report handler will format the exclusive times into a special log message and then send it via the super-quick and non-blocking Log::Syslog::Fast module to the UDP localhost syslog service, where it is mirrored to disk for debugging use, and then forwarded on to our main company-wide syslog server.

On our setup, we can then use an excellent commercial log analysis product called Splunk (limited free version also available if you want to try it out) to do our tracking and trending of the performance for the entire application across the entire cluster.

The nicest thing about the Aspect system is that it scales the performance cost and complexity risk directly to the complexity of the use case you are using it for.

If it turns out that the Aspect hooks are too intrusive and causing a performance drain, or accidentally causing some weird edge case behaviour, you can simply turn off that the Aspect and restart the servers and the performance penalty just vanishes.

Maintenance of the profiler zones is really easy, because they are all listed clearly in one place.

Depending on your use case, you could even define the performance zones in your website configuration, and then adjust the profiler zone boundaries directly on production (although I'm not sure I'd want to do that in our specific case, since we're fairly paranoid about untested code going into production).

This ZoneTimer is just the first of several Aspect-based tools I plan to release over the next few months. If you get a chance to try out any of these new toys, I'd love to hear feedback on how well they work for you.

In addition to the value of the ZoneTimer aspect itself, one other thing that some people might find of interest is just how little code it took to get this all working.

The entire Aspect::Library::ZoneTimer was implemented in about 80 lines of code, which you can see at http://cpansearch.perl.org/src/ADAMK/Aspect-Library-Timer-0.04/lib/Aspect/Librar y/ZoneTimer.pm

This is small enough that you could just clone the module and tweak it for your own use case, if you wanted to creat a new and tricky customised profiler.

How we deploy massive Perl applications at work

Alias — 2010-07-20T02:25:21+00:00

Every now and then, we hear people talking about mechanisms for doing Perl in a commercial environment and how they deal with packaging and dependencies.

This is mine.

At Corporate Express, our main Perl application is a 250,000 line non-public monster of a website that has over 100,000 physical users and turns over about a billion dollars. It implements huge amounts of complex business functionality, and has layer upon layer of security and reliability functions in it because we supply to multinationals, governments and the military (only the stuff that doesn't blow up of course). Our .pm file count is around 750, and our test suite runs for about 4 hours (and is only around one third complete).

Lest you suspect that 200,000 lines is wasted in re-implementing stuff, the main Build.PL script has around 110 DIRECT dependencies, and somewhere in the 300-500 range of recursive dependencies. Loading the main codebase into memory takes around 150-200meg of RAM.

When I joined the team, the build system was horribly out of date. The application was stuck on an old version of RedHat due to go out of support, and as a Tier 1 application we are absolutely forbidden from using unsupported platform.

So I took on the task of upgrading both the operating system and the build system for the project. And it's a build system with a history.

Once upon a time, long ago, the project went through a period where the development team was exceptionally strong and high skilled. And so of course, they created a roll-your-own build system called "VBuild".

They built their own Perl, and along with it they also built their own Apache, mod_perl, and half a dozen other things needed by the project. This is similar to many suggestions I hear from high-skilled people today, that at a certain point it's better just to build your own Perl. VBuild this created in the pre-commercial Linux era, so it's not an entirely unreasonable decision for that time period.

Unfortunately, a few years later the quality of the team dropped off and VBuild turned into a maintenance nightmare because it required a high-skill person to maintain it.

At the time, the Tier 1 "Must be supported" policy was coming into effect, and after the problems with custom-compiling they decided to go with the completely opposite approach of using only vendor provided packages, in a system called "UnVBuild".

Since their platform of choice was RedHat, this had become troublesome even before I arrived. Worse, in the change from RHEL 4 to RHEL 5, some of the vendor packages for things like XSLT support were dropped entirely leaving us in a bind.

My first instinct was to return to the build everything approach, but the stories (and commit commentary) from that time period reinforced the idea that complete custom build was a bad idea. Office supplies is hardly a sexy industry, and the ability to entice good developers into it is a quite legitimate risk.

So in the end, I went with an approach we ended up nicknaming "HalfBuild". The concept behind it is "Vendor where possible, build where needed".

We use a fairly reasonable chunk of vendor packages under this model. Perl itself, the Oracle client, XML::LibXML and a variety of other things where our version needs are modest and RHEL5 contains it. We also use a ton of C libraries from RHEL5 that are consumed by the CPAN modules, like all the image libraries needed by Imager, some PDF and Postscript libraries, and so on.

One RPM "platform-deps" meta-package defines the full list of these system dependencies, and that RPM is maintained exclusively by server operations so that we as developers are cryptographically unable to add major non-Perl dependencies without consulting them first.

On top of this is one enormous "platform-cpan" RPM package built by the dev team that contains every single CPAN dependency needed by all of our Perl projects.

This package lives in its own home at /opt/cpan and is built using a process rather similar to the core parts of Strawberry Perl. With PREFIX pointing into /opt/cpan, we first build a hand-crafted list of distribution tarballs to upgrade RHEL5 to a modern install of CPAN.pm (without overwriting any normal system files).

We then boot up the CPAN client from /opt/cpan, and tell it to install all the rest of the dependencies from a private internal CPAN mirror which is rsynced by hand specifically for each new CPAN build. This ensures that the build process is deterministic, and that we can fix bugs in the build process if we need to without being forced to upgrade the modules themselves).

The CPAN client grinds away installing for an hour, and then we're left with our "CPAN Layer", which we can include in our application with some simple changes to @INC at the beginning of our bootstrapping module.

The /opt/cpan directory for our project currently weighs in at about 110meg and contains 2,335 .pm files.

Updating /opt/cpan is still something of an exercise even under this model because of potential upgrade problems (Moose forbidding +attributes in roles hit us on our most recent upgrade) but the whole process is fully automated from end to end, and can be maintained by a medium-skill Perl hacker, rather than needing an uberhacker.

Over the last 2 years, we've upgraded it around once every 6 months and usually because we needed to add five or ten more dependencies. We tend to add these new dependencies as early as we can, when work that needs them is confirmed but unscheduled.

We also resort to the occasional hand-copied or inlined pure-Perl .pm file in emergencies, but this is temporary and we only do so about once a year when caught unprepared (most recently Text::Unidecode for some "emergency ascii'fication" of data where Unicode had accidentally slipped in).

While not ideal, we've been quite happy with the /opt/cpan approach so far.

It means we only have to maintain 5 RPM packages rather than 500, and updating it takes one or two man-days per 6 months, if there aren't any API changes in the upgrade.

And most importantly it provides us with much better bus sensitivity, which is hugely important in applications with working lives measured in decades.

Perl-Critic in Depth

Phred — 2010-07-12T23:18:48+00:00

Jeff Thalhammer will be back to speak about Perl-Critic again on July 27nd at 7pm, at the office of LookSmart.

As any Perl developer will tell you, There's More Than One Way To Do It. But over time, we all learn that Some Ways Are Better Than Others.

Perl-Critic is a static source code analyzer based on Damian Conway's book Perl Best Practices. In this presentation, you'll learn how to use Perl-Critic to improve the quality and maintainability of your code. We'll cover the perlcritic command-line tool and the Test::Perl::Critic module, as well as tips for policy selection and incorporating Perl-Critic into your development culture. We'll also demonstrate how to extend Perl-Critic with your own custom policies.

Perl-Critic - http://perlcritic.com/

Jeff Thalhammer's CPAN page - http://search.cpan.org/~thaljef/

Announcement posted via App::PM::Announce

RSVP at Meetup - http://www.meetup.com/San-Francisco-Perl-Mongers/calendar/14082571/

Padre Second Birthday Party 24th-25th of July

Alias — 2010-07-10T16:50:21+00:00

2 years ago this month, Gabor did the first Padre release.

The last 12 months has seen Padre mature from a high-end text editor to a low-end refactoring IDE. We've stolen a number of features from Ultraedit, Komodo and EPIC, and we've invented new features all of our own, making Padre a very fluid and natural place to write Perl in.

We've added support for Perl 6, Template Toolkit, remote file support, more languages, syntax checking, an interactive debugger, a regex editor, and our first half a dozen refactoring tools.

We've also greatly solidified the code. Window integration is now totally solid, we've added a resource locking API, a new filesystem API, a new search API, a new display API, rewritten the threading and background Task subsystem, heavily overhauled the Plugin Manager API and GUI, and added Advanced Preferences and the ability for advanced users to selective disable various Padre bloat/features.

The last couple of months have also seen great improvements in Padre's hackability as well. The new Task 2.0 API lets people write background logic and consume multiple cores of CPU without having to know how threading works, and the new wxFormBuilder plugin lets you build GUI code without having to know Wx (one of the biggest barriers to contributing to Padre).

On the weekend of the 24th-25th of July we would like to invite all Padre developers, users, friends and well-wishers to join us for Padre's Second Birthday Party and Hackathon in the Padre IRC channel at irc://irc.perl.org/#padre or via the Mibbit Web Client.

If you've always been curious about, or interested in hacking on, Padre we'll have a number of developers in channel to help you out.

Personally, I plan to debut the first public release of Padre::Plugin::FormBuilder, and to start ripping out all Padre's older fixed-size dialogs and replacing them with new shiny model-generated sizer-based dialogs that will work much better across all three operating systems.

If you'd like to help out in this effort, I'll be in channel most of the day on both days (Sydney timezone).

I look forward to seeing you all there.

Yahoo provides (awesome) alternative Geocoder to Google's

Alias — 2010-07-06T04:10:06+00:00

As far as I'm concerned, there are three critical things you need in a Geocoder service.

1. Global Coverage

Because when it doesn't have global coverage, it's basically means "America-only" and thus pointless for most of the world.

2. Multiple Matching

For ordinary humans, there's massive power in being able to just search for "1 Oxford Street" without listing any more details. From there, if it is a country-specific application you just change it to "1 Oxford Street, Australia" behind the scenes.

That results in a list of possible locations.

You show the results for the first result, and then a list of "Did you mean:" links for the other results. This lets the application do what people mean most of the time, while making it trivial to recover if it isn't accurate enough.

You can see the effect I'm talking about by running the example query here...

http://geo2gov.com.au/

3. No usage limitations

Google Maps Geocoder can only be used with Google Maps. Fail.

I'm willing to accept a volume limitation like "You have to pay after the first 50,000 requests" but I can't accept a usage limitation.

I'm happy to report that Yahoo's new PlaceFinder service is the first free Geocoder that meets all these primary criteria.

It's global in scope, lets you control the list of results with paging, and doesn't limit the usage to any particular domain.

Now all we need is an Geo::Coder plugin for it.

Please write one for me Lazyweb :)

The next challenge for Perl on Windows (et al)

Alias — 2010-07-01T03:32:16+00:00

Even at this early stage, before any actual installers have appeared, things are looking pretty good for Strawberry Professional Alpha 2.

Padre is starting to firm up as a usable editor, Frozen Bubble works quite well and is completely playable, and I think my wxFormBuilder work is coming along nicely (which bodes well for quickly and easily creating GUI apps in some Alpha 3...)

My biggest remaining concern at the moment though, is also one of the smallest and seemingly trivial issues.

Although we can much more easily build and install large, rich and good looking desktop applications there is still no way to launch these applications without resorting to the command line.

Any iconography that are filled into the start menu in Strawberry Professional will do so because Curtis has put them there himself.

A Perl installation, via the %Config information, lets an installer know where to put libraries, binaries, documentation etc. Shared files in the File::ShareDir model are really just a hack, putting them in an agreed location within lib.

File::HomeDir isn't any use to us either for this. It is designed to let programs deal with user-owned data at run-time, NOT with system integration at install time.

Without the ability to install programs in a way that desktop users can easily launch, our little nascent desktop revolution will never really be able to get up to speed.

Having a post-install step where you need to launch a Windows command line, and then run "padre --desktop" just to install a desktop icon is simply not good enough.

Likewise, having to run Frozen Bubble from the command line is silly as well.

So consider this a challenge to anyone out there that likes tackling tricky puzzles, to try and build a File::Launcher (or whatever you want to call it) that can locate install paths, and be integrated into module installers, so we can make proper use of the Start Menu, Desktop, Quick Launcher, and all the equivalent features on all three major desktop platforms (Windows, Mac, and FreeDesktop.org).

If you build it, Padre will use it.

I hereby report the success of my 2005 Perl Foundation Grant

Alias — 2010-06-15T12:41:57+00:00

In late 2005 I was awarded a grant from the Perl Foundation for "Extending PPI Towards a Refactoring Perl Editor". You can see the original grant here.

http://www.perlfoundation.org/adam_kennedy_extending_ppi_towards_a_refactoring_p erl_editor

My idea at the time was to take the recently completed PPI parser, fill in a variety of different supporting features, build some modules to improve platform integration, and then build a refactoring editor that used PPI and that would fire the imagination of what was possible in Perl.

A year and a half later in early 2007, I was forced to report the failure of the grant.

http://www.perlfoundation.org/march_25_2007_adam_kennedy_s_refactoring_editor_gr ant

Not having anything like the time or experience to build an editor from scratch, I had intended to build a thin refactoring layer over the top of an existing Perl editor. For various reasons, I couldn't make this workable.

To my great surprise and astonishment, the Perl Foundation decided to pay out the grant anyway. Their reasoning was that I had indeed completed all the secondary goals, and the benefits provided by the creation of Vanilla Perl (later to become Strawberry Perl) which was NOT in the grant made the grant worthwhile anyway.

As Curtis noted at the time "Given that he didn't complete his grant, some might be surprised that we've decided to make the final payment...".

And indeed, from time to time the subject has come up here and there about getting paid despite not succeeding. Not often, and not loudly, but occasionally and ongoing.

It has also eaten at me. I hate not delivering on what I say I'll do, and I wanted the refactoring editor for me as much as for everyone else. While Padre has grown and improved, it's never quite met my expectations of what a great Perl IDE should be like.

So it is with great joy that with the landing of the Task 2.0 API I think Padre finally meets the criteria that I set for myself in taking the grant, a Perl Refactoring IDE that would let me give up Ultraedit and present Perl in a modern GUI setting.

Task 2.0 makes Find In Files, Replace In Files, and Incremental Search only a small amount of coding away from being created. These are the last features I need to finally ditch Ultraedit once and for all. The time has arrived, I think, to draw a line in the sand.

And so almost 5 years after the original grant, it is with great joy that I would like to formally report I have completed my 2005 Perl Foundation grant.

I have built, on top of the shoulders of Gabor and the rest of the Padre gang, a refactoring editor that I think will fire the imagination of Perl developers.

And I hope that this corrects and draws to a close one of the most persistently annoying failures in my Perl career.

Please enjoy the features in the new Padre releases as they appear over the next month and a half, leading up to Padre's second birthday party.

I am extremely proud of what the whole Padre gang has achieved in the last year, and I can't wait to stabilise a new group of features just waiting for the background support to firm up a little more.

San Francisco Perl Mongers Twitter Feed

Phred — 2010-06-14T04:41:38+00:00

@sfperlmongers

Hudson for Everybody Else

Phred — 2010-06-10T20:01:52+00:00

Joe McMahon will be talking about Hudson on June 22nd at 7pm, at the office of Mother Jones.

"Continuous integration" sounds like a great idea: you automatically run your build on every checkin, so you know very soon after you've committed if you make a mistake or checked in a bug. However, like any
properly lazy Perl programmer, the last thing you want to do is write more code; you want to take advantage of work that's already done: that's Hudson.

Hudson is a continuous integration server that's easy to set up, customize, and use. Unlike other similar Java-based tools, Hudson is language-agnostic, even well-integrated with other tools.For Perl
projects, with a little assistance from CPAN, it's easy to set up and use for Perl projects. We'll look at a sample setup that covers most of the bases, including a few pointers on making it easy to build and track things
under Hudson, and finish up with a look at using Hudson to get your team involved - even enjoying - continuous integration.

Announcement posted via App::PM::Announce

RSVP at Meetup - http://www.meetup.com/San-Francisco-Perl-Mongers/calendar/13762958/

The CPAN just got a whole lot heavier, and I don't know why

Alias — 2010-06-09T05:49:49+00:00

According to the latest CPANDB generation run, and the associated Top 100 website at http://ali.as/top100, something big just happened to the CPAN dependency graph.

After staying relatively stable for a long time, with the 100th position coming in at around 145 dependencies and incrementing by 1 every 3 months, the "weight" of the entire Heavy 100 set of modules has jumped by 20 dependencies in the last week or so!

The 100th position is now sitting at 166 dependencies, and perennial leader of the Heavy 100 MojoMojo has skyrocketed to an astonishing 330 dependencies. The shape of the "Jifty Plateau" is also much less distinguishable, which suggests it might be more than just a pure +20 across the board.

The question is why?

Is this caused by the restoration of a broken META.yml uncovering formerly ignored dependencies? Or has someone added a new dependency somewhere important accidentally?

Padre::Task 2.0 - Making Wx + Perl threading suck faster

Alias — 2010-06-03T03:37:29+00:00

Some time Real Soon Now, I'll be landing the shiny new second-generation Padre Task API (which is used for background tasks and threading control) onto trunk.

A few people have asked me to give a quick high level overview on how it works, so this will be a quick bottom to top pass over how Padre will be doing threading.

The first-generation Padre::Task API was true frontier programming. While some of us knew Wx and others knew Perl's threads a bit, nobody really had any idea how the two interact. Indeed, the proliferation of bugs we found suggests that Padre has really been the first major application to use both at the same time.

Steffen Müller boldly led the Padre team into this wilderness, putting together a solid threading core loosely inspired by the Process API. He nursed a ton of bug reports and tracked upstream fixes in Wx.pm.

Some of the code was a bit ugly in places, and the threads burned a ton of memory, but it worked and worked well. It worked well enough that Andrew Bramble was later able to extend it to add Service support for long running background tasks with bidirectional communication.

But while we've reduced some of the worst of the memory problems with the "slave driver" methodology, the API has been at the end of its natural life for a while now. It's hard to write new background tasks without knowing both Wx and threads, which limited access to only three or four people.

The goals for the new Padre::Task 2.0 are threefold.

Firstly, to allow the creation of the three distinct background jobs we need in Padre, in a way that doesn't abuse either Wx or Perl's threads mechanism. These three background job types are Task (Request,Wait,Response), Stream (Request,Output,...,Response) and Service (Request,Input,Output,...,Response).

Second, to allow the implementation to have (or have in the future) the theoretically smallest possible memory consumption beyond the minimum overheads imposed by Perl's threading model.

Third, to allow the creation of Wx + threads tasks without the need to learn either Wx or threads. This should open up background tasks to the main body of Padre contributors, beyond the elites, and spur great improvements in Padre's ability to take full advantage of multi-core developer machines.

A fourth bonus goal is to allow us to migrate Padre's backgrounding implementation to something other than threads in the future, without having to change any of the code in the actual tasks next time around. This should also allow the people that don't like Perl threads and want us to use something else to move their arguments from bikeshedding to actual proof-by-code.

After several months of experimenting, I've gone with a somewhat unusual implementation, but one that is completely workable and meets the criteria.

The core of this implementation is a communications loop that allows messages from the parent thread to a child thread, and back again.

The odd thing about this particular communications loop is that the two halves of the loop are done using utterly different underlying mechanisms.

The parent --> child link is done using Perl threads shared variables, in particular Thread::Queue.

Each Padre thread is created via a a Padre::TaskThread parent abstraction, which governs the creation of the real threads.pm thread, but also provides a Thread::Queue "inbox" for each thread. This is inspired by the Erlang micro-threading model for message passing, but is way way heavier.

In this manner, the top level task manager can keep hold of just the queue object if it wants, feeding messages into the queue to be extracted at some unknown place in another thread it has no control over.

Once spawned, each worker thread immediately goes into a run-loop waiting on messages from its message queue. Messages are simply RPC invocations, with the message name being a method name, and the message payload becoming method parameters.

Every thread has the ability to clone itself if passed an empty Padre::TaskThread object to host it. This gets a little weird (you end up passing shared Thread::Queue objects as a payload inside of other shared Thread::Queue objects) but the end result is that you don't have to spawn Perl threads from the parent thread, you can spawn them from child threads but retain the parent's ability to send messages to them regardless of the location on the thread spawn graph they are actually running.

The end result of this trickery is that we can replicate the slave driver trick from the first-generation API. By spawning off an initial pristine thread very early in the process, when the memory cost is small, we can create new threads later by spawning them off this "master" thread and retain the original low per-thread memory cost.

And while we don't do it yet, we can be even tricksier if we want. If we have a thread that has had to load 5, 10, or 20 meg of extra modules, we don't need to load them again. Instead, we could choose to clone that child directly and have a new thread with all the same task modules pre-loaded for us.

The second half of this communications loop is the up-channel, which is done using a totally different Wx mechanism. For communicating messages up to the parent, the Wx documentation recommends the creation of different messages types for each message, and then the use of a thread event.

This care and feeding of Wx turns out to be difficult in practice for non-elites, because you end up registering a ton of different Wx event types, all of which need to be stored in Perl thread-shared variables. And each message needs to be pushed through some target object, and the choice of these can be difficult.

Instead, what we do instead is to register a single event type, and a single global event "conduit". As each message is received by the conduit, it filters down to just appropriately enveloped events and pass only those along to the task manager. The task manager removes the destination header and routes it to the correct task handle in the parent.

Again, messages are done as RPC style calls, with a message type being the method to call, and the payload being the params.

This lets you do a reasonable simple form of cross-thread message passing (at least from the child to the parent anyway).

sub in_child { my $self = shift; $self->message('in_parent', 'Some status value'); } sub in_parent { my $self = shift; my $status = shift; }

We've also put a Padre::ThreadHandle layer over the task itself to do eval'ing and other cleanliness work, in the same way we put a Padre::PluginHandle over every plugin object to protect us against them.

This handle adds some extra value as well, automatically notifying the task manager when a task has started and stopped, and generally keeping everyone sane.

Within this communications loop lives the actual tasks.

The Task API forces work into a very rigid structure. The rigid rules on behaviour is the cost of allowing all the threading magic to happen without the task having to know how it happens.

The basic API looks something like this.

package Some::Task; use strict; use base 'Padre::Task'; # Constructor, happens in the parent at an arbitrary time before the job is run. sub new { my $class = shift; my $self = bless { @_ }, $class; return $self; } # Get ready to serialize, happens in the parent immediately before being sent # to the thread for execution. # Returns true to continue the execution. # Returns false to abort the execution. sub prepare { return 1; } # Execute the task, happens in the child thread and is allowed to block. # Any output data should be stored in the task. sub run { my $self = shift; require Big::Module; $self->{output} = Big::Module::something($self->{input}); return 1; } # Tell the application to handle the output, runs in the parent thread after # the execution is completed. sub finish { my $self = shift; Padre::Current->main->widget->refresh_response( $self->{output} ); return 1; }

To fire off this task, in the widget which commissioned this background work you just do something like this.

sub refresh { my $self = shift; require Some::Task; Some::Task->new( input => 'some value', )->schedule; } sub refresh_response { my $self = shift; my $output = shift; $self->wx_thingy->SetValue($output); }

As you can see here, at no point do you need to care about threads, or event handling, or that kind of thing. You wrap a task class around the blocking part of your app, have the finish push the answer through to your wx component, and then handle it in the wx component.

Of course, this is still a little fidgety. So I'm currently in the process of adding a secondary layer for the common case where the task is created primarily for the purpose of a single parent owner.

The new API layer simplifies things even more. The task takes an "owner" param that represents the Wx component that commissioned it. It does some weaken/refaddr based indexing magic to map the task to the owner object, and then makes sure there is a default finish method to route the answer automatically back to the owner.

The neat part about this is that it takes care of synchronisation problems automatically. If the task finish is called at a time when the Wx component has been destroyed, the answer is automatically dropped and ignored.

The owner can also explicitly declare that the answers from any tasks currently in flight are no longer relevant, and those will be dropped as well.

With the new helper code, your task is shrunk to the following.

package Some::Task; use strict; use base 'Padre::Task'; sub run { my $self = shift; require Big::Module; $self->{output} = Big::Module::something($self->{input}); return 1; } 1;

And the code in the owner component is just this...

sub refresh { my $self = shift; # Ignore any existing in-flight tasks $self->task_reset; # Kick off the new task $self->task_request( task => 'Some::Task', input => 'some value', ); } sub task_response { my $self = shift; my $task = shift; my $output = $task->{output} or return; $self->wx_thingy->SetValue($output); }

This is not the 100% final API look and feel, but it demonstrates the volume of code that you will need to write to do something in the background in the Task 2.0 API.

There's also some interesting opportunities to make it smaller again for very simple cases, by using the generic eval task to inline the execution code.

sub refresh { my $self = shift; # Ignore any existing in-flight tasks $self->task_reset; # Kick off the new task $self->task_request( task => 'Padre::Task::Eval', input => 'some value', run => <<'END_TASK', my $self = shift; require Big::Module; $self->{output} = Big::Module::something($self->{input}); return 1; END_TASK ); } sub task_response { my $self = shift; my $task = shift; my $output = $task->{output} or return; $self->wx_thingy->SetValue($output); }

This would mean we don't need a task class at all, which could be useful in cases where we want to avoid generating lots of tiny 10 line task classes,
or where we want to generate the background code on the fly at run-time.

The downside would be that because the work is bunched inside a single task class, we lose the chance to do worker thread specialisation (where the workers track which tasks they have loaded into memory, so further tasks of the same type can be assigned to them in preference to other threads).

Perhaps we support all the different ways, so that we can pick and choose which option is best on a case by case basis and change them over time. That would certainly fit the Padre rule of "something is better than nothing, don't worry too much about having to break it later".

The downside of this approach, of course, is that breakage. The new Task API will basically kill any plugin currently doing their own background tasks.

The upside, of course, is that once they are fixed these plugins can do those background tasks much more efficiently that before.

The new Task 2.0 API will land in Padre 0.65, which should be out in about 2 weeks.

Aspect 0.90 - Aspect-Oriented Programming for Perl, rebooted

Alias — 2010-05-27T02:44:01+00:00

After two years of development at $work, we've finally completed our program of stability work. To the great credit of our 7 man Perl team, we've managed to reduce the (predicted and actual) downtime to about 1 hour per year for our billion dollar ecommerce site that you've never heard of.

With stability solved, our focus now turns to usability. And unlike downtime, which is easily measurable, we don't really have any decent metrics for usability. So in preparation for this massive new metrics development push, for about a year now I've been hacking away on Aspect.pm, Perl's Aspect-Oriented Programming framework.

My plan is to use this to write performance and metrics plugins for a generic Telemetry system, which would less us turn on and off metrics capturing for a library of different events, driven by configuration on the production hosts.

If you have never encountered Aspect-Oriented Programming before, you can think of it as a kind of Hook::LexWrap on steroids. The core "weaving" engine controls the selection and subjugation of functions, and the generation of the hooking code, and then two separate sugar-enhanced mechanisms are provided on top of this core to define what functions to hijack, and what to do when something we hit one of them.

The main power of AOP is in the ability to build up rich and complex hijacking conditions, and then write replacement code in a way that doesn't have to care about what is being hijacked and when.

This ability to manipulate large numbers of functions at once makes it a great tool for implementing "cross-cutting concerns" such as logging, tracing, performance metrics, and other kinds of functions that would normally result in having a bucketload of single lines scattered all through your implementation.

It also lets you do semi-evil "monkey-patching" tricks in a highly controlled manner, and potentially limited to only one single execution of one single process, without having to touch the target modules all the time everywhere.

For example, the following code shortcuts to false a particular named function only if it isn't being called recursively, and randomly for 1% of all calls, simulating an intermittent failure in the interface to some kind of known-unreliable hardware interface like a GPS unit.

use Aspect; before { $_->return_value(undef); } call 'Target::function' & highest & if_true { rand() < 0.01 };

While the original implementation of Aspect.pm wasn't too bad and did work properly, it had two major flaws that I've been working to address.

Firstly, it was trying to align itself too closely to the feature set of AspectJ. Unlike AspectJ, which does aspect weaving (installation) at compile time directly into the JVM opcodes, Perl's Aspect.pm does weaving at run-time using function-replacement technique identical to Hook::Lexwrap.

This difference means there's certain techniques that AspectJ does that we can't really ever do properly, but there are other options open to us that couldn't be implemented in Java like the use of closures.

For example, with Aspect.pm you can do run-time tricks like that rand() call shown above. Since the if_true pointcut is also a closure, you can even reach have it interact with lexical variables from when it was defined.

This makes for some very interesting options in your test code, allowing lighter alternatives to mock objects when you only want to mock a limited number of methods, because you can just hijack the native classes rather than building a whole set of mock classes and convincing your main code to make use of them.

use Aspect; # Hijack the first four calls to any foo_* method my @values = qw{ one two three four }; before { $_->return_value(shift @values); } call qr/^Target::foo_\w+$/ & if_true { scalar @values };

The second problem I've fixed is the speed. The original implementation made heavy use of object-orientation at both weave-time and run-time.

Logically this is a completely defensible decision. However, from a practical standpoint when you are going to be running something inline during function calls, you don't really want to have to run a whole bunch of much more expensive recursive methods.

So I've restructured much of the internals to greatly improve the speed.

I've dumbed down the inheritance structure a bit to inline certain hot calls at the cost of a bit more code.

I've also implemented a currying mechanism, so that run-time checks can safely ignore conditions that will always be true due to the set of functions that ended up being hooked.

And finally, rather than do method calls on the pointcut tree directly, we now compile the pointcuts down to either a string parameter-free equivalent functions.

In the ideal case the entire pointcut tree is now expressed as a single-statement Perl expression which makes no further function calls at all.

This not only removes the very expensive cost of Perl's function calls, but having everything in a single expression also exposes the boolean logic to the lower-level opcode compiler and allows culling and shortcutting at a much lower level.

With these two major improvements in place, I've also taken the opportunity to expand out the set of available conditions to include a number of additional rules or interest to Perl people that don't have equivalents in the Java world like tests for the wantarray'ness of the call.

use Aspect; # Trap calls to a function we know behaves badly in list or void contexts before { $_->exception("Dangerous use of function in non-scalar context"); } call 'Target::function' & ! wantscalar;

In addition to the expanded functionality, I've also massively expanded the test suite and written up full (or at least nearly full) POD for all classes in the entire Aspect distribution.

To celebrate this near-completion of the first major rewrite phase, I've promoted the version to a beta'ish 0.90.

There's still some work to do before release 1.00. In particular, at present the point context objects still work to a single unified set of method calls, regardless of the advice type.

This means that advice code can attempt to interrogate or manipulate things that are irrelevate or outright illegal in certain contexts.

Cleaning this up will elevate these bad context usages to "method not found" exceptions, and will also allow prohibiting the use of pointcut conditions in contexts where they are meaningless (such as exception "throwing" clauses for before { } advice when no such exception can exist).

It should also allow faster implementations of the point context code, because it won't need to take multiple situations into account and can specialise for each case.

Once this is completed, and some additional test scripts have been adding for particularly nasty cases, I'll be able to do the final 1.00 release.