DAxelrod's Friends' Journals

DBD::SQLite 1.31 releasing next week and may break your code

Alias — 2010-09-09T02:11:54+00:00

After 6 or 7 months (mainly waiting around for the next "recommended upgrade instruction" from the SQLite project) the latest DBD::SQLite release should occur next week.

You can get the 1.30_06 release candidate from the CPAN, or from the following URL if your mirror hasn't synced yet.

http://svn.ali.as/cpan/releases/DBD-SQLite-1.30_06.tar.gz

Apart from the normal batch of SQLite upgrades (from 3.6.22 to 3.7.2), bug fixes, and minor enhancements, this release has two changes that may break your code.

These changes have been in the dev releases for some time, but you may want to take the opportunity to test more intensively if you use either of the following features.

1. BLOB columns with UTF8 content

- Resolved #54271: Inserting a string with utf-8 flag on corrupts BLOB data; now BLOB data is always stored as bytes (without the utf-8 flag) even if it has the flag set (ISHIGAKI)

2. FTS3 queries

- Added support for FTS3 tokenizers written in Perl. Added tests and documentation on how to use FTS3. Changed compilation flag to use the recommanded -DSQLITE_ENABLE_FTS3_PARENTHESIS *** MAY POSSIBLY BREAK OLD APPLICATIONS THAT ALREADY USED FTS3 *** (DAMI)

If you are currently using FTS3, please see DBD::SQLite::FTS3Transitional which contains a helper function for automatically upgrading old FTS3 queries to the new syntax.

use Perl; Shutting Down Indefinitely

pudge — 2010-09-08T22:07:47+00:00

See here.

Should Module::Install move to explicit plugin declaration?

Alias — 2010-09-06T02:26:06+00:00

Module::Install has been through a long period of gradual stability over the last year, without any really dramatic improvements to the grammar or APIs.

With the more urgent "it doesn't work with blah" stuff mostly solved now, one of the big remaining issues is around error clarity and excessive magic.

For example, some random author that is trying to checkout a Catalyst project needs:

1. To have Module::Install installed.
2. To have Module::Install::Catalyst installed.

In the case of the former, you get the semi-cryptic but at least standard "Can't find inc/Module/Install.pm in @INC" message, so the error is resolvable.

But in the latter case, you're likely to get something like "Unknown command 'catalyst_ignore'", with no real obvious resolution mechanism.

I think this idea of automatic plugin discovery is starting to hit it's limits in terms of clarity.

And so I'd like to do something counter to my natural instincts here, and make M:I more verbose.

I'm thinking of something like the following for explicitly declaring the use of a non-core Module::Install extension.

use inc::Module::Install qw{ Catalyst XSUtil };

This would both allow M:I to error with a much more meaningful error when you don't have a plugin, and also prevent the loading of unused plugins which should prevent accidental plugin collisions (some of which I've seen occurring in the CPAN Testers machines).

Thoughts?

Speaking at Microsoft TechEd - Any issues you want raised?

Alias — 2010-08-20T03:20:46+00:00

Next week I will at Microsoft's TechEd Australia event, courtesy of Microsoft Australia and Microsoft Open Source Labs.

More specifically, I'll be attending the Open Source mini-conf and discussion day on Tuesday, and presenting in the Community Presentations to Microsoft session on the current state of Perl and Windows on Wednesday.

Likely topics will include a review of the first year of the CPAN Testing Lab and a second-generation based on their Cloud Services, free code signing certificates for open source developers, and what issues are slowing us down or blocking progress.

So consider this your opportunity to raise any outstanding issues you have with Microsoft and Perl. What problems are you still seeing, what would like fixed or changed, and what is on your want-to-have list?

I'll try to address as many of your issues as possible in the time I have available with them (which is actually pretty substantial).

Class::XSAccessor now even faster'er

Alias — 2010-08-16T03:04:05+00:00

The new 1.07 release of Class::XSAccessor mentions the use a new somewhat-evil technique for making the code even faster than it was previously.

But how much faster is it?

The following are being run on a fairly typical corporate Windows XP machine, with Strawberry Perl 5.10.1 and thread support.

First, some benchmarks using the previous 1.05 release (two runs)

Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true... accessor_get: 1 wallclock secs ( 2.51 usr + 0.00 sys = 2.51 CPU) @ 3976143.14/s (n=10000000) accessor_set: 2 wallclock secs ( 3.09 usr + 0.00 sys = 3.09 CPU) @ 3233107.02/s (n=10000000) constructor: 16 wallclock secs (15.67 usr + 0.00 sys = 15.67 CPU) @ 638080.65/s (n=10000000) false: 2 wallclock secs ( 1.91 usr + 0.00 sys = 1.91 CPU) @ 5243838.49/s (n=10000000) getter: 1 wallclock secs ( 2.34 usr + 0.00 sys = 2.34 CPU) @ 4266211.60/s (n=10000000) predicate: 1 wallclock secs ( 2.38 usr + 0.00 sys = 2.38 CPU) @ 4210526.32/s (n=10000000) setter: 2 wallclock secs ( 3.27 usr + 0.00 sys = 3.27 CPU) @ 3061849.36/s (n=10000000) true: 1 wallclock secs ( 1.80 usr + 0.00 sys = 1.80 CPU) @ 5564830.27/s (n=10000000) Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true... accessor_get: 3 wallclock secs ( 2.51 usr + 0.00 sys = 2.51 CPU) @ 3976143.14/s (n=10000000) accessor_set: 3 wallclock secs ( 3.14 usr + 0.00 sys = 3.14 CPU) @ 3183699.46/s (n=10000000) constructor: 15 wallclock secs (15.73 usr + 0.00 sys = 15.73 CPU) @ 635566.29/s (n=10000000) false: 2 wallclock secs ( 1.86 usr + 0.00 sys = 1.86 CPU) @ 5379236.15/s (n=10000000) getter: 3 wallclock secs ( 2.50 usr + 0.00 sys = 2.50 CPU) @ 4000000.00/s (n=10000000) predicate: 3 wallclock secs ( 2.47 usr + 0.00 sys = 2.47 CPU) @ 4050222.76/s (n=10000000) setter: 4 wallclock secs ( 3.13 usr + 0.00 sys = 3.13 CPU) @ 3200000.00/s (n=10000000) true: 2 wallclock secs ( 1.98 usr + 0.00 sys = 1.98 CPU) @ 5037783.38/s (n=10000000)

And now again with the new 1.07 release.

Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true... accessor_get: 2 wallclock secs ( 1.75 usr + 0.00 sys = 1.75 CPU) @ 5711022.27/s (n=10000000) accessor_set: 1 wallclock secs ( 2.69 usr + 0.00 sys = 2.69 CPU) @ 3721622.63/s (n=10000000) constructor: 15 wallclock secs (15.62 usr + 0.00 sys = 15.62 CPU) @ 640000.00/s (n=10000000) false: 1 wallclock secs ( 1.28 usr + 0.00 sys = 1.28 CPU) @ 7806401.25/s (n=10000000) getter: 1 wallclock secs ( 1.56 usr + 0.00 sys = 1.56 CPU) @ 6397952.66/s (n=10000000) predicate: 2 wallclock secs ( 1.92 usr + 0.00 sys = 1.92 CPU) @ 5205622.07/s (n=10000000) setter: 3 wallclock secs ( 2.50 usr + 0.00 sys = 2.50 CPU) @ 4000000.00/s (n=10000000) true: 2 wallclock secs ( 1.55 usr + 0.00 sys = 1.55 CPU) @ 6464124.11/s (n=10000000) Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true... accessor_get: 2 wallclock secs ( 1.78 usr + 0.00 sys = 1.78 CPU) @ 5614823.13/s (n=10000000) accessor_set: 3 wallclock secs ( 2.63 usr + 0.00 sys = 2.63 CPU) @ 3809523.81/s (n=10000000) constructor: 16 wallclock secs (15.69 usr + 0.00 sys = 15.69 CPU) @ 637429.88/s (n=10000000) false: 2 wallclock secs ( 1.22 usr + 0.00 sys = 1.22 CPU) @ 8203445.45/s (n=10000000) getter: 2 wallclock secs ( 1.53 usr + 0.00 sys = 1.53 CPU) @ 6535947.71/s (n=10000000) predicate: 2 wallclock secs ( 1.78 usr + 0.00 sys = 1.78 CPU) @ 5614823.13/s (n=10000000) setter: 2 wallclock secs ( 2.56 usr + 0.00 sys = 2.56 CPU) @ 3903200.62/s (n=10000000) true: 2 wallclock secs ( 1.48 usr + 0.00 sys = 1.48 CPU) @ 6738544.47/s (n=10000000)

The numbers are pretty impressive.

The 'accessor', 'setter', 'predicate' and 'true' methods are about 25% faster, while 'getter' is a whopping 60% faster and (curiously) 'false' is about 50% faster as well.

Constructors are really the only thing that hasn't changed.

Impressive work, even if the code is a bit risky.

Why does Object::Tiny only support getters

Alias — 2010-08-15T01:39:50+00:00

http://perlalchemy.blogspot.com/2010/08/objecttinyrw-and-moosexnonmoose.html

Zbigniew Lukasiak tries out Object::Tiny and wonders why it is that I didn't allow for the creation of setters when it is only a one line change.

Like most ::Tiny modules the reason is a bit complex and the result of compromises.

Object::Tiny began as an attempt to create a lighter, faster, version of Class::Accessor. A way to bulk-generate the accessor code I had to type over and over again.

However, where I differ is a strong preference for light and elegant API design.

And so I decided to implement mine with as little implementation code as possible, and as little API code as possible.

Once you have decided to go down the simplicity path, there's a couple of standard techniques you often end up using.

The first and most important is state reduction.

In their introduction to Erlang, the founders of that language describe state as one of the main sources of failures in programs. And so anything that removes state, at the very least unnecessary state, is a positive. Especially if the state reduction also results in code reduction, and a reduction in computation.

So take the following example, where we create an object with some attributes and then run some code that will use those object attributes..

my $object = Class->new; $object->foo(1); $object->bar(2); $object->do_something;

This is a use case that we see fairly often, but it's really quite horrible code. It is really only the object-oriented equivalent of something like the following.

our $Object::foo = 1; our $Object::bar = 2; do_something('Object');

It is especially bad code if the following code would throw an exception.

my $object = Class->new; $object->do_something;

If this blows up, then you are REALLY doing something wrong, because you have allowed the creation of completely invalid objects. Now anybody taking one of these objects as a parameters needs to do with following.

sub foo { my $object = shift; unless ( $object->isa('Class') and defined $object->foo and $object->foo > 0 and defined $object->bar and $object->bar > 2 ) { die "Invalid object"; } }

If you are going to create an object for something, you HAVE to be sure that the objects are trustworthy.

And so you should never allow objects to exist that are invalid. EVERY object should be a valid object.

At the absolute minimum objects should be able to default every attribute to something reasonable and unlikely to cause problems.

But this still results in excess and wasteful work, because the object has to transition through two or more states.

You start with an object with parameters and defaults, and you validate them. And then you change on of the attributes immediately, validating it AGAIN. In the mean time, your object exists in a state that it will never actually be used in.

And so everywhere you possibly can, you should be setting attributes in the constructor rather than afterwards.

my $object = Class->new( foo => 1, bar => 2, ); $object->do_something;

Less state, less complexity, less CPU, and less bugs.

If we accept this model of pushing all the configuration into the object up front to reduce state, then why change the object arbitrarily?

In fact, anything that you ARE going to change should be done under very controlled conditions.

It should require a dedicated method to apply the change, it should require validation, and work. It shouldn't be trivial, and it shouldn't be automatic.

If I had my way, Moose would set is => 'ro' by default, to make people think before they go about simply allowing stuff to change.

It also happens to let you shrink down the API markedly.

There are three potential use cases available when implementing accessors. Everything readonly, everything readwrite, or mixed.

With Object::Tiny, I was aiming for the smallest possible code.

Implementing either all-readonly or all-readwrite can be done with the following.

use Class qw{ foo bar };

By contrast, if we want to allow mixed readonly and readwrite, we would need some way of distinguishing. Something like the following.

use Class { readonly => [ 'foo' ], readwrite => [ 'bar' ], }; use Class [ qw{ foo } ], [ { bar } ]; use Class { foo => 'ro', bar => 'rw', };

No matter how you try, there's always an inherent additional element of complexity that results from the split between them.

And so the decision to go with all-readonly in Object::Tiny is a combination of these two issues.

If went with all-readwrite, I'm practically encouraging bad behaviour and more bugs. If I went with mixed accessors, the API would remain relative complex.

In the end, the best way to achieve both API simplicity and code safety is to only provide read-only accessors, and anything more complex should require both though and effort.

use Perl;

pudge — 2010-08-11T23:34:11+00:00

I am no longer working for Slashdot/Geeknet as of September 30. I am actively seeking new employment. Unless you want to hire me, you don't need to care, unless you also care about use Perl;, because it has been generously hosted by Geeknet since I started the site over 10 years ago, shortly after I was hired by Andover.Net to work on Slashdot.

Long story short, I have not done much with the site in recent years, so my options at this point are to do nothing; migrate the site to a new server and keep it running as-is; or take the data and do something with it on a new site. Or something I haven't thought of.

I am hereby accepting proposals for what to do with use Perl;. In short, I would like to donate it to someone who will give it a good home. If you're interested, give me your best pitch.

Cross-posted on <pudge/*>.

Matt Trout, aka mst, is insane

pudge — 2010-08-11T16:41:54+00:00

Wow. I occasionally, but not too often, go into #perl. Very busy with family and life. So I go in today, and for no reason, mst bans me and tells me to not come back.

What's up with him being such an irrational dick?

Trailer Theory - Reinvented for Ignite Sydney as Economics

Alias — 2010-08-04T05:40:53+00:00

Back in 2005 in only my fifth use.perl post ever, I outlined an idea I had been developing for a couple of years that I called "Trailer Theory".

A few years ago on my Portable Perl world hack'a'tour, I took with me a lightning talk version of the concept. It was a pretty crude talk but was received, it seemed, fairly well by the development community.

Since that trip, and inspired by the unexpected conversion of my "Perl is unparsable" claims in the PPI docs into a formal mathematical proof (complete with "Kennedy's Lemma") I've been wondering if this "Trailer Theory" idea could really be developed as a proper scientific proof, and if so what would that look like.

A couple of months ago I presented a new version of the talk at Ignite Sydney, speaking to an mixed audience of Twitterati, social media, advertising and journalist types.

I've rebuilt the talk from scratch and tried to outline the same idea, but in the form of a kind of layman's Economics Proof.

I hope you enjoy the result.

Adam Kennedy - Using Economics to make movies suck less

Notes for other speakers:

1. Ignite advances slides every 15 seconds, no clickers allowed. This turns out to take a shitload of practice to get right.

2. When someone says to you "Here's your mark, you need to stay on this line to be in the fixed spot" it helps to pay attention. FAIL :)

Protesting the Moose with sugary alternatives

Alias — 2010-08-03T05:09:02+00:00

At work, we've been experimenting with Moose because in our enormous and complex codebase we think we can probably benefit a lot from the extra rigour that it brings.

Before I continue, let me note that ours is a typical mod_perl enterprise setup with 6gig of memory per machine, so any memory consumed before the Apache fork is essentially free.

So none of the issues people (including me) have with startup code and memory consumption apply in this case, and I won't be addressing performance issues in this post.

The consensus of the half-dozen people at work is how Moose tries to look like a declarative extension, but doesn't actually act like it.

The following is what Moose seems to have been aiming for.

package Foo; use Moose; extends 'Bar'; with 'Role1'; with 'Role2'; has this => ( is => 'ro', isa => 'Str', ); has that => ( is => 'ro', isa => 'Str', ); 1;

Unfortunately, this is what we've had to do instead.

package Foo; use Moose; BEGIN { # When we use Catalyst extends 'Bar'; } has this => ( is => 'ro', isa => 'Str', ); has that => ( is => 'ro', isa => 'Str', ); with 'Role1'; with 'Role2'; no Moose; __PACKAGE->meta->make_immutable;

This "real Moose" code totally spoils the dream of what we felt like we were going to get when we started to play with it.

Most of our current options for fixing this amount to either.

a) Add this extra dependency that will unscrew one of the other of the problems (namespace::autoclean)

b) Use this SECOND heavy sugar layer on top of the FIRST sugar layer, on top of Class::MOP.

Is fixing the syntax or writing light weight sugar really so hard?

As a kind of protest, I tried it for myself and managed to create MooseX::Atom.

This still has some flaws, but the current equivalent of the above would just be this.

package Foo; use MooseX::Atom [ extends => 'Bar', with => 'Role1', with => 'Role2', has => [ this => ( is => 'ro', isa => 'Str', ), ], has => [ that => ( is => 'ro', isa => 'Str', ), ] ]; 1;

You can do the same thing for roles with MooseX::Role::Atom.

Now clearly, this might have some issues. It's the work of an hour and not a whole lot of thought.

But it's still light and clean, with all the class spec in one place up the top where people are used to seeing the declarative stuff in Perl modules.

Perhaps something like this might be a little better...

package Foo; use MooseX::Hash { extends => 'Bar', with => [ 'Role1', 'Role2' ], default => { is => 'ro' }, has => { this => { isa => 'Str' }, that => { isa => 'Str' }, }, ); 1;

Help?

Alias — 2010-07-29T13:23:44+00:00

(Before I begin, I should clarify I did not write this code, I'm just trying to maintain it)

The following error is the first thing spat out by make test for the Padre sync server, located at http://svn.perlide.org/padre/trunk/Madre-Sync.

Wasn't the move of Catalyst to Moose going to make things easier?

Can someone explain how you debug this?

I get the basics, I can see the "Can't locate Madre/Sync/Schema.pm". But that file should be, I think, automatically generated. And I don't really get how to dig down the 75 caller levels from the start to the end to work out where the actual functionality is failing...

not ok 1 - use Catalyst::Test; # Failed test 'use Catalyst::Test;' # at t\01app.t line 7. # Tried to use 'Catalyst::Test'. # Error: Couldn't load class (Madre::Sync) because: Couldn't instantiate component "Madre::Sync::Model::padreDB", "Can't locate Madre/Sync/Schema.pm i n @INC (@INC contains: blib\lib blib\arch C:/strawberry/perl/lib C:/strawberry/perl/site/lib C:\strawberry\perl\vendor\lib .). at C:\strawberry\perl\vendor \lib/Class/MOP.pm line 132 # Class::MOP::load_first_existing_class('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137 # Class::MOP::load_class('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Catalyst/Model/DBIC/Schema/Types.pm line 21 # Catalyst::Model::DBIC::Schema::Types::__ANON__[C:\strawberry\perl\vendor\lib/Ca talyst/Model/DBIC/Schema/Types.pm:21]('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Moose/Meta/TypeCoercion.pm line 63 # Moose::Meta::TypeCoercion::__ANON__[C:\strawberry\perl\vendor\lib/Moose/Meta/Ty peCoercion.pm:67]('Madre::Sync::Schema') called at C:\strawberry\per l\vendor\lib/Moose/Meta/TypeCoercion.pm line 97 # Moose::Meta::TypeCoercion::coerce('Moose::Meta::TypeCoercion=HASH(0x4a2b444)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Moose /Meta/TypeConstraint.pm line 90 # Moose::Meta::TypeConstraint::coerce('Moose::Meta::TypeConstraint=HASH(0x4a29854 )', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/M ooseX/Types/TypeDecorator.pm line 206 # eval {...} called at C:\strawberry\perl\vendor\lib/MooseX/Types/TypeDecorator.pm line 205 # MooseX::Types::TypeDecorator::AUTOLOAD('MooseX::Types::TypeDecorator=HASH(0x4a3 1424)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\l ib/Moose/Meta/Attribute.pm line 743 # Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x4b467 fc)', 'Madre::Sync::Schema', 'Madre::Sync::Model::padreDB=HASH(0x4db 7c64)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Attribute.pm line 398 # Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(0 x4b467fc)', 'Moose::Meta::Instance=HASH(0x4db7ed4)', 'Madre::Sync::M odel::padreDB=HASH(0x4db7c64)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Class.pm line 567 # Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/C lass.pm line 540 # Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Class.pm line 256 # Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Object.pm lin e 25 # Moose::Object::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at generated method (unknown origin) line 4 # Catalyst::Model::DBIC::Schema::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at C:\strawberry\perl\vendor\lib/MooseX/ Traits/Pluggable.pm line 131 # MooseX::Traits::Pluggable::new_with_traits('Madre::Sync::Model::padreDB', 'Madre::Sync') called at C:\strawberry\perl\vendor\lib/CatalystX/Componen t/Traits.pm line 146 # CatalystX::Component::Traits::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Cl ass/MOP/Method/Wrapped.pm line 48 # Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:49]('Madre::Sync::Model::padreDB', 'Madre::Sync', ' HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 89 # Catalyst::Model::DBIC::Schema::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/C atalyst.pm line 2502 # eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2502 # Catalyst::setup_component('Madre::Sync', 'Madre::Sync::Model::padreDB') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2416 # Catalyst::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 54 # Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:64]('Madre::Sync') called at C:\strawberry\perl\ven dor\lib/Class/MOP/Method/Wrapped.pm line 89 # Madre::Sync::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 1142 # Catalyst::setup('Madre::Sync') called at blib\lib/Madre/Sync.pm line 62 # require Madre/Sync.pm called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 114 # Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:118]() called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 74 # eval {...} called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 67 # Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125 # Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137 # Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24 # Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493 # Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424 # Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S ub/Exporter.pm line 742 # Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per l\vendor\lib/Catalyst/Test.pm line 112 # Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2 # main::BEGIN() called at blib\lib/Madre/Sync.pm line 0 # eval {...} called at blib\lib/Madre/Sync.pm line 0 # eval 'package main; # use Catalyst::Test @{$args[0]}; # 1; # # ;' called at C:/strawberry/perl/lib/Test/More.pm line 858 # Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p m line 833 # Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7 # at C:\strawberry\perl\vendor\lib/MooseX/Types/TypeDecorator.pm line 208 # MooseX::Types::TypeDecorator::AUTOLOAD('MooseX::Types::TypeDecorator=HASH(0x4a3 1424)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\l ib/Moose/Meta/Attribute.pm line 743 # Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x4b467 fc)', 'Madre::Sync::Schema', 'Madre::Sync::Model::padreDB=HASH(0x4db 7c64)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Attribute.pm line 398 # Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(0 x4b467fc)', 'Moose::Meta::Instance=HASH(0x4db7ed4)', 'Madre::Sync::M odel::padreDB=HASH(0x4db7c64)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Class.pm line 567 # Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/C lass.pm line 540 # Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Class.pm line 256 # Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Object.pm lin e 25 # Moose::Object::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at generated method (unknown origin) line 4 # Catalyst::Model::DBIC::Schema::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at C:\strawberry\perl\vendor\lib/MooseX/ Traits/Pluggable.pm line 131 # MooseX::Traits::Pluggable::new_with_traits('Madre::Sync::Model::padreDB', 'Madre::Sync') called at C:\strawberry\perl\vendor\lib/CatalystX/Componen t/Traits.pm line 146 # CatalystX::Component::Traits::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Cl ass/MOP/Method/Wrapped.pm line 48 # Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:49]('Madre::Sync::Model::padreDB', 'Madre::Sync', ' HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 89 # Catalyst::Model::DBIC::Schema::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/C atalyst.pm line 2502 # eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2502 # Catalyst::setup_component('Madre::Sync', 'Madre::Sync::Model::padreDB') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2416 # Catalyst::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 54 # Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:64]('Madre::Sync') called at C:\strawberry\perl\ven dor\lib/Class/MOP/Method/Wrapped.pm line 89 # Madre::Sync::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 1142 # Catalyst::setup('Madre::Sync') called at blib\lib/Madre/Sync.pm line 62 # require Madre/Sync.pm called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 114 # Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:118]() called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 74 # eval {...} called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 67 # Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125 # Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137 # Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24 # Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493 # Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424 # Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S ub/Exporter.pm line 742 # Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per l\vendor\lib/Catalyst/Test.pm line 112 # Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2 # main::BEGIN() called at blib\lib/Madre/Sync.pm line 0 # eval {...} called at blib\lib/Madre/Sync.pm line 0 # eval 'package main; # use Catalyst::Test @{$args[0]}; # 1; # # ;' called at C:/strawberry/perl/lib/Test/More.pm line 858 # Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p m line 833 # Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7"Compilation failed in require at C:\strawberry\perl\vendor\lib/Class /MOP.pm line 114. # at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 121 # Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:125]('Couldn\'t instantiate component "Madre::Sync::Model::padreDB"...') called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 98 # Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125 # Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137 # Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24 # Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493 # Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424 # Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S ub/Exporter.pm line 742 # Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per l\vendor\lib/Catalyst/Test.pm line 112 # Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2 # main::BEGIN() called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 2 # eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 2 # eval 'package main; # use Catalyst::Test @{$args[0]}; # 1; # # ;' called at C:/strawberry/perl/lib/Test/More.pm line 858 # Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p m line 833 # Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7 # BEGIN failed--compilation aborted at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2.

For people running Perl conferences

Alias — 2010-07-25T12:55:04+00:00

Leo Lapworth comments in "For speakers at Perl conferences" that you should record your own talks at conferences, because organisers cannot be trusted to release the videos they take of you.

I wholeheartedly agree. I've spoken at numerous conferences, including several large ones, and only in about 10-20% of cases have the videos of me EVER appeared.

I'm amazed that conference organisers can put in so much effort into recording (dozens of tapes, multiple cameras, multiple operators) and then never produce anything as a result.

The only time I'm aware of that a full talk of mine has appeared online was at a Linux.conf.au, who had a dedicated video team of eight people.

Where is all this footage, and why does it never get processed? Why even bother recording it? If you don't have time, send me the raw tape of my own talk and I will do it myself if needed.

YAPC::NA? You listening?

A reminder - Padre Second Birthday Hackathon this weekend

Alias — 2010-07-23T03:53:28+00:00

This is just a quick reminder that this weekend is the Padre's second birthday party and hackathon.

If you are a Padre user, please drop in and say hello to the team. We'd love to hear how you are using Padre and where your main needs are for the next year.

If you are interested in trying out Padre for the first time, or trying your hand at improving it for the first time, it's a great time to get started because we'll have plenty of people around to provide guidance and advice.

Some of the plans for this weekend are to bring all the plugins up to date with the latest versions of the plugin API, to start the merge of the ConfigSync branch, and if we can, to start on the Madre server that will serve as the ConfigSync and Telemetry server for Padre.

I look forward to seeing you there!

Profiling your website while live on a production cluster

Alias — 2010-07-22T02:27:33+00:00

As I mentioned in my last post, by day I wrangle a large web application that occasionally verges on being too complex to for mere humans to understand.

Curiously, because it is private and the income rate is high (we probably average something like $5 in gross turnover per page view) we don't have to deal with a lot of servers to deliver it, by internet standards anyway.

But the application is still way too big to profile easily by normal methods, and certainly on production it's way too heavy, even if we applied targeted profiling using something like Aspect::Library::NYTProf.

Between the web servers, transaction server, database, search engine and cache server, we are probably only dealing with 12 servers and 30 CPUs. Of course, these servers are all horribly expensive, because they are all server-virtualised, network-virtualised, doubly redundant (high-availability + disaster-recovery) and heavily monitored with high end support contracts.

One of our most sensitive issues is database load.

We have a ton of database tables (about 200) and lots of medium sized queries running across them. One of our main failure modes is that some deep change to code boosts the quantity of some significant query, which stresses the database enough to cause contention and lock-storms, leading to site failure.

Complicating things, big parts of some pages are embedded in other pages. So attributing load and lag to one part of the website, or to Oracle, is tricky and hard to benchmark in advance (although we do load test the main load paths to catch the worst cases).

For a long time, we've had a mechanism for zoned profiling the production site, so we can allocate wallclock costs to different general areas of the site.

But it is fragile and unreliable, requiring perfect maintenance and every developer to remember to write this kind of thing everywhere.

# Embed a foo widget in the current page $perf->push_timing('foo'); foo($bar); $perf->pop_timing;

Since you don't know this profiling system exists unless you've seen it somewhere else in the code before, and it's hard to care about something that is orthogonal to the problem you are actuall solving, this mechanism has degraded over time. While we still get some pretty cacti graphs showing load breakdown, they are highly unreliable and you can never be entirely sure if the load attribution is correct.

This kind of performance monitoring as a "cross-cutting concern" is a textbook use case for Aspect-Oriented Programming, and so in our Christmas 2009 code freeze window I set about trying to rewrite the push/pop_timing profiler using Aspect.pm.

Unfortunately, Aspect.pm turned out to be a bit too slow and naive for my needs. But after a 6 month delay to do a 90% rewrite of Aspect.pm, I now finally have something sophisticated enough (and fast enough) to meet my needs.

So today I'm releasing the shiny new Aspect::Library::ZoneTimer, which will serve as the main plank of our new production profiling system.

The idea behind ZoneTimer is to define each performance zone as a pointcut. The aspect will track the movement of your code across each zone boundaries and build a running total of the exclusive time spent in each performance zone.

When your program exits the top-most performance zone, the totals will be sent to a handler function.

A typical use of the ZoneTimer aspect looks something like this

use Aspect; aspect ZoneTimer => ( zones => { main => call 'MyProgram::main' & highest, search => call 'MyProgram::search', widgets => call qr/^MyProgram::widget_.*/, templates => call 'MyProgram::render', dbi => call qr/^DB[DI]::.*?\b(?:prepare|execute|fetch.*)$/, system => ( call qr/^IPC::System::Simple::(?:run|runx|capture)/ | call 'IPC::Run3::run3' | call qr/^Capture::Tiny::(?:capture|tee).*/ ) }, handler => sub { my $top = shift; # "main" my $start = shift; # [ 1279763446, 796875 ] my $stop = shift; # [ 1279763473, 163153 ] my $exclusive = shift; # { main => 23123412, dbi => 3231231 } print "Profiling from zone $top\n"; print "Started recording at " . scalar(localtime $start) . "\n"; print "Stopped recording at " . scalar(localtime $stop) . "\n"; foreach my $zone ( sort keys %$exclusive ) { print "Spent $exclusive->{$zone} microseconds in zone $zone\n"; } }, );

This example breaks out the cost of a typical small web application into a general zone, a special zone for the search page, and then splits the costs of generating widgets, rendering the HTML template, waiting for the database, and making calls to the underlying system.

Start and stop times are returned as a two element array exactly as returned by C, and the exclusive zone totals are returned as integer microseconds (all math is done in these integer microseconds to prevent floating point corruption).

The use of Aspect allows us to easily mix special cases via the pointcuts, such as the use of "highest" which makes sure that "main" only matches the first time it is seen, and any widget which that does a re-entry into main still has that cost attributed to the widget. We've also hooked into multiple system call modules to measure system call cost, because we know different modules our program consumes will use different methods for interacting with the system.

While the handler I've shown here will just print out a summary of the call, in our environment at work the profile report handler will format the exclusive times into a special log message and then send it via the super-quick and non-blocking Log::Syslog::Fast module to the UDP localhost syslog service, where it is mirrored to disk for debugging use, and then forwarded on to our main company-wide syslog server.

On our setup, we can then use an excellent commercial log analysis product called Splunk (limited free version also available if you want to try it out) to do our tracking and trending of the performance for the entire application across the entire cluster.

The nicest thing about the Aspect system is that it scales the performance cost and complexity risk directly to the complexity of the use case you are using it for.

If it turns out that the Aspect hooks are too intrusive and causing a performance drain, or accidentally causing some weird edge case behaviour, you can simply turn off that the Aspect and restart the servers and the performance penalty just vanishes.

Maintenance of the profiler zones is really easy, because they are all listed clearly in one place.

Depending on your use case, you could even define the performance zones in your website configuration, and then adjust the profiler zone boundaries directly on production (although I'm not sure I'd want to do that in our specific case, since we're fairly paranoid about untested code going into production).

This ZoneTimer is just the first of several Aspect-based tools I plan to release over the next few months. If you get a chance to try out any of these new toys, I'd love to hear feedback on how well they work for you.

In addition to the value of the ZoneTimer aspect itself, one other thing that some people might find of interest is just how little code it took to get this all working.

The entire Aspect::Library::ZoneTimer was implemented in about 80 lines of code, which you can see at http://cpansearch.perl.org/src/ADAMK/Aspect-Library-Timer-0.04/lib/Aspect/Librar y/ZoneTimer.pm

This is small enough that you could just clone the module and tweak it for your own use case, if you wanted to creat a new and tricky customised profiler.

How we deploy massive Perl applications at work

Alias — 2010-07-20T02:25:21+00:00

Every now and then, we hear people talking about mechanisms for doing Perl in a commercial environment and how they deal with packaging and dependencies.

This is mine.

At Corporate Express, our main Perl application is a 250,000 line non-public monster of a website that has over 100,000 physical users and turns over about a billion dollars. It implements huge amounts of complex business functionality, and has layer upon layer of security and reliability functions in it because we supply to multinationals, governments and the military (only the stuff that doesn't blow up of course). Our .pm file count is around 750, and our test suite runs for about 4 hours (and is only around one third complete).

Lest you suspect that 200,000 lines is wasted in re-implementing stuff, the main Build.PL script has around 110 DIRECT dependencies, and somewhere in the 300-500 range of recursive dependencies. Loading the main codebase into memory takes around 150-200meg of RAM.

When I joined the team, the build system was horribly out of date. The application was stuck on an old version of RedHat due to go out of support, and as a Tier 1 application we are absolutely forbidden from using unsupported platform.

So I took on the task of upgrading both the operating system and the build system for the project. And it's a build system with a history.

Once upon a time, long ago, the project went through a period where the development team was exceptionally strong and high skilled. And so of course, they created a roll-your-own build system called "VBuild".

They built their own Perl, and along with it they also built their own Apache, mod_perl, and half a dozen other things needed by the project. This is similar to many suggestions I hear from high-skilled people today, that at a certain point it's better just to build your own Perl. VBuild this created in the pre-commercial Linux era, so it's not an entirely unreasonable decision for that time period.

Unfortunately, a few years later the quality of the team dropped off and VBuild turned into a maintenance nightmare because it required a high-skill person to maintain it.

At the time, the Tier 1 "Must be supported" policy was coming into effect, and after the problems with custom-compiling they decided to go with the completely opposite approach of using only vendor provided packages, in a system called "UnVBuild".

Since their platform of choice was RedHat, this had become troublesome even before I arrived. Worse, in the change from RHEL 4 to RHEL 5, some of the vendor packages for things like XSLT support were dropped entirely leaving us in a bind.

My first instinct was to return to the build everything approach, but the stories (and commit commentary) from that time period reinforced the idea that complete custom build was a bad idea. Office supplies is hardly a sexy industry, and the ability to entice good developers into it is a quite legitimate risk.

So in the end, I went with an approach we ended up nicknaming "HalfBuild". The concept behind it is "Vendor where possible, build where needed".

We use a fairly reasonable chunk of vendor packages under this model. Perl itself, the Oracle client, XML::LibXML and a variety of other things where our version needs are modest and RHEL5 contains it. We also use a ton of C libraries from RHEL5 that are consumed by the CPAN modules, like all the image libraries needed by Imager, some PDF and Postscript libraries, and so on.

One RPM "platform-deps" meta-package defines the full list of these system dependencies, and that RPM is maintained exclusively by server operations so that we as developers are cryptographically unable to add major non-Perl dependencies without consulting them first.

On top of this is one enormous "platform-cpan" RPM package built by the dev team that contains every single CPAN dependency needed by all of our Perl projects.

This package lives in its own home at /opt/cpan and is built using a process rather similar to the core parts of Strawberry Perl. With PREFIX pointing into /opt/cpan, we first build a hand-crafted list of distribution tarballs to upgrade RHEL5 to a modern install of CPAN.pm (without overwriting any normal system files).

We then boot up the CPAN client from /opt/cpan, and tell it to install all the rest of the dependencies from a private internal CPAN mirror which is rsynced by hand specifically for each new CPAN build. This ensures that the build process is deterministic, and that we can fix bugs in the build process if we need to without being forced to upgrade the modules themselves).

The CPAN client grinds away installing for an hour, and then we're left with our "CPAN Layer", which we can include in our application with some simple changes to @INC at the beginning of our bootstrapping module.

The /opt/cpan directory for our project currently weighs in at about 110meg and contains 2,335 .pm files.

Updating /opt/cpan is still something of an exercise even under this model because of potential upgrade problems (Moose forbidding +attributes in roles hit us on our most recent upgrade) but the whole process is fully automated from end to end, and can be maintained by a medium-skill Perl hacker, rather than needing an uberhacker.

Over the last 2 years, we've upgraded it around once every 6 months and usually because we needed to add five or ten more dependencies. We tend to add these new dependencies as early as we can, when work that needs them is confirmed but unscheduled.

We also resort to the occasional hand-copied or inlined pure-Perl .pm file in emergencies, but this is temporary and we only do so about once a year when caught unprepared (most recently Text::Unidecode for some "emergency ascii'fication" of data where Unicode had accidentally slipped in).

While not ideal, we've been quite happy with the /opt/cpan approach so far.

It means we only have to maintain 5 RPM packages rather than 500, and updating it takes one or two man-days per 6 months, if there aren't any API changes in the upgrade.

And most importantly it provides us with much better bus sensitivity, which is hugely important in applications with working lives measured in decades.

Padre Second Birthday Party 24th-25th of July

Alias — 2010-07-10T16:50:21+00:00

2 years ago this month, Gabor did the first Padre release.

The last 12 months has seen Padre mature from a high-end text editor to a low-end refactoring IDE. We've stolen a number of features from Ultraedit, Komodo and EPIC, and we've invented new features all of our own, making Padre a very fluid and natural place to write Perl in.

We've added support for Perl 6, Template Toolkit, remote file support, more languages, syntax checking, an interactive debugger, a regex editor, and our first half a dozen refactoring tools.

We've also greatly solidified the code. Window integration is now totally solid, we've added a resource locking API, a new filesystem API, a new search API, a new display API, rewritten the threading and background Task subsystem, heavily overhauled the Plugin Manager API and GUI, and added Advanced Preferences and the ability for advanced users to selective disable various Padre bloat/features.

The last couple of months have also seen great improvements in Padre's hackability as well. The new Task 2.0 API lets people write background logic and consume multiple cores of CPU without having to know how threading works, and the new wxFormBuilder plugin lets you build GUI code without having to know Wx (one of the biggest barriers to contributing to Padre).

On the weekend of the 24th-25th of July we would like to invite all Padre developers, users, friends and well-wishers to join us for Padre's Second Birthday Party and Hackathon in the Padre IRC channel at irc://irc.perl.org/#padre or via the Mibbit Web Client.

If you've always been curious about, or interested in hacking on, Padre we'll have a number of developers in channel to help you out.

Personally, I plan to debut the first public release of Padre::Plugin::FormBuilder, and to start ripping out all Padre's older fixed-size dialogs and replacing them with new shiny model-generated sizer-based dialogs that will work much better across all three operating systems.

If you'd like to help out in this effort, I'll be in channel most of the day on both days (Sydney timezone).

I look forward to seeing you all there.

Yahoo provides (awesome) alternative Geocoder to Google's

Alias — 2010-07-06T04:10:06+00:00

As far as I'm concerned, there are three critical things you need in a Geocoder service.

1. Global Coverage

Because when it doesn't have global coverage, it's basically means "America-only" and thus pointless for most of the world.

2. Multiple Matching

For ordinary humans, there's massive power in being able to just search for "1 Oxford Street" without listing any more details. From there, if it is a country-specific application you just change it to "1 Oxford Street, Australia" behind the scenes.

That results in a list of possible locations.

You show the results for the first result, and then a list of "Did you mean:" links for the other results. This lets the application do what people mean most of the time, while making it trivial to recover if it isn't accurate enough.

You can see the effect I'm talking about by running the example query here...

http://geo2gov.com.au/

3. No usage limitations

Google Maps Geocoder can only be used with Google Maps. Fail.

I'm willing to accept a volume limitation like "You have to pay after the first 50,000 requests" but I can't accept a usage limitation.

I'm happy to report that Yahoo's new PlaceFinder service is the first free Geocoder that meets all these primary criteria.

It's global in scope, lets you control the list of results with paging, and doesn't limit the usage to any particular domain.

Now all we need is an Geo::Coder plugin for it.

Please write one for me Lazyweb :)

Perl 6 Design Minutes for 30 June 2010

chromatic — 2010-07-03T08:13:30+00:00

The Perl 6 design team met by phone on 30 June 2010. Allison, Patrick, and chromatic attended.

Allison:

working on Parrot packages for Debian experimental
seems like a good idea to do that before the 2.6 supported release
there was also a request for Rakudo packages
not sure if I'm the best person to do it

Patrick:

I'm sure we should package Rakudo Star

Allison:

Debian had a packager for those, but I haven't looked at the packages
this'd be an early run of what we'll do with Rakudo Star

Patrick:

we're not quite ready for packaging that yet
maybe a couple of weeks
finished the List and Iterator types for the #30 release
adjusted Rakudo's Associative and Positional roles
much cleaner implementation now
that'll require a few small spec changes
redid Rakudo's container types
more robust
preparing for autovivification of hashes and arrays
expect to finish those in the next couple of days
there was no container model previously; the code was consequently crufty
lots of cleanup of incorrect assumptions
Rakudo lists are now properly lazy
comment syntax fixed
ROADMAP updated
fixed the meaning of Nil; it's defined, not undefined
added the sink prefix (?)
fixed setting of $!
started fixing bugs and closing tickets on Monday, did 15 or 20
mostly already fixed in the previous couple of weeks
looking at the implementation of the series operator
spec is self-contradictory or ambiguous or both
waiting for Larry's clarification
fixed a bug in $*ARGFILES
had a nice contribution of that implementation last week
that behavior works on any set of files, not just those on the command line
working on autoviv
have some regex backtracking bugs to fix
will work on closures after that
put together three new YAPC presentations
the Rakudo Star presentation will become a video cast or a blog post or both

worked on a slew of Parrot optimizations for Rakudo
have a few more to go
might have to create a Rakudo branch temporarily
will try to help merge the new GC
working on a metamodel for Parrot objects, informed by Perl 6 and Moose

The next challenge for Perl on Windows (et al)

Alias — 2010-07-01T03:32:16+00:00

Even at this early stage, before any actual installers have appeared, things are looking pretty good for Strawberry Professional Alpha 2.

Padre is starting to firm up as a usable editor, Frozen Bubble works quite well and is completely playable, and I think my wxFormBuilder work is coming along nicely (which bodes well for quickly and easily creating GUI apps in some Alpha 3...)

My biggest remaining concern at the moment though, is also one of the smallest and seemingly trivial issues.

Although we can much more easily build and install large, rich and good looking desktop applications there is still no way to launch these applications without resorting to the command line.

Any iconography that are filled into the start menu in Strawberry Professional will do so because Curtis has put them there himself.

A Perl installation, via the %Config information, lets an installer know where to put libraries, binaries, documentation etc. Shared files in the File::ShareDir model are really just a hack, putting them in an agreed location within lib.

File::HomeDir isn't any use to us either for this. It is designed to let programs deal with user-owned data at run-time, NOT with system integration at install time.

Without the ability to install programs in a way that desktop users can easily launch, our little nascent desktop revolution will never really be able to get up to speed.

Having a post-install step where you need to launch a Windows command line, and then run "padre --desktop" just to install a desktop icon is simply not good enough.

Likewise, having to run Frozen Bubble from the command line is silly as well.

So consider this a challenge to anyone out there that likes tackling tricky puzzles, to try and build a File::Launcher (or whatever you want to call it) that can locate install paths, and be integrated into module installers, so we can make proper use of the Start Menu, Desktop, Quick Launcher, and all the equivalent features on all three major desktop platforms (Windows, Mac, and FreeDesktop.org).

If you build it, Padre will use it.

Modern Perl: The (Draft) Book

chromatic — 2010-06-28T23:43:33+00:00

This took longer than I expected, but the draft of the Modern Perl book is available for review. I'm especially interested in hearing from people who don't consider themselves expert Perl 5 programmers. The goal of the book is to explain how Perl 5 works (and how to write Perl 5 effectively) to help novices become adepts.

Perl 6 Design Minutes for 16 June 2010

chromatic — 2010-06-26T17:07:30+00:00

The Perl 6 design team met by phone on 16 June 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

documented TOP (again), and explained how parsing is initiated and how it actually works
series operator (...) now picks a monotonic function when using single characters as endpoints
STD can now catch duplicates involving protos as well as onlys
STD no longer advises removal of parens on spaceless sub() declaration
mostly advised sorear and pmichaud
Stefan is finishing the boostrap of the STD parser
also working on adding a parallel NFA and DFA engine
no, he doesn't want to generate all the states in advance
it works faster lazily

Allison:

working on chroot environments with something more secure than chroot
relevant to building Parrot packages
looking at some bugs for Will

Patrick:

Rakudo developers decided not to make extra special effort to make a June release of Rakudo Star
the calendar works against us
the new date for the release is July 29
we're I comfortable with hitting that target
we won't be happy with the results of moving heaven and earth to release in June
there are lots of advantages
one disadvantage is not having Rakudo Star at YAPC::NA
one big advantage is using the supported Parrot 2.6 release as the basis
I'll write a post outlining the plan in the next couple of days
otherwise working on lists and interators in Perl 6 and Rakudo
after deciding to make iterators immutable, Larry and I realized that solves many problems
everything works out as plain as day after that
very happy with that design
the incorrect assumptions of the old model were pervasive
replacing the old pieces is taking a while, which is no surprise
this approach feels right though
the new branch does things no previous version could do
slices work much better, for example
metaoperators work properly
map is lazy
slurpy arguments in lists are lazy by default
no weird binding or action at a distance problems
plenty of changes to Associative and Positional roles
those are now super clean and may be lazy
more features work
~30 failing tests (not test files, just tests) now, ~500 last night
most of the current failures are minor
will try to merge the branch before the release
replacing lots of ugly code with fewer lines of elegant code
Jonathan and others have worked on lots of other pieces
adding plenty of new features
looking forward to tomorrow's release

editing the Rakudo book
moving the Rakudo release date may let us have a printed book available about the same time
depends on how much there is left to write

Perl 6 Design Minutes for 09 June 2010

chromatic — 2010-06-24T12:24:33+00:00

The Perl 6 design team met by phone on 09 June 2010. Larry, Allison, Patrick, and chromatic attended.

Larry:

not much spec change this week
figured out a syntax for a regex block to return more than one cursor
based on gather/take
in STD hacking, continued to assist Stefan O'Rear in getting STD bootstrapped via viv
now that it's bootstrapped, we're refactoring things that make sense now
we're now starting to move bits of Cursor code from Perl 5 into Perl 6
refactoring the grammar for sanity of design
started upgrading STD to normal Perl 6 syntax where it previously catered to gimme5's limitations
for example, switched STD's old .<_from> and .<_pos> hash lookups to using .from and .pos accessors
started the prep work for moving EXPR out of STD to make it generally available to any grammar wanting operator precedence
in STD parsing, made Perl 5 $< detection have a longer token to avoid confusion with match variables
STD no longer attempts two-terms detection on infix_circumfix_meta_operator
STD now parses >>R~<< correctly, or at least dwimmily
STD doesn't complain about P5isms in printf formats like "%{$count}s"
STD was parsing /m and /s with the opposite semantics
termish now localizes $*MULTINESS in its scope so that inner declarations aren't accidentally multified
STD now carps about package Foo; as a Perl 5 construct

Allison:

talked to Chris Shiflett, a PHP developer, on someone from the PHP community to sit on the Parrot board
will be in the US for a few weeks

Patrick:

working on list simplification
had a couple of breakthrough ideas on Monday
working on the implementation now
worked out inversion lists for character class matching in regexes
will make them faster, especially with long ranges of character classes
fixed a half-dozen tickets in RT
fixed Rakudo hash constructors
fixed an intermittent bug with colon-pair signatures
two possible parses exist in STD, but we removed an unneeded one in Rakudo
fixed a bug with Parrot's exit opcode
NQP and PAST needed an update not to cheat with PASM constants
I fixed that too
Vasily added multisub and multimethod support to NQP, that was a big plus
fixed the ** quantifier in regexes to understand surrounding whitespace
regex engine tried to match beyond the end of a string, so I added guards for that
will work on lists furiously before the next release
I don't think it'll take long
closures are next, hope to have those in place by the weekend

released a new version of Pod::PseudoPod::LaTeX to support the various books in progress

Perl 6 Design Minutes for 02 June 2010

chromatic — 2010-06-22T01:12:29+00:00

The Perl 6 design team met by phone on 02 June 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

mostly, I supported sorear in bootstrapping STD to use viv instead of gimme5
his stage 2 and stage 3 now output identical Perl 5 versions of STD
produces a huge amount of warnings
appears to require Perl 5.12 at the moment
working on both of those
S03 refines hyper dwimminess to be more like APL, with modular semantics
S02 refines Blobs to simply be immutable Bufs, with similar generic characteristics
S02 now describes native blob types
implemented post-declaration checks for BEGIN and use, since those can't wait for end of file
STD no longer loses existing bindings when we go to a sublanguage
STD now uses $*GOAL variable only as informative, never as a "stopper"
instead, we create a <stopper> rule for $*GOAL if necessary
can check for that only, instead of that or $*GOAL
answering lots of questions on how STD and viv work besides that

Allison:

did a lot of research on graph color algorithms for register usage algorithms
will finish my finals on Monday

Will:

trying to herd the discussion of dynop libraries
a recent branch to close an old ticket broke a lot of assumptions
some bugs have become more visible because of these changes
hope to get that cleaned up this week

Allison:

I liked your suggestion of bringing back the getstderr and related opcodes

Will:

trying to resurrect Partcl
stuck on a TT #389 closing issue
not sure how to fix that, the way things are now

Patrick:

working on the iterator and list design
brainstorming the implementation
will implement somethine one way or another this week
people keep implementing workarounds for the current system
they'll bite us eventually
Moritz and I worked on making the regex engine returning real Perl 6 objects
that mostly works
exposes some places where lists don't work exactly right
the workarounds there made me replan the list and iterator implementation
answered some questions online
Jonathan added a better backtrace algorithm for Rakudo
reports Perl 6 source lines instead of PIR lines
I'll review his code
think I can borrow it for NQP for all HLLs
Jonathan reports that it was a lot easier in NQP than PIR

trying to answer a few Parrot design questions
looking at the continuation of design from Perl 1 - 4 to Perl 5 and Perl 6
hope to have coding time soon

Perl 6 Design Minutes for 26 May 2010

chromatic — 2010-06-20T19:40:02+00:00

The Perl 6 design team met by phone on 26 May 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

:() syntax is now always signature
we now use foofix:[...] as the general op form instead of foofix:(...)
refactored the sematics of :nth and :x
:nth() now only ever takes a monotonically increasing list
S03 now explains how "not-raising" works on != and ne
it now basically matches the intuitions of an English speaker via HOP definition of negate metaop
STD sometimes didn't require semi between statements
statement modifiers are expression terminators but not valid statement terminators
an unexpected statement modifier word like if could terminate one statement and start another
fixed up backslashes in character classes to allow \s etc and reject \u etc
STD was accidentally using the same lexpad for different multis
Cursor now treats :() on name extension as a signature always, never as a categorical
we shouldn't introduce the stopper for circumfix until we're in the circumfix, or we can't use the same char on both ends
placeholder messages error messages are now much more informative and correct
we now disallow use of placeholder after same variable has been used as a non-placeholder, even for an outer reference
renamed add_macro (which it doesn't) to add_categorical (which it does)
participating frequently in discussions on semantics both on irc and p6l
working closely with sorear++ as he brings viv closer to bootstrapping, yay!
soon can bootstrap past gimme5

Allison:

worked on Pynie this week in my limited spare time
one goal is to generate the parser directly from the Python grammar
wrote a small, lightweight PEG parser which generates a match tree from the Python 3 grammar
can generate a lexer directly
right now it creates a parse tree
looks similar to the match nodes of NQP-rx
dumps out a tree to the PIR parser
working on PaFo elections for next year, but trying to delegate those
will have more time after June 7

Will:

working on Perl 6 advent tests
many more people are doing more work than me
liasing with Rakudo folks for any important Parrot bugs before the Rakudo Star release
my current direction there is "don't break anything"

Patrick:

sorear added hash flattening to NQP
lots of work on closures in PAST and NQP
they properly clone
fixes some lexical problems
need to get that to work in Rakudo
that's tougher; Rakudo has to wrap Parrot subs
wrapper object needs cloning as well, along with its attributes
we'll add a new PAST node type to help
that node understands contexts
essentially a way to add void context optimizations to your AST
that solves many problems in Rakudo beyond closures
added a setting into NQP along with its test suite
not automatically loaded, but available
contains standard hash and array methods
Parrot's ops2c project uses those
other people can update and enhance that setting as necessary
NQP also has the ability to parse type names
NQP doesn't do anything with them yet
eventually they'll allow the use of multis
cleaning up some NQP bugs regarding lexicals and package storage of subs
Bruce Keeler enabled variable interpolations in regexes
working on some refactorings to simplify that approach
works in NQP and Rakudo now
that's a feature we've never had before
Rakudo's REPL now works better, thanks to sorear
HLLCompiler now written more in NQP as part of that
NQP now can do eval
NQP remembers lexicals in interactive mode now
adding that to Rakudo is more complex
working on that
pleased with the progress on #perl6

reviewing long term plans for GC and Lorito
should have more time free soon

Perl 6 Design Minutes for 19 May 2010

chromatic — 2010-06-16T21:38:09+00:00

The Perl 6 design team met by phone on 19 May 2010. Larry, Will, and chromatic attended. Patrick added his notes later.

Larry:

S03 makes more explicit that doctrine that ~~ topicalizes, and removes smartmatch table fossils that automatically fall out from that
S05 renames 'accent' to 'mark' for better Unicode conformance
:a and :aa changed to :m and :mm
S05 disrequires retroactive semantics on :samecase and :samemark
the method form must now explicitly add case or mark modifiers to the pattern
regularized mm// to ms// to avoid confusion with new :m ignoremark option
STD now does a bit better at diagnosing bogus ??!! constructs of various sorts
STD now correctly adds operators to symbol tables as subs
CORE.setting now has protos of all the operators so they can be recognized as subs too
Cursor now canonicalize operator names in the symbol table
btw, not quite like specced
STD now reads user's mind on 'Str $toto' to intuit missing declarator
STD now properly diagnoses a typename between routine declarator and sub name

Will:

working on code for Carl Masak, trying to get his poker code example running on Rakudo
both fun and frustrating
some stuff doesn't quite work yet
going through the Advent examples
adding them to spectests
make sure we won't regress on such public examples
other people are helping with that now

will get back to editing the Rakudo book soon
hope to have it in print by YAPC, but no guarantee

Patrick:

fixed closures in NQP, as a precursor for fixing them in Rakudo
worked with sorear on REPL in Rakudo and PCT in general
ported the NQP "standard library" done by japhb++, bacek++, and many others into the nqp-rx repository and made it part of the standard build sequence for nqp and Parrot
decided we need a new "context sensitive" node type in PAST, will be used to create proper closures and to handle sink context
worked with bacek on adding better multimethod support to PAST and nqp-rx
discovered a problem with lexical subs in NQP being automatically entered into the package namespace (and some existing code relying on this behavior)
did some initial fixes to at least get things entered properly, but a complete fix may require a deprecation cycle
plan to review others' patches this week
plan to fix REPL, closures, and sink context in Rakudo (since those are currently large pain points)
plan to work on loops and iterators after that

Perl 6 Design Minutes for 12 May 2010

chromatic — 2010-06-16T01:47:02+00:00

The Perl 6 design team met by phone on 12 May 2010. Larry, Allison, Patrick, and Will attended.

Larry:

clarified usage of brackets around infixes
added various 128-bit types to the spec; we might make them arbitrarily extensible via role
at least LLVM could support this, even to non-powers-of-two sizes
modernized the paleolithic grammatical category description in S02
STD now uses double-quote rules for interpolating @foo[] into regex
STD now gives better message on 1__3
added the specced 128-bit types to CORE.setting
added minmax function to CORE.setting
implemented circumfix:«X Y» as grammar derivation
currently only allows a >> inside
now also recognizes foofix:("\x[face]") and foofix:("\c[YOUR CHARACTER HERE]") without actually evaluating
playing with factoring yaml out of gimme5, since viv is not likely to go that route.
mostly just answered a lot of questions on irc
egged people on about concurrency issues

Patrick:

thought on handling closures properly
have a solution, just need some time to implement
discussion on changes to CodeString
work on compiler toolkit to avoid CodeString, using StringBuilder instead where possible, in PCT, NQP, and rakudo. Pretty easy, no downstream projects block on a deprecation issue
after that, lists
also been answering questions on interactive mode (REPL) for rakudo et al. (the issue with losing lexicals)

Allison:

resolved the git conversation pretty well (for Parrot's repo migration)
worked on a pure PEG parser (following the paper), straight PIR, single day; now self-parsing. Interesting project, is lightweight. currently has memoization, but that might not be right for us because of backtracking. With some more effort, could probably handle EBNF form (useful for python)
could be setup for developer status for Debian which will improve our packaging status for Debian and Ubuntu

Will:

Parrot CodeString performance improvements
we're definitely faster in branch, but some feedback from pmichaud should help us clean up the API a bit as well, look for those to hit trunk in the next few days
Parrot makefile deps cleanup

I hereby report the success of my 2005 Perl Foundation Grant

Alias — 2010-06-15T12:41:57+00:00

In late 2005 I was awarded a grant from the Perl Foundation for "Extending PPI Towards a Refactoring Perl Editor". You can see the original grant here.

http://www.perlfoundation.org/adam_kennedy_extending_ppi_towards_a_refactoring_p erl_editor

My idea at the time was to take the recently completed PPI parser, fill in a variety of different supporting features, build some modules to improve platform integration, and then build a refactoring editor that used PPI and that would fire the imagination of what was possible in Perl.

A year and a half later in early 2007, I was forced to report the failure of the grant.

http://www.perlfoundation.org/march_25_2007_adam_kennedy_s_refactoring_editor_gr ant

Not having anything like the time or experience to build an editor from scratch, I had intended to build a thin refactoring layer over the top of an existing Perl editor. For various reasons, I couldn't make this workable.

To my great surprise and astonishment, the Perl Foundation decided to pay out the grant anyway. Their reasoning was that I had indeed completed all the secondary goals, and the benefits provided by the creation of Vanilla Perl (later to become Strawberry Perl) which was NOT in the grant made the grant worthwhile anyway.

As Curtis noted at the time "Given that he didn't complete his grant, some might be surprised that we've decided to make the final payment...".

And indeed, from time to time the subject has come up here and there about getting paid despite not succeeding. Not often, and not loudly, but occasionally and ongoing.

It has also eaten at me. I hate not delivering on what I say I'll do, and I wanted the refactoring editor for me as much as for everyone else. While Padre has grown and improved, it's never quite met my expectations of what a great Perl IDE should be like.

So it is with great joy that with the landing of the Task 2.0 API I think Padre finally meets the criteria that I set for myself in taking the grant, a Perl Refactoring IDE that would let me give up Ultraedit and present Perl in a modern GUI setting.

Task 2.0 makes Find In Files, Replace In Files, and Incremental Search only a small amount of coding away from being created. These are the last features I need to finally ditch Ultraedit once and for all. The time has arrived, I think, to draw a line in the sand.

And so almost 5 years after the original grant, it is with great joy that I would like to formally report I have completed my 2005 Perl Foundation grant.

I have built, on top of the shoulders of Gabor and the rest of the Padre gang, a refactoring editor that I think will fire the imagination of Perl developers.

And I hope that this corrects and draws to a close one of the most persistently annoying failures in my Perl career.

Please enjoy the features in the new Padre releases as they appear over the next month and a half, leading up to Padre's second birthday party.

I am extremely proud of what the whole Padre gang has achieved in the last year, and I can't wait to stabilise a new group of features just waiting for the background support to firm up a little more.

The CPAN just got a whole lot heavier, and I don't know why

Alias — 2010-06-09T05:49:49+00:00

According to the latest CPANDB generation run, and the associated Top 100 website at http://ali.as/top100, something big just happened to the CPAN dependency graph.

After staying relatively stable for a long time, with the 100th position coming in at around 145 dependencies and incrementing by 1 every 3 months, the "weight" of the entire Heavy 100 set of modules has jumped by 20 dependencies in the last week or so!

The 100th position is now sitting at 166 dependencies, and perennial leader of the Heavy 100 MojoMojo has skyrocketed to an astonishing 330 dependencies. The shape of the "Jifty Plateau" is also much less distinguishable, which suggests it might be more than just a pure +20 across the board.

The question is why?

Is this caused by the restoration of a broken META.yml uncovering formerly ignored dependencies? Or has someone added a new dependency somewhere important accidentally?

Padre::Task 2.0 - Making Wx + Perl threading suck faster

Alias — 2010-06-03T03:37:29+00:00

Some time Real Soon Now, I'll be landing the shiny new second-generation Padre Task API (which is used for background tasks and threading control) onto trunk.

A few people have asked me to give a quick high level overview on how it works, so this will be a quick bottom to top pass over how Padre will be doing threading.

The first-generation Padre::Task API was true frontier programming. While some of us knew Wx and others knew Perl's threads a bit, nobody really had any idea how the two interact. Indeed, the proliferation of bugs we found suggests that Padre has really been the first major application to use both at the same time.

Steffen Müller boldly led the Padre team into this wilderness, putting together a solid threading core loosely inspired by the Process API. He nursed a ton of bug reports and tracked upstream fixes in Wx.pm.

Some of the code was a bit ugly in places, and the threads burned a ton of memory, but it worked and worked well. It worked well enough that Andrew Bramble was later able to extend it to add Service support for long running background tasks with bidirectional communication.

But while we've reduced some of the worst of the memory problems with the "slave driver" methodology, the API has been at the end of its natural life for a while now. It's hard to write new background tasks without knowing both Wx and threads, which limited access to only three or four people.

The goals for the new Padre::Task 2.0 are threefold.

Firstly, to allow the creation of the three distinct background jobs we need in Padre, in a way that doesn't abuse either Wx or Perl's threads mechanism. These three background job types are Task (Request,Wait,Response), Stream (Request,Output,...,Response) and Service (Request,Input,Output,...,Response).

Second, to allow the implementation to have (or have in the future) the theoretically smallest possible memory consumption beyond the minimum overheads imposed by Perl's threading model.

Third, to allow the creation of Wx + threads tasks without the need to learn either Wx or threads. This should open up background tasks to the main body of Padre contributors, beyond the elites, and spur great improvements in Padre's ability to take full advantage of multi-core developer machines.

A fourth bonus goal is to allow us to migrate Padre's backgrounding implementation to something other than threads in the future, without having to change any of the code in the actual tasks next time around. This should also allow the people that don't like Perl threads and want us to use something else to move their arguments from bikeshedding to actual proof-by-code.

After several months of experimenting, I've gone with a somewhat unusual implementation, but one that is completely workable and meets the criteria.

The core of this implementation is a communications loop that allows messages from the parent thread to a child thread, and back again.

The odd thing about this particular communications loop is that the two halves of the loop are done using utterly different underlying mechanisms.

The parent --> child link is done using Perl threads shared variables, in particular Thread::Queue.

Each Padre thread is created via a a Padre::TaskThread parent abstraction, which governs the creation of the real threads.pm thread, but also provides a Thread::Queue "inbox" for each thread. This is inspired by the Erlang micro-threading model for message passing, but is way way heavier.

In this manner, the top level task manager can keep hold of just the queue object if it wants, feeding messages into the queue to be extracted at some unknown place in another thread it has no control over.

Once spawned, each worker thread immediately goes into a run-loop waiting on messages from its message queue. Messages are simply RPC invocations, with the message name being a method name, and the message payload becoming method parameters.

Every thread has the ability to clone itself if passed an empty Padre::TaskThread object to host it. This gets a little weird (you end up passing shared Thread::Queue objects as a payload inside of other shared Thread::Queue objects) but the end result is that you don't have to spawn Perl threads from the parent thread, you can spawn them from child threads but retain the parent's ability to send messages to them regardless of the location on the thread spawn graph they are actually running.

The end result of this trickery is that we can replicate the slave driver trick from the first-generation API. By spawning off an initial pristine thread very early in the process, when the memory cost is small, we can create new threads later by spawning them off this "master" thread and retain the original low per-thread memory cost.

And while we don't do it yet, we can be even tricksier if we want. If we have a thread that has had to load 5, 10, or 20 meg of extra modules, we don't need to load them again. Instead, we could choose to clone that child directly and have a new thread with all the same task modules pre-loaded for us.

The second half of this communications loop is the up-channel, which is done using a totally different Wx mechanism. For communicating messages up to the parent, the Wx documentation recommends the creation of different messages types for each message, and then the use of a thread event.

This care and feeding of Wx turns out to be difficult in practice for non-elites, because you end up registering a ton of different Wx event types, all of which need to be stored in Perl thread-shared variables. And each message needs to be pushed through some target object, and the choice of these can be difficult.

Instead, what we do instead is to register a single event type, and a single global event "conduit". As each message is received by the conduit, it filters down to just appropriately enveloped events and pass only those along to the task manager. The task manager removes the destination header and routes it to the correct task handle in the parent.

Again, messages are done as RPC style calls, with a message type being the method to call, and the payload being the params.

This lets you do a reasonable simple form of cross-thread message passing (at least from the child to the parent anyway).

sub in_child { my $self = shift; $self->message('in_parent', 'Some status value'); } sub in_parent { my $self = shift; my $status = shift; }

We've also put a Padre::ThreadHandle layer over the task itself to do eval'ing and other cleanliness work, in the same way we put a Padre::PluginHandle over every plugin object to protect us against them.

This handle adds some extra value as well, automatically notifying the task manager when a task has started and stopped, and generally keeping everyone sane.

Within this communications loop lives the actual tasks.

The Task API forces work into a very rigid structure. The rigid rules on behaviour is the cost of allowing all the threading magic to happen without the task having to know how it happens.

The basic API looks something like this.

package Some::Task; use strict; use base 'Padre::Task'; # Constructor, happens in the parent at an arbitrary time before the job is run. sub new { my $class = shift; my $self = bless { @_ }, $class; return $self; } # Get ready to serialize, happens in the parent immediately before being sent # to the thread for execution. # Returns true to continue the execution. # Returns false to abort the execution. sub prepare { return 1; } # Execute the task, happens in the child thread and is allowed to block. # Any output data should be stored in the task. sub run { my $self = shift; require Big::Module; $self->{output} = Big::Module::something($self->{input}); return 1; } # Tell the application to handle the output, runs in the parent thread after # the execution is completed. sub finish { my $self = shift; Padre::Current->main->widget->refresh_response( $self->{output} ); return 1; }

To fire off this task, in the widget which commissioned this background work you just do something like this.

sub refresh { my $self = shift; require Some::Task; Some::Task->new( input => 'some value', )->schedule; } sub refresh_response { my $self = shift; my $output = shift; $self->wx_thingy->SetValue($output); }

As you can see here, at no point do you need to care about threads, or event handling, or that kind of thing. You wrap a task class around the blocking part of your app, have the finish push the answer through to your wx component, and then handle it in the wx component.

Of course, this is still a little fidgety. So I'm currently in the process of adding a secondary layer for the common case where the task is created primarily for the purpose of a single parent owner.

The new API layer simplifies things even more. The task takes an "owner" param that represents the Wx component that commissioned it. It does some weaken/refaddr based indexing magic to map the task to the owner object, and then makes sure there is a default finish method to route the answer automatically back to the owner.

The neat part about this is that it takes care of synchronisation problems automatically. If the task finish is called at a time when the Wx component has been destroyed, the answer is automatically dropped and ignored.

The owner can also explicitly declare that the answers from any tasks currently in flight are no longer relevant, and those will be dropped as well.

With the new helper code, your task is shrunk to the following.

package Some::Task; use strict; use base 'Padre::Task'; sub run { my $self = shift; require Big::Module; $self->{output} = Big::Module::something($self->{input}); return 1; } 1;

And the code in the owner component is just this...

sub refresh { my $self = shift; # Ignore any existing in-flight tasks $self->task_reset; # Kick off the new task $self->task_request( task => 'Some::Task', input => 'some value', ); } sub task_response { my $self = shift; my $task = shift; my $output = $task->{output} or return; $self->wx_thingy->SetValue($output); }

This is not the 100% final API look and feel, but it demonstrates the volume of code that you will need to write to do something in the background in the Task 2.0 API.

There's also some interesting opportunities to make it smaller again for very simple cases, by using the generic eval task to inline the execution code.

sub refresh { my $self = shift; # Ignore any existing in-flight tasks $self->task_reset; # Kick off the new task $self->task_request( task => 'Padre::Task::Eval', input => 'some value', run => <<'END_TASK', my $self = shift; require Big::Module; $self->{output} = Big::Module::something($self->{input}); return 1; END_TASK ); } sub task_response { my $self = shift; my $task = shift; my $output = $task->{output} or return; $self->wx_thingy->SetValue($output); }

This would mean we don't need a task class at all, which could be useful in cases where we want to avoid generating lots of tiny 10 line task classes,
or where we want to generate the background code on the fly at run-time.

The downside would be that because the work is bunched inside a single task class, we lose the chance to do worker thread specialisation (where the workers track which tasks they have loaded into memory, so further tasks of the same type can be assigned to them in preference to other threads).

Perhaps we support all the different ways, so that we can pick and choose which option is best on a case by case basis and change them over time. That would certainly fit the Padre rule of "something is better than nothing, don't worry too much about having to break it later".

The downside of this approach, of course, is that breakage. The new Task API will basically kill any plugin currently doing their own background tasks.

The upside, of course, is that once they are fixed these plugins can do those background tasks much more efficiently that before.

The new Task 2.0 API will land in Padre 0.65, which should be out in about 2 weeks.

Summary of Celtics vs. Lakers NBA Finals History

pudge — 2010-05-30T05:25:50+00:00

There have been 60 NBA Finals. The Boston Celtics have won 17, and the Minneapolis/Los Angeles Lakers have won 15, which amounts to those teams winning more than half of all NBA Finals. This year, they play each other in the Finals again, making it 33 champions out of 61 being either the Lakers or the Celtics.

Additionally, 39 of the 61 Finals have included either the Lakers or the Celtics. Including this year, the Celtics and Lakers have played each other in the Finals a whopping 12 times (just under one-fifth of all Finals have been these two teams).

The Celtics have made 21 total Finals appearances, so have faced the Lakers more than half of the times they've been in the Finals. The Lakers have made 30 Finals appearances, facing the Celtics in two-fifths of those.

The Celtics won nine of those against the Lakers, which accounts for more than half of all their 17 championships. The Lakers' two victories over the Celtics came after the Celtics won their first eight encounters.

The longest streak without either team in the Finals was eight years from '92-'99. The longest streak with either the Lakers or Celtics in the Finals was 10 years, done twice (from '57-'66, in which the Celtics appears all 10 years, and the Lakers five of those; and '80-'89, in which the Lakers appeared eight times, the Celtics five).

The longest streak for one team appearing in the Finals was, as noted, Boston, in the 10 years from '57-'66. Boston won nine of those 10 years, including eight in a row (the longest winning streak from any one team) from '59-'66, and also won 10 in 12 years, from '57-'69.

The Lakers' longest appearance streak is "only" four, from '82-'85, winning twice; but they have also appeared three times in a row five additional times (including the current three-year streak). In two of those, they won all three years; in one, they lost all three.

The only other team to "threepeat" was the Bulls, winning three years in a row twice in eight years ('91-'98). No other team but the Celtics and Lakers have had four consecutive appearances. The only other team to have three consecutive appearances was the Knicks, losing all three from '51-'53.

The 2010 NBA Finals begin in Los Angeles on Thursday.

Cross-posted on <pudge/*>.