Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Protesting the Moose with sugary alternatives

Posted by Alias on 2010.08.03 0:09 (#40477)
17 Comments
User Journal

At work, we've been experimenting with Moose because in our enormous and complex codebase we think we can probably benefit a lot from the extra rigour that it brings.

Before I continue, let me note that ours is a typical mod_perl enterprise setup with 6gig of memory per machine, so any memory consumed before the Apache fork is essentially free.

So none of the issues people (including me) have with startup code and memory consumption apply in this case, and I won't be addressing performance issues in this post.

The consensus of the half-dozen people at work is how Moose tries to look like a declarative extension, but doesn't actually act like it.

The following is what Moose seems to have been aiming for.

package Foo;
 
use Moose;
 
extends 'Bar';
with    'Role1';
with    'Role2';
 
has this => (
    is  => 'ro',
    isa => 'Str',
);
 
has that => (
    is  => 'ro',
    isa => 'Str',
);
 
1;

Unfortunately, this is what we've had to do instead.

package Foo;
 
use Moose;
 
BEGIN { # When we use Catalyst
    extends 'Bar';
}
 
has this => (
    is  => 'ro',
    isa => 'Str',
);
 
has that => (
    is  => 'ro',
    isa => 'Str',
);
 
with    'Role1';
with    'Role2';
no Moose;
__PACKAGE->meta->make_immutable;

This "real Moose" code totally spoils the dream of what we felt like we were going to get when we started to play with it.

Most of our current options for fixing this amount to either.

a) Add this extra dependency that will unscrew one of the other of the problems (namespace::autoclean)

b) Use this SECOND heavy sugar layer on top of the FIRST sugar layer, on top of Class::MOP.

Is fixing the syntax or writing light weight sugar really so hard?

As a kind of protest, I tried it for myself and managed to create MooseX::Atom.

This still has some flaws, but the current equivalent of the above would just be this.

package Foo;
 
use MooseX::Atom [
    extends => 'Bar',
    with    => 'Role1',
    with    => 'Role2',
    has     => [
         this => (
             is  => 'ro',
             isa => 'Str',
         ),
    ],
    has     => [
         that => (
             is  => 'ro',
             isa => 'Str',
         ),
    ]
];
 
1;

You can do the same thing for roles with MooseX::Role::Atom.

Now clearly, this might have some issues. It's the work of an hour and not a whole lot of thought.

But it's still light and clean, with all the class spec in one place up the top where people are used to seeing the declarative stuff in Perl modules.

Perhaps something like this might be a little better...

package Foo;
 
use MooseX::Hash {
    extends => 'Bar',
    with    => [ 'Role1', 'Role2' ],
    default => { is => 'ro' },
    has     => {
        this => { isa => 'Str' },
        that => { isa => 'Str' },
    },
);
 
1;

Help?

Posted by Alias on 2010.07.29 8:23 (#40470)
3 Comments
User Journal

(Before I begin, I should clarify I did not write this code, I'm just trying to maintain it)

The following error is the first thing spat out by make test for the Padre sync server, located at http://svn.perlide.org/padre/trunk/Madre-Sync.

Wasn't the move of Catalyst to Moose going to make things easier?

Can someone explain how you debug this?

I get the basics, I can see the "Can't locate Madre/Sync/Schema.pm". But that file should be, I think, automatically generated. And I don't really get how to dig down the 75 caller levels from the start to the end to work out where the actual functionality is failing...

not ok 1 - use Catalyst::Test;
 
#   Failed test 'use Catalyst::Test;'
#   at t\01app.t line 7.
#     Tried to use 'Catalyst::Test'.
#     Error:  Couldn't load class (Madre::Sync) because: Couldn't instantiate component "Madre::Sync::Model::padreDB", "Can't locate Madre/Sync/Schema.pm i
n @INC (@INC contains: blib\lib blib\arch C:/strawberry/perl/lib C:/strawberry/perl/site/lib C:\strawberry\perl\vendor\lib .). at C:\strawberry\perl\vendor
\lib/Class/MOP.pm line 132
#       Class::MOP::load_first_existing_class('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137
#       Class::MOP::load_class('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Catalyst/Model/DBIC/Schema/Types.pm line 21
#       Catalyst::Model::DBIC::Schema::Types::__ANON__[C:\strawberry\perl\vendor\lib/Ca talyst/Model/DBIC/Schema/Types.pm:21]('Madre::Sync::Schema') called
at C:\strawberry\perl\vendor\lib/Moose/Meta/TypeCoercion.pm line 63
#       Moose::Meta::TypeCoercion::__ANON__[C:\strawberry\perl\vendor\lib/Moose/Meta/Ty peCoercion.pm:67]('Madre::Sync::Schema') called at C:\strawberry\per
l\vendor\lib/Moose/Meta/TypeCoercion.pm line 97
#       Moose::Meta::TypeCoercion::coerce('Moose::Meta::TypeCoercion=HASH(0x4a2b444)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Moose
/Meta/TypeConstraint.pm line 90
#       Moose::Meta::TypeConstraint::coerce('Moose::Meta::TypeConstraint=HASH(0x4a29854 )', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/M
ooseX/Types/TypeDecorator.pm line 206
#       eval {...} called at C:\strawberry\perl\vendor\lib/MooseX/Types/TypeDecorator.pm line 205
#       MooseX::Types::TypeDecorator::AUTOLOAD('MooseX::Types::TypeDecorator=HASH(0x4a3 1424)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\l
ib/Moose/Meta/Attribute.pm line 743
#       Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x4b467 fc)', 'Madre::Sync::Schema', 'Madre::Sync::Model::padreDB=HASH(0x4db
7c64)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Attribute.pm line 398
#       Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(0 x4b467fc)', 'Moose::Meta::Instance=HASH(0x4db7ed4)', 'Madre::Sync::M
odel::padreDB=HASH(0x4db7c64)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Class.pm line 567
#       Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/C
lass.pm line 540
#       Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Class.pm
line 256
#       Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Object.pm lin
e 25
#       Moose::Object::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at generated method (unknown origin) line 4
#       Catalyst::Model::DBIC::Schema::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at C:\strawberry\perl\vendor\lib/MooseX/
Traits/Pluggable.pm line 131
#       MooseX::Traits::Pluggable::new_with_traits('Madre::Sync::Model::padreDB', 'Madre::Sync') called at C:\strawberry\perl\vendor\lib/CatalystX/Componen
t/Traits.pm line 146
#       CatalystX::Component::Traits::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Cl
ass/MOP/Method/Wrapped.pm line 48
#       Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:49]('Madre::Sync::Model::padreDB', 'Madre::Sync', '
HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 89
#       Catalyst::Model::DBIC::Schema::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/C
atalyst.pm line 2502
#       eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2502
#       Catalyst::setup_component('Madre::Sync', 'Madre::Sync::Model::padreDB') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2416
#       Catalyst::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 54
#       Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:64]('Madre::Sync') called at C:\strawberry\perl\ven
dor\lib/Class/MOP/Method/Wrapped.pm line 89
#       Madre::Sync::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 1142
#       Catalyst::setup('Madre::Sync') called at blib\lib/Madre/Sync.pm line 62
#       require Madre/Sync.pm called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 114
#       Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:118]() called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 74
#       eval {...} called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 67
#       Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125
#       Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137
#       Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24
#       Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a
t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493
#       Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call
ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424
#       Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S
ub/Exporter.pm line 742
#       Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per
l\vendor\lib/Catalyst/Test.pm line 112
#       Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2
#       main::BEGIN() called at blib\lib/Madre/Sync.pm line 0
#       eval {...} called at blib\lib/Madre/Sync.pm line 0
#       eval 'package main;
# use Catalyst::Test @{$args[0]};
# 1;
#
# ;' called at C:/strawberry/perl/lib/Test/More.pm line 858
#       Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p
m line 833
#       Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7
#  at C:\strawberry\perl\vendor\lib/MooseX/Types/TypeDecorator.pm line 208
#       MooseX::Types::TypeDecorator::AUTOLOAD('MooseX::Types::TypeDecorator=HASH(0x4a3 1424)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\l
ib/Moose/Meta/Attribute.pm line 743
#       Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x4b467 fc)', 'Madre::Sync::Schema', 'Madre::Sync::Model::padreDB=HASH(0x4db
7c64)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Attribute.pm line 398
#       Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(0 x4b467fc)', 'Moose::Meta::Instance=HASH(0x4db7ed4)', 'Madre::Sync::M
odel::padreDB=HASH(0x4db7c64)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Class.pm line 567
#       Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/C
lass.pm line 540
#       Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Class.pm
line 256
#       Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Object.pm lin
e 25
#       Moose::Object::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at generated method (unknown origin) line 4
#       Catalyst::Model::DBIC::Schema::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at C:\strawberry\perl\vendor\lib/MooseX/
Traits/Pluggable.pm line 131
#       MooseX::Traits::Pluggable::new_with_traits('Madre::Sync::Model::padreDB', 'Madre::Sync') called at C:\strawberry\perl\vendor\lib/CatalystX/Componen
t/Traits.pm line 146
#       CatalystX::Component::Traits::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Cl
ass/MOP/Method/Wrapped.pm line 48
#       Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:49]('Madre::Sync::Model::padreDB', 'Madre::Sync', '
HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 89
#       Catalyst::Model::DBIC::Schema::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/C
atalyst.pm line 2502
#       eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2502
#       Catalyst::setup_component('Madre::Sync', 'Madre::Sync::Model::padreDB') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2416
#       Catalyst::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 54
#       Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/M ethod/Wrapped.pm:64]('Madre::Sync') called at C:\strawberry\perl\ven
dor\lib/Class/MOP/Method/Wrapped.pm line 89
#       Madre::Sync::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 1142
#       Catalyst::setup('Madre::Sync') called at blib\lib/Madre/Sync.pm line 62
#       require Madre/Sync.pm called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 114
#       Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:118]() called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 74
#       eval {...} called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 67
#       Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125
#       Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137
#       Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24
#       Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a
t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493
#       Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call
ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424
#       Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S
ub/Exporter.pm line 742
#       Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per
l\vendor\lib/Catalyst/Test.pm line 112
#       Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2
#       main::BEGIN() called at blib\lib/Madre/Sync.pm line 0
#       eval {...} called at blib\lib/Madre/Sync.pm line 0
#       eval 'package main;
# use Catalyst::Test @{$args[0]};
# 1;
#
# ;' called at C:/strawberry/perl/lib/Test/More.pm line 858
#       Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p
m line 833
#       Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7"Compilation failed in require at C:\strawberry\perl\vendor\lib/Class
/MOP.pm line 114.
#  at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 121
#       Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:125]('Couldn\'t instantiate component "Madre::Sync::Model::padreDB"...') called at
C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 98
#       Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125
#       Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137
#       Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24
#       Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('Ca talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a
t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493
#       Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call
ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424
#       Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S
ub/Exporter.pm line 742
#       Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Cat alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per
l\vendor\lib/Catalyst/Test.pm line 112
#       Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2
#       main::BEGIN() called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 2
#       eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 2
#       eval 'package main;
# use Catalyst::Test @{$args[0]};
# 1;
#
# ;' called at C:/strawberry/perl/lib/Test/More.pm line 858
#       Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p
m line 833
#       Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7
# BEGIN failed--compilation aborted at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2.

Strawberry Perl install rolled back

Posted by jdavidb on 2010.07.26 15:34 (#40462)
3 Comments
User Journal

Strawberry Perl 5.12.0 was almost completely installed when suddenly it flashed some message I didn't see into the install wizard and the progress bars started moving backward! I have never seen anything like it. I realized the progress bar caption had been changed to simply "Rolling Back Action" and watched as at least three anonymous "actions" were rolled back, progress bar by progress bar. Then the install wizard simply told me "Strawberry Perl Setup Wizard ended prematurely Strawberry Perl Setup Wizard ended prematurely because of an error. Your system has not been modified. To install this program at a later time, run Setup Wizard again. Click the Finish button to exit the Setup Wizard."

I wish it would tell me what the error was so I might have some hope of correcting it.

For people running Perl conferences

Posted by Alias on 2010.07.25 7:55 (#40461)
18 Comments
User Journal

Leo Lapworth comments in "For speakers at Perl conferences" that you should record your own talks at conferences, because organisers cannot be trusted to release the videos they take of you.

I wholeheartedly agree. I've spoken at numerous conferences, including several large ones, and only in about 10-20% of cases have the videos of me EVER appeared.

I'm amazed that conference organisers can put in so much effort into recording (dozens of tapes, multiple cameras, multiple operators) and then never produce anything as a result.

The only time I'm aware of that a full talk of mine has appeared online was at a Linux.conf.au, who had a dedicated video team of eight people.

Where is all this footage, and why does it never get processed? Why even bother recording it? If you don't have time, send me the raw tape of my own talk and I will do it myself if needed.

YAPC::NA? You listening?

A reminder - Padre Second Birthday Hackathon this weekend

Posted by Alias on 2010.07.22 22:53 (#40460)
0 Comments
User Journal

This is just a quick reminder that this weekend is the Padre's second birthday party and hackathon.

If you are a Padre user, please drop in and say hello to the team. We'd love to hear how you are using Padre and where your main needs are for the next year.

If you are interested in trying out Padre for the first time, or trying your hand at improving it for the first time, it's a great time to get started because we'll have plenty of people around to provide guidance and advice.

Some of the plans for this weekend are to bring all the plugins up to date with the latest versions of the plugin API, to start the merge of the ConfigSync branch, and if we can, to start on the Madre server that will serve as the ConfigSync and Telemetry server for Padre.

I look forward to seeing you there!

Profiling your website while live on a production cluster

Posted by Alias on 2010.07.21 21:27 (#40457)
2 Comments
User Journal

As I mentioned in my last post, by day I wrangle a large web application that occasionally verges on being too complex to for mere humans to understand.

Curiously, because it is private and the income rate is high (we probably average something like $5 in gross turnover per page view) we don't have to deal with a lot of servers to deliver it, by internet standards anyway.

But the application is still way too big to profile easily by normal methods, and certainly on production it's way too heavy, even if we applied targeted profiling using something like Aspect::Library::NYTProf.

Between the web servers, transaction server, database, search engine and cache server, we are probably only dealing with 12 servers and 30 CPUs. Of course, these servers are all horribly expensive, because they are all server-virtualised, network-virtualised, doubly redundant (high-availability + disaster-recovery) and heavily monitored with high end support contracts.

One of our most sensitive issues is database load.

We have a ton of database tables (about 200) and lots of medium sized queries running across them. One of our main failure modes is that some deep change to code boosts the quantity of some significant query, which stresses the database enough to cause contention and lock-storms, leading to site failure.

Complicating things, big parts of some pages are embedded in other pages. So attributing load and lag to one part of the website, or to Oracle, is tricky and hard to benchmark in advance (although we do load test the main load paths to catch the worst cases).

For a long time, we've had a mechanism for zoned profiling the production site, so we can allocate wallclock costs to different general areas of the site.

But it is fragile and unreliable, requiring perfect maintenance and every developer to remember to write this kind of thing everywhere.

# Embed a foo widget in the current page
$perf->push_timing('foo');
foo($bar);
$perf->pop_timing;

Since you don't know this profiling system exists unless you've seen it somewhere else in the code before, and it's hard to care about something that is orthogonal to the problem you are actuall solving, this mechanism has degraded over time. While we still get some pretty cacti graphs showing load breakdown, they are highly unreliable and you can never be entirely sure if the load attribution is correct.

This kind of performance monitoring as a "cross-cutting concern" is a textbook use case for Aspect-Oriented Programming, and so in our Christmas 2009 code freeze window I set about trying to rewrite the push/pop_timing profiler using Aspect.pm.

Unfortunately, Aspect.pm turned out to be a bit too slow and naive for my needs. But after a 6 month delay to do a 90% rewrite of Aspect.pm, I now finally have something sophisticated enough (and fast enough) to meet my needs.

So today I'm releasing the shiny new Aspect::Library::ZoneTimer, which will serve as the main plank of our new production profiling system.

The idea behind ZoneTimer is to define each performance zone as a pointcut. The aspect will track the movement of your code across each zone boundaries and build a running total of the exclusive time spent in each performance zone.

When your program exits the top-most performance zone, the totals will be sent to a handler function.

A typical use of the ZoneTimer aspect looks something like this

use Aspect;
 
aspect ZoneTimer => (
    zones => {
        main      => call 'MyProgram::main' & highest,
        search    => call 'MyProgram::search',
        widgets   => call qr/^MyProgram::widget_.*/,
        templates => call 'MyProgram::render',
        dbi       => call qr/^DB[DI]::.*?\b(?:prepare|execute|fetch.*)$/,
        system    => (
            call qr/^IPC::System::Simple::(?:run|runx|capture)/
            |
            call 'IPC::Run3::run3'
            |
            call qr/^Capture::Tiny::(?:capture|tee).*/
        )
    },
    handler => sub {
        my $top       = shift; # "main"
        my $start     = shift; # [ 1279763446, 796875 ]
        my $stop      = shift; # [ 1279763473, 163153 ]
        my $exclusive = shift; # { main => 23123412, dbi => 3231231 }
 
        print "Profiling from zone $top\n";
        print "Started recording at " . scalar(localtime $start) . "\n";
        print "Stopped recording at " . scalar(localtime $stop)  . "\n";
        foreach my $zone ( sort keys %$exclusive ) {
            print "Spent $exclusive->{$zone} microseconds in zone $zone\n";
        }
    },
);

This example breaks out the cost of a typical small web application into a general zone, a special zone for the search page, and then splits the costs of generating widgets, rendering the HTML template, waiting for the database, and making calls to the underlying system.

Start and stop times are returned as a two element array exactly as returned by C, and the exclusive zone totals are returned as integer microseconds (all math is done in these integer microseconds to prevent floating point corruption).

The use of Aspect allows us to easily mix special cases via the pointcuts, such as the use of "highest" which makes sure that "main" only matches the first time it is seen, and any widget which that does a re-entry into main still has that cost attributed to the widget. We've also hooked into multiple system call modules to measure system call cost, because we know different modules our program consumes will use different methods for interacting with the system.

While the handler I've shown here will just print out a summary of the call, in our environment at work the profile report handler will format the exclusive times into a special log message and then send it via the super-quick and non-blocking Log::Syslog::Fast module to the UDP localhost syslog service, where it is mirrored to disk for debugging use, and then forwarded on to our main company-wide syslog server.

On our setup, we can then use an excellent commercial log analysis product called Splunk (limited free version also available if you want to try it out) to do our tracking and trending of the performance for the entire application across the entire cluster.

The nicest thing about the Aspect system is that it scales the performance cost and complexity risk directly to the complexity of the use case you are using it for.

If it turns out that the Aspect hooks are too intrusive and causing a performance drain, or accidentally causing some weird edge case behaviour, you can simply turn off that the Aspect and restart the servers and the performance penalty just vanishes.

Maintenance of the profiler zones is really easy, because they are all listed clearly in one place.

Depending on your use case, you could even define the performance zones in your website configuration, and then adjust the profiler zone boundaries directly on production (although I'm not sure I'd want to do that in our specific case, since we're fairly paranoid about untested code going into production).

This ZoneTimer is just the first of several Aspect-based tools I plan to release over the next few months. If you get a chance to try out any of these new toys, I'd love to hear feedback on how well they work for you.

In addition to the value of the ZoneTimer aspect itself, one other thing that some people might find of interest is just how little code it took to get this all working.

The entire Aspect::Library::ZoneTimer was implemented in about 80 lines of code, which you can see at http://cpansearch.perl.org/src/ADAMK/Aspect-Library-Timer-0.04/lib/Aspect/Librar y/ZoneTimer.pm

This is small enough that you could just clone the module and tweak it for your own use case, if you wanted to creat a new and tricky customised profiler.

How we deploy massive Perl applications at work

Posted by Alias on 2010.07.19 21:25 (#40453)
4 Comments
User Journal

Every now and then, we hear people talking about mechanisms for doing Perl in a commercial environment and how they deal with packaging and dependencies.

This is mine.

At Corporate Express, our main Perl application is a 250,000 line non-public monster of a website that has over 100,000 physical users and turns over about a billion dollars. It implements huge amounts of complex business functionality, and has layer upon layer of security and reliability functions in it because we supply to multinationals, governments and the military (only the stuff that doesn't blow up of course). Our .pm file count is around 750, and our test suite runs for about 4 hours (and is only around one third complete).

Lest you suspect that 200,000 lines is wasted in re-implementing stuff, the main Build.PL script has around 110 DIRECT dependencies, and somewhere in the 300-500 range of recursive dependencies. Loading the main codebase into memory takes around 150-200meg of RAM.

When I joined the team, the build system was horribly out of date. The application was stuck on an old version of RedHat due to go out of support, and as a Tier 1 application we are absolutely forbidden from using unsupported platform.

So I took on the task of upgrading both the operating system and the build system for the project. And it's a build system with a history.

Once upon a time, long ago, the project went through a period where the development team was exceptionally strong and high skilled. And so of course, they created a roll-your-own build system called "VBuild".

They built their own Perl, and along with it they also built their own Apache, mod_perl, and half a dozen other things needed by the project. This is similar to many suggestions I hear from high-skilled people today, that at a certain point it's better just to build your own Perl. VBuild this created in the pre-commercial Linux era, so it's not an entirely unreasonable decision for that time period.

Unfortunately, a few years later the quality of the team dropped off and VBuild turned into a maintenance nightmare because it required a high-skill person to maintain it.

At the time, the Tier 1 "Must be supported" policy was coming into effect, and after the problems with custom-compiling they decided to go with the completely opposite approach of using only vendor provided packages, in a system called "UnVBuild".

Since their platform of choice was RedHat, this had become troublesome even before I arrived. Worse, in the change from RHEL 4 to RHEL 5, some of the vendor packages for things like XSLT support were dropped entirely leaving us in a bind.

My first instinct was to return to the build everything approach, but the stories (and commit commentary) from that time period reinforced the idea that complete custom build was a bad idea. Office supplies is hardly a sexy industry, and the ability to entice good developers into it is a quite legitimate risk.

So in the end, I went with an approach we ended up nicknaming "HalfBuild". The concept behind it is "Vendor where possible, build where needed".

We use a fairly reasonable chunk of vendor packages under this model. Perl itself, the Oracle client, XML::LibXML and a variety of other things where our version needs are modest and RHEL5 contains it. We also use a ton of C libraries from RHEL5 that are consumed by the CPAN modules, like all the image libraries needed by Imager, some PDF and Postscript libraries, and so on.

One RPM "platform-deps" meta-package defines the full list of these system dependencies, and that RPM is maintained exclusively by server operations so that we as developers are cryptographically unable to add major non-Perl dependencies without consulting them first.

On top of this is one enormous "platform-cpan" RPM package built by the dev team that contains every single CPAN dependency needed by all of our Perl projects.

This package lives in its own home at /opt/cpan and is built using a process rather similar to the core parts of Strawberry Perl. With PREFIX pointing into /opt/cpan, we first build a hand-crafted list of distribution tarballs to upgrade RHEL5 to a modern install of CPAN.pm (without overwriting any normal system files).

We then boot up the CPAN client from /opt/cpan, and tell it to install all the rest of the dependencies from a private internal CPAN mirror which is rsynced by hand specifically for each new CPAN build. This ensures that the build process is deterministic, and that we can fix bugs in the build process if we need to without being forced to upgrade the modules themselves).

The CPAN client grinds away installing for an hour, and then we're left with our "CPAN Layer", which we can include in our application with some simple changes to @INC at the beginning of our bootstrapping module.

The /opt/cpan directory for our project currently weighs in at about 110meg and contains 2,335 .pm files.

Updating /opt/cpan is still something of an exercise even under this model because of potential upgrade problems (Moose forbidding +attributes in roles hit us on our most recent upgrade) but the whole process is fully automated from end to end, and can be maintained by a medium-skill Perl hacker, rather than needing an uberhacker.

Over the last 2 years, we've upgraded it around once every 6 months and usually because we needed to add five or ten more dependencies. We tend to add these new dependencies as early as we can, when work that needs them is confirmed but unscheduled.

We also resort to the occasional hand-copied or inlined pure-Perl .pm file in emergencies, but this is temporary and we only do so about once a year when caught unprepared (most recently Text::Unidecode for some "emergency ascii'fication" of data where Unicode had accidentally slipped in).

While not ideal, we've been quite happy with the /opt/cpan approach so far.

It means we only have to maintain 5 RPM packages rather than 500, and updating it takes one or two man-days per 6 months, if there aren't any API changes in the upgrade.

And most importantly it provides us with much better bus sensitivity, which is hugely important in applications with working lives measured in decades.

Hiveminder: personal RT, for free

Posted by jdavidb on 2010.07.19 11:58 (#40452)
0 Comments
User Journal

You mean someone will provide RT for me to use for free on the web? And they've built an awesome AJAX-y frontend for it? And they allow me to tag tasks and they encourage me to keep my work todo list and as many personal todo lists as I want in here? And they give me awesome search utilities for figuring this out and keeping it organized?

It's almost like a dream come true.

Padre Second Birthday Party 24th-25th of July

Posted by Alias on 2010.07.10 11:50 (#40441)
0 Comments
User Journal

2 years ago this month, Gabor did the first Padre release.

The last 12 months has seen Padre mature from a high-end text editor to a low-end refactoring IDE. We've stolen a number of features from Ultraedit, Komodo and EPIC, and we've invented new features all of our own, making Padre a very fluid and natural place to write Perl in.

We've added support for Perl 6, Template Toolkit, remote file support, more languages, syntax checking, an interactive debugger, a regex editor, and our first half a dozen refactoring tools.

We've also greatly solidified the code. Window integration is now totally solid, we've added a resource locking API, a new filesystem API, a new search API, a new display API, rewritten the threading and background Task subsystem, heavily overhauled the Plugin Manager API and GUI, and added Advanced Preferences and the ability for advanced users to selective disable various Padre bloat/features.

The last couple of months have also seen great improvements in Padre's hackability as well. The new Task 2.0 API lets people write background logic and consume multiple cores of CPU without having to know how threading works, and the new wxFormBuilder plugin lets you build GUI code without having to know Wx (one of the biggest barriers to contributing to Padre).

On the weekend of the 24th-25th of July we would like to invite all Padre developers, users, friends and well-wishers to join us for Padre's Second Birthday Party and Hackathon in the Padre IRC channel at irc://irc.perl.org/#padre or via the Mibbit Web Client.

If you've always been curious about, or interested in hacking on, Padre we'll have a number of developers in channel to help you out.

Personally, I plan to debut the first public release of Padre::Plugin::FormBuilder, and to start ripping out all Padre's older fixed-size dialogs and replacing them with new shiny model-generated sizer-based dialogs that will work much better across all three operating systems.

If you'd like to help out in this effort, I'll be in channel most of the day on both days (Sydney timezone).

I look forward to seeing you all there.

Yahoo provides (awesome) alternative Geocoder to Google's

Posted by Alias on 2010.07.05 23:10 (#40436)
1 Comment
User Journal

As far as I'm concerned, there are three critical things you need in a Geocoder service.

1. Global Coverage

Because when it doesn't have global coverage, it's basically means "America-only" and thus pointless for most of the world.

2. Multiple Matching

For ordinary humans, there's massive power in being able to just search for "1 Oxford Street" without listing any more details. From there, if it is a country-specific application you just change it to "1 Oxford Street, Australia" behind the scenes.

That results in a list of possible locations.

You show the results for the first result, and then a list of "Did you mean:" links for the other results. This lets the application do what people mean most of the time, while making it trivial to recover if it isn't accurate enough.

You can see the effect I'm talking about by running the example query here...

http://geo2gov.com.au/

3. No usage limitations

Google Maps Geocoder can only be used with Google Maps. Fail.

I'm willing to accept a volume limitation like "You have to pay after the first 50,000 requests" but I can't accept a usage limitation.

I'm happy to report that Yahoo's new PlaceFinder service is the first free Geocoder that meets all these primary criteria.

It's global in scope, lets you control the list of results with paging, and doesn't limit the usage to any particular domain.

Now all we need is an Geo::Coder plugin for it.

Please write one for me Lazyweb :)