Once I finished writing PPI (a process that spanned 3 years) it occured to me that I had reached my scaling limit. There was just no way I was ever going to be able to write anything larger than that on my own and remain effective.
Ever since, my favourite area of computing has become what we call "Programming in the Large". Dealing with the effective design of code and implementation of concepts far larger that you could ever hope to actually write yourself.
It means learning as you can about things like iteration and evolution can be intentionally applied to your design, learning how to treat concepts like "trust" and "property" as fundamental elements of design. And learning how to anticipate areas where, if you just give it a little bit of a push (like the Win32 Perl effort), you can get things past a sticking point and the natural inventiveness of others will take over and do the rest of the work for you (and a big thankyou to everyone that has been putting so much work in lately finding and fixing CPAN modules on Vanilla/Strawberry Perl)
Because I really like this area, I've also gotten much more heavily involved in the CPAN. After a couple of years immersed in it, I think I mostly "get" the CPAN. Exactly why it works so well, when others have failed, and the areas it should be doing better.
But one issue has remained a bit of a challenge for me, and that is the concept of "choice" as an element of design.
When it comes to common areas, CPAN often has more than one module that does the same thing. As a result, people searching for modules need to choose. That we have choice, and that people can upload any old crap they like, is one of the ultimate reasons for the CPAN's success, because in an iterative and evolutionary environment, betting on any single solution exclusively can be very risky.
Just look at Maypole, who could never have predicted that the ORM module they chose (Class::DBI) would so suddenly fall out of favourite with developers and suddenly turn Maypole from the cutting edge into a second-tier choice amoung the Perl technorati.
It is notable that its logical successor Catalyst embraced choice and diversity, and so when the Class::DBI debacle occured, it was quite a different story. While maintaining Class::DBI for compatibility, they have switched their default choice to DBIx::Class and have moved on with relatively little negative effect.
But one of the perplexing problems with choice is the problems of too much choice.
I've had it put to me by several Perl trainers that teach university students that having 4 modules for roman numerals is completely silly. It causes a problem for them, because students are trained to look for the one correct answer, so any university assignment featuring roman numerals causes much consternation.
The message from trainers on this issue of choice has been very consistent.
And one need only look at the number of modules for parsing Windows
Sometimes having 5 modules to choose from, each for a different situation or that makes different trade-offs, is good. Sometimes 5 is too many...
So I was curious to see today that there is work being done on this very problem. On exactly how MUCH choice is a good level of choice. And how too much choice can make people sick.
For all those Programming in the Large devotees, the following video is recommended viewing.
Sorry about the RealPlayer format, but thanks go to the ABC for making it available to the world for free, from their (co-incidentally named) Catalyst science TV program.