Important caveat before you read: please don't read this blog post as Ovid saying "don't evangelize Catalyst, don't write tutorials, etc." I'm not saying that. These things are important and we should continue them. I'm saying that we need a core team to get together to really understand the nature of the problem we face rather than relying on anecdotal evidence. Now on with the show.
Yesterday I wrote Perception is Reality - we need a director of marketing. After a few hours I was caught on IRC by some people who said the main problem why companies are not picking Perl or why they are leaving Perl is that they cannot find good Perl programmers. As reaching programmers to learn perl is not the job for a marketing director hence we don't need a marketing person.
Gábor doesn't buy this argument and he's right not to because this reflects a very fundamental and common error of identifying the solution as the problem. The reasoning works like this: "companies aren't finding enough Perl programmers so we need to teach more programmers Perl." The error can be illustrated by a slight change to that: "companies aren't finding enough COBOL programmers so we need to teach more programmers COBOL."
The problem is identified as "not enough Perl programmers", but that's actually a component of the solution because it completely fails to address the problem of why we don't have enough Perl programmers. Since this error has been made, I really have to attribute a large part of that to myself because I've failed to communicate as effectively as I can, so I'll try to take another swing at it. Let's look at a few proposed solutions to what I'm calling P3 ("Perl Perception Problem") and see what new (or old) perceptions might result. To be fair, many of these "solutions" might very well reflect a perception of a different problem.
(Note: read the next paragraph carefully. I've been accused of trashing Perl and I expect that's because people are skimming what I write.)
Notice that I am not saying I agree with any of those hypothetical responses. Nor am I saying that those are the responses we'd receive. Further, all of those proposed solutions have merit, but the one (or ones) which will gain us the most bang for our buck are unknown. Heck, parts of the solution might be things we've not even considered yet.
One danger of proposing solutions without understanding the underlying problem is that we may come off as self-serving and we don't want to do that. To avoid it, one thing we must realize is that not all of our critics are wrong. Consider PHP. They mopped the floor with us on what many considered our home turf. They've obviously done something right. We might make vague claims of "technical superiority", but tell that to SmallTalk and Scheme developers. They also think they have "technical superiority", but superiority for what? If we don't understand the problem, we can't say what we're superior for! If our solutions are truly self-serving, they deserve to be ignored as there's a good chance we're solving the wrong problem. Just look at the RIAA. Many people involved with that really believe they're doing the right thing just as we're convinced they're not. It's easy to hear what one wants to hear.
In regards to understanding the problem, chromatic wrote:
I think we can both agree that marketing activities should reflect reality. That's why I base my conclusions [regarding the Perl 5 development process] on (and refer to) publicly accessible raw data, such as timelines, release dates, bug reports, patch submissions, commit logs, documentation, and mailing lists are all public information.
Relying on raw data is important and I'm happy chromatic is doing that, but few, if any of those sources really tell us why we have a problem (the mailing lists sound tempting, but we're really in an echo chamber). For example, let's say that you're consulting your access logs and you find many searches for the term "email". You might conclude that you have a support problem and people are looking for contact information. It might be spambot email harvesters. We're the BBC. We might have people thinking they can find celebrities email addresses online. The thing is, you can't know by just looking at raw data. You can only guess. If we can gather better information about P3, we still won't know the answer, but we can make better guesses.
To understand P3, we need to do research. Some people have mentioned the financial aspect and that's a worthwhile concern, but we don't necessarily have to spend money -- though we will need to spend time. I doubt many of us are experts in market research but maybe someone is and is willing to volunteer services? Maybe we can team up with another open-source group to facilitate this? Maybe we can find a university teaching marketing and propose an interesting research project and domain expertise? Do we have contacts for any of this? If we want to understand this problem, we can. We just need to be sufficiently creative and motivated to get to the bottom of it.
Dave Cross and I (and Rozallin Thompson, I believe) will be meeting in Lisbon at YAPC::EU 2009 to discuss this issue. I encourage others to join us. Let's tackle this beast once and for all. Our careers depend on it.