Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

btilly (5037)

btilly
  (email not shown publicly)

Journal of btilly (5037)

Monday June 11, 2007
02:54 AM

Interesting insights from Software Estimation

Note: This is a letter that I wrote to Steve McConnell after I read his most recent book. I'm publically sharing it because I learned something that may be of general interest from thinking about the topic, and I'd like to see more people read the book.

I finally got around to reading your most recent book. As always with your books, I found it fascinating. So fascinating that I'm writing you an open letter which I am going to publicly post elsewhere.

First of all, as I expected, you hit it out of the park. You've done a very good job on a difficult topic that our industry normally does a horrible job on. Very few of your readers have any clue how poorly they judge what is 90% likely. It is incredibly helpful to be conscious of how people confuse estimates, targets and commitments. You were absolutely right to back up your advice on how to create accurate estimates with advice on how to defend those estimates from organizational pressures to replace them with wishful thinking. And, as always, all of the advice is backed up by invaluable compiled (and meticulously referenced) data on everything from how uncertain the best possible estimates are at various stages in the software lifecycle to how wide the productivity variation is from company to company.

As with any book, it is not perfect. However the overall quality is extremely high and the remaining imperfections are small. Furthermore you took into account all of my criticisms for the one chapter that I reviewed. Since I had the opportunity to review the rest of the book and didn't, I feel that any oversights that I notice are more my fault than yours.

Needless to say I highly recommend this book to everyone involved in the software development process. And my main difficulty now is identifying who in my immediate environment I should lend it to first. (ie Who would create the biggest positive impact on the company I'm in.)

As is often the case with your work, of even higher value is how close reading leads to or reinforces insights on other parts of software development. Sometimes this is presented in an understated way. Such as the paragraph on page 64 that says, "...individual performance varies by a factor of 10 or more. Within any particular organization, however, your estimates probably won't need to account for that much variation because both top-tier and bottom-tier developers tend to migrate toward organizations that employ other people with similar skill levels." (A fact which you then provide 2 references to.) I laughed aloud at that one.

Sometimes the tangential gems are presented very directly. For a random example on page 69 you point out that multi-site development increases needed effort an average of 56%. As you say, this effect should be carefully considered by organizations considering outsourcing. And while most software professionals understand that this is a significant factor, very few of us can quantify it. Which makes it hard for us to get businesses to take it seriously.

And sometimes the insights are not directly presented. They are just implicit in the copious data that you've presented, waiting to reward the careful reader who can spot them. I'd like to talk about one of those.

It has long been a mantra among people who like dynamic languages that developers are more productive in small groups, and so there is great value in delivering languages that make small groups as productive as possible. I cannot count how many times I have seen variations on this theme, nor can count how many times I have personally repeated it. Supporting anecdotal evidence is easy to find. However until I read your book, I'd never seen concrete quantitative evidence that I could quote to support what is common knowledge in some circles.

Well I'd long known evidence for part of that assertion. Variations on the chart that you reproduce as table 5-3 on pages 64-65 have been circulating for ages. And while I agree with your conclusion that it is more productive to use a language such as Java instead of a language like C, I'd also point out that it is more productive to use a language such as Perl instead of a language such as Java. Interpolating from that chart with too much precision, about 2.4 times. I hadn't before seen the more detailed table 18-3 that you offer on page 202. Judging from that, average Java programmers need 2.75 times as much code as average Perl programmers to do the same task. Those estimates agree since neither is very precise - Java takes somewhere between 2 and 3 times as much work for the same task as Perl.

Of course coding is but one of the tasks that needs to happen in software development. If only half of your development time would have gone to coding (a reasonable estimate based on table 21-4 on page 236), then reducing coding time to 40% of what it was only saves you 30% of overall effort. Still that is a significant reduction. Why don't people pay more attention to it?

The catch is, of course, that Java has many features that make it much better than Perl for handling the challenges of development in large teams. Therefore it is easy to dismiss the productivity benefit because "Perl is not scaleable." And it is easy to likewise dismiss the anecdotal accounts of exactly how productive small teams are because common sense keeps us from accepting that 6 people do more than a dozen.

Which is part of the reason why I am grateful to you for reproducing figure 20-3 on page 229. I've heard estimates before that it takes a team of about 20 people to match the output of a team of 6-7, but I'd never before seen concrete data backing that up.

Anecdotaly the primary cause is well-understood: people are most effective in a flat team, but that only works for teams up to about 6-8 people. With that structure you have little to no overhead from having to manage process, or from people not being able to find out what they need to know when they need to know it. But that falls apart when there are too many lines of communications. The solution to that problem is to introduce process to cut down who needs to talk to whom, when. However adding process drops productivity per person significantly, meaning you have to add more people. And this cascades until you get to the same productivity with a far larger team. But then you can scale for a lot longer, but at far higher cost.

There are secondary issues that are also well understood. For instance you're likely to find a higher portion of good developers in the small team environment? Why? Well there are a lot of reasons. First of all it is clear that it is easier for an individual to be productive in the small team than in the large one. People who are drawn to productive environments are likely to be people who value their personal productivity, who are therefore likely to be productive people. Conversely it is much harder for an incompetent developer to hide in a small group than a large one, so the worst developers don't stay. Additionally, given comparable turnover rates, one can maintain staffing levels in a small group while being more selective about candidates than one can in a large group. And finally a company that understands the cost benefits of having a small group of good people can justify higher individual salaries for those people.

So the 3-1 individual productivity difference in lines of code between small teams and large teams has a number of causes. It really isn't as simple as saying, "Move 2/3 of your 20 person team away and you'll get the same productivity." However that said, the line of code measure may be hiding some more dramatic productivity differences.

Some are very hard to quantify. For example common sense tells us that a team of 6 people that all talk to each other is going to have more consistency across 57,000 lines of code than a team of 20 people who are deliberately being kept from talking all the time. That lack of consistency is going to show up in all sorts of bad ways, from re-invented wheels to misunderstood internal APIs.

But one is easy to quantify: the small team is much more likely to be using a productive interpreted language than the large one. So the 57,000 line project delivered by the 7 person team might well have 2-3 times the functionality of the 57,000 line project delivered by a 20 person team in about the same time. (As I've noted, the productivity difference comes from a combination of factors, including having better people.) Even if you're paying those programmers 50% more per person, your productivity per dollar is about 5 times better with the small team than the large one. That's a pretty dramatic difference. While I'll be the first to admit that there are limits to what small teams of good people can do, I'll also stand in line to point out that those limits are farther out than most people realize, and there is a very good business case for relying on small teams whenever you can.

Anyways, congratulations on yet another excellent book, and I'm sure that I'll be digesting its consequences for a long time to come.

Cheers,
Ben

Thursday June 07, 2007
06:22 PM

What does DBD::Pg not liked named parameters?

At $work I'm getting a few basic tools set up. Since we use PostgreSQL, I'm using DBD::Pg. With DBD::Oracle I grew to like some of the more flexible ways of entering parameters, so I look for that in DBD::Pg. I find that there are three options.

1. The DBI standard ?. If I was happy with that, I wouldn't be reading this documentation.

2. They allow positional parameters with $1, $2, $3. Given that this is inside of Perl, I'm going to constantly be wondering whether I'm accidentally interpolating in variables. Better than ? but I don't really like the syntax. The visual disambiguation from variable interpolation was a reason to prefer DBD::Oracle's :1, :2 and so on.

3. They support full named parameters. :foo. :bar. Yay! I like it! But they go on to say, While this syntax is supported by DBD::Pg, its use is highly discouraged.

Huh? Does anyone know why they discourage the cleanest and most flexible solution? Personally whenever I get the chance to work by name rather than by position, I jump at the opportunity. But I'm somewhat reluctant to use a feature that the software authors don't want me to use without knowing the reason for that dislike...

Tuesday May 22, 2007
10:49 AM

No longer a programmer

I never intended to be a programmer. By personality I'm a poor fit for being a programmer. And now I'm somewhat officially no longer a programmer.

It has been an interesting trip.

About a decade ago (only a decade ago? Seems longer) my wife and I left grad school and she entered medical school. I needed to make money. I had no work experience and so was willing to take any job I could get. Given my math background, and the fact that I'd be living in New York city, I figured that my best options were to be an actuary, something in finance, or a programmer. Of the three I thought that the least likely was to be a programmer. I'd done some programming when I was younger, and had never liked it much.

It took me a month to find a job. And the first job that I found was as a programmer working for a churn and burn consultancy. I can't say enough about how horrible the experience was. But I did wind up learning databases (mostly Access) and VB, and I was given the opportunity to learn Perl. While programming still wasn't a perfect fit for me, Perl was a good fit. And so, less than a year later, I went to another company.

At that company I had the fortune to work with an extremely talented programmer named Frank, who went from someone who was consulting with the company, to a fellow programmer, to my boss. Along the way he, directly and indirectly, taught me most of what I know about programming. (Note, not most of what he knew about programming.)

That job ended several years later because I moved. When I moved to the west coast I considered either a finance or a programming job, and found the programming job first at an excellent company named Rent.com. Not too long after I joined, we were purchased by eBay. I've done well here, and can absolutely recommend this as a very good place to work at. Certainly the best that I've worked at. In fact I'd recommend that anyone good who wants to work in Santa Monica should ask me how to apply there.

Along the way I joined some online communities, made friends, learned a lot more, and generally enjoyed the experience. And I've become a competent programmer. However an ongoing issue is that, no matter how capable I might become, I've really got the wrong personality to be a programmer. The key problem is that I'm too extroverted. Sit me down to work on any significant project, and before long I need to surface for air and talk to someone. After a week of this I can feel my motivation level slip. Which means that I actively avoid projects with long periods of heads down development. For similar reasons, unlike most programmers I know, I simply don't wind up taking on personal programming projects.

Despite this issue I've been productive. And there are ways to accomodate me. Certainly Rent has bent over backwards to do so. However how many companies will do so? And when I combine that with the ageism that some older friends have experienced, I've long doubted that programming is a sustainable long-term career path for me.

The problem has been what could come next.

Well at Rent I've wound up in a reporting role. This works out well for me. Lots of people need data. They don't always understand what data they need, so they need to talk with someone who can talk their needs through with them. And most of the time finding ways to get that data tends to be a short project with immediate results. In the process I've wound up learning a bit about how business decisions actually get made.

As this has happened, I've slowly drifted away from programming. That is not to say that I don't sometimes write or modify programs. I do, and that is unlikely to change. However I'm doing a lot less of it, and things like becoming better at programming or staying in touch with the current best environments, modules, and practices is feeling ever less relevant to my life. This is one of the reasons why I've drifted away from communities like Perlmonks. (A bigger reason is named Sam. It is amazing how much spare time vanishes when you're a parent. It is worth every moment, but it is still a lot of time.)

But what has happened unofficially has now become somewhat official. I've just given notice at Rent.com. This was a very hard decision for me. I love the company, the organization, the technology, etc. However I have an opportunity to work with friends much closer to home at a company with good opportunities (but admittedly also with some problems) which will give me more of an opportunity to explore how much I like being engaged with the business side. So, with considerable regrets, I'm going from an official title of senior software engineer at eBay to reporting architect at Pictage. To me this is the clearest milestone in my slow drift out of programming saying that I'm no longer a programmer.

Ironically I actually will be programming more (at least at first) in my new role. There is a lot of basic infrastructure that is missing and I'm going to have to create that. But this will be a fairly limited project. And, if all goes well, I'll wind up doing less and less programming over time. And I will no longer be reflexively describing myself as a programmer.

For whomever reads to this point, thank you for letting me ramble.

Friday March 23, 2007
11:57 PM

MySQL has row-level locking? Really?

One of the DBAs here was playing around with MySQL. He'd heard that the InnoDB engine has rowlevel locking. So he created an InoDB table called cjg (his initials) with two columns, col1 is a number and col2 is text. He inserted a few rows. He set autocommit off. He then ran:

update cjg
set col1 = 2
where col2 = 1;

Then in another session he tried to insert the value (3, test) into cjg. It blocked.

He tried a variety of things, but was unable to get MySQL 5 to demonstrate that it knew how to do row-level locking.

So..can anyone come up with a demonstration where MySQL clearly does row-level locking? And can you explain why the example he tried to run locked the whole table?

Thanks,
Ben

Friday April 21, 2006
10:04 PM

Goodbye, Bill

My uncle Bill died today. He is survived by his wife Lorna of 58 years, 6 sons, a daughter, many grandchildren and more memories. He will be missed.

Pretty much by accident, my wife and son met him April 2 and 3. I'm glad they did.

Funeral details are not set yet.

Ben

Saturday August 20, 2005
01:02 PM

The story about ping

Scroll down to and read the first review. :-)
Monday August 08, 2005
01:25 AM

Are there videos of the lightning talks at YAPC::NA?

I saw Luke Closs' hilarious juggling lightning talk. I despair of describing it to co-workers. Therefore I'd like to send a video around instead.

Unfortunately I can't find one.

He says that it was filmed at YAPC::NA, but he has not seen the video anywhere.

Is there hope that someone knows where such a thing might be found?

Thanks,
Ben

Thursday July 21, 2005
06:01 PM

If O'Reilly cooperates, I'll go to OSCON

I was asked if I was interested a couple of months ago, but it wasn't until yesterday that approval came through for the company to pay for it. Of course then I had to OK it with my wife (because she's working and it will leave her with extra child care she wasn't prepared for). That's OK as well, so I'm going!

So I bought my airline ticket, then tried to buy a conference ticket. Wrong order. O'Reilly can't process my order. :-( I sent them an email, got no reply. So I waited a while, tried again from the same form and got no confirmation page. My credit card didn't show a charge, so I figured that there was a timeout on their end. I tried from the start again, got to the end and again they can't process my order.

So right now I'm flying to Portland at the same time as a conference that I can't purchase a registration at. Wonderful.

In other news, my employer is looking for another developer. If you're good at Perl and are interested in working for Rent.com (a subsidiary of eBay) in Santa Monica, I can put you in touch with my boss.

Saturday April 09, 2005
07:04 AM

Sometimes the solution is simple

Many moons ago I had an idea for a useful module. Unfortunately I ran into a small roadblock and I dropped the project.

Well tonight I realized that instead of installing into UNIVERSAL::AUTOLOAD I could just install into AUTOLOAD where requested and document the problems that AUTOLOAD always has and let people request their own AUTOLOADs, and accept the problems that come with it.

I have no idea why I missed it before.

Wednesday March 16, 2005
02:58 AM

Never believe what you see in the news

It seems that every time anything that I know about gets near a reporter, I have cause to wince. Witness that I appear to have moved to Michigan.

Of course slashdot is worse yet, they claim that the University of Minnesota is now Southwest Missouri State University. Um, not quite.

Seriously, friendly complaining aside, the publicity is unexpected and great to see. But brief exposures like this make me wonder at how little I should trust the news that I normally see. Even when the government doesn't deliberately manipulate it.