In a recent study, Princeton University researchers asked 200 volunteers to play leader in a 2-country wargame, predicting their own chances before they started, and recording the results of the games (which were played anonymously, and for a $10 cash prize).
Self-confident people that rated themselves highly were more likely to make unprovoked attacks (attacking was optional, you could also win by economic means) and were less likely to emerge victorious.
According to psychological tests also made on the volunteers, those self-confident people were also more narcissistic (had a higher intrinsic sense of self-worth). But there was no relationship between gender and likelyhood of attack.
Their conclusion, narcissism, and not maleness, makes some people overly optimistic and aggressive.
As someone with a keen interest in understanding (and hopefully eventually predicting) the medium to long term success of software projects, I can't help but wonder if we have analogies in the software world.
Could "attacking a problem" and subsequent victory against the problem be seen in a similar light?
I certainly see some tentative evidence for supporting a relationship between high lead developer humility and pragmatism, and phenomenal long term success (take Perl, SQLite and Linux for example).
Similarly, one might posit that the opposite exists for a number of modules that start with high cpanratings, and then gradually get worse over time (I know of 3 obvious ones, but there are more less obvious ones) or that have consistently high bug counts.
Is there something we can learn from this?
Should developers consider the idea of testing themselves for narcissism as warnings to themselves that they are naturally too optimistic? It's certainly one of the more unusual ideas for improving yourself as a developer.
Certainly I tend to be of the opinion that the best software designs are the ones that have been reviewed by a lot of people willing and happy to tear your idea to shreds and ruin all your nice assumptions. #perl is particularly useful for this task
Just as good is getting in the habit of routinely and systematically tearing your own ideas apart. Many design decisions you make are likely to be wrong, but you aren't yet aware why. God knows I've made dozens and dozens of them.
If you get in the habit of second-guessing yourself (by actively consulting with experts in any area you aren't an expert) you can save yourself massive amounts of pain and work in a year's time. There's almost certainly dozens of problems with your new idea, you just don't see them yet.