On Perl Monks once in a while I get a message that states "Ack! You just lost 1 experience point." That means someone disliked what I wrote somewhere enough to downvote me.
That's OK. If I were the cheeriest optomist in the world I wouldn't expect everyone to like or agree with what I write. The thing that nags me is the 'somewhere'. I haven't written a tremendous amount on Perl Monks, but I have written enough so that it is not worth the time to go looking for the node that someone downvoted.
The reason I would like to know what the anonymous monk disliked is to give me a chance to improve my writing, either technique or content. To know that that was the second (or thirty-second) time a node I wrote was downvoted would be a help toward improving.
But no one will write to tell you why they disliked what you wrote, whether you were an idiot, or wrong or just a bad writer. Sort of like wandering blindfolded through a dark house where people run up and hit you with sticks. Its not the pain, its the not knowing.
I have never downvoted any node myself. It is not nobility or a forgiving nature, heaven knows. It is more the fact that my knowledge of Perl is not great enough for me to be sure that any particular node is wrong enough to earn a downvote. And stupidity is not the monopoly of anyone, as well vote against the rising tide. As for trolls, I leave them to the editors and the gods. Why waste a vote?