Matts (email not shown publicly) I work for MessageLabs [messagelabs.com] in Toronto, ON, Canada. I write spam filters, MTA software, high performance network software, string matching algorithms, and other cool stuff mostly in Perl and C.
Why don't web crawlers send Accept-Encoding: gzip. It's very frustrating to have AxKit web sites and the biggest bandwidth suckers of them all are crawlers. Simple web browsers are using 1/10th of the bandwidth of these beasties. Very annoying.