Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

dws (341)

dws
  (email not shown publicly)
http://www.davewsmith.com/

Journal of dws (341)

Sunday August 18, 2002
11:44 AM

Playing with Graham's Bayesian filter

[ #7149 ]

I spent yesterday in blissful play, at least at first, running a bayesian spam filter against a big backlog of mail. I'm getting no false positives on good mail, and ~5% false negatives on message in my spam file. Exploring this, I first found that there were some messages in the spam pile that shouldn't have been there. Oops. Fortunately, none were job offers.

The other false negatives where more interesting. One class of them are very short spams. Here, I'm suspecting that Graham's algorithm needs to be tuned to use less than 15 weights for short messages. Gotta play with this one.

The other class of false negatives leads me to suspect that Graham simplified a few things for purposes of writing a compelling article. His examples show that he's cannonified tokens to lower case. My first guess was that he'd done this to save space, and I dutifully translitered. But this misses obvious things like "25 MILLION EMAILS". From my sampling, "25" carries a weight of 0.3935, "million" weighs 0.2794 and "emails" weighs 0.7363. Assuming these are even selected by the algorithm, they tilt toward non-spam. The obvious thing to do is retain case, but this blows the data way up, and scanning a 1Mb weighting table for each email has got to suck. I'm going to try converting to lower case if and only if there's already at least one lower-case character in the token.

However, a couple of folks have requested code, so next up is to get this working with procmail. Then a thousand experiments can bloom.

Fun. Much fun.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.