use Perl

• Similar code... (Score:3, Insightful)

  by Matts (1087)

  I also posted very similar code [sourceforge.net] to do this to the SpamAssassin-Talk mailing list yesterday, in case anyone is interested in a slightly different encoding of the algorithm.
  
  I use my own mail parser class that doesn't use memory (it uses temp files instead), and decodes all the MIME stuff for you. Might be worth checking out too in case anyone is interested.
  
  We'll probably plug this into SA 2.41+ or SA3 (whichever comes first).
  • Re:Similar code... (Score:1)

    by dws (341)

    If you've got SpamAssasin covered, I'll keep going on a Mail::Audit plugin (which also handles MIME). I've reworked the algorithm to scan the weighting file after tokenizing the message body. No more sucking everything into memory.
    
    By the way, a simple tokenizer tweak cut my falst negatives in half. I only force a token to lowercase if at least one character is already lowercase. This has the effect of keeping a separate (high) weights for "MILLION" and "EMAILS" than for "million" and "emails", which have l
    • Re:Similar code... (Score:2)

      by Matts (1087) on 2002.08.20 10:20 (#11988) Journal

      I've reworked mine to use SQLite. Seems to work well. Database is 17MiB though, so I think I need to investigate other options too.
      
      The upper/lower case thing didn't make one squat of a difference for me.

      Reply to This

      Parent

      • Re:Similar code... (Score:1)

        by dws (341)

        Are you sure you're tokenizing consistently? I got bit by that once.
        
        When querying, are you going after tokens one at a time, batching up requests using IN (), or trying to get them all at once using a JOIN?
        
        • Re:Similar code... (Score:2)

          by Matts (1087)

          Inserting into a temporary table, and using a LEFT OUTER JOIN.