Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

ooh, search engine fun 2 Comments More | Login | Reply /

 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.

  • Cool it looks like I could build a funky vector space search engine using the mysql cosine function.

    Not quite. MySQL has a scalar cosine function, like Perl and damn near every other programming language you can name. Because this is a vector based search engine, you need to take the cosine of the two vectors, as the article clearly states:

    The formula for calculating the cosine is this:

    cos = ( V1 * V2 ) / ||V1|| x ||V2||

    Where V2 and V2 are our vectors, the vertical bars indicate the

    • I guessed it wouldn't be that easy ;)

      That confirms my initial concerns that it wouldn't scale well, as you would have to do a lot of mathematical grunt work in Perl (or more likely the C PDL library), as opposed to using a specialised tool like a relational database to the bulk of the processing.

      I don't think MySQL would manage the cosine math involved here. Oracle probably would using horrid PL/SQL perversions, but that would subtract rather from the initial elegance.

      It would be nice to be able to do simple / fast similar document matches and scoring, but reverse index still offers advantages such as boolean matches and search term weighting and phrase matching that could be hit and miss with vector based searching.

      --

      @JAPH = qw(Hacker Perl Another Just);
      print reverse @JAPH;
Stories, comments, journals, and other submissions on use Perl; are Copyright 1998-2006, their respective owners.