use Perl

• And that works really well... (Score:1)

  by Elian (119) on 2004.09.29 13:56 (#34812) Homepage Journal

  as long as you're using an 8-bit character set. Unicode kinda makes this sub-optimal :)

  Reply to This

  • Re:And that works really well... (Score:2)

    by Matts (1087)

    s/char/wchar/ :-) (I know this doesn't really work)
  • Re:And that works really well... (Score:2)

    by bart (450)

    It should do just as well on UTF8, no? Except for its probable confusion on string length vs. number of bytes..
    • Re:And that works really well... (Score:2)

      by Matts (1087)

      In a multibyte string you get potential clashes between characters with different surrogate bytes but similar character bytes, meaning invalid results.
    • Re:And that works really well... (Score:1)

      by Elian (119)

      Nope. UTF-8 is particularly bad. In addition to potentially having combining characters in the match, which is a Unicode issue, you can run into lots of cases where the second (and third, and fourth, and fifth...) byte of two multibyte characters are identical but the characters aren't the same because the first byte isn't.
      
      This technique only works on fixed-width characters in sets with no combining characters (In which case it works really well) or in cases where you can ensure that you're only using the
      • Re:And that works really well... (Score:2)

        by Matts (1087)

        Though to be fair it would be easy to fix for UTF-8. You just need to use a NEXT() macro (or something - NEXT() is what libxml uses internally) instead of iterating through an array.
        • Re:And that works really well... (Score:1)

          by Elian (119)

          Oh, sure -- it's not that big a deal, and you could certainly handle it straightforwardly enough. It'd be a bit of a pain, though.
  • Re:And that works really well... (Score:1)

    by waitman (6469)

    I had an idea that seems to work on big strings. Kind of like Edward Scissor Hands cutting out the "bad stuff" in a bunch and leaving only the "good stuff". If you have two strings a and b, chop up a into pieces that have *characters* in common with b, and vice-versa. then chop up all the pieces. Like salad or a block of ice. Then it is quick to find the real matches. Should work with other things besides 8 bit text. I did it in Python, here-> http://www.livejournal.com/users/bramcohen/22069.html [livejournal.com]