Stories
Slash Boxes
Comments

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

gav (2710)

gav
  (email not shown publicly)
http://www.estey.com/
AOL IM: flufflegavin (Add Buddy, Send Message)
Hacker in NYC.

Journal of gav (2710)

Saturday October 16, 2004
02:33 PM

Regexps for reverse engineering?

[ #21374 ]

I got roped into trying to help a friend of a friend extract some reports from their billing application data. I got given a 26 meg data file to play with, and some digging with 'strings' helped me find bits of data mixed in with the binary gibberish. I came up with the following code:

my $person  = qr/[\x08\x09]([A-Z]{8})/;
my $lo      = qr/[\x00-x20]{1,3}/;
my $id      = qr/OH${lo}7${lo}(\d{5})${lo}AA/;
my $date    = qr/\x06(\d\d)(\d\d)(\d\d)/;
my $time    = qr/\x08(\d\d):(\d\d):(\d\d)/;
my $money   = qr/([1-9]\d+\.\d\d)/;
my $chars   = qr/[\x20-\x7E]+?/;
my $desc    = qr/\x17($chars)\x00/;

while ($data =~ /$person.*?$id.*?$date.*?$desc.*?$time.*?$time.*?$money/gs) {
    my ($p_id, $r_id, $d, $text, $t1, $t2, $cost)
        = ($1, $2, "$3/$4/$5", $6, "$7:$8:$9", "$10:$11:$12", $13);
    print "$p_id: $r_id -- $d $t1 -> $t2 -- [$text] \$$cost\n";
}

I was happy, I'd got back a bunch of records that looked sensible. The problem was that I wasn't getting some of the records that I saw in the file. Changing my $person to qr/[\x08\x09](G[A-Z]{7})/ or qr/[\x08\x09](S[A-Z]{7})/ gives me a bunch of different records. But why would (S[A-Z]{7}) give different results than ([A-Z]{8})?. I'm stumped.


List all Journal entries
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Regexps for reverse engineering? 5 Comments More | Login | Reply /

 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.

  • > Changing my $person to
    >    qr/[\x08\x09](G[A-Z]{7})/
    > or
    >    qr/[\x08\x09](S[A-Z]{7})/
    > gives me a bunch of different records. But why would
    >    (S[A-Z]{7})
    > give different results than
    >    ([A-Z]{8})
    > ?. I'm stumped.

    I'm not sure if I'm misreading this, but it looks like you have three different regular expressions there:

        qr/  G [A-Z]{7}  /x
        qr/  S [A-Z]{7}  /x
        qr/   

    • > I would expect the three of them to match different values. The first one matches eight-letter uc words starting with 'G'. The second matches eight letter uc words starting with 'S'. The final regex matches eight letter uc words starting with any letter.

      Sorry for not being clear. I'd expect /S[A-Z]{7}/ to be a subset of the matches from /[A-Z]{8}/ but instead the latter isn't returning some of the results the former does.
      • That would match my expectation as well. Are you 100% positive that there is no other difference in the two regexen (or the two scripts)?

        If so, I'd be stumped too :/

        -matt
      • Glad to see /x modifier there. /S[A-Z]{7}/ should be a subset of /[A-Z]{7}/, in particular the subset /(?=S)[A-Z]{7}/. If it isn't, it could be a bug in the backtracking logic ... or an issue with binmode?

        If there is any possibility of accented 'national' characters (which there always is in unconstrained data) '\w' is much preferred to [A-Za-z] or [A-Z]/i.

        I'd worry that some 'persons' might actually be shorter than 8 chars, or have spaces or lower case in some systems. (van Helsing etc)

        What strings(1)
        --
        Bill
        # I had a sig when sigs were cool
        use Sig;
  • I suspect part of the problem could be that you're not getting the correspondence between person, ID, and so on that you expect because .*? is matching more than you want it to. You could be matching a person from one record and then skipping over parts of that record to another one because some of the other patterns don't match for that record.

    It may not be what's happening in this case, but a common oversight in constructing regexes is to assume that .*? will match the shortest possible sequence of chara
Stories, comments, journals, and other submissions on use Perl; are Copyright 1998-2006, their respective owners.