use Perl

Of course, this is based on an Evil Hack. The Evil Hack is to parse the output of perl -c -DTp. (You'll need a perl compiled with -DDEBUGGING for this to work.) These debug flags make perl output traces for any tokenizer and parser actions.

So, based on the traces, I can reconstruct the functioning of an LALR(1) parser, that "shadows" perl's parser (you know, shifts, reduces, and reading a new token symbol).

Drawbacks :

• You need -DDEBUGGING.
• Perl's tokenizer is very clever. It can produce fake (zero-length) tokens or permute some token in the input stream. You don't want to know about how it tokenizes "abc$def".
• For the moment I can't always get the part of the input source that's associated with some tokens. (If I find out that some information is missing, I'll patch the debug traces in the core!)
• If your Perl script outputs something like "yydebug: after reduction, shifting from state 23 to state 79" to stderr during execution of a BEGIN block, this will confuse my parser.

Now I have to design an API for it. Basically I can trigger any callback on shifts, reduces and reads. Those sets of callbacks are conveniently packaged as, well, packages. So I was thinking about something like

• a Perl::ShadowParser that implements the parser
• Perl::ShadowParser::* backend plugins that provides the callbacks
• a little program for your convenience that runs the parser on any script with any callback(s) you've provided :

  perlshadowparser -b backend1 -b backend2=option1,option2 perlscript

Lots of tests will be needed, too.

If you have any ideas of something cool to do with it (ideas for backends...), I'm listening.