use Perl

I recently received a bug report for HOP::Lexer, the lexer described in Higher Order Perl. One concern expressed in the bug report is that the lexer returns an array reference in the form [ TOKEN, VALUE ] with undef as the EOS (End Of Stream) value. However, if one doesn't use HOP::Parser but instead uses Parse::Yapp or similar tools, they expect a list in the form ( TOKEN, VALUE ) with the EOS indicated by ( '', undef ).

The suggested fix was to modify HOP::Lexer to optionally allow this. To my mind, this was the wrong fix. Since HOP is all about functional techniques -- applying functions to functions -- a much cleaner is to not create a special case, but to use a function!

In other words, what we have is this:

my $lexer = make_lexer( $iter, @input_tokens );

while ( my $next = $lexer->() ) {
    my ( $token, $value ) = @$next;
    ...
}

But we can now do this:

my $lexer = make_lexer( $iter, @input_tokens );

my $yapp_lexer = sub {
    my $token = $lexer->();
    return ( '', undef ) unless $token;
    return @$token;
};

use MyParser;

my $parser = MyParser->new;
my $value  = $parser->YYParse( yylex => \&yapp_lexer, yyerror => \&error_sub );

That's the correct solution to this problem and, I think, an elegant one.

Of course, the other part of the bug report was line numbers from the lexer. I'll have to think about that.