Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Alias (5735)

  (email not shown publicly)

Journal of Alias (5735)

Wednesday December 16, 2009
07:28 PM

The slippery Devel::Declare slope for Perl tool support

[ #40026 ]

Devel::Declare vexes me greatly.

On one hand, it's interesting to see the kinds of things you can do when you have the ability to temporarily take over parsing of the language.

On the other hand, this ability to just parse whatever the hell you want means the gradual decline in utility for the entire Perl development toolset. It's like someone has taken source filters (which everyone at least KNEW were "evil") and made them popular and "safe".

The real problem here is not that these problems are occurring, but they are occurring for what often seems like (or actually is) a very good reason. And as anyone who has seen my "Nothing Can Possibly Go Wrong" talk should know, the most dangerous failures are the ones we are seduced into because it seems like such a good idea.

For example, take this cool implementation of URLs via Devel::Declare.

Mixed that with LWP::Simple and you can do things like this, that actually run.

my $content = GET;

Or how about inline SQL, as featured in a couple of other languages. I'm fairly certain that by adding hooks for, say, SELECT, INSERT, UPDATE and DELETE I could easily make the following work.

my $arrayref_of_hashes = SELECT * FROM table_name;

This stuff is supremely shiny, and if you're anything like me as soon as you look at it you want it badly.

But these features aren't free.

In the general case (looking at ARBITRARY modules rather than specific modules) you lose the following.

1. No/degraded syntax highlighting.
2. No/degraded PPI parsing.
3. No/degraded shiny Padre refactoring tools.

You also lose support for things like Perl::Critic almost entirely. PPI's statement classifier drives a lot of critic functionality. It only exists because PPI can reliably find the ends of statements.

If you break the ability to find the end of a statement (as many D:D modules do to define their own block keywords like "class { }") then the parse tree starts to get really really whacky and critic will inevitably start to spit out false positives.

Once you start spewing false positives in enough volume, you undermine confidence in the tool and we might as well stop using it (or at least turn off entirely rule subsets).

There are proposals for ways to extend PPI so somehow support these new keywords (I've even agreed with Matt on how we would go about it) but I continue to suspect they don't work in the general case because you have to have installed the PPI extensions in advance, before you attempt to parse a document containing those D:D modules (or you allow arbitrary execution of user code, at which point Acme::BadExample will complete pwn your machine).

This process gradually grinds onwards. D:D gets more mature and more used, Perl 5.12 contains core enhancements that help make it work better, further legitimising the concept.

Each step along the way is completely rational, and completely defensible. And it all looks great in blog code snippits. To paraphrase one prominent Perl core dev "Should we stop making the language better because it breaks PPI?"

And yet, at the risk of bringing down Godwin's Law on myself, I can help but see the kind of effect described in the famous poem First They Came.

First they came for the communists, and I did not speak out—because I was not a communist;
Then they came for the trade unionists, and I did not speak out—because I was not a trade unionist;
Then they came for the Jews, and I did not speak out—because I was not a Jew;
Then they came for me—and there was no one left to speak out for me.

The difficulty level to modify the parser and break the development tools is already starting to move from "a large effort by experts" to "check out this neat URL hack".

At some point, the use of D:D-style parser hacks becomes routine and normal, and people start implementing new extensions just for their one specific project.

And at some point the primary interface to Moose or Catalyst is using D:D or something like it, because it's SO much more elegant. And so any code written with Moose isn't compatible with Perl Critic, and so on and so forth.

Some time soon, before Perl turns into Smalltalk (for which there are basically NO tools at all that don't also involve running the code), I think we need to have this discussion.

How many useful development tools are we willing to sacrifice (making development harder) in order to make development "easier".

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.
  • How many useful development tools are we willing to sacrifice (making development harder) in order to make development "easier".

    I'd have some sympathy if there were any chance of getting (for example) function signatures, the method keyword, or the class keyword into Perl 5 this side of the heat death of the universe, but that's not going to happen because "that would make maintaining existing code harder".

    • You know, that's funny, because working method signatures in Perl 5 are one of my new favourite things.

      use MooseX::Method::Signatures;
      method morning (Str $name) {
          $self->say("Good morning ${name}!");

      • Which demonstrates my point that there will be an increasingly widespread use of source filters... (I'm assuming that's what that it implemented using).

        • Nope, MooseX::Method::Signatures is Devel::Declare and a heaping spoonful of other magic.

          • ... and Devel::Declare is a source filter, albeit (as peregrin put it on #p5p) "keyhole source filters".

            • While its true DD and traditional source filters both change the source code, it is unfair to simply lump DD in with source filters. "Source filter" is a dirty word. DD does not share the traditional problem with source filters, that of trying to parse Perl and their global nature. For example, a source filter might decide to alter code-like strings and regexes. DD won't be fooled by that. It gets around this problem by (ab)using Perl's own parser.

              DD brings in its own bag of problems, but calling it a

              • I agree that it does not share some of the side-effects, and the level of collateral damage is greatly reduced.

                But it still makes arbitrary changes to the lexical structure of the language. From a static parsing point of view, THAT is the critical problem here.

                The problems with collateral damage are only a problem from the point of view of an executing parser.

      • I meant in the Perl 5 core.

  • And now the primary interface to Moose is using D:D or something like it ...

    Actually, MooseX::Declare is just another way to use Moose, but very surely not the primary.

    Many people are using MooseX::Declare, and find it very nice and elegant, but many others (myself included) are not willing to use MooseX::Declare or Devel::Declare in production because it is not yet mature enough. Personally, my reasoning is that I feel it still exposes too much internals when it errors and makes for some difficult d

    • I have to agree with chromatic, the language itself cannot be a test bed for experimentation....

      I agree, but that's not my main point here.

      It is a mistake to canonize features before they're ready, unless you have some mechanism of indicating that they may change in subsequent releases. No one wants the chaos of frequent churn.

      It's also a mistake, possibly more severe, to refuse to add features widely-requested and often reimplemented in subtly incompatible ways.

      Having D::D available with core support is

    • I've updated the post to remove the suggestion that the CURRENT Moose API is D:D, or that it is specific to just Moose

  • I seem to recall something Nat said in his "What's With Those Python Fucks Anyway?" rant that "circles should be square so they're easier to draw and measure".

    While I sympathize the the folks that write tools to do static analysis of Perl code, it seems rather putting the cart before the horse to restrict language development to serve those tools. It should be the other way around. It would be nice if we had formal Perl 6 style grammars to make life easier for the toolsmiths, but I'm not holding my breath

    • While I sympathize the the folks that write tools to do static analysis of Perl code, it seems rather putting the cart before the horse to restrict language development to serve those tools.

      If you could create the best language in the universe, but it could only be written using Notepad (or an equivalent), would anyone actually use it?

      Support for good toolchains appears to be correlative with the success of a language (although proving this, and then further proving causality, would be an interesting exerci

      • If you could create the best language in the universe, but it could only be written using Notepad (or an equivalent), would anyone actually use it?

        Sadly, yes. A less panicky question would be to say if you could do without syntax highlighting and indentation, which is really all the Perl specific stuff I use. I'd miss indentation, but syntax highlighting I can live without.

        That said, its not EITHER we have tools OR we get to play with the language. The tools have already had to reverse engineer the entir

  • Considering this:

    my $arrayref_of_hashes = SELECT * FROM table_name;

    I think it gains me nothing. Nothing at all.

    Considering this:

    method morning (Str $name) {
        $self->say("Good morning ${name}!");

    If I can get that (amongst other things), I will cheerfully give up:

    • No/degraded syntax highlighting.
    • No/degraded PPI parsing.
    • No/degraded shiny Padre refactoring tools.

    Different people value different things. I'd hate to give up the items on that list, but I'd much prefer to work with a program

    • That's why D::D should only be a prototyping tool and eventually such features should be available from the core.

      Note, however, that making the new syntax available with a "feature" invocation carries virtually the same problems as D::D. You never know from static analysis (and non-recursive) whether it's on or not. But at least this would mean it's blessed and to be considered part of the language.

      • Well, the difference with use feature is that at least in that case it's official, in the core, works the same on every operating system and every host, and doesn't require the parser to leave the document.

        So it's possible to add the relevant special casing in, where in the generic DD case that is not true.

        • Not really. Consider Modern::Perl, common::sense, and their ilk. And besides the ones on CPAN, any codebase can contain its own equivalent. Sure, you can special-case strict.

    • That example of MooseX::Declare like code is well intentioned but I (personally) think *horrific* for proving your point.

              function morning (Str $name) {
                      say "Good Morning $name"

      This won't piss off the people who think you're pushing OO at them.

      • Both can be made available, but don't forget that Perl doesn't really know the differences between functions and methods. That being said, I want a modern language and P5P is too conservative to provide that to me. Regrettably, they may be right.