use Perl

Below is the milestone status report I submitted for my Hague Grant; the original grant description is at http://news.perlfoundation.org/2008/11/tpf_awards_first_hague_grant_t.html .

----------

Rakudo Perl and PCT improvements
Hague grant status report, milestones M1/M2

This is a milestone report for the "Rakudo Perl and PCT improvements"
Hague Grant. Rakudo Perl [1] is a Perl 6 [2] implementation built on top
of the Parrot Virtual Machine [3].

The overall purpose of this grant is to design and implement the
components needed to align Rakudo Perl's parsing parsing more closely
with the current Perl 6 specification and to increase the level of
Perl 6 code used to implement the compiler itself.

This report focuses on the completion of the deliverables needed
for milestones M1 and M2 in the grant description [4], and marks
the "halfway" checkpoint for the grant. Originally this checkpoint
was expected to be reached in early 2009, but significant changes
to Parrot, Rakudo, and Perl 6 in the months shortly after this
grant was awarded delayed progress on some M1 and M2 items until
later in 2009. However, progress on many other items in the
grant were on time or accelerated, such that even though this
is officially the "halfway" checkpoint report, in reality almost
all grant deliverables are completed, and the "final" report
for this grant can be expected in December 2009.

A significant event during this grant period has been the planning
and announcement of "Rakudo Star", a "usable and useful Perl 6 release"
to occur in April 2010 [5]. Rakudo Star is not intended to be a
complete implementation of Perl 6, but rather to be an official
"interim" release that provides sufficient features to be a viable
platform for consideration by application developers. Much of the
overall Rakudo project effort, including work to be performed
under this grant and other Hague grants, has been prioritized
and focused on ensuring the successful release of Rakudo Star
in April 2010. Indeed, nearly half of the "critical must-have"
features identified in the Rakudo Star ROADMAP [6] depend on
the deliverables from this grant; the current state of
progress described by this milestone grant report is exactly
as planned and required by the Rakudo Star ROADMAP.

The specific items to be achieved under milestones M1 and M2 are:

      M1-1: PGE internal refactors and initial protoregex implementation
      M1-2: Selected protoregex constructs added to Rakudo's grammar
      M1-3: Interface design for pre-compilation and external libraries

      M2-1: Completed protoregex implementation
      M2-2: Initial implementation of longest token matching in PGE
      M2-3: Completed Rakudo grammar migration to protoregexes
      M2-4: Initial implementations of external HLL library support

Items M1-3 and M2-4 concern handling of library precompilation and
external interfaces; these were achieved in late 2008 and early 2009.
Rakudo Perl allows library modules to be precompiled to Parrot .pir
and/or .pbc modules for faster loading and execution; indeed, the
Test.pm module is precompiled to substantially reduce the time needed
for Rakudo to run the Perl 6 official test suite ("spectests"). Many
applications and libraries written for use with Rakudo similarly
pre-compile the modules to achieve better load performance.

An interface for loading external libraries (including those written
in other Parrot high level languages) was prototyped and added to
Parrot and Rakudo in early 2009; this interface is being continually
refined and improved in response to changes in the Perl 6 specification.

The remaining milestone items concern the implementation of protoregexes
and longest token matching (LTM), and the integration of these features
into Rakudo. The initial expectation of the grant was that these features
would be added to the existing Parrot Grammar Engine (PGE), and
subsequently used by Rakudo through PGE. However, as work progressed
it became apparent that the changes needed to PGE would significantly
impact backwards compatibility and run counter to Parrot's official
stability goals. Therefore, instead of modifying PGE I have built
a new regex engine essentially from scratch and embedded the engine
directly into a new version of NQP (the lightweight Perl6-like
language used to build the Rakudo compiler).

The new engine directly supports protoregexes (M1-1 and M2-1), a
limited form of longest token matching (M2-2), and is much more
consistent with the Perl 6 specification and STD.pm implementation [7]
as they have evolved over the past couple of years. In addition,
instead of compiling regular expressions directly to PIR (as PGE does),
the new engine compiles regular expressions to the same abstract
syntax tree representation (PAST) used by other HLL compilers in Parrot.
This allows better integration of regex support with higher level
languages (e.g., HLL closures and variables embedded in regexes).
It also may facilitate migrating the regex engine to backends other
than Parrot at some point in the future. Another key feature is
that the new NQP and regex implementations are largely self-hosted
-- that is, the source code is written in NQP itself, further
enhancing maintenance and retargetability. The source code repository
for this new version of NQP and regular expression support is
currently hosted on GitHub at [8].

Earlier this year Jonathan Worthington and I reviewed the tasks
needed for Rakudo to migrate to protoregexes and the standard
grammar (STD.pm), as well as achieve the other critical features
needed for Rakudo Star. Because the Perl 6 specification has evolved
substantially from when Rakudo's existing grammar and compiler
were first started, we decided that we would likely be more
successful rebuilding Rakudo from a "fresh grammar" (including
protoregexes) rather than try to incrementally refactor the
old grammar.

This rebuilding work has been proceeding in the "ng" ("new grammar")
branch of the Rakudo repository, and thus far I am extremely
pleased with our progress and results. The new grammar in this
branch makes full use of protoregexes (M1-2 and M2-3), and is
extremely consistent with the parsing model used by STD.pm .
Furthermore, in the new branch we have already been able to
implement critical Perl 6 features (lazy lists, proper lexical
handling, lexical subs) that were extremely difficult to achieve
in the previous implementation.

All of the items listed for milestones M1 and M2 of the grant have
now been realized. Over the next few weeks I expect to continue
work on (re)building the Rakudo-ng branch; when it is passing
a comparable percentage of the test suite as the existing Rakudo
releases we will officially redesignate Rakudo-ng as the mainline
development trunk. (The ng branch is effectively the mainline
for development already -- we simply want newcomers to Rakudo to get
the "more complete implementation" in the old master branch instead
of the "rapidly catching up" version in the ng branch.)

In the process of completing this migration I also expect to
complete the remaining deliverables (D1-D7) for this grant. To
briefly review where we are with respect to each grant deliverable:

D1: (Implementation of protoregexes and longest token matching)
        Essentially complete, with a few more improvements needed
        for better longest token matching semantics inside of
        regular expressions.

D2: (Alignment of Rakudo's parser and grammar with STD.pm.)
        About 70% complete. Rakudo's grammar in the ng branch
        is very close to STD.pm, with only minor additions and updates
        (and redesignation of the branch as 'master') needed to
        complete this task. It's worth noting that the goal for D2
        is "alignment" of the Rakudo and STD.pm grammars as opposed
        to "adoption"; indeed, in several areas STD.pm is changing
        to reflect some of the ideas being pioneered by the Rakudo
        grammar implementation.

D3: (Develop Perl 6 builtins for Rakudo, p6-based "Prelude".)
        Complete. Perl 6 now uses the term "core setting" instead
        of "Prelude" for its builtin classes and functions. Since
        early 2009 Rakudo has increasingly relied on Perl 6-based
        definitions of its builtin classes. In the Rakudo-ng branch,
        most of the builtins not already written in Perl 6 are
        placed in the "src/cheats/" directory in the expectation
        that they will eventually be rewritten as Perl 6.

D4: (Develop and improve Rakudo's ability to use external libraries.)
        Essentially complete, although further work is ongoing to
        bring Rakudo and the compiler tools up-to-date with recent
        changes to the Perl 6 specification in this area, and to
        provide further documentation.

D5: (Continue developing official Perl 6 test suite.)
        An ongoing task, but complete for the purposes of the grant.
        At the time the grant proposal was written (September 2008),
        the test suite contained approximately 7,000 tests and Rakudo
        was passing 2,700 of these (38%). As of this writing, the
        test suite contains over 38,000 tests and Rakudo master
        passes over 32,000 (85%).

D6: (Create additional tests for regex engine and library components.)
        About 50% complete. The NQP repository contains tests for the
        new regex implementation and protoregexes, some additional tests
        need to be written for the library interoperability components.

D7: (Publish weekly reports on Perl 6 implementation progress.)
        Ongoing and on track. Although reports have not come out weekly,
        there has been regular communication through our monthly release
        process, updates to the Rakudo ROADMAP [6], the Rakudo twitter
        feed [9], regular posts to use.perl and mailing lists, updates
        to the perl6.org and rakudo.org websites, and other channels.
        The goal of D7 has been to ensure increased visibility into
        Perl 6 and Rakudo progress, this has been largely achieved
        and will become even more apparent as we near the Rakudo Star
        release in April 2010.

Thus the primary tasks remaining for completion of this grant come
from deliverables D2, D4, and D6:
    1. increased alignment between Rakudo's grammar and STD.pm,
    2. improved longest token matching support, and
    3. update HLL library interfaces, documentation, and tests
    4. write final grant summary and report

Of these, only #2 represents any significant or time-consuming
effort; the others are likely to be achieved in the normal course of
ongoing Rakudo development. The remaining work on longest token
matching will be performed with the primary goal of meeting the
needs of the Rakudo Star release.

In summary, all of the items listed for milestones M1 and M2
of the grant have now been realized, and many items in the
remaining M3 and M4 milestones have also been completed. The
few remaining deliverables for this grant are expected to
be achieved before the end of 2009.

[1] http://rakudo.org/
[2] http://perl6.org/
[3] http://parrot.org/
[4] http://news.perlfoundation.org/2008/11/tpf_awards_first_hague_grant_t.html
[5] http://use.perl.org/~pmichaud/journal/39411
[6] http://github.com/rakudo/rakudo/blob/master/docs/ROADMAP
[7] http://svn.pugscode.org/pugs/src/perl6/STD.pm
[8] http://github.com/perl6/nqp-rx
[9] http://twitter.com/rakudoperl

As I wrote a few weeks ago, the period since Rakudo's October release has been one of a lot of refactoring and rebuilding to a new grammar and core implementation. So far everything has gone almost exactly as I outlined at the end of that post, so this is just a status update to let people know where we are.

The new version of NQP ("nqp-rx") has been substantially completed and includes many new features that weren't available in the previous regex engine. In particular, Perl 6 grammars can now support protoregexes, and we also have the ability to execute code blocks, declare variables, and add parameters to regexes. The new version of NQP is also bootstrapped, which means that the code for all of NQP (and its regex engine) is mostly written in NQP.

On October 30, Jonathan and I started a new branch (called "ng" for "new grammar" or "next generation") for refactoring Rakudo to use the new NQP and regex engine implementation. Nearly all of our work since then has been in this new branch, and I think it is going exceedingly well. Today we were able to get the sanity tests and core compiler to where it can again begin compiling and running the test suite.

Outwardly this probably seems like very slow progress for 10 days of effort, but running the test suite doesn't tell the whole story. In the process of getting to this point, we also knocked off several of the critical features needed for Rakudo Star that weren't possible in the previous version of Rakudo, including lazy lists, proper array element vivification, dynamically generated metaoperators, correct handling of containers and constant values, better dispatch semantics, and a whole host of other features that make the Rakudo core much stronger and easier to work with as we move forward from here.

Not only that, but we're already at the level where we are implementing builtin classes and methods in Perl 6 (as opposed to PIR). And we're also taking this as an opportunity to rewrite some of the previous PIR-based components into Perl 6.

So, over the next couple of weeks we will continue to focus on bringing the new "ng" branch up to the same level as the current "master" branch. When we do that, the "ng" branch will likely become the new master branch. I'm still a little reluctant to give a time estimate for when we'll make this switch, but given our current momentum (and the ease in which things are coming together) I suspect it won't be far off -- a few weeks at most. More importantly, unless we hit any major roadblocks, I expect we will have nearly all of the critical components needed for Rakudo Star (and certainly the most challenging ones) implemented by the end of December. We'll then use the remaining months until April to add other important but less-critical features, continue improving performance, and work on packaging and documentation issues.

Most of the progress reports (and Twitter messages) I'll be writing over the next few weeks will be about progress in nqp-rx and in the "ng" branch as opposed to the Rakudo "master" branch. Stay tuned!

Pm

It's been two weeks since I last did a blog post; and in that couple of weeks there's been an incredible flurry of activity and development. In fact, so productive that I barely have had time to write about it, although that will happen soon.

As a simple indication of the speed in which things are moving, I thought I would just post a copy of my report to today's #parrotsketch meeting. This represents things that have been achieved since October 27, and what we expect to be doing this upcoming week.


# parrotsketch report for 2009-11-03:
What I did:
* NQP stuff:
** Added contextual variables, named arguments, modules, class
      declarations, private class attributes, methods, 'make' statement,
      pir::op access to PIR opcodes, grammars, tokens, rules, regexes,
      INIT blocks, unified with , lexical subroutines,
      parameterized regexes, :my declarations in regexes, codeblocks
      in regexes, code assertions in regexes, \x and \o escapes in
      regexes and double-quoted strings, variable interpolation in
      double-quoted strings, a 'make install' target, a --parsetrace
      option, perl 6 pod comments, warnings for unsupported or NYI features
** ... and made the nqp parser and compiler self-hosting.

* Rakudo stuff:
** Started the new implementation of Rakudo based on the nqp-rx
      engine, that is going very well.
** Have a new implementation of Lists, Parcels and Arrays,
      all of which can now have lazy semantics.
** Fixed constants and containers handling, Rakudo no longer
      allows assignment to constant values.
** Implemented "real" assignment metaoperator parsing (e.g., &infix:);
      Rakudo-ng now builds assignment metaoperators only when needed.
** Changed subroutines to be stored with the & sigil.
** Changed Rakudo operators to be "&infix:<+>" instead of "infix:+".
** Have many of the sanity tests running again; Test.pm compiles
      but doesn't run completely yet

* Plumage:
** Updated Plumage Configure and code to work with nqp-rx, passes
      Plumage's test suite.

What I'm doing this week:
* Continuing to work on Rakudo-ng, get us running spectests again
    and compiling setting (now called "CORE")
* More minor nqp-rx updates, error message improvements and better
    syntax checking
* Profiling the regex engine to find some speed improvements

What I'm blocking on:
* Useful programming time


As you can see, things are happening quickly, and we're all pleased with the progress. Together with Jonathan's work on dispatch, we appear to have overcome the two major hurdles identified for Rakudo Star. From here on out it's basically just fleshing out the rest of the implementation and adding feature coverage according to the ROADMAP.

I'll write more details soon; I have to get back to a few other tasks right now. Hope you enjoy reading the above list as much as I enjoyed writing it.

It's been quite a while since I've written any articles about Rakudo's progress, but the delay in articles has been because I've been really focused on code development for a number of things we're going to need quickly for Rakudo Star.

At long last I've made the time to make substantial progress on my Hague Grant, which will enable us to bring Rakudo's grammar and parser much more in line with the current STD.pm grammar. In fact, looking at the Rakudo ROADMAP one can see that a significant number of the critical tasks needed for Rakudo Star are depending on the "PGE refactors" identified in the grant.

This brings me to one of the major points of this post: In the weeks that follow this month's release we expect that Rakudo will be quite unstable as we undertake some much-needed refactoring and redevelopment of some of Rakudo's core pieces. The biggest change will be a complete replacement of Rakudo's underlying grammar; the grammar we have today is still largely based on the Perl 6 grammar as it existed in January 2008, but STD.pm and the Perl 6 specification have evolved significantly since then.

Jonathan and I believe that now's the time to bite the bullet and make another big refactor to bring Rakudo in line with the spec, even though it will likely involve a rework of many features and perhaps a few significant (but temporary) regressions. So, if you see some chaos and upheaval in Rakudo development in the next few weeks, it's a planned and necessary sort of mayhem.

Many of the needed grammar changes will be possible because of the grant work on protoregexes and a new operator precedence parser. Originally the plan was to build these features into the Parrot Grammar Engine (PGE), but after thinking long and hard about it I concluded that it would be better to redesign and reimplement a new regex engine than to try to fix PGE. For one, I think maintaining backwards compatibility would be a significant challenge (and a drain on my energy and resources). Another reason favoring a rewrite is that we now have better language tools available for Parrot, and a rewrite can take advantage of those tools.

Thus, instead of compiling directly to PIR, the new regex engine compiles to Parrot's abstract syntax tree representation (PAST). In addition, the source code for the regex engine is written in NQP instead of PIR.

For those not familiar with NQP, it's a Perl 6-like language I designed for Parrot in conjunction with the Parrot Compiler Toolkit. NQP acts like a "mini Perl 6", it understands a subset of Perl 6 language constructs and can generate Parrot code that doesn't rely on additional runtime libraries. Most of the HLL compiler authors for Parrot have been using NQP to generate PAST, and it's proven to be much easier to write and maintain than PIR.

Since the regex engine will now be written using NQP, it also seemed fitting that NQP would receive the ability to use Perl 6 regexes and grammars directly. Adding regexes and grammars to the NQP language means that a compiler writer can write nearly all of the components (parser, ast conversion, runtime libraries) using NQP. This is in contrast to the existing setup that requires multiple languages and APIs.

The new version of NQP is currently called "nqp-rx" ("NQP with regexes"); I may come up with another name for the bundle but I'm somewhat attached to "NQP". This new version also has a new source code repository (separate from Parrot) -- it's hosted on GitHub at http://github.com/perl6/nqp-rx .

Since mid-September I've been working on nqp-rx, and I'm very pleased with how it's all coming together.

For example, late last week I completed most of the work on the new regex engine. This first version includes a very naive implementation of protoregexes, which PGE lacked, and ultimately should perform pattern matching and parsing more efficiently than PGE does. It now compiles to PAST instead of directly to PIR, which means it will fit more cleanly with the rest of Rakudo, especially with being able to handle lexical variables and code blocks in regexes.

More importantly, the regex compiler is self-hosted (or "bootstrapped"). In other words, the regex engine is able to parse and compile the specification that was used to build itself. Stated another way, the regex engine is written using Perl 6 grammars and regular expressions that it knows how to compile.

Since completing the regex bootstrap I've been working on creating the new version of NQP based on the new regex engine. Over the weekend I created some common rules for parsing number and quoted string tokens, and yesterday and today I completed a new operator precedence parser (all of these based on the STD.pm equivalents). Now all of the pieces are in place to create a new NQP compiler, which I plan to do over the next couple of days. And, like the regex engine, I'm planning to make this new version of NQP self-hosted as well.

So, when all of this is completed, NQP will continue to be a "Perl 6 lite" language, but it will also support grammars, regular expressions, protoregexes, longest token matching, a very powerful operator precedence parser, attributes on classes, and a fair bit more. It should also be a bit faster than the previous NQP, and have a few additional optimizations (such as inlining of blocks).

Thus, here's a quick rundown of the status and plan for the next couple of weeks:

1. Parrot 1.7 was released today (October 20).
2. Jonathan has just completed a significant refactor of Rakudo's signature binding code and merged it into Rakudo's master.
3. Scott Duff ("PerlJam") will be cutting the October Rakudo release on Thursday, based on the Parrot 1.7 release.
4. Immediately following the Parrot release, new code for the Parrot Calling Conventions is set to be merged to the Parrot trunk. This is one of the major tasks (B) listed in Rakudo's ROADMAP.
5. In the days following the Rakudo release, we'll be working to synchronize the multidispatch and binding algorithms in Rakudo with the new Parrot calling conventions.
6. Also in the next few days, we'll complete implementation of nqp-rx, or at least bring it to the point that it can be used instead of the previous compiler tools for building Rakudo.
7. When we're comfortable that the Parrot calling conventions work has sufficiently stabilized, we'll start a new branch for the major refactor of Rakudo's internals, switching to the new compiler tools, and updating the grammar to be much closer to STD.pm.
8. We don't know how long this last piece will take, but it could easily occupy most of the month before the November Rakudo release.
9. During the time that work is taking place in the branch, we don't expect much progress or changes to be made in Rakudo's master branch, so that people can continue to use and test the "more functional" version.
10. If work bogs down in the branch, we'll regroup and come up with an alternate plan. But I don't think this likely.

It looks to be an exciting couple of weeks! I'll be writing more articles with details about the new regex engine, NQP implementation, and Rakudo conversion to the new tools. I hope and expect that by the November release we'll be completely switched over to the new regex engine and have knocked out a large number of the "critical" items on the ROADMAP for Rakudo Star.

Pm

As I mentioned in my previous post, I'm doing two Rakudo days per week for my Vienna.pm grant to make up for some of the weeks I missed during conferences and summer travel. This week I focused on adding a rational (Rat) data type, support for contextual variables, and fixing up the handling of environment variables via %*ENV.

After the work that was done on operator overloading last week, implementing a basic Rat data type ended up being surprisingly easy. In fact, I just wrote it as a straightforward Perl 6 class and added it to the setting -- you can see the easy-to-read results in src/setting/Rat.pm. I did just enough of the implementation to get the basics in place, at which point Solomon Foster, Moritz Lenz, and others started working out a lot of the other details such as conversions to and from Rat and other MMD considerations. As a result we have quite a few clarifications and improvements to the specification, a lot of new tests have been written, and Rakudo passes a lot more tests in the test suite.

While adding Rat was relatively easy, adding contextual variables to Rakudo was quite a bit more challenging. Contextual variables are similar to lexicals, except that searching for a contextual variable involves looking at a block's callers instead of the block's outer lexical scopes. Contextuals have many similarities to the way that Unix environment variables work, but they also fill many of the niches that would have otherwise been handled by global or environment variables.

Contextual variables in Perl 6 are denoted using the * twigil. Here's a simple example that uses a contextual:

    sub foo() {
        say $*PHRASE;
    }

    sub bar() {
        my $*PHRASE = 'world';
        foo();
    }

    my $*PHRASE = 'hello';
    foo();
    bar();

When run, this outputs "hello\nworld\n". When foo() needs to look up the value of $*PHRASE, it does so by first checking its local lexpad, then checking its caller's lexpad, then its caller's caller's lexpad, and so forth until it finds a declared $*PHRASE. Thus in the first call to foo() above, foo()'s caller is the mainline and so foo() finds the mainline's definition of $*PHRASE ("hello"). The call to bar() declares a new contextual named $*PHRASE, and thus the second call to foo() sees bar()'s value of $*PHRASE ("world").

In other words, contextual variable lookups always use the (dynamic) caller chain instead of (static) lexical scoping.

This turns out to be incredibly useful in a number of situations where we want to provide called functions with an idea of the dynamic context in which they're being called. One of the most common uses for contextuals can be to override the default output for print, say, and other builtin I/O functions. The print and say builtins output to $*OUT, a contextual variable. So, to change the output destination for any calls to say or print, simply declare a new contextual $*OUT:

    say "Hello world on standard output";

    {
        my $*OUT = open("outfile.txt", :w);
        say "This text is going to outfile.txt";
        say "Report follows:";
        print_report();
        $*OUT.close;
    }

    say "...and this also goes to standard output";

So, in the bare block in the above code, the declaration of a new $*OUT contextual causes all of the calls to say and print executed inside of that block to send their output to outfile.txt instead of the default standard output. This behavior holds even for nested function calls such as print_report() -- its calls to say and print also get sent to outfile.txt.

This ability to establish a dynamic context value that can be quickly looked up from within nested function calls is key to simplifying many compiler implementation details. If you look at STD.pm you'll quickly see how much it relies on contextuals. Indeed, one of my next tasks will be to add contextuals to NQP and PGE so that we can move even closer to the way STD.pm handles parsing.

As I alluded above, contextual variables also fill some of the niches held by global and environment variables. If a search for a contextual variable doesn't find it declared in any of the callers' lexpads, we then fall back to looking in the GLOBAL and PROCESS packages. Thus a $GLOBAL::foo variable can be quickly accessed using $*foo if no caller has declared its own $*foo.

In fact, most of the predefined contextuals such as $*OUT, $*ERR, %*ENV, $*PID, etc., are actually defined in the PROCESS package. This gives us a lot of flexibility in deciding things:

    $*OUT          # the standard output of my caller
    $PROCESS::OUT  # the standard output for this process

The implementation for contextual variables in Rakudo required creating new Parrot operations for finding variables in the dynamic caller chain instead of the static outer scope chain, creating a (!find_contextual) sub in Rakudo to handle the contextual lookups, refactoring the compiler to treat the * twigil as a contextual instead of a global variable, and migrating the previous "contextual globals" we had into the PROCESS namespace. These later steps took the longest by far to accomplish.

Finally for this week's Rakudo days, there have been a number of RT tickets and requests for fixing up the handling of %*ENV -- previously it was based on Parrot's Env PMC and some of the Parrot-isms leaked out (e.g. RT #57400). The implementation of contextual variables opened the door to cleaning up %*ENV to work more appropriately, so I went ahead and did that.

Overall I'm very pleased by the progress made in this week's pair of Rakudo days: we got a good start on rational datatypes and cleaning up the relationships among the built-in scalar types, we now have a working contextual model that we'll need for the next phases of compiler development, and environment variables are now working more like they are supposed to.

Many thanks to Vienna.pm for sponsoring this work.

Pm

July and August have been really busy months with conferences and vacations, so I've fallen a bit behind on Rakudo days for my Vienna.pm grant. However, now my travels are largely finished and the kids are back in school again, so I'm hoping I can manage at least two Rakudo days per week until I'm caught up with the original schedule.

This week's major task did indeed need two days: I've now enabled operator overloading for many of Rakudo's builtin operators. Previously Rakudo has allowed some custom operators to be defined, but overloading the builtin operators (such as infix:<+>) would generally result in "Null PMC in find_method" errors.

The main approach I ended up taking for this was to re-implement most of the existing operators (previously written in PIR) as Perl 6 multisub definitions with inline PIR. This ended up being somewhat more difficult than it might sound or appear from simply reading the patch -- part of the challenge is that if any variant of an operator is written using Perl 6 multisubs, then all of them must be written that way. But since we had been generating PIR for the Whatever and Junction variants of some builtins, that meant rewriting those versions as well (and figuring out a way to get Rakudo to create WhateverCode objects from the setting).

Anyway, the hardest pieces are now done, so that many of the built-in operators can be overloaded with custom variants. I'm sure a lot of people will start taking advantage of those; for example, I'm hoping that SF will be able to update his Vector class to overload the builtin operators now.

There are still some operators that need to be moved to the setting; although I've been able to migrate prefix:<-> and prefix:<~>, attempting to do the same with prefix:<+> causes a failure in one of the spectests. We'll keep plugging away at it until we figure it out.

I"m also seeking clarification about the definitions of some of the relational ops; for example, infix:<==> and infix:<eq> are defined in terms of the infix:<===> operator, but I'm curious about the definitions of the other relational ops.

One item that greatly concerned me about moving the operators into the settings was the possibility that it would significantly slow down Rakudo, because subs compiled from Perl 6 result in a lot more code than hand-written PIR subs do. I even explored some ways to be able to automatically rebless PIR multisubs into Rakudo equivalents. However, I decided to just try the Perl 6 approach (we'll ultimately need to do that anyway), and I was pleasantly surprised that the changes didn't result in significant speed hit to the spectests. Indeed, my timings show that the use of the Perl 6 versions may in fact be slightly faster. I'm not sure how this can be possible, but my best guess at this point is that Rakudo's custom multisub dispatcher (which Jonathan wrote as part of his Hague Grant) is somehow significantly faster than Parrot's default multi-dispatch. But that's just a guess on my part...

Speaking of speed, yesterday I also finished and committed another significant change to Parrot's "isa" functions for testing if a PMC is an instance of a given type. For a long time Parrot has used type name (string) comparisons to check isa membership; this is not only a bit slow, but it can also result in unwanted type name collisions. Ideally the comparison should test the identities of the class objects; but getting this to work required cleaning a few other pieces of Parrot's object and class handling. When it was all finished we obtained a ~4.5% overall speed improvement in Rakudo's spectests.

Lastly, I've done some more work on improving Rakudo's build and install environment. We now have Rakudo building from an installed Parrot, and yesterday I improved Rakudo's Configure.pl so that it warns with a more useful error message if the files needed from a Parrot devel installation aren't present.

Next I'll need to review the ticket queue again to see how many tickets have been resolved by the above changes. I'll also want to see about adding the tighter/looser/equiv traits to user-defined operators. But mainly I'm glad that we can now start to do some more advanced operator handling and overloading in Rakudo.

Thanks as always to Vienna.pm for sponsoring this work.

Pm

In a comment to my post announcing Rakudo Star, gdonald writes:

Here, I'll explain since you don't seem to get it. When people ask "When will Perl6 be finished?", they want to know when there will be a Perl 6.0 release.

Actually, I really think I do "get it", although I admit I may not be clearly expressing that. Later in the same comment thread gdonald follows up with:

Seriously, the Perl6 spec isn't even nailed down yet? What is the hold-up on that? Design by committee?

To me, that sounds an awful lot like a variation of "When will Perl 6 be finished?", especially the "What is the hold-up..." part. In short, the thread quickly went from "you don't seem to get it" to ultimately saying something very like the question I used to start my original post.

Neither this reply nor my original article are intended to imply that people shouldn't be asking such questions or that they're wrong for doing so. I call out gdonald's messages not because I wish to ridicule him or prove him wrong, but simply because I think they illustrate well the "disconnect" that exists in discussions about Perl 6, and why I'm hoping to change the discussions to something more useful than "finished" and "not finished". Because the end result of the "finished/not finished" discussion is often:

Seems like Perl6 might be going the way of GNU/Hurd, eternally under development, and never to land.

I don't believe that Perl 6 will be eternally under development -- but I am concerned that the perception of "eternally under development" is potentially a significant blocker to continued progress on Rakudo and Perl 6. That's one of the major issues that Rakudo Star is intended to address.

I also completely "get it" that for most people the real thing they want to know is "When can I start using Perl 6?" in the sense of "apt-get install perl6". But I think even that question has many hidden assumptions that must be exposed before there can be any sort of useful answer. Indeed, the answer changes depending on who is asking the question and what they want to do. So another purpose for Rakudo Star is to illuminate the development process and our ongoing status so that people can begin answering that question for themselves.

As the quotes above illustrate, somewhere there's an implicit assumption that the specification should be finished already. When we then say that the spec isn't finished (and it isn't), people often conclude that Perl 6 is suffering from a "design by committee process flaw" that is preventing the specification from being finished, and in turn that's what is blocking "apt-get install perl6". I think these misconceptions arise from assuming a "waterfall" development model where it's a one-way path from specification to implementation. But just because (unlike Perl 5) Perl 6 has a spec that is separate from its implementations doesn't mean the specification comes first.

Looking back, it's not too surprising to me that people have assumed a waterfall model of development -- those of us working on Perl 6 haven't given a good public picture of the true story. The various development roadmaps we've provided have been accurate as far as they went, but they didn't really give insight into the underlying processes. And for many of us, myself included, we're only now learning how to put words to parts of the process to be able to tell others about them. Here's my version...

For the past few years, changes to the specification have been made almost exclusively in response to the concerns and issues that have come about from compiler implementations and application programs written in Perl 6. For example, while implementing a feature in Rakudo we often discover a place where the specification is self-contradictory, so the specification is changed to resolve the contradiction. Or, someone will be using Perl 6 to write an application, and as a result we find places where the specification is deeply sub-optimal and needs to be cleaned up. These days it's very rare that the design team says "wouldn't it be nice/better if Perl 6 changed to do X" on its own initiative; our discussions are nearly always "implementations are having trouble with this part of the specification, what do we need to change to improve that?"

So the specification isn't finished, but that's mainly because it's evolving in response to implementations and applications, and not due to a tendency to over-design it. Because the specification is evolving based on implementation issues, simply freezing what exists now as "6.0.0" will make things harder on implementors, not easier. In other words, freezing the existing spec will paradoxically delay implementations of Perl 6, not expedite them.

We are rapidly entering a phase where what we will need most is for more people to be creating real use cases in Perl 6, testing the soundness of the design and implementation. That is, we need to see more applications to be written in Perl 6 so we can harden the specification even further. For many people and applications Rakudo is ready for use today, but there are still enough issues that I'm hesitant to call it anything other than a "development release" for a more general audience. The problem then is that many people rightly take "development release" to mean "not ready to use yet", and that's also counterproductive to what we need.

This is where Rakudo Star comes in, and this is what I mean by "a useful implementation of Perl 6". It's intended to recognize that a Perl 6 release can be useful to many even though it may be incomplete. It's intended to provide a bright line where we can say "here's what is working now" and "here's what is not working yet". It's intended to help people determine when Perl 6 has been sufficiently realized to be ready for their needs. And it's intended to make it possible for more people to start writing programs in Perl 6, because one of the things we are needing is real-world use cases to test, refine, and extend the existing design.

At the same time, we need to be very careful about using the label "finished" or "1.0" for anything that isn't all of what has been promised for Perl 6.0. (See KDE 4 development for why this is a bad thing.) In fact, I'd like us avoid the notion of "finished" altogether. Instead, I want us to regularly deliver something that is useful and usable, make it clear what we are delivering, make it clear what we're not delivering, and enable people to see when Perl 6 is likely be ready for them (as opposed to "ready for all").

Ultimately Rakudo Star is intended to give some justifiable support and clarity to phrases like "Perl 6 exists" and "you can now write usable applications in Perl 6", without the distractions that arise from the "When will [Perl 6 | Rakudo] be finished?" sorts of questions. It's not that I think people are somehow wrong for asking these questions -- I think I do very much understand what leads people to ask them -- it's just that I'd like us to start finding ways to move our discussions beyond the "finished / not finished" trap that we seem to have fallen into. I'd like to help us all escape this trap because (1) I don't think it reflects reality, and (2) if "Perl 6 is finished" remains the primary criteria that most people use to decide whether or not to write applications in Perl 6 (and the criteria that we hold ourselves to), then we'll never get there.

Pm

The most common question I get from people who aren't generally involved with Perl 6 development is:

"When will Perl 6 be finished?"

In some ways the wording of this question bugs me a bit, because the word "finished" implies there's a point at which we all say "We're done" and development ceases (or at least moves to some other phase). But there really isn't a "finish line" for Perl 6, there are just stages of development at which more and more people are able to make use of whatever is currently available.

So, once we eliminate the notion of "finished", the wording is often changed to try to make it more tractable, often by asking when there will be a "stable release", or when the specification will be frozen so an implementation can be completed, or many other variations on the theme. I understand the assumptions behind questions like these, but at the same time part of me thinks "Huh? Those questions don't really fit with the way things really happen in language development..."

The truth is that language design is an evolutionary process, with the design and implementation efforts serving to influence and guide further progress in the other. (See "whirlpool model".)

But there's another important input to the process: "real-world" application programs. We need to know how Perl 6 is actually being used in order to finish parts of the specification and implementation. Indeed, there are parts of Perl 6 (e.g., concurrency) where the specification is incomplete or underspecified precisely because we need input from people writing Perl 6 applications.

But this poses a problem of sorts, because if programmers are waiting for Perl 6 to "finish" before they start using it to write programs, and if Perl 6 is blocking on feedback from applications and implementations before it can progress, then we have a deadlock of sorts.

So, we need a way to break the deadlock. To me, one good answer is to start making releases of Perl 6 that may not implement the entire Perl 6 specification, but that application writers will feel comfortable enough to start using in their projects. I've started to call these "useful releases" or "usable releases". While it might not have every feature described in the Perl 6 synopses, enough features will be present that can make it a reasonable choice for application programs.

In doing this, I'd like to also refocus conversations to avoid words like "finished" and "stable", because they have such varied and strong meanings in this context.

So, here's what the Rakudo Perl project is going to do:

We will make an "official", intermediate, useful and usable release of Perl 6 (an appropriate subset) by Spring 2010.

Of course, we have to decide what will will be included (and excluded) in this intermediate "official release". At the Rakudo BOF on Monday we held a lively discussion about what the release could look like, what needed to be present, and how it could be packaged. During the hackathons and days following YAPC::EU we'll be drafting and publishing the more detailed blueprint for the release. But one of our guiding principles will be to "under-promise and over-deliver", to make it clear what can be done with the release, and to make it very clear which parts of Perl 6 are not yet supported in the release.

A short list of things we know will be in the release (that Rakudo doesn't already have): use of the STD.pm grammar for parsing, laziness, better support for modules, fewer bugs, better error messages, better speed. Again, our goal is to make something that is reasonable for people to start using, even if it doesn't meet all of the requirements for Perl 6.0.0.

We've also had discussions about what to call the intermediate release. We've considered tagging it as "Rakudo 1.0", but some of us think that the "1.0" name might tend towards "overpromise". We also considered things like "0.5" or "0.9", but these come with the message of "not ready for use", and that's not what the impression we want to make either.

So, yesterday morning I finally got around to thinking about it as "Rakudo 'whatever'". In Perl 6 the * term is used to signify "whatever", so that leads to a working name of "Rakudo *" (or "Rakudo Star").

So, the focus of the Rakudo project is to release "Rakudo Star" in Spring 2010 as a useful (but incomplete) implementation of Perl 6. More details about the features, milestones, and roadmap for this release will be forthcoming over the next few days.

Pm

P.S.: Several of our "down-under" community members have pointed out that "Spring 2010" can be a bit ambiguous. I'm using a season (instead of a month) to leave a month or two of wiggle room, but my intention is April 2010. As we get a bit more detail into the plan, we'll identify a specific month.

YAPC::EU Day 1

During today's lightning talks, Geoffrey Avery added a new "lightning advertisements" feature to the session. Lightning advertisements are 30-second "promotional spots" that occur between each pair of talks; these help to fill the "dead air time" that typically occurs between each talk as the next lightning talk speaker works to get his or her presentation synced with the projector.

The purpose of a lightning advertisement is to simply create awareness or buzz about something happening at the conference. For example, I used my lightning advertisement to remind people of the Rakudo BOF taking place immediately after the lightning talk session. I'm certain that this directly increased the number of people who made it to the BOF, so I'm very pleased that Geoffrey added this feature to the session.

Others used the 30-second lightning advertisement slots to make announcements about other BOFs taking place and items being put for sale at the auction.

In order for things to run smoothly, it's important that there be a second microphone available for advertisers to use while the next lightning talk speaker is setting up. Also, the advertisers need to be at the front "ready to go" when one talk finishes and the next begins. With only 30 seconds available, there's not really time for slides or any visuals -- it's just the person making whatever announcement there is to be made.

In today's session several advertisers had difficulty finding the "on/off" switch on the microphone (it was really tiny); I suspect things can go a bit smoother if there's someone selected as the "microphone manager" to make sure it's correctly configured for each advertiser.

With eight lightning talk speakers, there are seven slots for lightning advertisements. When the session began there were only three "advertisers" at the front of the room, but as the lightning talks progressed (and people discovered how advertisements worked) the remaining four slots were quickly filled. I suspect it would be no problem for some slots to remain empty if there aren't sufficient advertisers, but I also suspect that as the idea catches on there will be no lack of advertisers. It also didn't seem to require much formal organization; the advertisers simply designated themselves by sitting at the front of the room, and we JIT-ted the ordering of advertisers as the session progressed.

I found the lightning advertisements to be an incredibly useful addition to the lightning talks session; kudos to Geoffrey Avery for adding them. I hope they become a regular feature of future lightning talk sessions.

[ironman perl]

Since I was busy at OSCON all this week, it was difficult to find a single day to dedicate to my Vienna.pm grant. But I did manage to get several major tasks done from my hotel room, so I'm going to bundle those together and count them as my "Rakudo day" for the week.

The biggest accomplishment was to finally get Rakudo so that it can build from an installed Parrot. Prior to Parrot 1.4.0 this has been exceedingly difficult, as an installed copy of Parrot did not provide all of the tools needed to compile dynamic PMCs and dynamic opcodes on our target platforms. But over the past few weeks Allison Randal and Will Coleda have gotten it to work for ParTcl, and now I've been able to adapt those techniques to work with Rakudo. The current state of building Rakudo from an installed Parrot is in the "ins2" branch of the github repository, if you wish to give it a try. (See below for instructions.) Note that some of the spectests will fail if you try "make spectest"; because the "ins2" branch is using an older version of Rakudo and some spectests have been added to the test suite since then. Since we're really just testing build/install, "make test" is sufficient here (and I'll clean up any spectest issues when I merge it back to the master branch).

Some may ask why we don't simply merge it back to master now; I haven't wanted to merge the ins2 branch back into trunk until we have verification that it builds properly on a variety of platforms. So far I've only had it fully tested on a couple of versions of Linux; I don't want to end up cutting out other operating systems from playing with Rakudo.

One of the downsides of building Rakudo from an installed Parrot is that we effectively lose the ability to easily build Rakudo from a build-tree copy of Parrot (like we do now). Part of the problem is that the filesystem layout of a build-tree copy of Parrot is very different from the filesystem layout of an installed Parrot. So at a minimum we would need a lot of code that says "if installed Parrot use this path, if Parrot build use this other path". This is true not only for file locations, but also for the tools used to build dynamically loaded PMCs and opcodes. Instead of trying to support both layouts, I'd prefer to just stick with using an installed Parrot for now.

(Note that it doesn't have to be a system-installed Parrot, the --gen-parrot option to Rakudo's Configure.pl will make a local install of Parrot and then build Rakudo from that.)

We're not abandoning the ability to build Rakudo from a build-copy of Parrot, we're just switching gears for a while. Based on conversations with other Parrot team members at YAPC::NA Pittsburgh and online, we've decided that of building (but not installing) Parrot should result in something that directly mimics the filesystem layout of an installed Parrot. When this is done, it will be easier for HLL languages and other tools built on Parrot to work from either a build tree or install tree version of Parrot.

In other areas, during Monday's OSCON tutorial sessions I sat in on Damian Conway's "Perl 6: Why? What? How?" tutorial. I wanted to see the tutorial itself, but I was also curious to know what problems would arise during the tutorial so that I could work on fixing them quickly. One of the problems that was quickly identified was that the << ... >> quoting operator wasn't handling comments properly. In other words, the following code

    my $a = <<
        do         # a deer
        re         # a drop of golden sun
        mi         # a name I call myself
    >>;

would act like a list of eighteen words instead of three. To be honest, I had overlooked that comments were allowed here, so that evening I quickly updated the parser to skip over comments as well as whitespace, and now the above works the way it is supposed to. Note that it even skips embedded and pod comments:

    my $a = << do #(a deer) re #(a drop of golden sun) >>;  # ("do", "re")

We had quite a few tickets dealing with places where operations end up returning Parrot String objects instead of Perl 6 Str objects. The easiest way to detect when this happens is to attempt to perform .trans on the string -- the .trans method for Parrot strings doesn't work the same as Rakudo's .trans method. So I converted a few settings functions (e.g., .uc, .flip, .lc, etc.) to explicitly call prefix:<~> on the result value; this guarantees that we end up with a Perl 6 Str object.

When we ultimately switch Rakudo to use HLL mapping of Parrot types, these explicit coercions won't be needed. However, at the moment using HLL mapping imposes a significant speed penalty on Rakudo (we're working on this), and given that things are on the slow side already I'd rather keep the speed and maintain workarounds for the time being.

I also fixed up stringification of several of the builtin types, especially Int, Num, and Junction. Previously printing a Junction object would produce a string like "Junction()<0x7fb898cb42b0>", which is almost certainly not what is wanted. So I updated Junction.Str to simply return its .perl representation.

Finally, Rakudo had been misparsing function names that began with a keyword followed by an apostrophe or hyphen. For example:

    sub do-something() { say 'hello'; }

    do-something();

Because do is a keyword, Rakudo would often end up parsing the above as do -something(), which of course wouldn't work properly. Similar issues existed with other keywords such as if, for, while, etc.

Having a longest-token matcher in the parser can avoid a lot of these misparses, but it's not always a complete solution. The STD grammar (as well as Rakudo's grammar) has a special <.nofun> lookahed subrule that can be used to verify that the keyword we just scanned is actually a keyword and not simply the lead-in to a function call. I went ahead and added a few <.nofun> calls to Rakudo's grammar, and now subroutine names that begin with keywords work like they're supposed to. (Thus making them a lot more fun. :-)

There were other fixes here and there throughout the week, and of course Moritz Lenz did the Rakudo #19 release on Thursday (which I describe in another post. I also worked with Jonathan on improving our internal object metamodel and introspection capabilities, and he and I worked out some ideas for refactoring our handling of lexicals. And all of this took place while I was attending OSCON, giving various presentations, and engaging in useful hallway discussions with other Perl folks. So it's been a busy and good week.

As always, my thanks go to Vienna.pm for sponsoring the work I did on the above tasks. Because of all of the travel and conferences I'm a bit behind on Rakudo days, so I will likely try to double-up on them for a few weeks until I'm caught up.

Pm

To test the ins2 branch:

    $ git clone git://github.com/rakudo/rakudo.git
    $ git branch ins2 origin/ins2
    $ git checkout ins2
    $ perl Configure.pl --gen-parrot
    $ make test