Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

pmichaud (6013)

pmichaud
  (email not shown publicly)
http://www.pmichaud.com/

Patrick Michaud is the pumpking for the Rakudo Perl 6 compiler. He holds a Ph.D. in Computer Science and was formerly a Professor of Computer Science at Texas A&M University-Corpus Christi. He is currently a software developer and consultant focused on open source development and applications, including Perl, PmWiki, and Linux.

Journal of pmichaud (6013)

Wednesday September 01, 2010
07:36 AM

Rakudo Star 2010.08 released

[This announcement was made last week on rakudo.org -- I'm reposting to use.perl.org so it will show up in the various Perl aggregators. --Pm]

On behalf of the Rakudo and Perl 6 development teams, I'm happy to announce the August 2010 release of "Rakudo Star", a useful and usable distribution of Perl 6. The tarball for the August 2010 release is available from http://github.com/rakudo/star/downloads.

Rakudo Star is aimed at "early adopters" of Perl 6. We know that it still has some bugs, it is far slower than it ought to be, and there are some advanced pieces of the Perl 6 language specification that aren't implemented yet. But Rakudo Perl 6 in its current form is also proving to be viable (and fun) for developing applications and exploring a great new language. These "Star" releases are intended to make Perl 6 more widely available to programmers, grow the Perl 6 codebase, and gain additional end-user feedback about the Perl 6 language and Rakudo's implementation of it.

In the Perl 6 world, we make a distinction between the language ("Perl 6") and specific implementations of the language such as "Rakudo Perl". The August 2010 Star release includes release #32 of the Rakudo Perl 6 compiler [1], version 2.7.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

This release of Rakudo Star adds the following features over the previous Star release:
* Nil is now undefined
* Many regex modifiers are now recognized on the outside of regexes
* Mathematic and range operations are now faster (they're still slow, but they're significantly faster than they were in the previous release)
* Initial implementations of .pack and .unpack
* MAIN can parse short arguments
* Removed a significant memory leak for loops and other repeated blocks

This release (temporarily?) omits the Config::INI module that was included in the 2010.07 release, as it no longer builds with the shipped version of Rakudo. We hope to see Config::INI return soon.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Thus, we do not consider Rakudo Star to be a "Perl 6.0.0" or "1.0" release. Some of the not-quite-there features include:
* nested package definitions
* binary objects, native types, pack and unpack
* typed arrays
* macros
* state variables
* threads and concurrency
* Unicode strings at levels other than codepoints
* pre and post constraints, and some other phasers
* interactive readline that understands Unicode
* backslash escapes in regex character classes
* non-blocking I/O
* most of Synopsis 9
* perl6doc or pod manipulation tools

In many places we've tried to make Rakudo smart enough to inform the programmer that a given feature isn't implemented, but there are many that we've missed. Bug reports about missing and broken features are welcomed.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. An updated draft of a Perl 6 book is available as in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC channel #perl6 on freenode.

Rakudo Star releases are created on a monthly cycle or as needed in response to important bug fixes or improvements. The next planned release of Rakudo Star will be on September 28, 2010.

[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Thursday July 29, 2010
07:28 AM

Rakudo Star - an "early adopter" distribution of Perl 6

On behalf of the Rakudo and Perl 6 development teams, I'm happy to announce the July 2010 release of "Rakudo Star", a useful and usable distribution of Perl 6. The tarball for the July 2010 release is available from http://github.com/rakudo/star/downloads.

Rakudo Star is aimed at "early adopters" of Perl 6. We know that it still has some bugs, it is far slower than it ought to be, and there are some advanced pieces of the Perl 6 language specification that aren't implemented yet. But Rakudo Perl 6 in its current form is also proving to be viable (and fun) for developing applications and exploring a great new language. These "Star" releases are intended to make Perl 6 more widely available to programmers, grow the Perl 6 codebase, and gain additional end-user feedback about the Perl 6 language and Rakudo's implementation of it.

In the Perl 6 world, we make a distinction between the language ("Perl 6") and specific implementations of the language such as "Rakudo Perl". "Rakudo Star" is a distribution that includes release #31 of the Rakudo Perl 6 compiler [1], version 2.6.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community. We plan to make Rakudo Star releases on a monthly schedule, with occasional special releases in response to important bugfixes or changes.

Some of the many cool Perl 6 features that are available in this release of Rakudo Star:

  • Perl 6 grammars and regexes
  • formal parameter lists and signatures
  • metaoperators
  • gradual typing
  • a powerful object model, including roles and classes
  • lazy list evaluation
  • multiple dispatch
  • smart matching
  • junctions and autothreading
  • operator overloading (limited forms for now)
  • introspection
  • currying
  • a rich library of builtin operators, functions, and types
  • an interactive read-evaluation-print loop
  • Unicode at the codepoint level
  • resumable exceptions

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Thus, we do not consider Rakudo Star to be a "Perl 6.0.0" or "1.0" release. Some of the not-quite-there features include:

  • nested package definitions
  • binary objects, native types, pack and unpack
  • typed arrays
  • macros
  • state variables
  • threads and concurrency
  • Unicode strings at levels other than codepoints
  • pre and post constraints, and some other phasers
  • interactive readline that understands Unicode
  • backslash escapes in regex <[...]> character classes
  • non-blocking I/O
  • most of Synopsis 9
  • perl6doc or pod manipulation tools

In many places we've tried to make Rakudo smart enough to inform the programmer that a given feature isn't implemented, but there are many that we've missed. Bug reports about missing and broken features are welcomed.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources.

Rakudo Star also bundles a number of modules; a partial list of the modules provided by this release include:

  • Blizkost - enables some Perl 5 modules to be used from within Rakudo Perl 6
  • MiniDBI - a simple database interface for Rakudo Perl 6
  • Zavolaj - call C library functions from Rakudo Perl 6
  • SVG and SVG::Plot - create scalable vector graphics
  • HTTP::Daemon - a simple HTTP server
  • XML::Writer - generate XML
  • YAML - dump Perl 6 objects as YAML
  • Term::ANSIColor - color screen output using ANSI escape sequences
  • Test::Mock - create mock objects and check what methods were called
  • Math::Model - describe and run mathematical models
  • Config::INI - parse and write configuration files
  • File::Find - find files in a given directory
  • LWP::Simple - fetch resources from the web

These are not considered "core Perl 6 modules", and as module development for Perl 6 continues to mature, future releases of Rakudo Star will likely come bundled with a different set of modules. Deprecation policies for bundled modules will be created over time, and other Perl 6 distributions may choose different sets of modules or policies. More information about Perl 6 modules can be found at http://modules.perl6.org.

Rakudo Star also contains a draft of a Perl 6 book -- see "docs/UsingPerl6-draft.pdf" in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

Rakudo Star releases are created on a monthly cycle or as needed in response to important bug fixes or improvements. The next planned release of Rakudo Star will be on August 24, 2010.

[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Saturday June 19, 2010
12:04 PM

Rakudo Star (a "usable Perl 6") to be released by July 29

As many of you know, last summer we announced that we would be releasing a "usable release of Rakudo Perl 6" to be called "Rakudo Star" in the second quarter of 2010. We later refined our target release date to be April 2010.

Until March of this year we were well on track to meet the April 2010 release date, but then I had an family medical emergency that took me away from Perl 6 development. As a result of my situation, the Rakudo and Perl 6 team met online in early March and decided that an April release date would be unrealistic, and we instead focused our efforts on trying to make a June release for Rakudo Star, to keep with our original "second quarter 2010" goal.

Ultimately it ended up being twelve weeks before I was able to return to active Perl 6 development (i.e., late May). During my absence the others on the Rakudo and Perl 6 team made incredible progress on Rakudo Perl 6; I think their progress shows that a truly capable (and growing) team of developers has coalesced around Rakudo Perl. Thanks to their efforts, as of late May the compiler had nearly everything we identified as critical for Rakudo Star in the ROADMAP, with only a few key features blocking on my personal participation. We therefore felt we still had a good likelihood of meeting the June 2010 target, and continued to work with that goal in mind.

As part of planning this week's Parrot and Rakudo releases, we all met online to solidify our plans for the Rakudo Star release. After much discussion, we decided that although we could likely make some sort of Rakudo Star release in June, there was too much risk that releasing in June would fall well short of our vision of what we want Rakudo Star to be.

Therefore, we've decided to to let the release date slip one more month and release Rakudo Star not later than July 29, 2010. We are firmly committed to the July 29 date; whatever we have available then, that's what we release. I know that another delay will be frustrating to many (it is very frustrating to me), and that some will undoubtedly cite this delay as yet more "evidence" that there will never be a release of Perl 6. But given the circumstances, I think we feel that we promote Perl 6 better by moving the release date by a month than we would by releasing something less than our vision.

For those who might claim that we should "release early", we are still continuing to make regular monthly compiler releases. The most recent release (#30, "Kiev") comes with a lot of improvements over previous releases, and I truly expect the next release (#31, "Atlanta") to continue the trend. As always, we continue to invite people to try out the compiler releases and to visit the Perl 6 website to see what Perl 6 is already doing today.

Finally, on a personal note, my wife and I sincerely appreciate the ongoing support, prayers, and understanding we have received from the Perl community (and especially the Rakudo and Perl 6 teams) during these difficult times. While my wife is still not "out of the woods" yet, things are far better now than they were in the Spring, and we continue to work towards and pray for her full recovery.

More details about the Rakudo Star release will be forthcoming over the next couple of weeks.

Pm

Friday April 23, 2010
06:38 PM

More Perl 6 Anti-FUD

Thursday April 22, 2010
04:47 PM

A wholly inadequate reply to an Anonymous Monk

[Reposted from http://www.perlmonks.org/?node_id=836349. Normally I wouldn't repost but I'd like it to appear in my normal Perl 6 journal and rss feeds as well as on PerlMonks. --Pm]

An Anonymous Monk on perlmonks.org recently started a discussion on "The current state of Perl 6" in which he (she? they?) offers his views on various aspects of Perl 6 development. Personally, I find nearly all of Anonymous Monk's comments in the thread to be so devoid of logic and so far removed from reality/facts that it's hard to imagine providing an adequate reply to them.

Beyond that, I have other priorities in my life right now. Uppermost is that my family is going through a very difficult period at present -- I need to take care of them first. Rakudo and Perl 6 come in at a somewhat distant second to that. Responding to garbage being spouted by anonymous person(s) probably ought to not even be anywhere on my radar.

But at least one post in the thread bugs me sufficiently that I'd like to respond, even if it's an inadequate response. And I don't want my response buried in a thread somewhere, so it gets its own post.

Anonymous Monk writes:

Oh I'll tell you how you do that [write a grammar engine capable of supporting Perl 6]. It's very simple. You get people skilled for this exact task ! Those skills are acquired in universities(preferably good ones) where professors teach courses like "Formal Languages and Automata" or "Compiler theory". If you have a bunch of people who are open-source contributors but don't have the knowledge or haven't studied the right things ... [t]hey don't know it in theory and they're trying to put it in practice! (emphasis in original)

Who does Anonymous Monk think is working on Perl 6? Personally, I have a Ph.D. in Computer Science and taught university language courses for nearly 14 years. Damian Conway certainly also has a fairly strong university background (Monash University) and understands the theory and practice of language and compiler design. Jonathan Worthington has a degree from the University of Cambridge, where he specialized in compiler construction and formal languages. Larry Wall has a degree in Natural and Artificial Languages with two years of graduate work in Linguistics at U.C. Berkeley. Many of our other key contributors also have degrees and backgrounds in language theory and practice. I'd be willing to compare the academic credentials of this group against any other dynamic language team Anonymous Monk might care to postulate.

If you can't get real compiler people to use Perl6 and help with it, the average open-source rookie won't be able to deal with this.

Somehow I have trouble classifying Larry, Damian, Allison, etc. as being "not real compiler people". And they along with many other Perl 6 contributors have a lot of experience in nurturing open source developers and projects. If you feel Larry et al. are indeed unqualified to be working in the field of programming language design and implementation, you probably don't want to be using Perl at all, much less Perl 6.

People like Anonymous Monk spew lots of opinions about how long they think Perl 6 development should take, and then speculate on the reasons why it has taken longer than they estimate it should be taking. The speculations that crop up most often are things like "the people involved are clueless about language design/implementation" (see above), "the design process itself is flawed", "they're building on an inadequate toolset like Parrot", etc. For such people it's easy to toss out random thoughts about "the problems with Perl 6" without bothering to look at the obvious evidence to the contrary, as Anonymous Monk does above. Indeed, I'm often amused when people suggest that we should be doing things that we're already doing .

Returning to the topic of developing a grammar engine, which Anonymous Monk claims above as "very simple" and just needing "people skilled for this exact task", it's interesting to contrast his opinions with the actual development of the Perl 6 standard grammar (STD.pm6). I think STD.pm6 is also indicative of the challenges confronting all Perl 6 implementors. Consider:

  • Larry is the primary implementor for the standard grammar.
  • The standard grammar doesn't need a frozen specification to be implemented, because its implementation is considered part of the spec.
  • The implementation is built entirely on top of Perl 5 -- there are no "unproven virtual machines" involved or holding things back.

I think one could consider this to be an almost perfect situation for developing a new language implementation -- an experienced language designer/implementor working without significant external restrictions on top of an advanced programming platform like Perl 5. Yet it's been three years since Larry started working on this implementation of the standard grammar and parser, and while STD.pm6 is very powerful, it's also certainly not "finished", and has been through a number of significant refactors in its development. This says to me that the amount of time and effort involved is more likely due to the sheer audacity and ambition of what Perl 6 seeks to accomplish.

(Some will undoubtedly look at the above and knee-jerk respond with something like "If it's taken three years just to create the parser, then finishing a compiler will take far longer still and therefore Perl 6 will never see the light of day." This argument ignores the fact that other pieces are being implemented in parallel with Larry's work, that software development is not a strictly sequential process, and the obvious fact that there are already "working" Perl 6 implementations available.)

Anyone who thinks that Perl 6 is fundamentally based on traditional compiler construction techniques taught in universities frankly has no clue as to what a fundamental paradigm shift Perl 6 represents to language design and implementation. It's this fundamental change that ultimately gives Perl 6 its power, but it's also why Perl 6 development is not a trivial exercise that can be done by a few dedicated undergraduates. As TimToady on #perl6 says, "We already have lots of those kinds of languages."

Personally, I'm impressed and honored to be associated with the people who are working on Rakudo and Perl 6. I understand that people are frustrated (and even feel burned by) the long wait for something as cool as Perl 6; I share the frustration and like to think that I'm doing something constructive about it. But I also find the vast majority of suggestions, comments, and conclusions coming from Perl 6's anonymous detractors to be (1) things we've already done or are doing, (2) ungrounded in reality, (3) in direct contradiction to reasonably observable facts, or (4) attempts to discredit a product they have little interest in ever using themselves. And far too many of the comments, like the ones I've highlighted in this post, are so easily refuted with just a little bit of fact digging and common sense, it's often hard to believe anyone can be seriously making them in the first place. Yet there they are.

Returning to my original theme, I think my response here is inadequate because it leaves so many other of Anonymous Monk's claims in the thread unrefuted. I could undoubtedly spend many more hours analyzing and responding to the many fallacies and untruths in the thread, but frankly I don't believe that's the best use of my time. People such as Moritz Lenz, chromatic, Michael Schwern, and others are also writing extremely well-reasoned posts refuting the garbage, for which I'm grateful, but it's far easier for Anonymous Monk and his like to spin garbage than it is for a small number of us to clean up after it. And it does need to be cleaned up, otherwise it festers and results in even more public garbage that someone has to clean up.

I hope that this post will at least encourage more people in the Perl community to critically examine the things they hear and read regarding Perl 6, especially when coming from sources with no apparent standing or reputation within the community. And maybe a few more people will even assist in publicly refuting the garbage ("many hands make light work"), so that the sloppy thinking, analysis, and dialogue that people like Anonymous Monk post doesn't spread to infect all of Perl.

Pm

P.S.: Some may reasonably conclude from what I've written above that Perl 6 is somehow "aiming too high", that our goals should be scaled back to make something ready "right now". I have two responses to this: (1) we are making things ready 'right now', just grab any of the available packages and start working and reporting bugs, and (2) there are already 'scaled back' versions of Perl 6 appearing and being used, such as NQP or even the components that execute the standard grammar. Some of these other projects, like NQP, are being used for "real" programs and applications today; they're not simply theoretical exercises or research projects.

Others often claim that all this effort on Perl 6 would be better spent on improving Perl 5. In my experience, those of us working on Perl 6 have absolutely no qualms with seeing Perl 5 improve and continue to grow -- we welcome it. Indeed, many of the features appearing in Perl 5 today come directly from ideas and things first attempted in Perl 6, and we're genuinely happy to see that. But just because Perl 6 developers also like Perl 5 doesn't mean that doing Perl 5 core development is interesting to us, or that (in my case at least) we'd even be qualified to do Perl 5 core changes. We aren't commodity programmers that are interested in simply being unplugged from one project and into some other project that others think is more worthwhile. Personally, I'd prefer to see people who are really into Perl 5 improvements continue to work to make them happen, and that the surrounding ecosystem continue to evolve to enable that to happen more easily for more people. Indeed, this is my wish for all open-source projects, even the ones I don't find interesting or otherwise disagree with.

Tuesday March 16, 2010
11:32 PM

Bad news, revisited

For those of you who are wondering where I've been lately with Perl 6 and Rakudo development, here's the story. As many of you know, my wife was diagnosed in early 2008 with ovarian cancer and remission [1,2]. Over the past few weeks various tests have indicated a probable cancer recurrence; and on March 2nd my wife was diagnosed with a small-bowel obstruction requiring immediate hospitalization. Since then nearly all of my time and energy has been essentially dedicated to caring for her and her needs. As of this writing (Mar 16) we're still in the hospital and hope she will be able to return home in the next couple of days.

The most obvious question relating to Rakudo development is "How does this affect scheduling of the Rakudo Star release in April?" Unfortunately, I don't have a good answer for this at the moment, as I don't yet have a firm date of when her recovery might be sufficiently far along that I can devote significant time again to Perl 6. I could be available again in just a few days, or it could take many weeks.

For now I've asked the core Rakudo and Perl 6 developers to work out what they think will be the best course of action for Rakudo Star given my current situation. As far as I'm concerned, all options are on the table (e.g., delaying the release a bit), but I think that whatever is decided needs to be a consensus among the entire development team. I suspect we will collectively announce an updated plan sometime in the next few days.

As I did in 2008, I'll tend to limit any further public posts about the current medical situation to major developments. But if anyone has any questions, I'll be glad to answer them.

As always, my wife and I send our thanks to everyone for your prayers, support, and understanding.

Pm

1. http://use.perl.org/~pmichaud/journal/35389
2. http://use.perl.org/~pmichaud/journal/37315

Tuesday November 24, 2009
09:49 AM

Hague grant status report, milestones M1/M2

Below is the milestone status report I submitted for my Hague Grant; the original grant description is at http://news.perlfoundation.org/2008/11/tpf_awards_first_hague_grant_t.html .

----------

Rakudo Perl and PCT improvements
Hague grant status report, milestones M1/M2

This is a milestone report for the "Rakudo Perl and PCT improvements"
Hague Grant. Rakudo Perl [1] is a Perl 6 [2] implementation built on top
of the Parrot Virtual Machine [3].

The overall purpose of this grant is to design and implement the
components needed to align Rakudo Perl's parsing parsing more closely
with the current Perl 6 specification and to increase the level of
Perl 6 code used to implement the compiler itself.

This report focuses on the completion of the deliverables needed
for milestones M1 and M2 in the grant description [4], and marks
the "halfway" checkpoint for the grant. Originally this checkpoint
was expected to be reached in early 2009, but significant changes
to Parrot, Rakudo, and Perl 6 in the months shortly after this
grant was awarded delayed progress on some M1 and M2 items until
later in 2009. However, progress on many other items in the
grant were on time or accelerated, such that even though this
is officially the "halfway" checkpoint report, in reality almost
all grant deliverables are completed, and the "final" report
for this grant can be expected in December 2009.

A significant event during this grant period has been the planning
and announcement of "Rakudo Star", a "usable and useful Perl 6 release"
to occur in April 2010 [5]. Rakudo Star is not intended to be a
complete implementation of Perl 6, but rather to be an official
"interim" release that provides sufficient features to be a viable
platform for consideration by application developers. Much of the
overall Rakudo project effort, including work to be performed
under this grant and other Hague grants, has been prioritized
and focused on ensuring the successful release of Rakudo Star
in April 2010. Indeed, nearly half of the "critical must-have"
features identified in the Rakudo Star ROADMAP [6] depend on
the deliverables from this grant; the current state of
progress described by this milestone grant report is exactly
as planned and required by the Rakudo Star ROADMAP.

The specific items to be achieved under milestones M1 and M2 are:

      M1-1: PGE internal refactors and initial protoregex implementation
      M1-2: Selected protoregex constructs added to Rakudo's grammar
      M1-3: Interface design for pre-compilation and external libraries

      M2-1: Completed protoregex implementation
      M2-2: Initial implementation of longest token matching in PGE
      M2-3: Completed Rakudo grammar migration to protoregexes
      M2-4: Initial implementations of external HLL library support

Items M1-3 and M2-4 concern handling of library precompilation and
external interfaces; these were achieved in late 2008 and early 2009.
Rakudo Perl allows library modules to be precompiled to Parrot .pir
and/or .pbc modules for faster loading and execution; indeed, the
Test.pm module is precompiled to substantially reduce the time needed
for Rakudo to run the Perl 6 official test suite ("spectests"). Many
applications and libraries written for use with Rakudo similarly
pre-compile the modules to achieve better load performance.

An interface for loading external libraries (including those written
in other Parrot high level languages) was prototyped and added to
Parrot and Rakudo in early 2009; this interface is being continually
refined and improved in response to changes in the Perl 6 specification.

The remaining milestone items concern the implementation of protoregexes
and longest token matching (LTM), and the integration of these features
into Rakudo. The initial expectation of the grant was that these features
would be added to the existing Parrot Grammar Engine (PGE), and
subsequently used by Rakudo through PGE. However, as work progressed
it became apparent that the changes needed to PGE would significantly
impact backwards compatibility and run counter to Parrot's official
stability goals. Therefore, instead of modifying PGE I have built
a new regex engine essentially from scratch and embedded the engine
directly into a new version of NQP (the lightweight Perl6-like
language used to build the Rakudo compiler).

The new engine directly supports protoregexes (M1-1 and M2-1), a
limited form of longest token matching (M2-2), and is much more
consistent with the Perl 6 specification and STD.pm implementation [7]
as they have evolved over the past couple of years. In addition,
instead of compiling regular expressions directly to PIR (as PGE does),
the new engine compiles regular expressions to the same abstract
syntax tree representation (PAST) used by other HLL compilers in Parrot.
This allows better integration of regex support with higher level
languages (e.g., HLL closures and variables embedded in regexes).
It also may facilitate migrating the regex engine to backends other
than Parrot at some point in the future. Another key feature is
that the new NQP and regex implementations are largely self-hosted
-- that is, the source code is written in NQP itself, further
enhancing maintenance and retargetability. The source code repository
for this new version of NQP and regular expression support is
currently hosted on GitHub at [8].

Earlier this year Jonathan Worthington and I reviewed the tasks
needed for Rakudo to migrate to protoregexes and the standard
grammar (STD.pm), as well as achieve the other critical features
needed for Rakudo Star. Because the Perl 6 specification has evolved
substantially from when Rakudo's existing grammar and compiler
were first started, we decided that we would likely be more
successful rebuilding Rakudo from a "fresh grammar" (including
protoregexes) rather than try to incrementally refactor the
old grammar.

This rebuilding work has been proceeding in the "ng" ("new grammar")
branch of the Rakudo repository, and thus far I am extremely
pleased with our progress and results. The new grammar in this
branch makes full use of protoregexes (M1-2 and M2-3), and is
extremely consistent with the parsing model used by STD.pm .
Furthermore, in the new branch we have already been able to
implement critical Perl 6 features (lazy lists, proper lexical
handling, lexical subs) that were extremely difficult to achieve
in the previous implementation.

All of the items listed for milestones M1 and M2 of the grant have
now been realized. Over the next few weeks I expect to continue
work on (re)building the Rakudo-ng branch; when it is passing
a comparable percentage of the test suite as the existing Rakudo
releases we will officially redesignate Rakudo-ng as the mainline
development trunk. (The ng branch is effectively the mainline
for development already -- we simply want newcomers to Rakudo to get
the "more complete implementation" in the old master branch instead
of the "rapidly catching up" version in the ng branch.)

In the process of completing this migration I also expect to
complete the remaining deliverables (D1-D7) for this grant. To
briefly review where we are with respect to each grant deliverable:

D1: (Implementation of protoregexes and longest token matching)
        Essentially complete, with a few more improvements needed
        for better longest token matching semantics inside of
        regular expressions.

D2: (Alignment of Rakudo's parser and grammar with STD.pm.)
        About 70% complete. Rakudo's grammar in the ng branch
        is very close to STD.pm, with only minor additions and updates
        (and redesignation of the branch as 'master') needed to
        complete this task. It's worth noting that the goal for D2
        is "alignment" of the Rakudo and STD.pm grammars as opposed
        to "adoption"; indeed, in several areas STD.pm is changing
        to reflect some of the ideas being pioneered by the Rakudo
        grammar implementation.

D3: (Develop Perl 6 builtins for Rakudo, p6-based "Prelude".)
        Complete. Perl 6 now uses the term "core setting" instead
        of "Prelude" for its builtin classes and functions. Since
        early 2009 Rakudo has increasingly relied on Perl 6-based
        definitions of its builtin classes. In the Rakudo-ng branch,
        most of the builtins not already written in Perl 6 are
        placed in the "src/cheats/" directory in the expectation
        that they will eventually be rewritten as Perl 6.

D4: (Develop and improve Rakudo's ability to use external libraries.)
        Essentially complete, although further work is ongoing to
        bring Rakudo and the compiler tools up-to-date with recent
        changes to the Perl 6 specification in this area, and to
        provide further documentation.

D5: (Continue developing official Perl 6 test suite.)
        An ongoing task, but complete for the purposes of the grant.
        At the time the grant proposal was written (September 2008),
        the test suite contained approximately 7,000 tests and Rakudo
        was passing 2,700 of these (38%). As of this writing, the
        test suite contains over 38,000 tests and Rakudo master
        passes over 32,000 (85%).

D6: (Create additional tests for regex engine and library components.)
        About 50% complete. The NQP repository contains tests for the
        new regex implementation and protoregexes, some additional tests
        need to be written for the library interoperability components.

D7: (Publish weekly reports on Perl 6 implementation progress.)
        Ongoing and on track. Although reports have not come out weekly,
        there has been regular communication through our monthly release
        process, updates to the Rakudo ROADMAP [6], the Rakudo twitter
        feed [9], regular posts to use.perl and mailing lists, updates
        to the perl6.org and rakudo.org websites, and other channels.
        The goal of D7 has been to ensure increased visibility into
        Perl 6 and Rakudo progress, this has been largely achieved
        and will become even more apparent as we near the Rakudo Star
        release in April 2010.

Thus the primary tasks remaining for completion of this grant come
from deliverables D2, D4, and D6:
    1. increased alignment between Rakudo's grammar and STD.pm,
    2. improved longest token matching support, and
    3. update HLL library interfaces, documentation, and tests
    4. write final grant summary and report

Of these, only #2 represents any significant or time-consuming
effort; the others are likely to be achieved in the normal course of
ongoing Rakudo development. The remaining work on longest token
matching will be performed with the primary goal of meeting the
needs of the Rakudo Star release.

In summary, all of the items listed for milestones M1 and M2
of the grant have now been realized, and many items in the
remaining M3 and M4 milestones have also been completed. The
few remaining deliverables for this grant are expected to
be achieved before the end of 2009.

[1] http://rakudo.org/
[2] http://perl6.org/
[3] http://parrot.org/
[4] http://news.perlfoundation.org/2008/11/tpf_awards_first_hague_grant_t.html
[5] http://use.perl.org/~pmichaud/journal/39411
[6] http://github.com/rakudo/rakudo/blob/master/docs/ROADMAP
[7] http://svn.pugscode.org/pugs/src/perl6/STD.pm
[8] http://github.com/perl6/nqp-rx
[9] http://twitter.com/rakudoperl

Tuesday November 10, 2009
11:57 PM

The Rakudo-ng branch

As I wrote a few weeks ago, the period since Rakudo's October release has been one of a lot of refactoring and rebuilding to a new grammar and core implementation. So far everything has gone almost exactly as I outlined at the end of that post, so this is just a status update to let people know where we are.

The new version of NQP ("nqp-rx") has been substantially completed and includes many new features that weren't available in the previous regex engine. In particular, Perl 6 grammars can now support protoregexes, and we also have the ability to execute code blocks, declare variables, and add parameters to regexes. The new version of NQP is also bootstrapped, which means that the code for all of NQP (and its regex engine) is mostly written in NQP.

On October 30, Jonathan and I started a new branch (called "ng" for "new grammar" or "next generation") for refactoring Rakudo to use the new NQP and regex engine implementation. Nearly all of our work since then has been in this new branch, and I think it is going exceedingly well. Today we were able to get the sanity tests and core compiler to where it can again begin compiling and running the test suite.

Outwardly this probably seems like very slow progress for 10 days of effort, but running the test suite doesn't tell the whole story. In the process of getting to this point, we also knocked off several of the critical features needed for Rakudo Star that weren't possible in the previous version of Rakudo, including lazy lists, proper array element vivification, dynamically generated metaoperators, correct handling of containers and constant values, better dispatch semantics, and a whole host of other features that make the Rakudo core much stronger and easier to work with as we move forward from here.

Not only that, but we're already at the level where we are implementing builtin classes and methods in Perl 6 (as opposed to PIR). And we're also taking this as an opportunity to rewrite some of the previous PIR-based components into Perl 6.

So, over the next couple of weeks we will continue to focus on bringing the new "ng" branch up to the same level as the current "master" branch. When we do that, the "ng" branch will likely become the new master branch. I'm still a little reluctant to give a time estimate for when we'll make this switch, but given our current momentum (and the ease in which things are coming together) I suspect it won't be far off -- a few weeks at most. More importantly, unless we hit any major roadblocks, I expect we will have nearly all of the critical components needed for Rakudo Star (and certainly the most challenging ones) implemented by the end of December. We'll then use the remaining months until April to add other important but less-critical features, continue improving performance, and work on packaging and documentation issues.

Most of the progress reports (and Twitter messages) I'll be writing over the next few weeks will be about progress in nqp-rx and in the "ng" branch as opposed to the Rakudo "master" branch. Stay tuned!

Pm

Tuesday November 03, 2009
01:25 PM

A brief report on progress

It's been two weeks since I last did a blog post; and in that couple of weeks there's been an incredible flurry of activity and development. In fact, so productive that I barely have had time to write about it, although that will happen soon.

As a simple indication of the speed in which things are moving, I thought I would just post a copy of my report to today's #parrotsketch meeting. This represents things that have been achieved since October 27, and what we expect to be doing this upcoming week.


# parrotsketch report for 2009-11-03:
What I did:
* NQP stuff:
** Added contextual variables, named arguments, modules, class
      declarations, private class attributes, methods, 'make' statement,
      pir::op access to PIR opcodes, grammars, tokens, rules, regexes,
      INIT blocks, unified with , lexical subroutines,
      parameterized regexes, :my declarations in regexes, codeblocks
      in regexes, code assertions in regexes, \x and \o escapes in
      regexes and double-quoted strings, variable interpolation in
      double-quoted strings, a 'make install' target, a --parsetrace
      option, perl 6 pod comments, warnings for unsupported or NYI features
** ... and made the nqp parser and compiler self-hosting.

* Rakudo stuff:
** Started the new implementation of Rakudo based on the nqp-rx
      engine, that is going very well.
** Have a new implementation of Lists, Parcels and Arrays,
      all of which can now have lazy semantics.
** Fixed constants and containers handling, Rakudo no longer
      allows assignment to constant values.
** Implemented "real" assignment metaoperator parsing (e.g., &infix:);
      Rakudo-ng now builds assignment metaoperators only when needed.
** Changed subroutines to be stored with the & sigil.
** Changed Rakudo operators to be "&infix:<+>" instead of "infix:+".
** Have many of the sanity tests running again; Test.pm compiles
      but doesn't run completely yet

* Plumage:
** Updated Plumage Configure and code to work with nqp-rx, passes
      Plumage's test suite.

What I'm doing this week:
* Continuing to work on Rakudo-ng, get us running spectests again
    and compiling setting (now called "CORE")
* More minor nqp-rx updates, error message improvements and better
    syntax checking
* Profiling the regex engine to find some speed improvements

What I'm blocking on:
* Useful programming time

As you can see, things are happening quickly, and we're all pleased with the progress. Together with Jonathan's work on dispatch, we appear to have overcome the two major hurdles identified for Rakudo Star. From here on out it's basically just fleshing out the rest of the implementation and adding feature coverage according to the ROADMAP.

I'll write more details soon; I have to get back to a few other tasks right now. Hope you enjoy reading the above list as much as I enjoyed writing it.

Tuesday October 20, 2009
11:56 PM

Hague grant work: the new regex engine and NQP

It's been quite a while since I've written any articles about Rakudo's progress, but the delay in articles has been because I've been really focused on code development for a number of things we're going to need quickly for Rakudo Star.

At long last I've made the time to make substantial progress on my Hague Grant, which will enable us to bring Rakudo's grammar and parser much more in line with the current STD.pm grammar. In fact, looking at the Rakudo ROADMAP one can see that a significant number of the critical tasks needed for Rakudo Star are depending on the "PGE refactors" identified in the grant.

This brings me to one of the major points of this post: In the weeks that follow this month's release we expect that Rakudo will be quite unstable as we undertake some much-needed refactoring and redevelopment of some of Rakudo's core pieces. The biggest change will be a complete replacement of Rakudo's underlying grammar; the grammar we have today is still largely based on the Perl 6 grammar as it existed in January 2008, but STD.pm and the Perl 6 specification have evolved significantly since then.

Jonathan and I believe that now's the time to bite the bullet and make another big refactor to bring Rakudo in line with the spec, even though it will likely involve a rework of many features and perhaps a few significant (but temporary) regressions. So, if you see some chaos and upheaval in Rakudo development in the next few weeks, it's a planned and necessary sort of mayhem.

Many of the needed grammar changes will be possible because of the grant work on protoregexes and a new operator precedence parser. Originally the plan was to build these features into the Parrot Grammar Engine (PGE), but after thinking long and hard about it I concluded that it would be better to redesign and reimplement a new regex engine than to try to fix PGE. For one, I think maintaining backwards compatibility would be a significant challenge (and a drain on my energy and resources). Another reason favoring a rewrite is that we now have better language tools available for Parrot, and a rewrite can take advantage of those tools.

Thus, instead of compiling directly to PIR, the new regex engine compiles to Parrot's abstract syntax tree representation (PAST). In addition, the source code for the regex engine is written in NQP instead of PIR.

For those not familiar with NQP, it's a Perl 6-like language I designed for Parrot in conjunction with the Parrot Compiler Toolkit. NQP acts like a "mini Perl 6", it understands a subset of Perl 6 language constructs and can generate Parrot code that doesn't rely on additional runtime libraries. Most of the HLL compiler authors for Parrot have been using NQP to generate PAST, and it's proven to be much easier to write and maintain than PIR.

Since the regex engine will now be written using NQP, it also seemed fitting that NQP would receive the ability to use Perl 6 regexes and grammars directly. Adding regexes and grammars to the NQP language means that a compiler writer can write nearly all of the components (parser, ast conversion, runtime libraries) using NQP. This is in contrast to the existing setup that requires multiple languages and APIs.

The new version of NQP is currently called "nqp-rx" ("NQP with regexes"); I may come up with another name for the bundle but I'm somewhat attached to "NQP". This new version also has a new source code repository (separate from Parrot) -- it's hosted on GitHub at http://github.com/perl6/nqp-rx .

Since mid-September I've been working on nqp-rx, and I'm very pleased with how it's all coming together.

For example, late last week I completed most of the work on the new regex engine. This first version includes a very naive implementation of protoregexes, which PGE lacked, and ultimately should perform pattern matching and parsing more efficiently than PGE does. It now compiles to PAST instead of directly to PIR, which means it will fit more cleanly with the rest of Rakudo, especially with being able to handle lexical variables and code blocks in regexes.

More importantly, the regex compiler is self-hosted (or "bootstrapped"). In other words, the regex engine is able to parse and compile the specification that was used to build itself. Stated another way, the regex engine is written using Perl 6 grammars and regular expressions that it knows how to compile.

Since completing the regex bootstrap I've been working on creating the new version of NQP based on the new regex engine. Over the weekend I created some common rules for parsing number and quoted string tokens, and yesterday and today I completed a new operator precedence parser (all of these based on the STD.pm equivalents). Now all of the pieces are in place to create a new NQP compiler, which I plan to do over the next couple of days. And, like the regex engine, I'm planning to make this new version of NQP self-hosted as well.

So, when all of this is completed, NQP will continue to be a "Perl 6 lite" language, but it will also support grammars, regular expressions, protoregexes, longest token matching, a very powerful operator precedence parser, attributes on classes, and a fair bit more. It should also be a bit faster than the previous NQP, and have a few additional optimizations (such as inlining of blocks).

Thus, here's a quick rundown of the status and plan for the next couple of weeks:

  1. Parrot 1.7 was released today (October 20).
  2. Jonathan has just completed a significant refactor of Rakudo's signature binding code and merged it into Rakudo's master.
  3. Scott Duff ("PerlJam") will be cutting the October Rakudo release on Thursday, based on the Parrot 1.7 release.
  4. Immediately following the Parrot release, new code for the Parrot Calling Conventions is set to be merged to the Parrot trunk. This is one of the major tasks (B) listed in Rakudo's ROADMAP.
  5. In the days following the Rakudo release, we'll be working to synchronize the multidispatch and binding algorithms in Rakudo with the new Parrot calling conventions.
  6. Also in the next few days, we'll complete implementation of nqp-rx, or at least bring it to the point that it can be used instead of the previous compiler tools for building Rakudo.
  7. When we're comfortable that the Parrot calling conventions work has sufficiently stabilized, we'll start a new branch for the major refactor of Rakudo's internals, switching to the new compiler tools, and updating the grammar to be much closer to STD.pm.
  8. We don't know how long this last piece will take, but it could easily occupy most of the month before the November Rakudo release.
  9. During the time that work is taking place in the branch, we don't expect much progress or changes to be made in Rakudo's master branch, so that people can continue to use and test the "more functional" version.
  10. If work bogs down in the branch, we'll regroup and come up with an alternate plan. But I don't think this likely.

It looks to be an exciting couple of weeks! I'll be writing more articles with details about the new regex engine, NQP implementation, and Rakudo conversion to the new tools. I hope and expect that by the November release we'll be completely switched over to the new regex engine and have knocked out a large number of the "critical" items on the ROADMAP for Rakudo Star.

Pm