use Perl

For the past couple of weeks I've been mostly providing feedback and guidance to others on IRC, as well as trying to keep up with $otherjob. But this week I managed to get some coding in and continue progress.

Of course, it's great news that Jonathan++ has received a grant from Vienna.pm to continue his excellent work on Rakudo. He's making so much progress lately that I'm finding it difficult to keep up with him. Of course, that's a very good "problem" to have, so I'm not at all complaining.

Chromatic indicated that he had a goal of reducing the number of open RT tickets for parrot to below 700, so I spent some time over the last couple of days closing out some of my older tickets. Earlier today were briefly down to 698, but then I found some additional Parrot bugs that brings us back to 700 as I'm writing this. But I'm sure we'll be able to close a few more quickly.

The Parrot developers could really use some help with reviewing the Parrot RT queue. Personally I think that a good number of those 700 open tickets must be irrelevant by now or otherwise already addressed. Coke wrote a nice set of guidelines for this at http://www.parrotblog.org/2008/05/700-ticket-challenge.html, if you're interested in helping.

Several RT tickets had to do with updating PGE to include some syntax changes in Synopsis 5. Of course, changing PGE's syntax also means we have to update the various grammars that are using the older syntax, which takes a little while. I think I managed to convert all of the existing languages except for Plumhead, which looked a little more involved than most. So I'm hoping Bernhard will be able update that one soon.

I also cleared out a few RT tickets from the "perl6" queue, applying several useful patches and closing some out-of-date tickets. This leaves us with 22 new/open tickets for Rakudo, although I expect that number to grow as more people start playing with it and trying out various tests.

Tonight I updated Rakudo's interface into the operator precedence parser so that the 'fatarrow' syntax now works properly. So, we can use fat arrows for named parameters in subroutine and method calls. Soon I'll want to work on the hash and list composers, and then start to tackle list context and list assignment.

For the upcoming weekend I'm planning to update the spectests so that "make spectest" in Rakudo doesn't give back a ton of not-very-useful-error messages. As a result, we can't easily determine if a change to Rakudo causes something to break that was working previously. Fortunately, the 'fudge' script makes it much easier for us to maintain this. In an upcoming post I'll give an overview of spectests are being marked in Rakudo.

Spinclad on #parrot was remarking that was working on kjs' Squaak tutorial, and wanted to initialize the @?BLOCK variable using pure NQP instead of the PIR List class that the tutorial currently requires. A couple of weeks ago I updated Parrot's ResizablePMCArray class to automatically provide the methods that the PIR List class was using, so I figured we'd be able to use those directly. Turns out it's not quite as simple as that, but NQP still lets us work around it a bit. What we ended up with was the following:

        Protomaker.new_subclass('ResizablePMCArray', 'List');
        our @?BLOCK := List.new();

The first line uses PCT's "Protomaker" object to create a new "List" class that is a subclass of Parrot's ResizablePMCArray. The second line then creates a new List object, and binds it to @?BLOCK. Since List inherits from ResizablePMCArray, we automatically get the shift/unshift methods that Squaak needs to do its work.

In fact, another good task will be to update the Squaak tutorial (or create another tutorial) and use some of the more recent PCT and NQP features that were added in April.

However, it's probably not a good idea to rely too strongly on Protomaker just yet. Currently we have two protoobject implementations -- one for PCT and another for Rakudo -- and after discussions with Jonathan and others I've decided that they really ought to be merged into a single implementation that is used by both. Protomaker may very likely disappear in the merged version (but we'll undoubtedly have something equivalent).

Anyway, lots to work on as usual, and I'm glad I have a few weeks of available coding time again.

Pm

Last week I heard that someone was having trouble finding the Rakudo sources, and it was suggested that blogging about it here might provide an additional pointer. So, if you're looking for the sources to Rakudo Perl, it's currently held in the languages/perl6/ directory of the Parrot repository.

Eventually Rakudo and other languages on Parrot will likely get separate repositories, but for now we all find it easier to keep everything in a single repository.

The architecture and layout of the Rakudo source code is described in docs/compiler_overview.pod. This says what each source file does and how it all fits together.

Also, by popular demand, I've created a list of "milestones" for Rakudo Perl development and stuck them in the ROADMAP. Reproducing the 2008-04-28 version here, we have:

* list context, list assignment
* return and control exceptions
* class, role, objects
* regex, token, rule, grammar
* selected libraries written in Perl 6
* modules
* junctions
* lazy lists
* slices
* multi sub & multi-method dispatch
* captures and signature handling
* operator overloading
* other S09 features (typed arrays, sized types)
* heredocs
* macros
* module versioning

While the milestones roughly in priority sequence, this list isn't meant to be rigid or strictly sequential in nature. If someone wants to work on a later milestone even though we haven't completed the earlier ones, that's certainly okay. This list just gives us a way to see where things are fitting in my head.

Suggestions and additions to the above list are welcome.

Pm

This is a long-overdue report on the state of Perl 6 on Parrot (now "Rakudo Perl") development. In late 2007 I had been writing progress reports on a regular basis as as part of a grant from the Mozilla Foundation (MF) and The Perl Foundation (TPF), but in early January I had to pause work on the grant and my reporting to take care of a family emergency. I'm very glad that things have now settled down and that I can focus energy back on Perl again.

In spite of my distractions, a lot of progress has been made on Rakudo since last the last major report I wrote in December 2007. The work performed in November and December of last year has opened the door for many others to participate in Perl 6 and Parrot development, and as you'll see in this report, many have done so. I'm very pleased with the team of developers that has arisen and the quality of tools we've created for compiler development on Parrot. If anything, the hiatus has shown that our ability to advance isn't dependent on a single individual.

Progress, December 2007 to March 2008

An early draft of this report started by going into substantial detail about the progress that has been made in recent months, but I soon realized that so much has happened that the resulting report would be far too lengthy to be interesting or useful. Thus for this report I've decided to briefly highlight some of the events and milestones reached to give a good sense of the progress being made, and then write later articles that provide more detail. If I've overlooked an important contributor or event, please accept my apologies and let me know so I can highlight them appropriately.

* In January, the Perl 6 on Parrot compiler was renamed to "Rakudo Perl", or "Rakudo" for short. "Rakudo" is short for "Rakuda-do", or "The Way of the Camel" in Japanese. ("Rakudo" also happens to mean "paradise" in Japanese, which is nice.)

* A tool created by chromatic and Jerry Gay (with contributions from Will Coleda, Bernhard Schmalhofer, Andy Lester, and Francois Perrad) makes it possible to build an executable "perl6" binary. As a result, many people can now use Rakudo to execute Perl 6 programs by simply typing "./perl6 script.pl".

* In December we substantially completed the implementations for the Parrot Compiler Toolkit (PCT) and Not Quite Perl (NQP). The end result is that many of Parrot's HLL compilers are now written using Perl 6 source code instead of PIR.

* Jeff Horwitz continued to extend the implementation of mod_parrot to include scripting in Perl 6 (Rakudo) and PHP (Plumhead).

* We've increased the quantity and quality of documentation on Parrot's compiler tools to make them more accessible to a wider range of programmers. As part of this, we also created a "make a new Parrot language shell" tool that populates a directory with a basic setup for implementing a new compiler in Parrot.

* As an unplanned demonstration of how well the tools work, in January Will Coleda and Simon Cozens used the new tools to create a working LOLCODE implementation (http://lolcode.com/) from scratch with just a few hours' effort. Simon wrote about the experience and LOLCODE has served as a great example for others who are interested in compiler development under Parrot.

* We developed and initiated a plan for reviewing and refactoring the Perl 6 tests of the Pugs repository to become the "official" Perl 6 test suite (http://svn.pugscode.org/pugs/t/spec/).

* Larry Wall and Jerry Gay designed and implemented tools to allow the official test suite to be easily used by multiple Perl 6 implementations, including those that may be incomplete in the early stages of development. Both Pugs and Rakudo are now using these tools (fudge/fudgeall).

* Jonathan Worthington added junctions, classes, objects, roles, given/when statements, and many other significant features to Rakudo. He presented his work at several conferences and workshops, and wrote an article about his positive experience using the new tools to add features to Rakudo.

* Jerry Gay added support for radix conversions, command line argument handling, new test harnesses, many builtin functions and operators, and a perl6doc parser. He has also continued to serve as a mentor and advisor for other Rakudo developers.

* Recently Klaas-Jan Stol completed a draft tutorial for building compilers using the various compiler tools in Parrot.

* We added a compiler overview that describes the architecture and layout of the components of the Rakudo Perl 6 compiler.

* Since December at least twenty individuals contributed patches for new features to Rakudo Perl and the Parrot Compiler Toolkit. In addition, the Parrot subversion repository received a record number of commits in the months of December (1175) and January (1025).

Where things are headed next

The overarching goal for the next couple of weeks will be to again boost development momentum on Rakudo Perl and Parrot compilers to the levels we were seeing in December and early January. Because we now have many of the compiler tools and core code components in place, parallel development efforts for Rakudo are much more possible than they have been in the past. A key task for the next couple of weeks will be to recruit and support additional individuals who can help with the implementation effort.

Specific tasks for the upcoming weeks will include:

* Continue building and refining documentation for the compiler tools and Rakudo implementation
* Review code added to Rakudo in the past couple of months
* Perform a refactor of Rakudo's base classes and namespace handling to better support junctions, classes, and objects in Perl 6.
* Add control exceptions
* Add more hash and list operations
* Continue improving and refactoring the official test suite
* Update test system to provide graphical views of passing/failing tests in Rakudo

Thanks for reading!

Pm

We've finally come up with a name for the Perl 6 on Parrot compiler, it's now "Rakudo Perl", or just "Rakudo" for short. This name was suggested by Damian -- he writes:

> Some years ago, Con Wei Sensei introduced a new martial art: "The Way
> Of The Camel". Or, in Japanese: "Rakuda-do". This name quickly became
> abbreviated to "Rakudo", which happens to mean "paradise" in Japanese.
>
> Perhaps "rakudo" would suit, since:
>
> * "Of The Camel" clearly connotes Perl
> * Perl on Parrot is definitely the Way
> * The name meets Hugo's Obscurity-for-Search criterion
> (at least for non-Japanese-language searches)
> * It's nevertheless a real word (in one language)
> * It may help us steal back mindshare from Ruby in its home market

For the time being Rakudo will continue to live in the languages/perl6/ subdirectory of the Parrot repository, and we'll continue to build the bytecode and executable as "perl6.pbc" and "perl6(.exe)". My current expectation is that someday Rakudo will live in its own repository separate from Parrot, and we can decide then if any file renaming needs to take place.

So, we're now at the point where we can say that the term "Perl 6" strictly refers to a language specification, while terms such as "Pugs", "Rakudo", and "kp6" refer to implementations of Perl 6. Hopefully this will reduce some confusion.

Several people have also inquired about a release numbering scheme for Rakudo. My current position on this topic is to postpone any decision until we start making releases that are separate from Parrot releases. I think a postponement here is especially appropriate since we're still in the "rapid expansion phase" of the implementation. In the meantime, whenever we need to reference a specific version of Rakudo we can use either a specific date or a subversion revision number from the Parrot repository.

Pm

In case people are noticing a sudden slowdown in the frequency of my journal posts and in my commits to the compiler, I wanted to give people some indication of what has happened.

For the last half of December and the first week of January my wife hadn't been feeling well and was getting progressively worse. Finally on this past Tuesday things got bad enough that we took her to the emergency room. After a few hours of waiting and tests we learned that she has ovarian cancer. Since then most my attention has obviously been on taking care of her and the rest of our family.

How I think this will affect Perl 6 compiler development: Based on the research we've done and what we're hearing from her oncologist, I'm thinking that this will be just a temporary slowdown for a couple of weeks. The compiler is continuing to progress without my direct involvement -- Jerry Gay, Cosimo Streppone, Simon Cozens, and others have continued to add features without having to check with me for every little detail. In fact, making it possible for others to work on the compiler is one of the outcomes we've been hoping to achieve. Jerry Gay in particular has been picking up some of the leadership for the compiler, so from my perspective I don't feel we're losing too much momentum. (Read Jerry's journal posts to get an idea of the progress that is being made.)

Beyond that, even with this week's "surprises" there have been more than a few times when it's convenient (and somewhat therapeutic) for me to continue to think, develop, and write about Perl 6. This journal post is but one example. So, my current expectation is that as our family gets another week or so along the road to recovery, I'll again have time to work on Perl 6 development. Of course, if things take a big turn for the worse then all bets are off.

How this affects the Mozilla Foundation and TPF grants: I'm very grateful to the grant managers for their generosity, understanding, and support over the past week. The grant status report for December 2007 is obviously delayed, and barring any sudden changes I'm expecting to be able to write it up this week. And as soon as I have a better feel for events to come I'll be working with the grant managers on any timeline changes or restructuring that needs to be made. I expect any overall negative impact on the grant outcomes to be minimal.

After this post my writings will tend to remain focused on the Perl 6 progress and not much about the medical issues, unless major events take place. But if anyone has any questions, I'll be glad to answer them.

Thanks again to everyone for being understanding and helpful -- we greatly appreciate it.

Pm

Progress continues on perl6 for Parrot and the Parrot compiler tools. Things are starting to happen so quickly now that it's hard to remember them all for these journal posts. I'll need more frequent posts. In the meantime, if anyone notices that I've overlooked something newsworthy then let me know and I'll add it to the next post.

I think the biggest splash over the past couple of days appears to be that we now have a "make perl6" target that directly creates an executable binary for perl6 on Parrot. So now someone with a working Parrot can do:

        $ make perl6
        $ ./perl6 hello.pl

There are still some limitations -- in particular, the perl6 executable is dependent on files in the Parrot build installation (the binary requires a few hard-coded paths to Parrot libraries). Fixing this is likely going to require getting Parrot's 'make install' target to work properly, which is a long-standing issue of its own. So, it's not a perfect solution yet. Also, the process for building the standalone executable may not work on all platforms -- we need more feedback from different platforms. Still, being able to treat perl6 as a standalone executable goes a long way to making it easier use and experiment with.

The magic for creating an executable binary from Parrot bytecode comes from chromatic and Jerry Gay's pbc_to_exe script. A number of important contributions and improvements also came from Will Coleda, Bernhard Schmalhofer, Andy Lester, and Francois Perrad. It's impressive to see how quickly it all came together.

In fact, based on this Andy Lester is already asking for the "-e" option to be implemented in perl6. And the question about Perl 6 standard command line options also came up during today's Perl 6 design team meeting. Larry says he will be looking at what options every Perl 6 implementation should be providing, and that will give us something solid to work from. In the meantime someone will likely implement the '-e' option. :-)

Yesterday I added the "defined-or" operator and the @*ARGS array to the perl6 compiler, so now the Towers of Hanoi example works with perl6 on Parrot:

    $ ./perl6 hanoi.pl
    ndisks = 3
    AB
    AS
    BS
    AB
    SA
    SB
    AB
    $

In other news on perl6 compiler improvements, Jonathan Worthington added a preliminary version of 'given'/'when', and even performed the commit "live on stage" at the Israeli Perl Workshop. Jonathan has also added more to his Junctions implementation. Personally, I had feared that Junctions was likely to be one of the more difficult pieces for us to add -- especially autothreading support -- but based on Jonathan's implementation and some IRC conversations with him I think it might go a lot easier than I originally expected. Time will tell...

Jonathan also implemented a 'set_outer' method that should let us implement a workable eval() function, but I think we'll also need a couple of PCT/PAST improvements for it.

Jerry Gay has been adding radix conversions to perl6, and today started on named/optional/slurpy parameters to subroutines. He's already implemented the same features for NQP, so he has a good model to work from when doing the perl6 implementation.

Yesterday I started a long-awaited refactor of the "official Perl 6 test suite", which is currently housed in the Pugs repository. Essentially I'm hoping to reorganize the tests along the lines of the chapters and sections given in the Synopses and the Perl 5 camel book, to make it easier to locate tests and determine where to add new ones. So far I've only done a few of the operator tests from S03, but have notes about other tests that belong in S02 and S04. As part of refactoring the tests into more a more sane organization, we're also verifying that they are still correct with the latest synopses and that all of the details are covered. Where things need more work, we're adding "XXX" notes so that others can come along and improve the tests even further.

The "make spectest" target is now active again in perl6 on Parrot, and as more tests make it through the refactor we'll be adding them to the spectest suite in Parrot. Of course, others are very welcome to contribute tests, help with the test refactoring, and the like. All you need is a Pugs commit bit, which is very easy to obtain.

Larry, Jerry, and I also had an excellent conversation last night about ways different Perl 6 implementations can mark the test suite with todo/skip flags without stepping all over each other. I very much like what we ended up with... but since this post is already getting a bit long I'll describe it all in a subsequent post (probably also when there are some examples to point to).

The thing I haven't done yet that I want to do is to have an example of using Perl 6 to create built-in functions for the perl6 on Parrot compiler. If anyone can write up an example of a Perl 6 builtin function or method that is itself written in Perl 6, then I'll be very glad to get it integrated into the compiler so we can use it as a template/example for others to follow.

In other news, Will Coleda is working on a web interface to quickly summarize tickets in RT that are of high-priority to the perl6 compiler (or other efforts). When it's finished I hope it will make it easier for us to point people to where effort is needed.

Jeff Horwitz has written an interesting article on the Parrot wiki that describes how mod_parrot works. It's worth a look if you'd like to see more about mod_parrot, especially since mod_parrot integrates the whole Parrot stack from HLL languages down to embedding and NCI interfacing.

In fact, this is a good place for me to point out that the Perl 6 wiki has also gotten received quite a bit of reorganization and improvement this past week; many thanks go to Conrad Schneiker, Aaron Trevena, Herbert Breunung, and others for the updates. Please feel free to help us flesh it out -- even just asking questions or saying "this is confusing" is very helpful!

Finally, in the -Ofun department: Tonight Will Coleda used my "create a Parrot language" script to start a LOLCODE implementation for Parrot. Then Simon Cozens (lathos) extended it to include variable support. Given that it didn't even exist four hours ago, it's a pretty good demonstration that Parrot's compiler and language tools are working the way we want.

Update: Simon has also blogged about his experience in writing compilers for Parrot using Perl 6.

You can see that there is a huge amount of activity taking place with Perl 6 and Parrot; in fact, most of what I've written about here has occurred in the 72 hours since my last journal post (December 31). And there's undoubtedly some stuff that I've overlooked. Next on my list of things to work on is more test suite improvements, implementing eval(), addressing a few RT tickets, making sure that people working on the compilers can continue to be productive, and adding more documentation and details so that others can more easily join in the fun.

As always, more to come...

Pm

I hope everyone is having a good holiday, I know that I am! However, even with the holidays work has continued on perl6/Parrot, so here's some of the latest tidbits.

Several people have commented that the plethora of tools and acronyms used for building the perl6/Parrot compiler seems a bit overwhelming (PAST, PCT, PIR, PGE, NQP, ...). I can totally understand the sentiment expressed, so this weekend I spent some time drafting some documentation that I hope will start to blaze a trail through the jungle. The document is available in the Parrot repository as languages/perl6/docs/compiler_overview.pod (or follow the link to view it in a browser).

I also started a perl6-specific glossary of acroynms and terms for the compiler. Parrot has its own glossary that defines these items, it also includes a lot of terms and features that I think might be distracting to a perl6 hacker, so I created the perl6-specific one.

Of course, both documents are first drafts, so they may still not be totally clear, and there are some sections to be written still). The main goal for the documents is to highlight the important components, so that newcomers will be less distracted by the things of lesser importance.

Any feedback or comments on the drafts would be greatly appreciated. I'm hoping that these can also serve as a start to the generic compiler tools documentation, or "How to write a compiler for Parrot".

As to making it easier for people to contribute code, there's still one piece that needs to be in place first: we need to decide on the file structure we'll use to organize the Perl 6 source files in the compiler, and how they'll be linked in to the perl6 runtime environment. I expect to have this in place sometime tomorrow, and after that I think it will be much more obvious how it all fits together.

Several have expressed that they're looking to replace all of the existing PIR code with Perl 6 equivalent...but I don't think that's realistic (or necessary). The PIR code that exists now is mainly for implementing Perl 6 primitive types and operations (things like Str, List, infix:, etc.), and these are likely to remain in PIR because they're, well, primitives. Once built, the primitives don't need much hackery anyway, and then other functions and features can be written using Perl 6 sources on top of those primitives.

Anyway, if you're at all interested, read the documents, ask questions, and give it another day or two and I think (hope) it will all become much clearer, or that we can put the pieces in place to make it so. And I expect to have some good detailed examples soon as well.

In other news, chromatic++ and others have made some excellent progress on getting the pbc_to_c translator working. I haven't had a chance to play with it much yet, but once all of the kinks are worked out we should be able to compile perl6 into its own executable, so that someone can type "perl6 hello.pl" and have it work the way we all expect.

Also, Jerry Gay did a lot of work last week on adding optional and named parameters to NQP. While this doesn't translate directly to perl6, it does give us the basic outline for the perl6 implementation, which I think we will also have this week.

My other immediate task for the week is to get cracking on the test suite reorganization, because I know a lot of people are interested in helping with that. As always, there's just so much to do!

Lastly, there have been quite a few opinion posts on perlbuzz this past week about the past, present, and future of Perl 6. As part of the conversation Andy Lester asked if there was any way to describe what had been accomplished thus far in Perl 6, and in response I wrote up my view of "Here's what we've done". I think it shows that we're a lot farther along than many people realize, and not only that but the pace of development is picking up dramatically. It's worth a look.

Happy new year, everyone!

Pm

Happy birthday, Perl!

On this 20th birthday of Perl, I'd like to express my appreciation to Larry Wall (and his family) for the many years of leadership, dedication, wit, wisdom, and sacrifice that he has put into Perl. Of course, there are many more people who deserve our thanks for Perl, too many to list here, but Larry has always been a fixture of the community and I'm so thankful for his efforts.

Today Parrot 0.5.1 (Hatachi) was released, and quite frankly it's another big leap in Parrot and Perl 6 development. So much has happened since my last journal post of a week ago that it's hard to know where to begin! The big news is that as of today the perl6 compiler is once again passing all of the 00-parrot and 01-sanity tests, so we can begin attacking the official test suite again. In other news:

* Jerry Gay made even more progress on the Punie (Perl 1) compiler under Parrot, using PCT and NQP he managed to get slurpy arguments working, and all of the tests passing. To me this is a signature achievement, and it shows that the compiler tools (PCT, PAST, and NQP) are working as we had intended.

* Jerry has also been instrumental in getting perl6 to pass its test suites again, particularly in adding constructs to the grammar, including "for", regexes, "END", and the like. So, we're passing the basic test suite again.

For those who might be concerned that we seem to have done a lot of work "just to get back to where we already were", I can only say that from an ease-of-development the perl6 compiler we have now is far ahead of what we had before. Unlike the previous compiler, we're finding that new features are easily added to the system, and that many changes aren't requiring me to shepherd many of the low-level details. For example...

* Flavio Glock joined the Parrot effort this week in a big way, and has been exploring NQP and perl6 and how they can fit in with the KindaPerl 6 (kp6) effort. Flavio has brought us many useful ideas and insights from kp6, and today he added BEGIN blocks to the perl6 compiler. We've also got some initial ideas about handling eval() in perl6, which will be very useful. Right now we're blocking on some issues dealing with lexical variable handling, but Jonathan Worthington has said he'll take a look at that.

* Jonathan Worthington has also been eager to play with junctions in Perl 6, so earlier this week he added an early implementation of a Junction object type. It even works for some simple things, although it doesn't yet do any sort of autothreading. On top of that, Jonathan also added postcircumfix:( ) (subroutine call), postcircumfix:{ } (hash index), and a working implementation of the '&' sigil. Of course, what I really liked best about this were his frequent comments about how easy and fun it was to work with the perl6 compiler. That makes me feel good.

Oh, and by the way, Jonathan also managed the Parrot 0.5.1 release today. Great work!

* chromatic implemented the "copy" opcode that we'll be needing to properly handle assignment in perl6 and other compilers. That was likely to become a huge obstacle to getting things to work properly, and I'm glad the ball for it is back in the PAST/perl6 court so that we can finish that off.

As for me, most of what I've been doing is answering questions on IRC and doing any refactors or cleanups that need to be done to allow others to implement features. So, I worked on things like
      * getting :slurpy to work in PAST (needed for Punie and perl6)
      * getting perl6 to talk directly to PGE again for regex matching
      * refactoring block generation in perl6 to better handle formal params
      * refactoring the PCT::HLLCompiler object and PAST/POST compilers to be a bit saner
      * describing various changes/improvements we're likely to want in Parrot

So, it's been a very productive week, and I'm hoping this next week can be equally productive. The things I plan to focus on for this week:

* some more documentation and articles about PCT, PAST, and NQP, and getting started with perl6 (including more detailed notes for new hackers)

* Scott Duff (PerlJam) has an article half-written about Perl 6 regexes that is waiting for a usable implementation, so I'd like to see perl6 get far enough along that he can publish his article based on perl6

* Jeff Horwitz is doing lots of amazing things with mod_parrot, and he says that if we can get modules working in perl6 that he'll be able to write mod_perl6 in Perl 6. So, 'module' is high on my list of things to accomplish.

* Of course, we're going to return to the effort of getting perl6 to pass as many of the official Perl 6 tests (from the pugs test suite) as we can, and refactoring tests as appropriate.

Stay tuned, take a look at the pieces that are coming together, and if you can, come join the fun!

Pm

More and more interesting things are happening in the Parrot world -- it's hard to keep up. Here's a laundry list of things that have happened since my last journal post.

* In perl6, Jerry Gay and I have re-enabled arrays, try blocks, testing for defined values, if/unless statement modifiers, booleans, ranges, and a whole lot more. More details about perl6 in an upcoming post.

* Last week I had to go out of town on business for a day, but while I was on the planes I was able to convert the fledgling Python on Parrot compiler ("pynie") to use NQP and the compiler toolkit. This reduces the compiler to using only 473 lines of PIR, including comments -- the rest is all written in Perl 6 (809 lines).

* Bernhard Schmalhofer has been updating the PHP on Parrot implementation ("Plumhead") to use PCT, and then he and Jeff Horwitz surprised us all with PHP running in mod_parrot !

* Jerry Gay is also making great progress in updating the Perl 1 compiler ("punie") to be written in NQP+PCT instead of PIR... in fact, as I write this he has just merged his development branch back into the trunk ( languages/punie/ ). Yay! Punie is also passing some TODO tests that weren't passing in the previous version. Yay again!

* There have been a lot of minor improvements to the compiler toolkit, especially in the area of error reporting and the overall API. More will undoubtedly come as we continue to explore the compiler space.

For any interested C programmers out there, I think that a patch for RT#47992 would be very appreciated. I wouldn't think this would require a lot of knowledge of parrot internals -- just some changes to Parrot's command-line argument handling.

For Perl (5) hackers, the language implementors on Parrot could really use some improvements to the test harness(es). Right now several languages (perl6, punie, pynie, pheme) are able to implement their own 'make test' target, but we don't have a good way to incorporate those into Parrot's top-level 'make languages-test' or 'make languages-smoke' targets. It has to do with the fact that the test files for each language aren't written in Perl 5 (the harness is, but not the individual tests). In October Colin Kuskie and a few others were working on this issue under the "unified-testing" branch, but that seems to have stalled. With compiler and language development accelerating it would be really good to see that effort picked up again.

Finally, a brief note that every Tuesday at 18:30 UTC the Parrot committers hold an online status meeting on irc.perl.org/#parrotsketch . Observers are welcome to attend, and the logs of the meetings are archived at http://www.parrotcode.org/misc/parrotsketch.html .

Pm

This is an update on the status of Perl 6 on Parrot development
as of December 4, 2007.

Progress in November 2007
-------------------------
The last report was published in November as a "road map" [1]
for continued development of the Perl 6 on Parrot compiler (perl6).
Since then we've basically been following the steps outlined
on the road map and we are seeing significant progress on the
compiler.

The big news for this month is that we are now using a
lightweight Perl 6 translator ("NQP") to implement the
translation components of the perl6 compiler. Making this
transition means we are currently failing quite a few tests
in the test suite, but Jerry Gay and I are rapidly cleaning
those up. This is also a good place for others to join in if
you're interested -- see "How to hack and contribute" below.

So, as mentioned in the previous status report, the perl6 compiler
has four major components: the parse grammar, specialized parsing
routines, the ast transformation, and the runtime support. Now
that we're using NQP for the ast transformation, this means that
the bulk of the compiler--the parse grammar and the ast
transformation--are written using Perl 6 syntax. If you want to
take a quick peek at these without downloading Parrot, they're
currently available at:

    http://svn.perl.org/parrot/trunk/languages/perl6/src/parser/grammar.pg
    http://svn.perl.org/parrot/trunk/languages/perl6/src/parser/actions.pm

We're also looking into the possibility of rewriting much of
the runtime support as Perl 6. One of the significant things
we added this past week is the ability to embed PIR code directly
inside subroutines compiled using NQP. This has turned out to be
extremely useful for development -- we use Perl 6 syntax for most
things, but can drop in and out of PIR whenever we (temporarily)
need it.

Another major item completed yesterday is a new set of routines
for parsing and transforming Perl 6's quote syntax. Of course, the
primary reference for this is the STD.pm grammar [2], and if
you take a look you'll see that it's a fairly complex component
because of the many options, interpolation forms, and extensions
that are available. It will likely be a while before we have all
of the pieces to implement quotes exactly as STD.pm has them,
so in the meantime I've written up a couple of parse subroutines
(in PIR) that handle most of the quoting forms and options we're
likely to encounter, including interpolation. The only major
forms not available at present are the «...» and heredocs, and
the «...» form is waiting only on some Unicode fixes in PGE
and Parrot (RT#48108).

While I'm on the topic of STD.pm and parsing: I do need to
compliment Larry, Damian, and others who have been building
STD.pm and the language specification for the improvements
made in the language syntax over the past year or so, such
as in quoting. The version of Perl 6 we have now is *much*
easier to parse--by both humans and machines--than what we
had a year ago.

Where things are headed next
----------------------------

There are two major items we will be focusing on for the
next couple of weeks. The first will be continued development
of the perl6 compiler -- getting it back to passing the tests
it was previously passing, as well as adding new features and
tests. Progress on this is already being made and should
accelerate nicely.

The other area of focus will be in writing a lot of prose --
documents that will help new developers figure out where
things are and how it's all put together. For instance, we
really need a "getting started" document for perl6 that
describes how to build the compiler, how to invoke it,
and how/where to report bugs and patches. Much of this
information is currently in the ROADMAP, but would probably
be better placed in a getting started document. We also want
a developer's roadmap of some sort.

I'm also planning to publish a "developer's glossary" somewhere
to identify many of the various terms that get thrown around
in development (PCT, PGE, NQP, STD.pm, regex, PAST, etc.).

Also on the horizon we will start looking at reviewing and
possibly restructuring the Perl 6 test suite, and coming
up with a mechanism to quickly update/publish the status
of various language features in the compiler (hopefully with
references back to the language spec).

How to hack and contribute
--------------------------

If you're interested in hacking on the compiler, this is
a unique time to start getting involved. There are a lot
of relatively easy features to be tested and added, and
the discussions are about the basics of the compiler and
how it's put together.

As mentioned above, we need a good "getting started" document,
but for now here's a quick overview. First, you'll want to
download and build Parrot -- instructions for this are at
http://www.parrotcode.org/. Once you have Parrot built,
you can change to the languages/perl6/ directory and type
'make' to build the perl6 compiler. Then use the command

        $ parrot perl6.pbc hello.pl

to run a hello.pl program (written in Perl 6).

To learn more about how compilers are being structured
in Parrot in general, I recommend looking at languages/abc/ [3].
This is an implementation of the bc(1) language using the
same tools we're using to implement perl6, with the advantage
of being a lot simpler language than Perl 6. We've also tried
to put a lot of comments into the source files for abc that
explain what each component is doing.

After that, you can look at perl6's source code files and
running some tests (either from the suite or files that
you build). If you want to see perl6's intermediate compilation
results, use --target=parse, --target=past, or --target=pir
immediately after the "perl6.pbc" part.

In the next couple of weeks we'll be writing up the documents
that describe the details of the compiler implementation, but
in the meantime please ask us questions! We'll use the questions
to help author the documents. The perl6 and Parrot developers
tend to hang out on irc.perl.org/#parrot, and questions are
welcome on the perl6-compiler mailing list [4].

If you find a bug or have a suggested patch, send it to
perl6-bug@perl.org. Please use the moniker [PATCH], [BUG],
or [TODO] (including the brackets) at the start of the subject
so that RT can appropriately tag the item.

Thanks for reading!

Pm

1. http://svn.perl.org/parrot/trunk/languages/perl6/ROADMAP
2. http://svn.pugscode.org/pugs/src/perl6/STD.pm
3. http://svn.perl.org/parrot/trunk/languages/abc/
4. http://dev.perl.org/perl6/lists/