use Perl

Some of you will recall that earlier in the year I posted an article with bad news that my wife developed ovarian cancer, so it's appropriate to share the good news: Paula's ovarian cancer is officially in remission. We completed her final chemotherapy treatment a couple of weeks ago, and the CT scan and other tests this week all show no detectable signs of cancer. So, no more treatments or medicines (yay!), and from here on we simply go for examinations every few months to watch for any recurrence (which is unfortunately quite common for ovarian cancer).

Although it took us a few extra months to get to this point (due to complications with finding a satisfactory chemotherapy mix), this is exactly the outcome we were aiming for at this point.

As always, we're glad to answer questions and provide more details for those who are interested -- but the bottom line is that things are looking very good right now.

Both Paula and I really appreciate and are grateful for the generous support, prayers, and understanding that everyone has shown us over these many months. You have all been a huge help to us, and we thank you.

Pm & Paula

Given that most of July and August for me was spent attending conferences, travel, vacation, and periods of limited network connectivity, it's been a while since I've been able to update Rakudo progress. So, this is a very brief update, to be followed by a longer post in a day or so.

First, we continue to make progress on passing more spectests. As of this writing Rakudo is passing 2278 spectests. A graph of our progress is available from http://www.pmichaud.com/perl6/rakudo-tests-2008-08-25.png. Much of the credit for passing tests over the past few weeks goes to Jonathan Worthington, Moritz Lenz, and Carl Mäsak (again, apologies if I've overlooked anyone). It's really good to see progress coming from so many sources.

OSCON and YAPC::EU were both excellent conferences, my compliments to the organizers of each of those. In particular, Jonathan, Allison, and I were able to use our time together at YAPC::EU to discuss and map out many of the outstanding issues for Parrot and Rakudo, which we can now begin documenting and implementing over the next few weeks and months. These include things like code initialization, handling Perl 6 parameter passing and signatures, MMD, strings, and many other issues.

One of the outcomes of this was that early last week I finally got the code in place to enable pre-compiled Perl 6 modules to function properly. At the moment this has a quite dramatic effect on running the spectest regression suite, because we aren't having to parse and recompile Test.pm on each test file execution (and parsing is still our biggest bottleneck). So, running the regression suite dropped from twelve minutes to under four minutes on my system, which isn't quite so interminable.

Of course, the next step will be to enable Rakudo to compiler Perl 6 programs into standalone PIR or PBC files that can automatically load the Perl 6 runtime. I expect to accomplish this sometime this week -- at present we need some refactors to the signature generation code that is blocking this from happening.

We're also working on enabling parts of the standard runtime library ("Prelude") to be written in Perl 6 and precompiled by Rakudo, instead of having it all written in PIR. As part of this we may implement a Perl 6-with-inline-PIR capability to help with the builtins, to make it easier to attach MMD signatures to the builtin functions and make sure they're exported properly.

We also have more of the interface for loading external modules (written in PIR or otherwise) specified, and will be working on that over the next couple of weeks.

At the YAPC::EU hackathon, Jesse Vincent and I also spent some time updating the Rakudo ROADMAP document. This newer version of the ROADMAP identifies some of the specific components that are left to be developed, along with estimates of their complexity and dependencies. If you look at the document you'll see some notations about how long it will take to develop each feature; these estimates are all in "idealized programmer units". An "idealized programmer unit" here assumes that the people involved have no interruptions or distractions, has all of the needed prerequisites in place, is mentally charged and ready for programming, doesn't have to wait to coordinate questions or answers with others, etc. As such, the times given should be taken only as relative estimates of task difficulty, and not how long a particular task will take in real-world time units.

The major subsystem redesign coming up for Rakudo will be changes to the parser and grammar engine to support protoregexes and longest token matching. This will enable us to support even more of the STD.pm "standard grammar". I expect much of this work to take place over the next four calendar months, as many elements are likely to require some intensive and sustained design and development effort (while trying to maintain progress in other areas). More on this as it progresses.

Pm

From a comment on rakudo.org:

dstar said:
  One thing I'm not clear on -- Rakudo is passing 1100 tests out of how many? IE, where's the goal line?

Well, the "goal line" is by definition a moving target. There's not a hard number we can cite -- even in Perl 5 the size of the test suite constantly changes as new features are added and new bugs are found. There will be a point at which we declare a particular set of tests as being the "official test suite" for Perl 6.0.0, but we're really not close to that point yet. Here's what I can say about the test suite thus far:

The "official test suite" is being built from the test suite that was created for Pugs -- the Pugs test suite has approximately 19,000 tests in it. But, the Pugs suite itself was somewhat disorganized and thus difficult to determine test coverage. It also has many tests that are incorrect, either because the language specification has changed since they were written or because they were incorrect in the original.

Thus, Adrian Kreher (mentored by Moritz Lenz and Jerry Gay) is in the process of conducting a complete review and reorganization of the Pugs test suite into the "official test suite", or what we call the "spectests" (in t/spec/ of the Pugs repository). We estimate that about 5,000 (25%-30%) of the tests in Pugs have been reviewed and migrated into t/spec/. And just because a test appears in t/spec/ doesn't mean it's necessarily correct -- we continue to find places where the test doesn't exactly match the language specification (or the language specification is ambiguous and needs clarification in the test).

Within the 5,000 or so spectests, there's a subset of 75 test files (~1,500 tests) that have been identified as the "spectest regression suite" for Rakudo. These are test files that we expect Rakudo to pass at any given moment in time, such that someone can do "make spectest_regression" and get back "All tests successful." The developers also use this to make sure that any changes to Rakudo don't break existing functionality. Of course, the spectest_regression suite is itself a rapidly moving target as Rakudo acquires more features and as the test suite evolves. For example, at the beginning of June there were 52 files (892 tests) in the spectest regression suite, today (July 1) we have 75 files (1522 tests) in the regression suite.

And beyond all of this have been comments from Larry and others that eventually the official test suite will likely be double or triple the current size of the Pugs suite -- I think I've heard estimates of 40,000-50,000 tests in the suite when we're "finished".

So, return to the original question of "Where's the goal line?", you can see that there's not really a straightforward answer. This is why I haven't been saying things like "passing 1100 out of N tests", because the exact value of N depends on what's being measured to depend on what one considers important, and until more of the official test suite is ready any useful N doesn't really correspond to the "goal line" of a released Perl 6. So, the number I tend to focus on at the moment is the absolute rate of increase in passing tests -- currently it's about 100 new passing tests per week. I expect that trend to continue or improve in the coming months. As long as the number of passing tests is increasing, we're making progress.

An obvious intermediate goal is to have Rakudo passing at least the tests in the Pugs test suite. This depends not only on Rakudo development progress, but also progress in reviewing the Pugs tests and migrating them into t/spec/ . This is also why the Perl 6 developers are constantly inviting and encouraging people to help with building and reviewing the test suite. As mentioned above, only about 25%-30% of the Pugs test suite has made it into t/spec/, and many of those have been simply moved into t/spec/ without significant review or verification against the Synopses. Once we have most of the Pugs test suite migrated to t/spec/ I'll probably start reporting passing tests relative to the size of overall suite. (But I'll also continue reporting the passing rate increase as well.)

Lastly, this is a good place for me to repeat an invitation I made at YAPC::NA: For those who are interested in helping with the test suite but don't know where to start, I suggest "adopting" a particular Perl 6 feature or Synopsis section and saying "I'm going to develop the tests for that feature." This doesn't require learning all of Perl 6, it only requires learning those parts of Perl 6 that are necessary for testing that particular feature. It also gives a sense of ownership for a part of Perl 6, as in "I did that part of the test suite".

I'll undoubtedly add more information about Perl 6 testing in the near future, as it's an area where many hands can help make light work.

Thanks to dstar for asking the question that prompted this post. :-)

Pm

This is the third "monthly" report for my development grant from the Mozilla Foundation and The Perl Foundation. As regular readers will have surmised by now, the definition of "month" has been stretched a bit for a variety of reasons, but as this report and the other reports will show, it's not been a hindrance to our progress.

To review, the primary goals of this grant are:

1. To have a Perl 6 on Parrot implementation that supports commonly-used Perl 6 constructs;
2. Improvements to the Perl 6 test suite;
3. To substantially complete the Parrot Compiler Toolkit, including documentation;
4. Increased community participation in Perl 6 and Parrot development, including development efforts on other languages utilizing Parrot and the Parrot Compiler Toolkit.

It's now clear that the work under this grant has been (or otherwise shortly will be) successful in meeting all of the above goals. As before, in this report I'll highlight the major events and milestones that have been reached since the previous report, and let my other article postings provide increased details.

Progress, April 2008 to June 2008

* Of course, one of the biggest news items is that in May 2008 The Perl Foundation received a $200,000 philanthropic donation from Ian Hague, roughly half of which will be used to support further development of Perl 6 and to build upon the work performed under this grant.

* In addition, Jonathan Worthington received a grant from Vienna.pm to continue his work on developing Rakudo; this grant led to implementations of type checking, multimethod dispatch, regex and grammar support, public and private methods, Ranges, scalar variables, runtime role composition, enums, and a lot more.

* During this period we continued to improve the documentation for the Rakudo (Perl 6 on Parrot) compiler, although more focus in this area is definitely needed. In April we published a list of Rakudo milestones as a basic "road map" to guide continued development. We also generated numerous articles and blog postings that describe the various features of the compiler and how it's all being put together.

* In May we started measuring progress on Rakudo by the number of passing tests from the official test suite. As of June 30, Rakudo Perl is passing 1126 tests from the official test suite, and we're averaging 100 new passing tests per week. I'm hoping this trend will continue.

* To facilitate development Moritz Lenz and Jerry Gay refactored the test harness to provide a "make spectest_regression" target. Now Rakudo developers can verify that changes to the compiler are not breaking any tests that were previously passing. Based on this we're able to to maintain a spectest-progress file, and Moritz created an excellent utility to display the progress as a graph:

http://www.pmichaud.com/perl6/rakudo-tests-2008-06-30.png

* Work on refactoring, improving, and updating the Perl 6 test suite is being handled by Adrian Kreher, Moritz Lenz, and Jerry Gay under a Google Summer of Code grant, along with test contributions and suggestions from many others. We now have a better process in place for reviewing and updating tests; this has enabled progress in other areas to also proceed more rapidly.

* Many of the basic Perl 6 statement types and constructs are now in place -- the primary notable exception being list assignment and lazy list operations. Implementing list assignment properly will require some modifications to the underlying grammar engine -- that work is expected to occur later this summer. Lazy lists and operators are awaiting some improvements to Parrot's exception subsystem (expected in early July).

* The Parrot Compiler Toolkit (PCT) and Not Quite Perl (NQP) tools developed in the first months of this grant continues to demonstrate its power and effectiveness. Most of the HLL translators for Parrot have either adopted or are planning to using PCT/NQP for their underlying code generation. In particular, both the Ruby and PHP implementations (Cardinal and Plumhead) have made significant progress by using the Parrot compiler tools. PCT now has support for basic "return" control exceptions -- other types of control exceptions will be added shortly.

* We continue to gather more active contributors to Parrot and Rakudo Perl. There has been a substantial increase in patch submissions -- so much so that we've held discussions about how we might improve our ability to respond to code contributions more quickly. I've given presentations about the recent improvements at FOSDEM 2008, The Texas Open Source Symposium, DFW Perl Mongers, and YAPC::NA 2008. Each of these presentations have increased participation and enthusiasm about Rakudo Perl and Parrot. More presentations about the project will be made at OSCON 2008 and YAPC::Europe 2008.

* At YAPC::NA 2008 Jim Keenan, Will Coleda, and others on the Parrot team organized a "Perl 6 and Parrot Workshop", in which we helped approximately 20 people download and build Parrot and Rakudo Perl and run through the basic test suite. This has further increased interest in the project; indeed, some of the participants came across some bugs (and filed bug reports and/or patches), and many are planning to hold similar workshops in their local user groups. We plan to repeat the workshop at other venues, including the Pittsburgh Perl Workshop, YAPC::EU, and likely other workshops and user groups.

* All of the "specific tasks" targeted in the end of the last grant report have been achieved.

Where things are headed next

The goal for the next few weeks will be to continue (and perhaps improve) the excellent development momentum achieved during the past couple of months. In particular, we will continue improving the test suite and Rakudo's ability to pass tests from the suite. In addition, as some new Parrot features become available (e.g., improved exception and lexical variable handling) we will be able to take advantage of them in the compilers and compiler toolkits. We will also begin identifying the specific individuals and tasks to be engaged under grants from the Ian Hague donation.

Specific tasks for the remainder of this grant:

• Continue improving the official test suite
• Develop a more complete implementation of Perl 6's exception model
• Implement basic lazy lists and operators
• More refactoring of basic operators, functions, and classes according to recent changes in the language specification
• Allow compiler and builtin library components to be written in Perl 6
• Continue convergence efforts with other Perl 6 implementations
• Write the final grant report, documenting the work performed, quantifying results achieved, and outlining the next phases of development

As things stand today, I expect to publish the final report for this grant sometime around OSCON 2008 (July 2008). Of course, I will continue to post articles at use.perl, rakudo.org, and parrotblog.org.

Thanks for reading!

Pm

[Rakudo spec regression status: 75 files, 1080 passing tests]

Rakudo Perl is now passing over 1000 tests in the official test suite!

http://www.pmichaud.com/perl6/rakudo-tests-2008-06-27.png

For those who are interested, the history of test results is available in CSV format from http://svn.perl.org/parrot/trunk/languages/perl6/docs/spectest-progress.csv .

Thanks to Jonathan, Moritz, Auzon, bacek, particle, Coke, chromatic, and everyone else who is helping to make this happen!

Yesterday at YAPC::NA 2008 we had a Parrot workshop, primarily organized by Jim Keenan. It went great -- Jim did a fantastic job. I think we had approximately 20 participants total, and several of the parrot folks came by to help.

During the workshop I think we managed to get nearly everyone with a working copy of Parrot and Rakudo Perl, and I even managed to take some notes on how we can improve things for others later.

We also found (and fixed) some bugs in the official test suite, and Deven Corzine even grabbed a Pugs commit bit and fixed a Windows bug in the fudgeall script!

I found the workshop to be very productive for me personally, and again my compliments and thanks to Jim Keenan and the other organizers for putting it together. (I was at the workshop to help, but I didn't do much of the work in putting it together. :-)

Pm

[Rakudo spec regression status: 64 files, 779 tests]

In my previous posts I reported on Rakudo's progress in passing the test suite. For those I had been using the output of Test::Harness to estimate the number of passing tests, but ultimately decided that it wasn't really giving me the information I want in the form I want it. So I wrote a custom test summarizer for Rakudo to make it easier to measure our test passing rate in the spectest suite.

Here's how things have been progressing over the last three weeks:

          Rakudo spectest regression daily results
                          files test pass fail todo skip
2008-05-22 00:00             31  564  223    0    0  341
2008-05-23 00:00             32  569  228    0    0  341
2008-05-24 00:00             32  569  228    0    0  341
2008-05-25 00:00             39  666  310    0    0  356
2008-05-26 00:00             39  666  310    0    0  356
2008-05-27 00:00             39  666  310    0    0  356
2008-05-28 00:00             39  666  317    0    0  349
2008-05-29 00:00             43  774  394    4   15  361
2008-05-30 00:00             43  775  415    0   15  345
2008-05-31 00:00             43  775  415    0   15  345
2008-06-01 00:00             52  892  518    0   15  359
2008-06-02 00:00             55 1012  623    0   15  374
2008-06-03 00:00             55 1012  623    0   15  374
2008-06-04 00:00             55 1012  624    0   14  374
2008-06-05 00:00             58 1107  668    0   15  424
2008-06-06 00:00             58 1110  674    0   14  422
2008-06-07 00:00             59 1139  682    0   16  441
2008-06-08 00:00             59 1139  697    0   17  425
2008-06-09 00:00             59 1139  699    0   15  425
2008-06-10 00:00             59 1139  699    0   15  425
2008-06-11 00:00             59 1145  705    0   15  425
2008-06-12 00:00             59 1145  705    0   15  425
2008-06-13 00:00             60 1148  707    0   15  426
2008-06-14 00:00             60 1148  711    0   15  422
2008-06-15 00:00             63 1201  754    0   15  432
2008-06-16 00:00             64 1226  779    0   15  432

The 00:00 in the above table represents midnight U.S. Central Time. The 'test' column indicates the number of tests run, and the 'pass' column shows how many tests were passing (excluding 'todo' and 'skip'). So, as of June 16 Rakudo is passing 779 tests in its spectest_regression suite.

In order to get a feel for our overall trend, I like to look at week-over-week progress instead of just the daily numbers. So, in the week from June 9 to June 16 we added 779 - 699 = 80 new passing tests. We'll see how things go in the future -- it will all depend on a combination of how quickly spectests can be reviewed and features added to Rakudo.

At the top of my future posts I plan to include a one-line summary of Rakudo's passing rate so that people can continually monitor our ongoing progress without having to scan the article (see the top of this post for an example).

Pm

Update: Moritz has generated a PNG graph of the above data, and is updating the script to take advantage of the CSV version of the data that we'll be maintaining in the repository. (I don't know how often the graph will be updated yet.)

[Rakudo spec regression status: 64 files, 779 tests]

Throughout YAPC::NA I'm hoping to make frequent posts about various events taking place, including the hackathons and other discussions. The Parrot hackathon took place yesterday and today, and we've had a lot of contributors and some excellent progress. A brief summary of some things that have occurred during the hackathon:

Moritz noticed a post on Perl Monks about Rakudo not yet supporting %*ENV; I forwarded it to rakudobug@perl.org to be filed as an RT ticket, and just a few minutes after that Jonathan had a working implementation.

Scott Duff (PerlJam) tested and closed a long-standing patch from Zev Benjamin to supply the 'run' function in Rakudo.

Bacek made some fixes to S29-list/minmax.t and so we've been able to add that as another set of tests to our spectest_regression suite. Jerry Gay also noticed some additional tests for spectest_regression. As of this writing Rakudo is now passing 779 tests in the spectest_regression suite.

Jerry Gay (particle) updated the mk_language_shell.pl script to automatically include the optable rules.

I did more refactoring and updating of the Range operators, as well as added the ternary ?? !! operator to Rakudo.

Olivier Mengué (dolmen) supplied patches to add the .perl method to Bool and Range types.

Chromatic has been cleaning up a lot of IMCC issues, improving constant string handling, and otherwise applying tons of patches to Parrot. Right now Parrot has 658 outstanding tickets -- we're hoping that can go below 650 by Tuesday's Parrot release.

Allison, chromatic, and I had a discussion about adding new branch opcodes to Parrot that PGE can use in lieu of bsr/ret for local subroutines. We came up with the "push and branch/jump" and "pop and branch/return" opcodes. These work like bsr/ret, but instead of using Parrot's internal control stack they allow the user to provide a stack such as ResizableIntegerArray, which can be much more efficient at storing and processing addresses.

Coke struggled a bit with Tcl on Parrot, but managed to update it to use the new control exception types in Parrot (and that Rakudo and NQP are now using for their 'return' statements).

I spent most of Saturday looking at trying to make a precompiled form of Rakudo's Test.pm script, so that we aren't re-compiling it upon each test execution. But I ran into some snags having to do with :load/:init handling, so I've stepped back from that for a day or two to re-think it (and there's been plenty of other good things going on).

Everyone at the hackathon seems to be having a terrific time, and we're getting lots done. I know that this report misses a few people who were working on documentation -- I'll try to get details in my next post.

Pm

Earlier today I realized that the numbers I reported yesterday for Rakudo's "make spectest_regression" were incorrect -- the Test module was not reporting the number of tests skipped (345), so the actual number of tests passed is only 430. That's still pretty good.

But on the plus side, since yesterday some excellent detective work and updating by moritz, bacek, particle, Auzon, and others means that we could add nine additional more test files into the spectest_regression target, and we're now passing 638 tests in the regression suite. Not bad for a 1-day improvement.

I suspect that there's more like this -- tests that can be moved into the "passing" category by simple fixes to the test and/or Rakudo itself. So, thanks to everyone who is helping to review the tests -- we're definitely making progress.

Pm