Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

pmichaud (6013)

pmichaud
  (email not shown publicly)
http://www.pmichaud.com/

Patrick Michaud is the pumpking for the Rakudo Perl 6 compiler. He holds a Ph.D. in Computer Science and was formerly a Professor of Computer Science at Texas A&M University-Corpus Christi. He is currently a software developer and consultant focused on open source development and applications, including Perl, PmWiki, and Linux.

Journal of pmichaud (6013)

Saturday September 05, 2009
02:56 PM

Rakudo day: Rats, contextual variables, and %*ENV fixes

As I mentioned in my previous post, I'm doing two Rakudo days per week for my Vienna.pm grant to make up for some of the weeks I missed during conferences and summer travel. This week I focused on adding a rational (Rat) data type, support for contextual variables, and fixing up the handling of environment variables via %*ENV.

After the work that was done on operator overloading last week, implementing a basic Rat data type ended up being surprisingly easy. In fact, I just wrote it as a straightforward Perl 6 class and added it to the setting -- you can see the easy-to-read results in src/setting/Rat.pm. I did just enough of the implementation to get the basics in place, at which point Solomon Foster, Moritz Lenz, and others started working out a lot of the other details such as conversions to and from Rat and other MMD considerations. As a result we have quite a few clarifications and improvements to the specification, a lot of new tests have been written, and Rakudo passes a lot more tests in the test suite.

While adding Rat was relatively easy, adding contextual variables to Rakudo was quite a bit more challenging. Contextual variables are similar to lexicals, except that searching for a contextual variable involves looking at a block's callers instead of the block's outer lexical scopes. Contextuals have many similarities to the way that Unix environment variables work, but they also fill many of the niches that would have otherwise been handled by global or environment variables.

Contextual variables in Perl 6 are denoted using the * twigil. Here's a simple example that uses a contextual:

    sub foo() {
        say $*PHRASE;
    }

    sub bar() {
        my $*PHRASE = 'world';
        foo();
    }

    my $*PHRASE = 'hello';
    foo();
    bar();

When run, this outputs "hello\nworld\n". When foo() needs to look up the value of $*PHRASE, it does so by first checking its local lexpad, then checking its caller's lexpad, then its caller's caller's lexpad, and so forth until it finds a declared $*PHRASE. Thus in the first call to foo() above, foo()'s caller is the mainline and so foo() finds the mainline's definition of $*PHRASE ("hello"). The call to bar() declares a new contextual named $*PHRASE, and thus the second call to foo() sees bar()'s value of $*PHRASE ("world").

In other words, contextual variable lookups always use the (dynamic) caller chain instead of (static) lexical scoping.

This turns out to be incredibly useful in a number of situations where we want to provide called functions with an idea of the dynamic context in which they're being called. One of the most common uses for contextuals can be to override the default output for print, say, and other builtin I/O functions. The print and say builtins output to $*OUT, a contextual variable. So, to change the output destination for any calls to say or print, simply declare a new contextual $*OUT:

    say "Hello world on standard output";

    {
        my $*OUT = open("outfile.txt", :w);
        say "This text is going to outfile.txt";
        say "Report follows:";
        print_report();
        $*OUT.close;
    }

    say "...and this also goes to standard output";

So, in the bare block in the above code, the declaration of a new $*OUT contextual causes all of the calls to say and print executed inside of that block to send their output to outfile.txt instead of the default standard output. This behavior holds even for nested function calls such as print_report() -- its calls to say and print also get sent to outfile.txt.

This ability to establish a dynamic context value that can be quickly looked up from within nested function calls is key to simplifying many compiler implementation details. If you look at STD.pm you'll quickly see how much it relies on contextuals. Indeed, one of my next tasks will be to add contextuals to NQP and PGE so that we can move even closer to the way STD.pm handles parsing.

As I alluded above, contextual variables also fill some of the niches held by global and environment variables. If a search for a contextual variable doesn't find it declared in any of the callers' lexpads, we then fall back to looking in the GLOBAL and PROCESS packages. Thus a $GLOBAL::foo variable can be quickly accessed using $*foo if no caller has declared its own $*foo.

In fact, most of the predefined contextuals such as $*OUT, $*ERR, %*ENV, $*PID, etc., are actually defined in the PROCESS package. This gives us a lot of flexibility in deciding things:

    $*OUT          # the standard output of my caller
    $PROCESS::OUT  # the standard output for this process

The implementation for contextual variables in Rakudo required creating new Parrot operations for finding variables in the dynamic caller chain instead of the static outer scope chain, creating a (!find_contextual) sub in Rakudo to handle the contextual lookups, refactoring the compiler to treat the * twigil as a contextual instead of a global variable, and migrating the previous "contextual globals" we had into the PROCESS namespace. These later steps took the longest by far to accomplish.

Finally for this week's Rakudo days, there have been a number of RT tickets and requests for fixing up the handling of %*ENV -- previously it was based on Parrot's Env PMC and some of the Parrot-isms leaked out (e.g. RT #57400). The implementation of contextual variables opened the door to cleaning up %*ENV to work more appropriately, so I went ahead and did that.

Overall I'm very pleased by the progress made in this week's pair of Rakudo days: we got a good start on rational datatypes and cleaning up the relationships among the built-in scalar types, we now have a working contextual model that we'll need for the next phases of compiler development, and environment variables are now working more like they are supposed to.

Many thanks to Vienna.pm for sponsoring this work.

Pm

Friday August 28, 2009
12:37 AM

Rakudo Day: operator overloading, faster isa, better build

July and August have been really busy months with conferences and vacations, so I've fallen a bit behind on Rakudo days for my Vienna.pm grant. However, now my travels are largely finished and the kids are back in school again, so I'm hoping I can manage at least two Rakudo days per week until I'm caught up with the original schedule.

This week's major task did indeed need two days: I've now enabled operator overloading for many of Rakudo's builtin operators. Previously Rakudo has allowed some custom operators to be defined, but overloading the builtin operators (such as infix:<+>) would generally result in "Null PMC in find_method" errors.

The main approach I ended up taking for this was to re-implement most of the existing operators (previously written in PIR) as Perl 6 multisub definitions with inline PIR. This ended up being somewhat more difficult than it might sound or appear from simply reading the patch -- part of the challenge is that if any variant of an operator is written using Perl 6 multisubs, then all of them must be written that way. But since we had been generating PIR for the Whatever and Junction variants of some builtins, that meant rewriting those versions as well (and figuring out a way to get Rakudo to create WhateverCode objects from the setting).

Anyway, the hardest pieces are now done, so that many of the built-in operators can be overloaded with custom variants. I'm sure a lot of people will start taking advantage of those; for example, I'm hoping that SF will be able to update his Vector class to overload the builtin operators now.

There are still some operators that need to be moved to the setting; although I've been able to migrate prefix:<-> and prefix:<~>, attempting to do the same with prefix:<+> causes a failure in one of the spectests. We'll keep plugging away at it until we figure it out.

I"m also seeking clarification about the definitions of some of the relational ops; for example, infix:<==> and infix:<eq> are defined in terms of the infix:<===> operator, but I'm curious about the definitions of the other relational ops.

One item that greatly concerned me about moving the operators into the settings was the possibility that it would significantly slow down Rakudo, because subs compiled from Perl 6 result in a lot more code than hand-written PIR subs do. I even explored some ways to be able to automatically rebless PIR multisubs into Rakudo equivalents. However, I decided to just try the Perl 6 approach (we'll ultimately need to do that anyway), and I was pleasantly surprised that the changes didn't result in significant speed hit to the spectests. Indeed, my timings show that the use of the Perl 6 versions may in fact be slightly faster. I'm not sure how this can be possible, but my best guess at this point is that Rakudo's custom multisub dispatcher (which Jonathan wrote as part of his Hague Grant) is somehow significantly faster than Parrot's default multi-dispatch. But that's just a guess on my part...

Speaking of speed, yesterday I also finished and committed another significant change to Parrot's "isa" functions for testing if a PMC is an instance of a given type. For a long time Parrot has used type name (string) comparisons to check isa membership; this is not only a bit slow, but it can also result in unwanted type name collisions. Ideally the comparison should test the identities of the class objects; but getting this to work required cleaning a few other pieces of Parrot's object and class handling. When it was all finished we obtained a ~4.5% overall speed improvement in Rakudo's spectests.

Lastly, I've done some more work on improving Rakudo's build and install environment. We now have Rakudo building from an installed Parrot, and yesterday I improved Rakudo's Configure.pl so that it warns with a more useful error message if the files needed from a Parrot devel installation aren't present.

Next I'll need to review the ticket queue again to see how many tickets have been resolved by the above changes. I'll also want to see about adding the tighter/looser/equiv traits to user-defined operators. But mainly I'm glad that we can now start to do some more advanced operator handling and overloading in Rakudo.

Thanks as always to Vienna.pm for sponsoring this work.

Pm

Saturday August 08, 2009
09:26 PM

"Perl 6 'finished'" isn't the story I want to tell

In a comment to my post announcing Rakudo Star, gdonald writes:

Here, I'll explain since you don't seem to get it. When people ask "When will Perl6 be finished?", they want to know when there will be a Perl 6.0 release.

Actually, I really think I do "get it", although I admit I may not be clearly expressing that. Later in the same comment thread gdonald follows up with:

Seriously, the Perl6 spec isn't even nailed down yet? What is the hold-up on that? Design by committee?

To me, that sounds an awful lot like a variation of "When will Perl 6 be finished?", especially the "What is the hold-up..." part. In short, the thread quickly went from "you don't seem to get it" to ultimately saying something very like the question I used to start my original post.

Neither this reply nor my original article are intended to imply that people shouldn't be asking such questions or that they're wrong for doing so. I call out gdonald's messages not because I wish to ridicule him or prove him wrong, but simply because I think they illustrate well the "disconnect" that exists in discussions about Perl 6, and why I'm hoping to change the discussions to something more useful than "finished" and "not finished". Because the end result of the "finished/not finished" discussion is often:

Seems like Perl6 might be going the way of GNU/Hurd, eternally under development, and never to land.

I don't believe that Perl 6 will be eternally under development -- but I am concerned that the perception of "eternally under development" is potentially a significant blocker to continued progress on Rakudo and Perl 6. That's one of the major issues that Rakudo Star is intended to address.

I also completely "get it" that for most people the real thing they want to know is "When can I start using Perl 6?" in the sense of "apt-get install perl6". But I think even that question has many hidden assumptions that must be exposed before there can be any sort of useful answer. Indeed, the answer changes depending on who is asking the question and what they want to do. So another purpose for Rakudo Star is to illuminate the development process and our ongoing status so that people can begin answering that question for themselves.

As the quotes above illustrate, somewhere there's an implicit assumption that the specification should be finished already. When we then say that the spec isn't finished (and it isn't), people often conclude that Perl 6 is suffering from a "design by committee process flaw" that is preventing the specification from being finished, and in turn that's what is blocking "apt-get install perl6". I think these misconceptions arise from assuming a "waterfall" development model where it's a one-way path from specification to implementation. But just because (unlike Perl 5) Perl 6 has a spec that is separate from its implementations doesn't mean the specification comes first.

Looking back, it's not too surprising to me that people have assumed a waterfall model of development -- those of us working on Perl 6 haven't given a good public picture of the true story. The various development roadmaps we've provided have been accurate as far as they went, but they didn't really give insight into the underlying processes. And for many of us, myself included, we're only now learning how to put words to parts of the process to be able to tell others about them. Here's my version...

For the past few years, changes to the specification have been made almost exclusively in response to the concerns and issues that have come about from compiler implementations and application programs written in Perl 6. For example, while implementing a feature in Rakudo we often discover a place where the specification is self-contradictory, so the specification is changed to resolve the contradiction. Or, someone will be using Perl 6 to write an application, and as a result we find places where the specification is deeply sub-optimal and needs to be cleaned up. These days it's very rare that the design team says "wouldn't it be nice/better if Perl 6 changed to do X" on its own initiative; our discussions are nearly always "implementations are having trouble with this part of the specification, what do we need to change to improve that?"

So the specification isn't finished, but that's mainly because it's evolving in response to implementations and applications, and not due to a tendency to over-design it. Because the specification is evolving based on implementation issues, simply freezing what exists now as "6.0.0" will make things harder on implementors, not easier. In other words, freezing the existing spec will paradoxically delay implementations of Perl 6, not expedite them.

We are rapidly entering a phase where what we will need most is for more people to be creating real use cases in Perl 6, testing the soundness of the design and implementation. That is, we need to see more applications to be written in Perl 6 so we can harden the specification even further. For many people and applications Rakudo is ready for use today, but there are still enough issues that I'm hesitant to call it anything other than a "development release" for a more general audience. The problem then is that many people rightly take "development release" to mean "not ready to use yet", and that's also counterproductive to what we need.

This is where Rakudo Star comes in, and this is what I mean by "a useful implementation of Perl 6". It's intended to recognize that a Perl 6 release can be useful to many even though it may be incomplete. It's intended to provide a bright line where we can say "here's what is working now" and "here's what is not working yet". It's intended to help people determine when Perl 6 has been sufficiently realized to be ready for their needs. And it's intended to make it possible for more people to start writing programs in Perl 6, because one of the things we are needing is real-world use cases to test, refine, and extend the existing design.

At the same time, we need to be very careful about using the label "finished" or "1.0" for anything that isn't all of what has been promised for Perl 6.0. (See KDE 4 development for why this is a bad thing.) In fact, I'd like us avoid the notion of "finished" altogether. Instead, I want us to regularly deliver something that is useful and usable, make it clear what we are delivering, make it clear what we're not delivering, and enable people to see when Perl 6 is likely be ready for them (as opposed to "ready for all").

Ultimately Rakudo Star is intended to give some justifiable support and clarity to phrases like "Perl 6 exists" and "you can now write usable applications in Perl 6", without the distractions that arise from the "When will [Perl 6 | Rakudo] be finished?" sorts of questions. It's not that I think people are somehow wrong for asking these questions -- I think I do very much understand what leads people to ask them -- it's just that I'd like us to start finding ways to move our discussions beyond the "finished / not finished" trap that we seem to have fallen into. I'd like to help us all escape this trap because (1) I don't think it reflects reality, and (2) if "Perl 6 is finished" remains the primary criteria that most people use to decide whether or not to write applications in Perl 6 (and the criteria that we hold ourselves to), then we'll never get there.

Pm

Wednesday August 05, 2009
01:42 PM

Building a "useful release of Perl 6"

The most common question I get from people who aren't generally involved with Perl 6 development is:

"When will Perl 6 be finished?"

In some ways the wording of this question bugs me a bit, because the word "finished" implies there's a point at which we all say "We're done" and development ceases (or at least moves to some other phase). But there really isn't a "finish line" for Perl 6, there are just stages of development at which more and more people are able to make use of whatever is currently available.

So, once we eliminate the notion of "finished", the wording is often changed to try to make it more tractable, often by asking when there will be a "stable release", or when the specification will be frozen so an implementation can be completed, or many other variations on the theme. I understand the assumptions behind questions like these, but at the same time part of me thinks "Huh? Those questions don't really fit with the way things really happen in language development..."

The truth is that language design is an evolutionary process, with the design and implementation efforts serving to influence and guide further progress in the other. (See "whirlpool model".)

But there's another important input to the process: "real-world" application programs. We need to know how Perl 6 is actually being used in order to finish parts of the specification and implementation. Indeed, there are parts of Perl 6 (e.g., concurrency) where the specification is incomplete or underspecified precisely because we need input from people writing Perl 6 applications.

But this poses a problem of sorts, because if programmers are waiting for Perl 6 to "finish" before they start using it to write programs, and if Perl 6 is blocking on feedback from applications and implementations before it can progress, then we have a deadlock of sorts.

So, we need a way to break the deadlock. To me, one good answer is to start making releases of Perl 6 that may not implement the entire Perl 6 specification, but that application writers will feel comfortable enough to start using in their projects. I've started to call these "useful releases" or "usable releases". While it might not have every feature described in the Perl 6 synopses, enough features will be present that can make it a reasonable choice for application programs.

In doing this, I'd like to also refocus conversations to avoid words like "finished" and "stable", because they have such varied and strong meanings in this context.

So, here's what the Rakudo Perl project is going to do:

We will make an "official", intermediate, useful and usable release of Perl 6 (an appropriate subset) by Spring 2010.

Of course, we have to decide what will will be included (and excluded) in this intermediate "official release". At the Rakudo BOF on Monday we held a lively discussion about what the release could look like, what needed to be present, and how it could be packaged. During the hackathons and days following YAPC::EU we'll be drafting and publishing the more detailed blueprint for the release. But one of our guiding principles will be to "under-promise and over-deliver", to make it clear what can be done with the release, and to make it very clear which parts of Perl 6 are not yet supported in the release.

A short list of things we know will be in the release (that Rakudo doesn't already have): use of the STD.pm grammar for parsing, laziness, better support for modules, fewer bugs, better error messages, better speed. Again, our goal is to make something that is reasonable for people to start using, even if it doesn't meet all of the requirements for Perl 6.0.0.

We've also had discussions about what to call the intermediate release. We've considered tagging it as "Rakudo 1.0", but some of us think that the "1.0" name might tend towards "overpromise". We also considered things like "0.5" or "0.9", but these come with the message of "not ready for use", and that's not what the impression we want to make either.

So, yesterday morning I finally got around to thinking about it as "Rakudo 'whatever'". In Perl 6 the * term is used to signify "whatever", so that leads to a working name of "Rakudo *" (or "Rakudo Star").

So, the focus of the Rakudo project is to release "Rakudo Star" in Spring 2010 as a useful (but incomplete) implementation of Perl 6. More details about the features, milestones, and roadmap for this release will be forthcoming over the next few days.

Pm

P.S.: Several of our "down-under" community members have pointed out that "Spring 2010" can be a bit ambiguous. I'm using a season (instead of a month) to leave a month or two of wiggle room, but my intention is April 2010. As we get a bit more detail into the plan, we'll identify a specific month.

Monday August 03, 2009
11:53 PM

Lightning advertisements

YAPC::EU Day 1

During today's lightning talks, Geoffrey Avery added a new "lightning advertisements" feature to the session. Lightning advertisements are 30-second "promotional spots" that occur between each pair of talks; these help to fill the "dead air time" that typically occurs between each talk as the next lightning talk speaker works to get his or her presentation synced with the projector.

The purpose of a lightning advertisement is to simply create awareness or buzz about something happening at the conference. For example, I used my lightning advertisement to remind people of the Rakudo BOF taking place immediately after the lightning talk session. I'm certain that this directly increased the number of people who made it to the BOF, so I'm very pleased that Geoffrey added this feature to the session.

Others used the 30-second lightning advertisement slots to make announcements about other BOFs taking place and items being put for sale at the auction.

In order for things to run smoothly, it's important that there be a second microphone available for advertisers to use while the next lightning talk speaker is setting up. Also, the advertisers need to be at the front "ready to go" when one talk finishes and the next begins. With only 30 seconds available, there's not really time for slides or any visuals -- it's just the person making whatever announcement there is to be made.

In today's session several advertisers had difficulty finding the "on/off" switch on the microphone (it was really tiny); I suspect things can go a bit smoother if there's someone selected as the "microphone manager" to make sure it's correctly configured for each advertiser.

With eight lightning talk speakers, there are seven slots for lightning advertisements. When the session began there were only three "advertisers" at the front of the room, but as the lightning talks progressed (and people discovered how advertisements worked) the remaining four slots were quickly filled. I suspect it would be no problem for some slots to remain empty if there aren't sufficient advertisers, but I also suspect that as the idea catches on there will be no lack of advertisers. It also didn't seem to require much formal organization; the advertisers simply designated themselves by sitting at the front of the room, and we JIT-ted the ordering of advertisers as the session progressed.

I found the lightning advertisements to be an incredibly useful addition to the lightning talks session; kudos to Geoffrey Avery for adding them. I hope they become a regular feature of future lightning talk sessions.

[ironman perl]

Sunday July 26, 2009
11:47 PM

Rakudo Day: install, quoting, and string operators

Since I was busy at OSCON all this week, it was difficult to find a single day to dedicate to my Vienna.pm grant. But I did manage to get several major tasks done from my hotel room, so I'm going to bundle those together and count them as my "Rakudo day" for the week.

The biggest accomplishment was to finally get Rakudo so that it can build from an installed Parrot. Prior to Parrot 1.4.0 this has been exceedingly difficult, as an installed copy of Parrot did not provide all of the tools needed to compile dynamic PMCs and dynamic opcodes on our target platforms. But over the past few weeks Allison Randal and Will Coleda have gotten it to work for ParTcl, and now I've been able to adapt those techniques to work with Rakudo. The current state of building Rakudo from an installed Parrot is in the "ins2" branch of the github repository, if you wish to give it a try. (See below for instructions.) Note that some of the spectests will fail if you try "make spectest"; because the "ins2" branch is using an older version of Rakudo and some spectests have been added to the test suite since then. Since we're really just testing build/install, "make test" is sufficient here (and I'll clean up any spectest issues when I merge it back to the master branch).

Some may ask why we don't simply merge it back to master now; I haven't wanted to merge the ins2 branch back into trunk until we have verification that it builds properly on a variety of platforms. So far I've only had it fully tested on a couple of versions of Linux; I don't want to end up cutting out other operating systems from playing with Rakudo.

One of the downsides of building Rakudo from an installed Parrot is that we effectively lose the ability to easily build Rakudo from a build-tree copy of Parrot (like we do now). Part of the problem is that the filesystem layout of a build-tree copy of Parrot is very different from the filesystem layout of an installed Parrot. So at a minimum we would need a lot of code that says "if installed Parrot use this path, if Parrot build use this other path". This is true not only for file locations, but also for the tools used to build dynamically loaded PMCs and opcodes. Instead of trying to support both layouts, I'd prefer to just stick with using an installed Parrot for now.

(Note that it doesn't have to be a system-installed Parrot, the --gen-parrot option to Rakudo's Configure.pl will make a local install of Parrot and then build Rakudo from that.)

We're not abandoning the ability to build Rakudo from a build-copy of Parrot, we're just switching gears for a while. Based on conversations with other Parrot team members at YAPC::NA Pittsburgh and online, we've decided that of building (but not installing) Parrot should result in something that directly mimics the filesystem layout of an installed Parrot. When this is done, it will be easier for HLL languages and other tools built on Parrot to work from either a build tree or install tree version of Parrot.

In other areas, during Monday's OSCON tutorial sessions I sat in on Damian Conway's "Perl 6: Why? What? How?" tutorial. I wanted to see the tutorial itself, but I was also curious to know what problems would arise during the tutorial so that I could work on fixing them quickly. One of the problems that was quickly identified was that the << ... >> quoting operator wasn't handling comments properly. In other words, the following code

    my $a = <<
        do         # a deer
        re         # a drop of golden sun
        mi         # a name I call myself
    >>;

would act like a list of eighteen words instead of three. To be honest, I had overlooked that comments were allowed here, so that evening I quickly updated the parser to skip over comments as well as whitespace, and now the above works the way it is supposed to. Note that it even skips embedded and pod comments:

    my $a = << do #(a deer) re #(a drop of golden sun) >>;  # ("do", "re")

We had quite a few tickets dealing with places where operations end up returning Parrot String objects instead of Perl 6 Str objects. The easiest way to detect when this happens is to attempt to perform .trans on the string -- the .trans method for Parrot strings doesn't work the same as Rakudo's .trans method. So I converted a few settings functions (e.g., .uc, .flip, .lc, etc.) to explicitly call prefix:<~> on the result value; this guarantees that we end up with a Perl 6 Str object.

When we ultimately switch Rakudo to use HLL mapping of Parrot types, these explicit coercions won't be needed. However, at the moment using HLL mapping imposes a significant speed penalty on Rakudo (we're working on this), and given that things are on the slow side already I'd rather keep the speed and maintain workarounds for the time being.

I also fixed up stringification of several of the builtin types, especially Int, Num, and Junction. Previously printing a Junction object would produce a string like "Junction()<0x7fb898cb42b0>", which is almost certainly not what is wanted. So I updated Junction.Str to simply return its .perl representation.

Finally, Rakudo had been misparsing function names that began with a keyword followed by an apostrophe or hyphen. For example:

    sub do-something() { say 'hello'; }

    do-something();

Because do is a keyword, Rakudo would often end up parsing the above as do -something(), which of course wouldn't work properly. Similar issues existed with other keywords such as if, for, while, etc.

Having a longest-token matcher in the parser can avoid a lot of these misparses, but it's not always a complete solution. The STD grammar (as well as Rakudo's grammar) has a special <.nofun> lookahed subrule that can be used to verify that the keyword we just scanned is actually a keyword and not simply the lead-in to a function call. I went ahead and added a few <.nofun> calls to Rakudo's grammar, and now subroutine names that begin with keywords work like they're supposed to. (Thus making them a lot more fun. :-)

There were other fixes here and there throughout the week, and of course Moritz Lenz did the Rakudo #19 release on Thursday (which I describe in another post. I also worked with Jonathan on improving our internal object metamodel and introspection capabilities, and he and I worked out some ideas for refactoring our handling of lexicals. And all of this took place while I was attending OSCON, giving various presentations, and engaging in useful hallway discussions with other Perl folks. So it's been a busy and good week.

As always, my thanks go to Vienna.pm for sponsoring the work I did on the above tasks. Because of all of the travel and conferences I'm a bit behind on Rakudo days, so I will likely try to double-up on them for a few weeks until I'm caught up.

Pm

To test the ins2 branch:

    $ git clone git://github.com/rakudo/rakudo.git
    $ git branch ins2 origin/ins2
    $ git checkout ins2
    $ perl Configure.pl --gen-parrot
    $ make test

Saturday July 25, 2009
09:24 PM

Behind the scenes of the Chicago Release

This week I learned just how well Rakudo development is able to continue even when I'm preoccupied with things not directly related to making commits to the Rakudo repository.

As you probably know, I spent this past week in San Jose at OSCON, which was as usual was a fantastic conference. I was able to give some presentations about Rakudo and Parrot (recruiting!), meet lots of new people, visit and make plans with others working on Perl 6, and otherwise just have a good time.

Prior to coming to San Jose I had been planning that I would have some time in the evenings to work directly on Rakudo and prepare for its #19 release on Thursday (Jul 23). In past years at OSCON I've been able to get a fair amount of hacking done in the evenings... but that didn't happen this year.

First of all, I found myself having far more hallway discussions and other sorts of meetings with folks than I have in past OSCONs. Then, on Tuesday night I decided that I really wanted to do a significant refactor of the "Hacking Rakudo Perl" talk that I would be giving on Wednesday. I knew this would take away from my tuits for preparing the Thursday Rakudo release, but I felt the benefit of having a better talk (which I will also be giving at YAPC::EU in Lisbon) was worth the risk of having trouble getting the release out on Thursday.

Fortunately I never had to run that risk. A couple of weeks ago I posted an article about getting release managers for Rakudo. On Tuesday (while I was busy at OSCON), Moritz Lenz was able to use the release guide to produce a "practice release" as described in the article.

As a result, on early Wednesday morning I was able to see that Moritz had produced a practice release, and knowing that I wanted to focus on my presentation I immediately asked him if he would like to do the official release on Thursday. (I also asked Jonathan if he could work on eliminating a Null PMC error that was showing up in backtraces and didn't look good for the release, which he of course took care of very quickly.)

Moritz accepted the release task, and I am extremely pleased with how everything turned out. First of course is the release itself, which is equal or better to anything I would've produced. Moritz, along with help from others on IRC #perl6, assembled the ChangeLog, drafted the release announcement (adding a new section for deprecations), and as far as I can tell executed the release flawlessly. As it turns out, my role in the release consisted only of reviewing the release announcement on Wednesday night, adding a couple of statistics to the announcement, and telling Moritz that he could issue the release whenever he felt it was ready. I didn't have any further involvement after that -- the next I heard about it was someone coming up to me at OSCON on Thursday afternoon to congratulate me on the Chicago release.

This past week saw more than just the release; during the course of the week several people added new features and bugfixes to Rakudo (with Moritz and others applying contributed patches). Rakudo's passing spectest count increased by over 100 during the course of the week.

So, my thanks go out to Moritz, Jonathan, Carl, Kyle, Scott, and everyone else who worked on the release and Rakudo this week. You all did a fantastic job, and tremendously simplified my life while I was at OSCON.

Of equal importance, we've now demonstrated that others besides myself can do Rakudo releases (although I had little doubt of this). As a result, I'm now issuing an official "call for release managers" for the remaining Rakudo releases in 2009. Ideally I'd like to see that the release responsibility is regularly rotated across multiple managers (as Parrot does), so that no single person ends up being burdened with producing releases. I'm also planning that I personally won't do any more Rakudo releases myself in 2009, unless we're completely unable to find release managers.

If you're interested in becoming a Rakudo release manager -- take a look at the release guide, make sure you've filed a Contributor License Agreement, and we'll put together a schedule for the 2009 releases.

Pm

Tuesday July 14, 2009
12:12 AM

improving "Hacking Rakudo Perl"

In about a week or so I'll be at OSCON 2009, giving a presentation on Hacking Rakudo Perl. I'll also be giving the talk at YAPC::EU.

I've given versions of this talk twice before -- once at the Nordic Perl Workshop and again at YAPC|10 in Pittsburgh; I think they went well, but the talk is probably due for an update. In particular, at OSCON and YAPC::EU I'll have less time than in the previous talks, so I'd like the talks to more directly focus on the items that will be of most use to the audience.

So, what are your ideas about what information attendees would like to know for "Hacking Rakudo Perl"? What should I definitely include in the updated versions of the talk?

For those who are interested, the slides from my previous talks are available at http://pmichaud.com/2009/pres .

Thanks!

Pm

Tuesday July 07, 2009
08:19 PM

Seeking future Rakudo release managers

Over the next couple of weeks I'm working on fleshing out "job openings" and descriptions for people who want to help advance Rakudo Perl 6. I'll write more about this in a later post, and they're likely to appear in my OSCON and YAPC::EU talks.

One of the jobs we've already identified is "Release Managers", following closely on the Parrot model. Release managers are people that are responsible for executing Rakudo releases according to the release schedule. And like Parrot, we want this responsibility to be widely spread among a team of managers, and not always fall to the same person for every release. So, over the next month or so I'll be recruiting people to be release managers, starting with the August 2009 release. (The July 2009 release is likely to have a few significant changes -- most notably a "make install" target -- so I think it's better for me to do that one last release myself. Also I may want to coordinate the release with my talk.)

However, yesterday it occurred to me that thanks to github, it's entirely possible today for people to independently go through all of the steps of cutting and publishing a Rakudo release. So now I'm eager to have others try it and see what happens. At present the release steps are targeted to work primarily from Unix environments -- I haven't tested or attempted to cut a release from other operating systems. (I'll accept patches and suggestions for doing so, though! :-)

For those who want to try it, here are the basic steps for creating a Rakudo "practice release":

  • Fork the Rakudo master repository on github.
  • Follow the steps in the docs/release-guide.pod file, substituting your own github repo clone for the Rakudo master.
  • Let us know "Hey, I just made a Rakudo release!" and point to your forked copy github repo. Or tell us where you ran into problems, so we can improve the process.
  • Profit.

Some notes about creating "practice releases":

  • It's okay to skip steps 1-3 of the release guide for practice releases
  • You can also safely skip any steps that involve communicating with others, such as posting messages to the mailing lists or updating the wiki
  • If you don't want to wait the 30+ minutes for a spectest run to complete, "make test" is sufficient for a practice release.
  • Yes, it's even possible for you to test uploading the tarball to github -- just make sure to put it in your own repository and not in Rakudo's master. :-)

So, anyone want to give it a try? If you do, please let us know how it works out, and places where you think things can improve. I know that we'll be updating the release guide in response to feedback from others, as well as adding things like testing of "make install" and the like.

Thanks!

Pm

Tuesday June 30, 2009
11:08 PM

Rakudo day: operators in setting and lots of RT bugfixes

At the beginning of June the Vienna.pm organization generously committed to funding me for 1-day-per-week of Rakudo effort, but because of the Rakudo release, Parrot Virtual Machine Workshop, YAPC::NA, and a short vacation, today is the first day that I had available to really dedicate to the task. In fact, to catch things up a bit I plan to do another Rakudo day tomorrow or Thursday.

Here's what I accomplished for today's Vienna.pm-funded Rakudo day.

The biggest task I tackled for the Rakudo day was to be able to write operators in the setting (Perl 6) instead of PIR (RT #66826). In fact, I had actually done most of this last week during the YAPC::NA hackathon day, but interruptions then and a few annoying Parrot bugs kept me from marking the task as completely accomplished then. What this means is that we can now begin defining operators directly in Perl 6 code (perhaps with some inlined PIR), which moritz++ has already been exercising for infix:<...>, infix:<eqv>, and a few other operators. Over the next few weeks I expect we'll move even more operators out of PIR and into the setting.

The rest of today's Rakudo day was spent reviewing and cleaning up the RT queue; it had grown to over 400 tickets but by the end of the day Jonathan and I have shrunk it back down to 387. I think we collectively closed about 16 tickets today, and I responded with requests for clarification or updates on several more. Here are some of the highlights:

RT #66060 noted a problem that the .uc method would fail on some strings where .lc worked. I tracked this down to a Parrot issue in its handling of Unicode strings when ICU wasn't present, and refactored the code to be a bit more robust there.

RT #66640 noted that the minmax operator wasn't implemented, so after some discussion about what it should do I added it to the setting (using the operator features mentioned above).

In RT #66624, the exception message coming back from not finding a substring within a string was particularly misleading; I adjusted .substr to provide a more useful error message.

For RT #66928 .WHAT would not work on some subs like &infix:<+>; this was because some of the builtin operators are still using Parrot's MultiSub PMC instead of the Perl6MultiSub PMC, and those didn't have a mapping to the type object. Eventually all of the operators will become Perl6MultiSub; in the meantime I set Parrot MultiSub PMCs to map to the Multi() type objects in the same manner that other Parrot PMC classes are mapped to Perl 6 types.

RT #66818 noted a problem with unwanted flattening of %*VM in a for statement; this was because the contents of %*VM were incorrectly bound to the Hash directly instead of going through a non-flattening reference (Perl6Scalar). Eventually I expect %*VM to be initialized in the setting, though, which will provide a more robust and direct solution to this problem.

In RT #66840 it was discovered that precedence errors in the ternary operator would cause Rakudo to issue an error message and exit completely, instead of throwing a catchable exception. I tracked this down to PGE's OPTable handling of the ternary operator, it was actually using "exit" when the error occurred (probably because it came from before Parrot's exception model was firmly in place). This was changed to throw an exception instead; the actual exception message needs a bit of work but I expect that will come from the much larger PGE refactoring that will be done as part of the Hague grant.

Lastly, today I spent a good bit of time discussing Rakudo and Parrot build/install issues with Allison, and I think we have basic agreement on the changes we'll be making in order to get those working. Hopefully we can get all of that done in time for the July release.

So, that's my first Vienna.pm Rakudo day -- lots of little pestering bug fixes, and a key bit of infrastructure to fully enable writing the builtin operators in Perl 6. Later this week I plan to do a long-needed refactor of container handling in Rakudo, and maybe to get a more complete implementation of BEGIN blocks (which we massively cheat on at the moment).

Thanks again to Vienna.pm for sponsoring this work.

Pm