Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

JonathanWorthington (6049)

JonathanWorthington
  (email not shown publicly)
http://www.jnthn.net/
Monday October 19, 2009
06:34 PM

The new Rakudo signature binder has landed

Since I got back from my travels in Asia - actually, as soon as I'd got a good night's sleep after the long flight - I've been busily hacking away on my latest Hague Grant. In fact, I've done rather little else. After a couple of weeks of work - with many late nights debugging - I've finally reached the first major milestone of the grant: the new signature binder has landed.

So what is a signature binder anyway? It's the thing that takes the signature on a block (or some more interesting type of code object, such as a sub or a method) along with the arguments that it was invoked with and - if possible - binds them against the signature, such that the arguments passed end up in the correct variables. It sounds relatively simple, and in some senses it is. However, Perl 6 signatures offer a lot of powerful features, making things rather less trivial.

If you've been using Rakudo before now, you'll probably have been writing things with signatures and passing arguments along to them quite happily, and be wondering, "why a new binder?" It's a fair enough question, and the simple answer is because we'd reached the point where the approach Rakudo had taken so far had gone as far as we could sensibly take it. We've often used the defaults provided by Parrot in the past to allow us to make progress on developing a useful compiler; we did so with multiple dispatch, method dispatch, objects, roles and so forth. The approach has paid off: people are writing cool stuff with Rakudo today (I played a board game online with masak++ earlier today; the game itself was implemented in Perl 6, used the Web.pm framework and was being served by a web server written in Perl 6 too). In the meantime, we've been replacing those Parrot defaults with things that get the more subtle and/or powerful parts of the Perl 6 specification correct, and usually that do so in a more efficient manner.

The main things that prompted development of a custom binder were:

  • While Parrot could bind the arguments to registers, we then had to follow that up with code to put those into lexical variables. We then had to make a second pass over the arguments to do type checks and enforce context. This was inefficient - we had to do a lot of extra lookups as well as two passes over all of the arguments. It was really also quite bad if we were just binding the signature because we had a multi-dispatch candidate that needed a bindability check: we'd have already got a lot of the work done before we could figure out that actually, the first argument had a constraint on it that immediately ruled it out.
  • Perl 6 allows binding of named parameters to positional variables - that is, even positional variables can be passed as named parameters too. Parrot didn't support this, and was going to be really quite hard to layer onto the existing model we had. Doing it efficiently was fairly out of the question. Getting Parrot to support it was also an option, but bulking up the VM for a feature only one language really wants is not really optimal.
  • If multi-dispatch had already decided that a candidate's types matched those of the arguments being passed, there wasn't a good way to avoid the argument type checks. At the same time, we had to be sure to enforce them if a multi candidate was somehow invoked without going through the multi-dispatcher.
  • If multi-dispatch depended on a bindability check, we'd like to just keep the candidate around, and not re-bind everything. There wasn't an easy way to do that either.
  • Nor was there a particularly easy way to implement nested signatures. These allow powerful functional-style matching as well as deep data structure unwrapping.
  • Signature binding in Perl 6 isn't just done when calling a routine. For example, you can use a signature to bind and unpack return arguments too. These return unpack signatures can be just as complex as those used when calling a routine. Heck, you can even have a signature literal and just smart-match against it to attempt binding. We needed something that was going to support these use cases too, which argued against having Parrot doing half the work and us patching it up later.

The new signature binder I have merged into master this evening either supports or is designed with later supporting all of these needs in mind. It handles binding named arguments to positionals right out of the box, for example. So now if you declare a sub:

sub plot($x, $y) { }

Then you can call it as any of:

plot(4,2);
plot(x => 4, y => 2);
plot(4, y => 2);
plot(2, x => 4);
plot(y => 2, x => 4);

Under the new signature binder, it was also trivial to add the code to avoid re-doing the type checks for a multi-dispatch candidate. While I didn't yet add the "don't re-bind when we did already passed a bindability check" improvement, that's mostly because it requires a slightly less trivial refactor to the multi-dispatcher - there's some stuff stubbed in to support it when I get that done though. And also there's a stub for nested signatures, which will be coming along later. Those are going to be way cool.

A couple of other things came up during the refactor that also led to some on-the-side bonus improvements. The most immediately useful one is that lexical variables declared outside of packages are now visible inside them - a source of much past frustration.

my $dips = 0;

class Chip {
    method dip() {
        $dips++;
    }
}

my @chips = Chip.new, Chip.new, Chip.new;
@chips>>.dip for 1..10;

say $dips; # 30

I also fixed the bug with junction auto-threading and for loops, so if the signature of your for loop declares a non-junctional type then you'll get the body run multiple times, once for each thingy in the junction. Also, signatures can now be written as literals, without getting weird errors.

Finally, I mentioned that one goal was getting better performance. I've not yet started to optimize the new binder - yes, it has some built-in by design performance improvements, but I didn't really dig in to the real effort of making it faster yet. Even so, the gains are notable. The figures below are the percentage of time it now takes to run some micro-benchmarks on specific language constructs compared to before I started working on the signature and binding improvements. So for example, 10,000 method dispatches now take about 26% of the time they used to.

Startup: 76%
10,000 sub dispatches: 45%
10,000 multi dispatches: 35%
10,000 method dispatches: 26%
10,000 multi-method dispatches: 20%
10,000 operator dispatches: 33%
postfix:<++> 10,000 times: 57%

These figures sure don't mean that Rakudo is fast yet - we've a long way to go until I'd say that - but I think for the "we didn't start optimizing the new binder yet" phase, these are an encouraging start.

More as the grant progresses. In the meantime, enjoy the improvements. :-)

Wednesday October 07, 2009
04:33 AM

Backpacking done, now back to Rakudo

I'm just back from my yearly autumn backpacking trip, and once again I took in a few Perl things amongst my vacation. I attended YAPC::Asia for the first time, and gave a couple of talks; you may be interested to see the slides. I think they were both quite well received, though one of them had a couple of bugs. I muchly enjoyed staying with the Pauleys - thanks for the hospitality...and introducing me to tonkatsu!

Other than Japan, I also visited South Korea. I gave a talk at Seoul.pm, and many of the Perl mongers there were kind enough to take time out to show me around and have dinner and beers with me, which was really great. As well as great Korean hospitality, I can't not mention that Korean food is SO good - I'm missing it lots already.

So, it was a great time (thanks to everyone who made it so - too many to mention here), and now I'm back, re-energized and ready to dig back into hacking on Rakudo. In fact, this is my last real vacation before Rakudo *, so I made the most of it. The next few months are going to be a serious amount of work - but never fear, I'm hacking on it already. :-)

My signatures Hague Grant got approved. I've started on that already, and I'll try and blog some progress in the next few days (for now I've done nothing more exciting than some refactors in preparation for the changes that will really make a difference...but with my current uncommitted bits they seem to shave roughly 10% off Rakudo startup time anyway, and that's before I begin the changes I expect to make more of a difference.) Also, during a Rakudo day before my trip and some hacking at YAPC::Asia, Blizkost made some more progress. I'll blog about that soon too. :-)

OK, back to hunting down bugs in my latest changes...

Tuesday August 25, 2009
06:46 PM

Starting on Perl 5 interop for Rakudo

I used the first chunk of my Rakudo day today to kick off a new project. At YAPC::EU, we had the Rakudo BOF where we collected together suggestions of features that mattered to people for Rakudo * - the major, usable Rakudo release we're planning for Q2 2010. One that came up was interoperability with Perl 5. This is something that I think had generally been assumed as something far off for Rakudo, or at least not really discussed in any great deal. The legacy is hardly great: the Ponie project, which tried to re-target Perl 5 to the Parrot Virtual Machine, was put out to pasture after plenty of effort for a variety of reasons, including Parrot being too much of a moving target at that point.

As far as I can see, re-targeting Perl 5 would be really quite a lot of work, and require a lot of guts knowledge. A lot of work in a specialized area presents a resourcing problem, and while I would happily cheer on a renewed effort to try and port Perl 5 to run on Parrot, I'm not at all expectant that anyone is going to jump in to such a task soon. Even if that happens, the chances of having something ready for Rakudo * seem to me fairly slim.

So, enter Blizkost. The Blizkost project is aiming to embed the Perl 5 interpreter and then build various bridges between it and Parrot. We'll start with the easy things - today I've done the easiest thing - and then build out from there. Rather than this being something built into Rakudo, instead Blizkost operates at the level of any other Parrot language implementation, meaning that it supports the HLLCompiler interface. This means that any other Parrot language that knows how to use this interface to eval code in another language, for example, will be able to make use of this.

The simplest thing I figured we could do is make it possible to eval a string of Perl 5 code. You can't get the value it returns, nor will it find any outer lexicals, but this means that you can now do (copy paste from the REPL):

> eval('print "Hello from Perl 5\n"', :lang<perl5>)
Hello from Perl 5

Rakudo itself needed no changes to deal with this (other than a minor fix to deal with a language inter-op regression that affected eval in any language). All I had to do was to configure, make and make install Blizkost, using the same installed Parrot as Rakudo is built against, and it Just Worked.

So, we have a starting point. It's a small one, but then I only spent several hours on this so far. :-) I really have little knowledge of Perl 5 guts, so my dream situation is that now I've seeded this, people will come, look in horror at what I've done and find themselves helplessly patching it. It's on GitHub so you can go fork it if you want to hack, and I'm also going to be giving out commit bits quite freely, so get in touch if you feel like joining in.

I spent remaining bit of my Rakudo day dealing with some bugs.

  • Fixed code generation bug when you tried to bind an attribute. Added a spectest.
  • Stopped a sprintf that is given too few arguments from exploding with a Null PMC Exception. Now it just returns a Failure. Another couple of spectests started passing.
  • Finally figured out how to successfully catch attempts to get hold of attributes on a type object. You can't, and you got a Null PMC Access exception if you tried (people hit this one a decent bit). However, my previous attempt to give a nicer error failed. Today I realized why it had failed, made a cup of tea, then implemented something that did work - it now tells you what you did and also the name of the attribute you tried to get hold of in order to aid debugging. I found and corrected a bogus spectest along the way, enabled the one that checks this doesn't die with a Null PMC Access, and managed to close two tickets from this. However, I have a suspicion there's at least another one or two that I didn't spot that are in the queue and boiled down to this same problem. Anyway, that's another of our major sources of Null PMC Access errors plugged.

I also spotted two other RT tickets that represented issues that I knew had been solved recently and closed them. I noted that a third was fixed and could be closed if somebody wrote tests to cover it and make sure it doesn't come back. Happily, we have some awesome test writers who take care of such things. :-)

Thanks to Vienna.pm for funding this Rakudo day.

Friday August 21, 2009
01:31 PM

Playing with traits

Fly on the wings of despair
No one is holding you back
The call of the wild is internal
Conquer the silence you fear
Tomorrow will not fade to black
Remember
No one can save you today
Wings Of Despair - Kamelot

Welcome to the final post in my Hague Grant updates. Yup, I've made it through, and will soon be submitting the final report. In the meantime, let me talk a bit about the remaining big topic in my grant that I have not yet discussed: traits.

Traits are the Perl 6 way of hooking in to various declarations. The mechanism is based upon multiple dispatch. That is, it's possible to write a trait handler by writing a multi-dispatch sub that declares what sort of declaration it applies to and how it is identified. The types of declarations that can be hooked into in this way are:

  • Classes and roles
  • Routines
  • Variables
  • Attributes

Generally, a trait handler will be called with some object representing the thing being declared, something representing the trait and, optionally, any arguments that were supplied.

A few days back, somebody (sorry, forget who) on #perl6 was asking about being able to introduce additional accessor methods for an attribute. If I can't remember their handle, I sure ain't going to remember their use case, so let me make one up. In Perl 6, we can write:

class ValueWithError {
    has $.min_value;
    has $.value;
    has $.max_value;
}

And we get a class that lets us represent a value as best we have it with minimum and maximum values that it could be, which we presumably compute by knowing the errors in some calculations. Thing is, everyone using our class keeps trying to call ".min" and ".minimum" instead of ".min_value", so we decide that we may as well support those too. Well, we could write some accessor methods ourselves, but instead we'll write a small bit of code that introduces a new trait called "aliased". After we've implemented this, we'll then be able to write:

class ValueWithError {
    has $.min_value is aliased<min minimum>;
    has $.value;
    has $.max_value is aliased<max maximum>;
}

So, let's get started. First, we need to declare a trait handler.

multi trait_mod:<is>(AttributeDeclarand $a, $names, :$aliased!) {

}

The first argument to a trait handler is always the declarand - that is, something representing the thing being declared. For attributes that is an instance of AttributeDeclarand, which gives us a reference to the container but also some extra bits of information, such as the attribute's name and the metaclass of the class it is defined in. Beyond this, we either need a required named argument matching the name of the trait, which we put as a named parameter at the end of the signature, or we need the second positional parameter to be the name of a type. When dispatching a trait, the compiler will see if it's a known type name, and dispatch with the type object for that type if so and a named parameter if not. Finally, we can optionally add another positional parameter if we want our trait to be able to accept arguments (here, that will contain the list of names we will alias to).

Now we'll start to fill out this stub a little bit.

multi trait_mod:<is>(AttributeDeclarand $a, $names, :$aliased!) {
    my $accessor_name = $a.name.substr(2);
    for $names.list -> $name {
        say "aliasing $name to $accessor_name";
    }
}

AttributeDeclarand has a .name property, which will hold the name of the attribute. Remember that all attributes have a real name of $!foo, even if declared $.foo to get an accessor generated. We take this and strip off the $! at the start, to get the name of the generated accessor method that we will forward to. Then we will take $names (which were the set of arguments that were passed to the trait), put it into list context and iterate over it. For now, I just added a say in there, so if we actually try creating the class I showed earlier, we'll get the following output:

aliasing min to min_value
aliasing minimum to min_value
aliasing max to max_value
aliasing maximum to max_value

So, what is the magic that we need to do in order to finish this? Well, we need to use the metaclass - accessible through the AttributeDeclarand object - and with it add a method. And what do we add? Well, we'll use the anonymous method syntax, the indirect method call syntax and closure semantics to build us a forwarder method, and then in the loop call add_method on the metaclass to add that method under each of the aliased names. Our final code looks like this - I also include the class and some example code.

multi trait_mod:<is>(AttributeDeclarand $a, $names, :$aliased!) {
    my $accessor_name = $a.name.substr(2);
    my $meth = method { self."$accessor_name" };
    for $names.list -> $name {
        $a.how.add_method($a.how, $name, $meth);
    }
}

class ValueWithError {
    has $.min_value is aliased<min minimum>;
    has $.value;
    has $.max_value is aliased<max maximum>;
}

my $v = ValueWithError.new(value => 42, min_value => 41.5, max_value => 42.8);
say $v.min_value;
say $v.min;
say $v.minimum;
say $v.max_value;
say $v.max;
say $v.maximum;

And here is the output:

41.5
41.5
41.5
42.8
42.8
42.8

And that's it, we wrote our first trait handler - and it was actually useful! And with that, I bring this series of blog posts on my current Hague Grant to an end. I hope they've been interesting. Worried you're going to miss them? Well, fear not - in a couple of days I'll be applying for my next Hague Grant. In the meantime, have fun playing with and finding creative ways of breaking the new features. :-)

Thursday August 20, 2009
08:38 PM

Ramblings on Rakudo startup

I try and blog about each Rakudo day on the day it happens or shortly thereafter. Occasionally I fail. This post is a result of such a failure. Worse, I was silly enough to leave myself the following notes:

  • Fixed a couple of minor RT tickets.
  • Worked on branch to investigate eliminating reblessing into Perl6MultiSub

Which are, uh, really great for writing a blog post from a couple of months later. Anyway, since exactly what RT tickets I fixed on that day are probably not of so much interest now, plus I'd have to go and figure out what the date was and what I committed and hope the commit messages had the RT ticket IDs in, I'll instead spend a while talking about what the branch was all about. It actually didn't end up getting merged, but some of the things discovered in it have been applied already, and one of the things I explored in it is going to happen in the future anyway.

Folks playing with Rakudo will probably have noticed that our startup time is, well, not exactly stellar. Part of the blame lay with Parrot for a while, but the Parrot team have done some really great work on improving things in that aspect, and a lot of it now is that Rakudo just does a lot of work at startup. Basically, at startup, for every block (including routines) we do some setup work. One of those bits of setup work is transforming the Parrot MultiSub data structure into a Perl6MultiSub, which implements the Perl 6 multiple dispatch semantics. What would be far better would be to create that in the first place. That needed a Parrot tweak first, which I did.

The thing is, Perl6MultiSub depends on the subs having a Perl 6 Signature object attached to them. A Signature object in Perl 6 stores details of the parameters that a block or routine takes, along with any type constraints, whether the parameter is optional or not and so forth. (Right now, some of that is duplicated in Parrot's own signature handling, but Rakudo will be moving away from that and binding entirely based upon its own Signature objects in the next couple of months.) And thus arose an issue: we were using multiple dispatch in various places in Rakudo, but specifying things using some Parrot syntax unrelated to the Perl 6 Signature objects. Some of them were guts, but others were things that really needed to end up with Perl 6 signatures as they'd be user facing.

Basically, the thing that really killed the branch was that there were just too many places that we'd have to tweak, and it wasn't worthwhile doing that yet. These days, we are writing more and more bits of Rakudo in Perl 6. We build a first stage compiler, then run it to compile the built-ins. All of those get a Perl 6 Signature object generated for them. Writing a signature object by hand in PIR, on the other hand, is a lot of effort. I did work out a way to generate them for a lot of cases, and that led to ideas about caching and possible sharing of signatures, which I expect we will implement in the future. But overall, it was too much pain, considering that if we waited to do the change for some months, it'd be a bunch less painful. It has been worth keeping in mind during development since then, though, and I am trying keep us on a path that will lead to this conversion eventually.

The other big cost at startup is the creation of the signature objects. This is something that I expect will become much cheaper in the next couple of months, as I review and refactor the way we build them. I'm also considering if - or how far - we can try and do more at compile time and less at runtime in this area.

Anyway, that was a quick little look through some Rakudo guts. Hope it wasn't excessively uninteresting. And thanks to Vienna.pm for funding this Rakudo day a couple of months ago. :-)

05:14 PM

Deferral

The ones who seek justice,
Will pray for it all their lives.
They can and they will skin us all one day,
Oh can you hear them cries.
As far as the man can run from us,
We're following the trail of blood,
So hunt my young ones,
The pack they have always feared is back.
Ain't Your Fairytale - Sonata Arctica

In this Ian Hague Grant update post, I'm going to discuss deferral. Deferral allows a method to pass control to the next best candidate that method dispatch would have called had the current method not existed. Control can either be passed on entirely - that is, we stop executing the current method and skip to the next one - or you can capture the results of the next method in the chain and do some post-processing on them. Additionally, you can massage the arguments before deferring, or alternatively just pass on the arguments as they were originally provided.

So what do I mean by next best candidate? Let's look at a simple example. Suppose I have a class that knows how to format and output some legal report.

class ReportRenderer {
    method render(Report $r) {
        ...
    }
}

This works fine for internal things, however when such reports are published on the web we need to redact certain things. Thus we can write a subclass with a report method that does this redaction by modifying the data structure, and then defers to the parent's render method to actually do the display work.

class PublicReportRenderer is ReportRenderer {
    method render(Report $r) {
        # Code to redact the report...
        nextwith($r);
    }
}

At this point you may be thinking, "hey, is this not just like a call to the superclass?" And in a way, yes, it is. However, a couple of things make deferral interesting. First of all, "who is my parent" gets more interesting in a multiple inheritance world; deferral instead walks the full method resolution order as seen from the object you initially dispatched on. Second, there's more than one way to defer - in fact, there are four:

  • nextwith($arg1, $arg2, ...) ends execution of the current method and passes control to the next method, using the supplied arguments.
  • nextsame() ends execution of the current method and passes control to the next method, using the arguments the current method was originally invoked with.
  • callwith($arg1, $arg2, ...) calls the next method, using the supplied arguments. It returns with the return value(s) of the method that was deferred to.
  • callsame() calls the next method, using the arguments the current method was originally invoked with. It returns with the return value(s) of the method that was deferred to.

Deferral isn't just up the inheritance hierarchy, however. It's also possible to defer to multiple dispatch variants that are less specific. For example, suppose that we have a web crawler, and one class in the system is responsible for persisting the interesting pages that we found while crawling. We get a report that long pages take too much processing later, and we are asked to store a summary. Imagining our class looks along the lines of...

class CrawlerPersistence {
    method save(Str $page) {
        # various code to break up the page and store it
    }
}

We could do a few things. One is that we could put the check and the logic inside of the save method to make the summary, but this may make the method longer than is really desirable. We could write a worker sub, do a length check in the save method and call that, which is better. However, in Perl 6 we have a third option: make the methods multis, and write an additional multi that handles the summarization process and defers.

class CrawlerPersistence {
    multi method save(Str $page) {
        # various code to break up the page and store it
    }

    multi method save(Str $page where { $page.chars > 10000 }) {
        # summarize
        nextwith($summary);
    }
}

Note that a multi with a constraint is considered to be narrower than one with the same type and no constraint, so we always will call the second of these if the constraint matches.

In general, deferral gives you a nice way of saying "call or pass on control to the next best thing". You can use it when you need to customize behavior, or in cases where you get data that you don't know what to do with but know a more general method accepting more general types or further up the inheritance hierarchy may be able to handle better.

Under the hood, every method dispatch saves enough information to be able to resume the dispatch and find the next candidate. It does not compute all of the possibilities in advance, since a lot of the time when we do a method invocation we won't defer. Thus we compute the candidates lazily.

So, that's deferral. Check in soon for my final Hague Grant update post for this grant, in which I'll talk about traits.

Wednesday August 19, 2009
06:31 PM

Rakudo pre-release bug hunting

Between vacations and conferences, I've not done a Vienna.pm Rakudo Day for a little while. Anyway, today I took the opportunity to spend the day doing Rakudo improvements before tomorrow's monthly release.

As more and more people explore Rakudo, we get more and more issues and corner cases reported. At the start of today we had 470 tickets either new or open in the Rakudo RT queue. I hadn't reviewed the ticket queue in a while; back when it was more like 150 tickets I did it every week, but it's a bit too long a job for doing it weekly now. Doing so today was very worthwhile, however: I was able to close six tickets that related to missing features that had already been added or bugs that had already been fixed, and masak++ was able to confirm that another two that he had filed were now dealt with and close them as well. 8 tickets down before I'd even got through my first cup of coffee of the day. Not bad. moritz++ also made inquiries to the perl6-language list over a ticket I brought up for discussion on the channel.

"Null PMC Access" is in many ways the Parrot equivalent of a segfault (though Parrot does actually segfault now and then too, so we get the best of both worlds... ;-)). Whenever one happens in Rakudo, it's Just Plain Wrong, and means we overlooked a check somewhere. Today I identified a group of tickets that all gave this error for very related reasons, patched them and unfudged various tests. That was another four tickets sorted out.

Another easy win was doing the last push on a ticket related to Failure object handling, which others had got mostly there. I did that and closed the ticket. Then I fixed a bug that has been causing some irritation for a while:

sub foo returns Int { fail("oh noes!") }

This is meant to work fine, since failure is always acceptable for any non-native type. However, in reality it did not, since the type check wasn't allowing the Failure to get out. It's fixed now. And as a bonus, I could now use fail on one of the typed routines in the setting to fix another bug: Rakudo was dying if you called ord('') with an empty string, when it fact it should only have been failing.

I spent a chunk of the day dealing with some multiple dispatch tweaks. The first was a trivial bug fix: we were missing something out in the PAST-generation for "multi foo() { ... }" but putting it in for "multi sub foo() { ... }". The second two tickets were trickier, and related to how we dealt with slurpy parameters and optional parameters when doing the narrowness analysis to sort candidates. First, the slurpy one.

multi foo (*@a){ 1 }
multi foo () { 2 }
say foo();

This was being treated as ambiguous before. To deal with this, we've implemented the rule that if two candidates are tied, they may be disambiguated if one has a slurpy parameter and the other does not. The one without the slurpy is considered narrower. The thinking behind this is that specifying exactly how many arguments you can take is more specific than saying you can take just any number. The upshot of this is that the above program now prints 2 instead of dying. The optional parameter one was a different issue.

multi foo (@a) { 1 }
multi foo ($a, %h?) { 2 }
say foo(<1 2 3>);

The narrowness analysis was looking at these two and saying "well, they have different numbers of items in the parameter list, so I can't compare them, so they're tied". I relaxed this rule a bit to allow comparison of these based on the non-optional parameters they have, and now since @a is narrower than $a (just out of the normal dispatch rules we already had in place), this example now outputs 1 instead of dying.

Finally, recently I've been helping __ash__, a new contributor to Rakudo, work on a patch to get us able to do method dispatches from the point of view of a different class. He showed up and said he was hacking on the patch, which was exciting because it's rather non-trivial as a way to get into Rakudo development and he was looking in the right places, but also challenging as I knew fairly well how to do it after some discussion with others on #perl6 about what the spec called for, but didn't want to end up just writing the patch for him, otherwise it'd be a crappy learning experience. Anyway, today __ash__++ made it, with a little more input, to a working patch, which I was very happy to review and apply. So now you can do this (the bottom line is the new thing):

class A { method foo { say 42 } }
class B is A { method foo { say 69 } }
my $x = B.new;
$x.foo; # 69
$x.A::foo; # 42

And that, along with the usual assorted discussions on #perl6 plus some updates to the ChangeLog ahead of the release, was what I got up to today. Thanks to Vienna.pm for funding it.

Thursday August 13, 2009
07:28 AM

A Little Reflection

I still remember the sunlight on your face that warm November day,
And I still remember my heartbeat quickened by desire, unaware of prices I would pay
I still remember the closing door the night we said goodbye,
And I still remember losing you for good and knowing that a part of me had died
Memory - Redemption

Sorry it's been a while since I last posted - YAPC::Europe and preparing for it were something of a distraction, as has been visiting some family and friends, though the real killer has been that in the last month I've managed to have myself a couple of colds/flu-ish things that left me without a great deal of energy during a lot of the time I've been here at home and working. Anyway, I think I've almost shaken off the current infection, and amongst it all I've been able to keep my Hague Grant moving along - in so far as the code, if not the blogging.

One of the big pieces of my grant that I've been working on is improving Rakudo's support for introspecting classes and roles. Before I started working on this, Rakudo didn't really have any such support. Of course, Parrot provided all of the primitives. But there was some work to be done in building up an implementation of the Perl 6 introspection interface around it.

To introspect an object, first you need to get hold of its metaclass. This class actually provides the various introspection-related methods. The metaclass can be obtained by using the .HOW macro (actually in Rakudo, implemented as a method, but we'll fix that in the future). All metaclass methods take as a first parameter the thing that you want to introspect. So for example, if I want to find the parents of the built-in List class, I would do (showing this using the REPL):

> say List.HOW.parents(List).perl
[Any, Object]

Where .perl gets a Perlish representation of a data structure (think Data::Dumper built into the language). At this point you may be thinking, gee, this sucks, I have to say List twice. Which is why there's a .^ operator, that evaluates what is on its left hand side once, calls .HOW on it, then calls the method you specify on the metaclass, fudging what was evaluated on the LHS in as the first argument to the call. Thus instead of the above you can just do:

> say List.^parents.perl
[Any, Object]

Which is much better. .^parents takes a few options too.

> class A { }
> class B { }
> class C is A is B { }
> say C.^parents.perl; # all, flat list
[A, B, Any, Object]
> say C.^parents(:local).perl; # immediate only
[A, B]
> say C.^parents(:tree).perl; # all, tree
[[A, [Any, [Object]]], [B, [Any, [Object]]]]

Of course, it's not just parents that can be introspected, but also methods and attributes - both on your own classes and built-in ones. For example, here we look at the methods defined in the Num class. Note the use of the parallel dispatch operator - implemented earlier on in this grant - to get the name property of each of the methods returned in the list from the methods method; we then join the list.

> say Num.^methods(:local)>>.name.join(", ");
tan, cos, sin, pred, acosec, sinh, asinh, cosech, acosech,
acotan, Str, asec, cotanh, acotanh, sech, ACCEPTS, asech, atan,
acos, tanh, asin, atanh, cosh, cosec, acosh, succ, WHICH, perl,
cotan, atan2, Scalar, sec

Here is a very similar example for introspecting attributes.

> say Pair.^attributes(:local)>>.name.join(", ");
$!key, $!value

While I have been using these with :local, they also support being called parameterless to get all attributes up the hierarchy, as well as :tree to build something similar to what you saw earlier for .^parents.

At this point you may have realized that all of this means that you can use a module at the REPL and then explore its classes using the introspection interface.

Anyway, for my final trick, here's some code that walks a class hierarchy and prints out an ASCII tree-ish thing depicting the class hierarchy. To stop it going and printing all the stuff in Any and Object - classes inherit from Any by default - I've put in a little check for that. It makes the output shorter and probably more useful.

multi describe($c, $prefix) {
    take "$prefix " ~ $c.perl ~ "\n";
    for $c.^attributes(:local) {
        take "$prefix + Attribute: " ~ .name ~ "\n";
    }
    for $c.^methods(:local) {
        take "$prefix + Method: " ~ .name ~ "\n";
    }
}

multi describe(@list, $prefix) {
    for @list -> $item {
        unless $item eq Any|Object {
            describe($item, $prefix ~ "    ");
        }
    }
}

multi describe($start) {
    return [~] gather {
        describe($start, "");
        describe($start.^parents(:tree)[0], "");
    }
}

This code demonstrates not only introspection, but also multiple dispatch by sigil, gather/take, the reduce meta-operator (which combined with gather/take lets us build up a string from a bunch of recursive routines without having to worry about building up and passing along a return value all the way up), junctions and a bunch of more basic features. Here's an example program using this.

class A {
    has $.a;
}
class B is A {
    has $!b;
    method foo() { }
}
class C is B {
    has $.c;
    method bar() { }
}
say describe(C);

And the output:

C
+ Attribute: $!c
+ Method: bar
+ Method: c
     B
     + Attribute: $!b
     + Method: foo
         A
         + Attribute: $!a
         + Method: a

Notice how it shows the auto-generated accessor methods for the non-private attributes too.

So, that's introspection. Have fun, send bug reports. :-)

Monday July 06, 2009
05:55 PM

The Great Method Dispatch Refactor

What don't kill you will make you more strong
Broken, Beat & Scarred - Metallica

The largest, trickiest part of my current Hague Grant has been an extensive refactor of method dispatch. It's the second major refactor of method dispatch that I have done in the course of Rakudo's development, and while I know there's going to be some tweaks from here to address some more subtle aspects of the semantics, I'm very hopeful that this will have been the last major refactor needed on the path to a Rakudo 1.0.

The last time I refactored method dispatch, it was to allow us to get a bunch of new features implemented, and to allow fast prototyping of many of the things we'd need. In that sense, it was a very successful refactor, and the ease with which I've been able to get all kinds of things in place - role punning, the handles stuff and junction dispatch from the last grant, correct multi-method dispatch semantics and more - has been very helpful for Rakudo's development. However, as we've come further along, a few things made it clear that this approach wasn't going to sustain us through to 1.0.

Firstly, it just didn't perform well enough. Method dispatch was running notably slower than sub dispatch. However, the real killer was multi-method dispatch. The previous refactor had opened the door to getting the semantics correct, but while multi sub dispatch was running nice and fast, there just wasn't a nice way to get the level of interplay between the method and multi dispatchers that I felt we really needed to have a shot at acceptable performance.

Secondly, my original thoughts on how we might implement deference - a more exception based model - turned out not to be the way we really needed to go. I will write in more detail about this in a future post with some examples, but in a nutshell deference is the idea that a method can defer to the next one in a superclass. However, if it's a multi then we defer to the next best multi candidate if there is one instead. Deference can keep the original arguments or use a new set, and can either fully defer (like a tailcall) or do something more akin to a call and get the results back to do further processing. Larry also felt much more strongly than I had expected that .wrap functionality on routines should operate through the same mechanism (I agreed that we really should unify the mechanism after pondering it for a while). Again, this issue involves the interplay between multi dispatch and method dispatch - there wasn't just a performance issue to worry about, but also a big semantic one.

Thirdly, the previous dispatch refactor had hurt our language interoperability a bit. Up until recently, the promised land of high level language interoperability in Parrot had been more promise than reality, but that picture has changed of late, with Tene++ posting concrete examples of calling between Cardinal (an early Ruby on Parrot compiler) and Rakudo. There's really not much point implementing Perl 6 on Parrot if we're not going to be able to interoperate decently with other high level languages on Parrot.

The performance and interop problems were partly the same issue: the previous dispatcher wasn't really built on top of the Parrot model for doing such things (which is, subclass Parrot's Object PMC override find_method). So one of the principals for the refactor was to build Perl 6 method dispatch semantics on the Parrot find_method/invoke model.

Figuring out how to build a method dispatcher as complex as the one needed for Perl 6, which would fit neatly into the Parrot model, have a good chance of working well when invoked from other high level languages, solve the deference problem, get the interplay with multiple dispatch right and on top of that lot actually have a chance of performing well, was non-trivial. I got us a bit of the way in small steps, but eventually I hit the point where it was time to do the big switch. It goes without saying that I was especially glad of the test suite at this point. The fact that I was able to do a complete switch in the way how something so fundamental worked and hear of almost no regressions to people's real world Perl 6 code when I committed it is a testament to the hard work that many have put into Perl 6 testing. So, a big thank you goes out to all involved!

So how does method dispatch look now in Rakudo? First, let me note the steps in invoking a method in Parrot. First, the find_method vtable method is called on the PMC representing the object. This hands back another PMC, which we call the invoke vtable method on to invoke it. Normally in Parrot, when you do a find_method, Parrot goes off and finds the PMC representing the chunk of code we need to execute (a Sub object or some subclass of that) and returns it. In my overridden find_method, we instead hand back a P6invocation PMC. This contains the method we will dispatch to. However, it also contains something else: the information needed to continue looking for more things to dispatch to in the event we later are asked to defer. That is, it doesn't find all of the things we might have to dispatch to. It just makes sure we have the information to do so at some future point. This lazy approach means that ordinary dispatch-and-we're-done calls aren't paying the cost of deference. However, in the event we need to find more methods, it can act like an iterator and provide them. This, conveniently, is exactly what we need to implement .can, which in Perl 6 returns a lazy iterator of all of the possible methods rather than just a boolean value (though I didn't implement this just yet - I'll get to it). One final thing: I think I can keep the initial P6invocation immutable, so we'd be able to cache it in a method cache at some future point and have even less work to do.

So anyway, that's what our find_method does. P6invocation's invoke, then, just knows the first result and goes and invokes it. But it has to do one little thing more than that: it needs to store itself in a place accessible to things like callsame/nextsame and friends which defer. It turns out that fudging it into the lexpad during the invoke is a very neat way to do this. For one because it means if people start doing evil things like returning a closure from within a method that defers, we'll probably do something sensible. It also means its lifetime is neatly managed for us, since it's tied to the "activation record".

So how far along is all of this? Well, the refactor is done, pushed and if you're using the last release of Rakudo or the latest git head then you're already using it now. I'm not yet done with building all of the bits of deference on top of it yet, so don't expect all of that to work at the moment. It's coming soon (I'm happy that the dispatcher fundamentals are correct, the issues are actually now in callsame and other such routines). Furthermore, I already refactored method wrapping to work in terms of candidate lists and P6invocation too. So that bit of my grant can be ticked off. (This refactor also meant that we started passing some more wrap tests too. In fact, until somebody - I forget who - recently discovered an odd bug in interaction between wrappers and lexical scopes, I really thought wrap was, uh, all wrapped up. Well, such is software development...

So, that's another installment in my Hauge Grant progress blog posts, and another significant step forward for Rakudo. Next time, I'll probably look at traits or deference. And yes, there'll be more code and less discussion of bird guts. :-)

Tuesday June 30, 2009
04:09 PM

Lots of little improvements

I'm back from a nice break in Italy and have been digging back in to Perl 6 stuff again. Today I've been doing a Vienna.pm-funded Rakudo day, and here's what I got up to.

First off, I went for a look through our RT queue. We now have over 400 tickets that are either new or open. While on the one hand that means we've a lot of work to do, it's also a sign that people, more and more, are playing with and exercising Rakudo. In just browsing through it, I found a bunch of things I could work on and hopefully resolve fairly easily during the day, and also another bunch of things that were already resolved. Just spotting the latter allowed me to mark 3 tickets resolved.

A couple of the things I worked on related to subtyping. Of note, the standard grammar accepted:

subset Foo of Int;

Without requiring a where clause. Rakudo now also accepts this, and our parsing is a little closer to STD.pm too (we parse traits on subtypes, but we don't do anything with them just yet). Next, I got Rakudo to support a neater syntax for declaring anonymous subtypes in signatures. If you just want to match a specific value, you can write the value in the signature, and that's it. For example, here is yet another way to do factorial (a recursive version).

multi factorial(0) { 1 }
multi factorial(Int $n) { $n * factorial($n - 1) }
say factorial(5); # 120

A signature :(0) is equivalent to :(Int $ where 0). This means that it will sort in the candidate list with Int. More generally, any literal value in where will get a nominal type based on the .WHAT of the value and have the value made into an anonymous subtype, so the signature :("tava") is just like :(Str $ where "tava"). I added some tests for all of this too.

Lyle++ had sent in a patch a while back for $*CWD and chdir. I took a look at these today. The $*CWD one looked pretty good, so I applied that with just a minor tweak. The chdir one needed some more attention and fixing up first, but I got that applied and extended the tests to better exercise it. Then I got both test files added to spectest.data. So now chdir and $*CWD are both functional. Here's some play with them in the REPL.

> my $fh = open("spectest.data", :r);
Unable to open filehandle from path 'spectest.data'
in Main (:1)
> say $*CWD;
C:\Consulting\parrot\trunk\languages\rakudo
> chdir "t";
> say $*CWD;
C:\Consulting\parrot\trunk\languages\rakudo\t
> my $fh = open("spectest.data", :r);
>

We had a couple of tickets relating to the interaction of //= and state variables. A little investigation, some discussion on #parrot and a fix later, I was able to unfudge tests and mark those resolved. A small inheritance bug was a similar story.

Finally, in preparation to improve type check failure error reporting and resolve at least one ticket in that area, I factored all type check error generation out to one routine, which we now call consistently. That means errors that previously missed out mentioning the expected and received types now do so, and the other issues I can fix - on some future Rakudo day - in one place, and everywhere that reports such errors will benefit.

In the course of the day, I also discovered a couple of other tickets that I had opened up to investigate at the start of the day were also already-fixed issues, so I made sure we had proper test coverage and got them closed up.

So, a pretty productive day. Thanks to Vienna.pm for funding!