Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

JonathanWorthington (6049)

  (email not shown publicly)
Thursday December 10, 2009
05:15 AM

Attribute sub-protocol and other role bits

Work in the ng branch continues; this post is an update on some of the work I've done on a slightly broken-up Rakudo Day (things have been a tad disrupted while I've come over to England for the month).

Perhaps the most exciting bit of news is that I've done an implementation of the attribute meta-protocol. The main upshot of this is that we now don't hard-code the creation of accessors, but rather call a "compose" routine on the Attribute object - the meta-object that represents an attribute - and it does the accessor creation. Amongst other things, this opens the door for custom accessor generation, which I believe is something that many Moose extensions do, and so will be useful for Perl 6 meta-programmers. As a bonus, the Attribute class is implemented in NQP, not PIR.

I've also continued working on roles. I'm happy to say that, unlike in master, we implement the does and but operators - which do runtime mix-ins of roles - in Perl 6 now. They don't actually do much - the heavy lifting is done by the RoleToObjectApplicator class. That one is implemented in NQP, with only one small Parrot-specific bit. It feels great to be writing these bits in something higher level, and - importantly - means they're all building on a fairly small set of primitives rather than having deep magic everywhere.

In other smaller bits of news, I got the dotty form of .= working, and also implemented the .^ dotty operator too (which makes metaclass calls prettier).

All in all, I'm fairly happy with how things are shaping up. I'm hoping another Rakudo-day sized push and we'll be able to run many of the roles related tests again, but this time atop a much cleaner and more correct implementation, a large part of which is in Perl 6 (or NQP, which is a subset of Perl 6). Thanks to for sponsoring this work.

Tuesday November 24, 2009
08:03 PM

Laying the role meta-foundations

I got back from Latvia yesterday. The first Baltic Perl Workshop was very small, but also very enjoyable. I also very much enjoyed meeting up with masak++, who shares my love of good food - curry included! Riga turned out to be a great place for having a good nom, both in terms of local stuff and nice hot curry. And of course, amongst the workshop and some sightseeing, some hacking got done too.

Today was Rakudo Day, and it was time to get back to meta-model matters. I actually spent a while during my time in Riga reading a good bit of Moose and Class::MOP - I'd seen bits of them before, but this time gave the parts relating to roles a much closer look and got myself a refresher on some other bits. I also spent the plane ride home (pretty much the entire flight!) reading an interesting paper on composing meta-objects from traits, which turned out to be a much easier read than I had expected, but left me wanting to read many of its references too.

The first thing I worked on today was getting more details into metamodel.pod, which is the currently under heavy construction specification for Rakudo's meta-model. It will in the future become a proposal for inclusion in the Perl 6 specification too, at which point it'll be time to co-ordinate with other implementations to get something that we can all agree on. Anyway, if you have an interest in meta-programming, you may be interested to give it a read and leave comments, or catch me on #perl6. For those just wanting to see some code rather than read a spec, skip straight down to the example of how writing a module to support something AOP-ish could look. It'll no doubt evolve somewhat, and we won't be able to run it for a little bit yet, but I think it's a nice example of how easy it should be to implement such things in Perl 6.

After that, I dug into starting to get some implementation work done, with a focus on working towards having role composition working as spec'd in the metamodel document. So far I've got a lot of the foundations laid, and I'm now starting to build up on them. So far, it's mostly just plumbing, but composing a role into a class and the class then gaining the role's methods does now work, and a bunch of what we need for required methods and conflict resolution is stubbed in, though untested. I also was able to write a chunk of this in NQP rather than PIR, something I'd like to see us increasingly do in the stage 1 compiler, where it makes sense to do so (the stage 1 compiler containing the things we need in order to compile the setting; the smaller we can keep this part, the better it'll be for us overall...for now the goal is mostly getting the ng branch landed rather than small-stage-1-nirvana though :-)). Anyway, I think I can probably progress pretty fast from here and get our role stuff up and running in the next couple of days.

Thanks to for sponsoring Rakudo Day.

Thursday November 19, 2009
05:41 AM

Progress and minor frustration

This is a summary of what I did on my Rakudo day on Monday as well as other little bits of random hacking since then on the Rakudo ng branch.

  • Corrected dispatches on proto-objects, and along the way fixed a few things that were quite likely bugs in master.
  • Got signature introspection back in place.
  • Switched back on @ requiring the Positional role and & requiring the Callable role when binding parameters.
  • Got parallel dispatch back in place, and we parse it a like STD now.
  • Added back parsing of fatarrow, and its ability to be used to supply named parameters
  • Added back submethods.
  • Made .= parse and work again; we almost certainly need to re-visit the way I did it, though, since while I didn't find a better way yet, myself and Pm both agree the current way is not ideal.
  • Put back Inf, -Inf, +Inf and NaN, which parse as literals.
  • Brought .arity and .count back over from Rakudo master and simplified them a bit along the way, thanks to other refactors.
  • Get placeholder variables parsing and mostly working again.

So, a whole grab back of little bits here and there. Things continue to come together nicely, but a source of some frustration is that our passing test count remains low. The issue here isn't that we aren't adding back lots and lots of things, but that the test files often require features aside from the things they're actually testing. This is part and parcel of having a Perl 6 test harness - indeed, one issue we have is that we didn't put back eval yet, so eval_dies_ok tests can't yet be run. But another issue is the temptation for test writers - and I'm just as guilty as anyone - to write tests using Perl 6 language features unrelated to the thing being tested themselves. For example, many of the tests for signature introspection rely on the parallel dispatch operator. The test file itself is beautiful and elegant, but unfortunately I'm still working towards putting back things we need - even though I have signature introspection back in place - because of other features it just assumes. So I expect we'll hit a point where we very suddenly get back loads of tests - the problem is that in the meantime, we don't have as much testing to tell us of regressions as would be ideal.

OK, time for me to fly to Latvia, for the first Baltic Perl Workshop, now! :-) Thanks to for funding this work.

Monday November 16, 2009
06:47 AM

Bringing back the setting, and bug hunting

This is a report on what I did on my Rakudo Day on Friday, but didn't manage to finish writing up until now.

It was time to switch focus from building up metamodel primitives in the Rakudo refactor for a bit and dig into the work of bringing back a bunch of the Perl 6 setting - the built-in classes and functions that are implemented in Perl 6. In doing so, I knew I'd discover lots of other little bits that would need putting back, which would guide my efforts. I chose to work towards bringing back first - various bits relating to numbers. Preparing to do that led me to fix up a whole raft of other bits.

  • Needed to bring the way we parsed method declarations more in line with
  • Needed to update the identifier rule to match so we could have identifiers with hyphens in.
  • Got basic support for when/default blocks in place; mathw++ had already done given blocks.

After that, I could also bring back into the setting. That went pretty smoothly, though needed a few tweaks since subs have "my" scope by default in Perl 6 now, something that we never updated master to handle, but have got right from the start in the ng branch. Adding that in allowed me to get rid of a few of the "cheats" that we'd had in place beforehand.

Continuing on the theme of number handling, my next act was to try and bring back Rat and Complex. Rat went in with a bit more hassle. I discovered that I'd managed to make a mess of declaration order mapping between source and the PAST tree we built. Happily, once I understood the problem, the fix was only a few lines of patching. However, then I hit on something else weird: we seemed to be calling wrong multi-candidates.

I was sure I'd made a mistake in the code-gen...but no, it all looked right. Bug in the signature generation? Nope, those looked good too. It wasn't going to be a multi-dispatcher bug really, given that this had been brought in practically unchanged from master - or at least, without any notable changes. After a lot of searching I discovered that we really weren't entering the multi-dispatcher at all, but instead always jumping to the same candidate - but only in other code in the setting, not in stuff done in the REPL. Realizing it was only an issue with code in the same compilation unit was a big clue. Due to changes in the way we're doing code generation now - in order to not create so many bogus objects of the wrong type at startup that we only then have to re-bless - a nasty Parrot issue had shown up to crash the party. Worse, while I'd call it an optimization bug, pmichaud++ correctly pointed out that a lot of existing code was almost certainly going to be relying on things working this way - the optimization was providing a feature, pretty much. Thus my proposal to just disable it wasn't quite going to fly. This issue still somewhat remains "to solve", but I'm hopeful we'll get some solution of some sort today, as I suspect it's going to become a bit of a serious blocker otherwise. Blocker as in, we probably can't pass any of the multi dispatch tests before it's fixed, even though the dispatcher itself is likely fine. What a PITA.

After hunting that down, I only really had time to get back in. There was much commenting out still, but that will guide me on what I needed to hack on next. And happily, having put these bits back led colomon++ to hack away on them a bit over the weekend. Thanks to for sponsoring my work on Rakudo.

Tuesday November 10, 2009
07:23 PM

Starting to get over the hump

So, a new week is here, and hopefully this will be a week of lots and lots of progress on the ng branch of Rakudo. We've made a pretty good start, and things are just about starting to hit the exciting stage now where we're starting - piece by piece - to bring back the Perl 6 setting. In there now we already have cases of operator and trait modifier definitions and well as some initial class declarations. Basically, that means that we've got a lot of the fundamentals in that we need for the rest of the setting. A lot of the bits we are still missing aren't actually that hard either. In fact, it's starting to feel like we might be getting to the point where this refactor is nearly "over the hump" and we can start winning back passing spectests quickly.

I continue to be happy with how things seem to be overall much cleaner now in the grammar and actions, as well as the general organization of the code base. My focus has been mostly on continuing to get things related to the object model and the type system back in place. Some of the highlights have been:

  • Adding the augment scope modifier, which is for adding extra bits to a class definition. This is especially useful when we want to define most of a class in Perl 6, but need to have some of the low-level bits available in the stage 1 compiler, which we use to compile the setting. This hadn't used to be a scope declarator, but rather a compiler-handled trait; we've regressed on various tests for quite a while since the spec changed, but we'll be able to have those back when the branch lands.
  • Getting multi-methods back in place again. This wasn't too much work at all, and I'm quite happy with the end result, which no doubt does some better error detection. We use these in the setting heavily, so they - along with augment - were another big roadblock removed.
  • Getting sub traits back in place, which unblocked pmichaud++ to work on export things. I'll leave him to blog the details, but we've made some nice wins there, in terms of handling things better than we have before.
  • Being able to pretty much copy-paste the error handling for bad type names in scoped declarators from into Rakudo's grammar. STD is really pretty good at giving decent diagnostics when things go wrong, and with the new grammar engine we're being able to bring a bunch of these into Rakudo. "In "my" declaration, typename Monkey must be predeclared" is such a nicer error to get than the rather less informative "Malformed declaration", which doesn't even tell you what sort of declaration it was, let alone what was wrong with it.
  • Adding back a couple of things that we were writing in PIR before but with Perl 6 implementations this time. Strangely enough, they're much nicer to look at. :-)

Thanks to for sponsoring this work. I'm now done with catching up the Rakudo days I didn't do in September while wandering around Asia, so now it's into the ones I didn't do in October. Given how much there is to do, I'm glad I've had a nice store of them to draw on at this point! :-)

Friday November 06, 2009
08:57 PM

More Rakudo ng hacking

I've spent most of my waking hours today (and a bit of yesterday) starting to build things out on top of the metamodel foundations in Rakudo's ng branch. It's feeling much cleaner that what we had before, and I've not found anywhere yet where it's feeling like I'm having to stick magic into the actions that is specific to a certain type of package. In fact, compared to before, the package_def action method is really quite compact and clean. The whole process of constructing a class (or role or grammar) is just a series of method calls on the metaclass.

Here's a list of things that I've got done since the last blog post.

  • Got classes inheriting from Any again by default, which took me on a detour into getting the add_parent method of the metaclass working. We can't quite do inheritance again yet though - that needs traits to be completed too.
  • Added handling of the grammar package declarator back in. Now the "inherit from Grammar by default and not Any" stuff is just done by us having a metaclass for grammars - GrammarHOW - that subclasses the metaclass for classes - ClassHOW. It simply overrides the compose method (what we call when we're finished defining a package) and checks if there are any explicit parents, and if not adds Grammar as a parent, then delegates up to the superclass to do the rest of the work.
  • Got us parsing trait modifiers and building AST for them. We have protoregexes now, so I could make it just pretty much just like STD. There were also some tweaks to synchronize with. Much of it was just brining code in from the existing actions and re-organizing it a bit.
  • One I had those in place, I got attribute accessor generation done. I had wanted traits so I could straight away add the "is rw" variant too. Again, the accessor method is added thorugh the metaclass, and this time it got a proper signature (I suspect there was probably a bug in the previous implementation to be found - maybe even a ticket - over this).
  • In doing the above, I realized that I needed to sort out handling of the invocant. Before, we were having Parrot handle it for us, which meant it was handled out of band with respect to the signature binder. In ng we just handle it like any other parameter, apart from we also make it available through "self" (or only make it available through self if the invocant is not specifically requested). This fixes the "can't put where clauses on the invocant" issue that came up a while back - now (in theory at least - didn't test it) you'll be able to write a bunch of multis that dispatch based on constraints on the invocant.
  • I started preparing to put roles - parametric ones too - back in place, again built around the metamodel approach. I'm hoping things get cleaner again as a result of that.
  • Helped Patrick with working out why compilation is so painfully slow. I timed all of the stages, and it turns out - at least on my box - that we're spending about 10/11ths of the time in the very final stage, turning POST (a tree representing the Parrot opcodes) to PIR. Since that just walks a tree and builds up a string, that's all rather strange. pmichaud++ is investigating.

Basically, my aim is to get us able to start bringing the Perl 6 setting back as soon as possible. Today Patrick did the infrastructural work for that, which was a great start. I was close to us having inheritance working by writing a trait modifier in there, but then realized that while lexical subs now work great (like, way cleaner than in master), package ones don't. Fail. Ah well, it's 3am and I need sleep. Thanks to for sponsoring this chunk of work.

Wednesday November 04, 2009
07:59 PM

Relaying the metamodel foundations

It's taken me pretty much a whole Rakudo day to figure out how to go about the package declarator support in the ng branch, but I think it's been worth it. The result is not only a design, but some initial code that starts to re-build our classes and modules support in terms of it.

There are a couple of big things to take away from this. The first is that we're going to have the vast majority of things in place for custom meta-classes when the ng branch merges. This is something we were rather more distant from in master, and something that people were hoping we'd get in place by Rakudo *, so I'm glad that we're going to be getting most of the way there on this. The really big deal here is that we're getting rid of a lot of stuff where things "just worked somehow" and re-building it all on top of calls to the meta-class. We're re-building how stuff happens with the new grammar and actions anyway, so it is a fitting time to get it in place.

So what are your options if you want to write a custom metaclass? First, you go and write the class - possibly subclassing one of the existing built-in metaclasses if you just want to do some small customization (for example, I expect GrammarHOW, the metaclass for grammar packages, will just be a very simple subclass of ClassHOW that at class composition time checks if we have a parent and if not makes Grammar the default parent rather than Any, then delegates to the super class). Second, you declare a sub-language and introduce a new package declarator to the grammar; if we want to support "controller" we'd add something a rule to our sub-language like:

token package_declarator:sym<controller> {
    :my $*PKGDECL := 'controller';
    <sym> <package_def>

Note that you're actually adding a new type of package to the language here that will be associated with your metaclass. So how do we make this association? There's a context variable %*HOW which contains the mapping from the value set in $*PKGDECL to the name of the metaclass to use. You'll notice this means that we could actually just want our controller keyword for documentation, and just map it straight through to ClassHOW, and not write a metaclass, or we can associate multiple package declarators with the same metaclass.

While this covers the common cases, there's a few times where we need to do some slightly different code generation in terms of different packages. There's some hooks for this too, and while those will not be spec, but rather Rakudo specific, it does mean that the overall implementation will get a lot cleaner, with the file that maps parse tree to abstract syntax tree now not having to know the details of different types of package.

Already it feels a lot cleaner with the few bits that are in place so far, and I think as we start to build out on top of this, it's going to lead to something that overall is a lot less magical, and a lot more maintainable and extensible, than what we had before. All of which make a pause for thought, rather than a charge into the code, very much worthwhile. Thanks to for making it possible for me to take the time to do these things. :-)

Sunday November 01, 2009
08:05 PM

A productive weekend in Rakudo land

I wanted to get a little rest and seeing friends in this weekend alongside helping move ng along - that is, the branch where we're re-building Rakudo atop of the new regex engine. Thus I took a Rakudo day and split it up over the two days, yesterday and today, so I could get a nice combination of work and play. Here's some of what I've been up to; I expect Patrick will blog later on today some of the outstanding work he has been doing too.

I guess the work we've done over this weekend could be divided into two overlapping tracks. Patrick has taken on a lot of the language fundamentals and had a key focus on us being able to run again, which will open up the rest of the sepctest suite to us. It's currently looking fairly likely that we'll reach that milestone within the next 24 hours, which is amazing progress. I've also been doing some things that will help us on the way to that, but my focus has on getting some key infrastructure in again so we can soon start bringing parts of the Perl 6 setting back into play.

One of the things I had to get back in place was type name handling, so we can differentiate our types from our other names. That was a lot neater this time, since it could just go in much more STD-ish as an assertion - written in NQP rather than PIR (there's a lot of this now). After that, it was easy to get typed signatures parsing again. I fixed up our name parsing along the way a bit too - the NQP grammar did something simpler, whereas we needed something a bit more complex for Perl 6.

One of the big changes we've had in from the start - in order to bring ourselves in line with the spec - is that subs are lexically scoped rather than package scoped by default. After some pondering, I realized that we could handle lexical multis in a far, far nicer way than we had been doing in the past, so I got that implemented. In fact, I'm now expecting I can follow the same pattern for methods too. It's a lot, lot cleaner than what we were doing before, and additionally avoids a lot of start-up reblessing, which was one of our big startup costs before. We really are taking the opportunity to apply our past experience and the new grammar engine features to build the grammars actions up in a much neater way, and I'm really happy with the results so far.

One of the things Patrick has been working to get in place is the Parcel data structure. Along with that, we've been reshaping our approach to issues like flattening. I've mostly been involved with that at the level of discussing the approach with Patrick, and he's led the implementation charge on it. Today he hit on the need for a small tweak to the method dispatcher, which was familiar ground for me, so I got it patched up. A silly fail on my part later, it was all working rather nicely. Finally, I got some pieces of roles patched back in to the new build.

One thing I've been preparing for is getting our class and role declaration support back in place. I'm planning to approach it in a rather different way than we have been up until now, the motivation partly being neater and more maintainable code, but also getting us on track to handle custom meta-classes. I'm not saying we'll actually have those when the ng branch lands, but equally I'm aiming for us to be a whole bunch closer. The other big difference in approach here is that I'm aiming for us to be able to keep a lot of the package information trivially introspectable during the building up of the class or role, which will make some things much easier and better encapsulated. Further, we can also keep that information around for assisting a future optimizer/type checker. (As an aside, I've been arranging signature things in a similar way during the course of my signatures grant. I guess the general principle is that while PAST is pretty much sufficient for expressing the semantics of the program, sometimes it's desirable to keep some higher level information around for entities that don't have a specific type of PAST node dedicated to them - mostly because languages tend to care about them in very different ways, so they don't belong in PAST anyway.)

Thanks to for funding my work at this important stage in Rakudo's development.

Friday October 30, 2009
08:29 PM

Rakudo Day: Starting to put Rakudo together again

Recently Patrick wrote about some major upcoming changes to Rakudo, which will be a huge step along the way towards Rakudo *. If you didn't read Patrick's post, I suggest you do so first and then come back to mine - it gives a lot of detail on why we're doing what we're doing at the moment. Anyway, after much hard work - on Patrick's part getting nqp-rx ready and on my part getting us building and running on top of the updated Parrot Calling Conventions - today we were finally ready to start on step 7 of the plan. Essentially, that involves a complete re-do of Rakudo's grammar and actions.

nqp-rx was already in a pretty awesome state at the start of the day. It was missing a make install target, which is the first thing I worked on. With that in place, I was curious to see if it could build the current Rakudo NQP files. Given that it was a ground-up re-implementation of NQP on top of a new implementation of the regex engine, which had changed in many ways, I was really quite impressed how little effort it took to build Rakudo's NQP parts on top of it and have us make it through the test suite mostly passing. This wasn't actually the goal - it was well beyond what Patrick and I had guessed we would manage - but it did shake out a couple of missing features in nqp-rx, which we were able to add quickly between us, as well as a bug, which I fixed. It was nice to have them out of the way early on, before we dug into really using nqp-rx to re-build Rakudo on top of.

With those fixes in place, Patrick then ripped a bunch of stuff out of the Rakudo makefile and added the NQP grammar and actions as a base for us to start building up from again. This means we've gone back to having something that won't even run the sanity tests completely yet. However, what's there so far is also far, far closer to STD than we were before, using protoregexes in various places as well as contextual variable declarations inside regexes and an operator precedence parser alike the one from STD. Certainly, the bits that I have put back in so far have gone in far, far cleaner than they appear in current Rakudo.

Unlike NQP, which just uses Parrot's default signature binder, Rakudo running on the new grammar just from today's efforts is already switched over to building Perl 6 signatures and is using the signature binder I've been developing as part of my current Hague Grant. Multi-dispatch and operator parsing should hopefully be landing tomorrow - since this is just a re-write of the way we parse and build up the AST, rather than all of the other guts and the setting - we're being able to put back major features in a short time-frame. All of that said, we're not shying away from using this as an opportunity to deal with other a couple of other refactors we've been wanting to do for a while along the way. Generally, they are fairly fundemental things that would have been quite a pain to deal with individually, and are easy to get in as assumptions from early on as we build things up again.

This is the first Rakudo day I've done in a while - I was wandering around Asia for a month, and then have been focused on my grant. That means I've got a nice little stash of Rakudo days to spend during the next couple of weeks, which will be great in getting Rakudo back on its feet, more featureful and stable than before, and with a grammar much closer to STD. Thanks to for funding Rakudo Days, and I'll keep you posted on how things are progressing in the ng branch. :-)

Tuesday October 27, 2009
12:45 PM

Signature introspection

One of my Hague Grant tasks was to get a signature introspection interface into the specification and get Rakudo conforming to it. While there may still be some minor tweaks to come, this part is essentially done now. You can read the spec additions in S06.

First off, you can call .signature on any block to get hold of its signature object - an instance of the Signature classs. Calling .params on a Signature object will in turn give you a List of Parameter objects, which describe each of the parameters specified in the signature. We are able to introspect the types, the names, whether the parameter is optional, get a closure that returns any default value and more. Here's an example.

sub example(Int $x, Str $y?) { }
my $sig = &example.signature;
for $sig.params -> $param {
    say "Name: {$}, Type: {$param.type}, Optional: {$param.optional}";

This gives the output:

Name: $x, Type: Int(), Optional: 0
Name: $y, Type: Str(), Optional: 1

To make sure that all of the information was really there and accessible, I then went and re-implemented the .perl method on a Signature in Perl 6 in the setting, using the Parameter objects. The new version lacked several of the bugs that the previous version had, which cleared up a few RT tickets. And finally, moritz++ wrote a bunch of spectests for signature introspection, following the new spec.