use Perl

Today plenty happened in Rakudo land - in fact, it was my most active day's Rakudo hacking in quite a while. colomon++ also made some great commits, and between us a lot of things moved forward today. For my part, hashes and pairs are in much better shape.

I wrote before that I'd got some Rakudo days left to write up; there are two of them, but I'll cover them both in this post, since some of the work crossed the two of them anyway. Here's what I got up to between them.

• Filled out attribute composition logic for role application. A good chunk of this was written in NQP - in fact, all of the role appliers are. :-) Along the way I brought roles up to speed with the attribute part of the meta-object protocol - I'd forgotten that when doing it for classes, though since we couldn't compose attributes at that time it wasn't so interesting anyway. The end result was that we could pass S14-role/attributes.t again.
• The specification states that if in a role you do inheritance, then this is just passed on to the class that the role is eventually composed in to, and added to that class's parents. We never had any support for this in master; with a neat meta-model approach it became rather easier to get it in place in ng.
• Got BUILD/CREATE fixed up a bit and added back support for "has $.answer = 42" style declarations, again through the new attribute sub-protocol.
• Got us handling non-block where blocks again, and added Block.ACCEPTS back - in Perl 6.
• We had various "helpers" to let us do some of the low-levelish stuff in PIR. This is mostly for the places where we need those things in place in order to be able to compile the rest of the built-ins that are written in Perl 6. However, a couple of these helpers knew too much about Parrot and too little about the meta-model, which abstracts it away. So, I re-wrote some of those in terms of the meta-model. Much cleaner.
• Before we relied entirely on Parrot for our "do we do this role" checks. However, given the unfortunate semantic mis-match between Parrot's built-in role support and what we need for Perl 6 (I did try and influence things in a different direction back when we were doing Parrot's role support, but failed), I've been gradually working us towards not relying on those for Perl 6's role support. (In master, it felt to me like we have almost as much code working around the semantics of Parrot roles as we'll need to have to not use them.) Anyways, the divorce isn't quite complete yet, and it's not even a goal for the ng branch. However, I did make a notable step towards it by getting our .does checks implemented entirely in terms of the meta-model. In the long run, I'm hoping we may be able to write the entire role implementation in NQP, which helps with the even-longer-run dreams I have of Rakudo having additional backends. But that's for The Future. :-)
• Cleaned up and re-enabled sigil based type checking in signatures.

Thanks to Vienna.pm for sponsoring me to hack on Rakudo, not only for these two days, but also throughout 2009!

Several days before Christmas, encouraged by my mum asking, "when you're going to start your Christmas break", I stopped working and hacking on stuff and started relaxing. Until then, I hadn't realized just how tired I was. I slept quite a few ten hour nights in the following week, and had an enjoyable Christmas break. I'd figured I'd maybe take a week or so's break, and then get straight back to things, but a week on I had no motivation or energy to dig in again whatsoever. So, I decided my break would go on through New Years. New Year's celebrations this year involved curry - something I certainly wouldn't mind it involving again.

Early January brought several days in Sweden, part of planning for an upcoming refactoring of my work/location - there's details on my personal blog, but the short version is that I've accepted a job at a Swedish startup and will be moving there in March. It's not full time, so I'll continue to have time for Perl 6 development. They know about and, happily, are supportive of my involvement in Perl 6 and my continued attendance of Perl conferences.

I spent a weekend in Prague on the way home. I did it by train rather than flying, which was enjoyable. It snowed almost my entire time in Prague, and I caught a cold in the following week, but it was kinda worth it to wander around this beautiful city. Didn't bother studying Czech at all, and sorta got by with speaking Slovak, though some folks heard me speak and immediately concluded English would be easier. :-) Somehow it kinda felt like I was back somewhere I belonged, even though I'd never been there before. I love central Europe, and excited as I am about Sweden, I know I'll miss this part of the world a lot.

Anyway, I eased back into some work in January, but mostly took it quite easy. The happy result is that, come February, I'm finding myself recharged and ready to dig back into things again. I got some nice commits done to Rakudo yesterday, and today I meant to, but instead participated on an interesting thread on p6l and did some other useful meta stuff (like this post). Tomorrow should have plenty of hacking time though, and I'm looking forward to it. I also have a couple of blog posts to do about Vienna.pm-funded Rakudo Days I did in December, but never got around to writing up; thankfully I did make notes on what I did on them. :-) My main focuses from here on will be on:

• Continuing to get Rakudo's ng branch into shape - we'll make it master soon. A lot is missing, but things are going back fast and often very neatly. It's easy to focus on what it doesn't yet do that master does, but it has many things right that master does not - now including laziness!
• Finishing up my signatures grant. I really, really want to do that within the next couple of weeks.

Anyway, that's what's been up with me. If you take away anything, it's that you may not realize how much you need a break from something until you take it, and if it's not the only thing putting food on the table, then it's probably better to take the needed amount of break and come back revitalized. I guess the other option is to dig back in regardless, but I suspect that's the path to burnout, something I'm quite keen to avoid.

More technical blabbering here soon. :-)

Work in the ng branch continues; this post is an update on some of the work I've done on a slightly broken-up Rakudo Day (things have been a tad disrupted while I've come over to England for the month).

Perhaps the most exciting bit of news is that I've done an implementation of the attribute meta-protocol. The main upshot of this is that we now don't hard-code the creation of accessors, but rather call a "compose" routine on the Attribute object - the meta-object that represents an attribute - and it does the accessor creation. Amongst other things, this opens the door for custom accessor generation, which I believe is something that many Moose extensions do, and so will be useful for Perl 6 meta-programmers. As a bonus, the Attribute class is implemented in NQP, not PIR.

I've also continued working on roles. I'm happy to say that, unlike in master, we implement the does and but operators - which do runtime mix-ins of roles - in Perl 6 now. They don't actually do much - the heavy lifting is done by the RoleToObjectApplicator class. That one is implemented in NQP, with only one small Parrot-specific bit. It feels great to be writing these bits in something higher level, and - importantly - means they're all building on a fairly small set of primitives rather than having deep magic everywhere.

In other smaller bits of news, I got the dotty form of .= working, and also implemented the .^ dotty operator too (which makes metaclass calls prettier).

All in all, I'm fairly happy with how things are shaping up. I'm hoping another Rakudo-day sized push and we'll be able to run many of the roles related tests again, but this time atop a much cleaner and more correct implementation, a large part of which is in Perl 6 (or NQP, which is a subset of Perl 6). Thanks to Vienna.pm for sponsoring this work.

I got back from Latvia yesterday. The first Baltic Perl Workshop was very small, but also very enjoyable. I also very much enjoyed meeting up with masak++, who shares my love of good food - curry included! Riga turned out to be a great place for having a good nom, both in terms of local stuff and nice hot curry. And of course, amongst the workshop and some sightseeing, some hacking got done too.

Today was Rakudo Day, and it was time to get back to meta-model matters. I actually spent a while during my time in Riga reading a good bit of Moose and Class::MOP - I'd seen bits of them before, but this time gave the parts relating to roles a much closer look and got myself a refresher on some other bits. I also spent the plane ride home (pretty much the entire flight!) reading an interesting paper on composing meta-objects from traits, which turned out to be a much easier read than I had expected, but left me wanting to read many of its references too.

The first thing I worked on today was getting more details into metamodel.pod, which is the currently under heavy construction specification for Rakudo's meta-model. It will in the future become a proposal for inclusion in the Perl 6 specification too, at which point it'll be time to co-ordinate with other implementations to get something that we can all agree on. Anyway, if you have an interest in meta-programming, you may be interested to give it a read and leave comments, or catch me on #perl6. For those just wanting to see some code rather than read a spec, skip straight down to the example of how writing a module to support something AOP-ish could look. It'll no doubt evolve somewhat, and we won't be able to run it for a little bit yet, but I think it's a nice example of how easy it should be to implement such things in Perl 6.

After that, I dug into starting to get some implementation work done, with a focus on working towards having role composition working as spec'd in the metamodel document. So far I've got a lot of the foundations laid, and I'm now starting to build up on them. So far, it's mostly just plumbing, but composing a role into a class and the class then gaining the role's methods does now work, and a bunch of what we need for required methods and conflict resolution is stubbed in, though untested. I also was able to write a chunk of this in NQP rather than PIR, something I'd like to see us increasingly do in the stage 1 compiler, where it makes sense to do so (the stage 1 compiler containing the things we need in order to compile the setting; the smaller we can keep this part, the better it'll be for us overall...for now the goal is mostly getting the ng branch landed rather than small-stage-1-nirvana though :-)). Anyway, I think I can probably progress pretty fast from here and get our role stuff up and running in the next couple of days.

Thanks to Vienna.pm for sponsoring Rakudo Day.

This is a summary of what I did on my Rakudo day on Monday as well as other little bits of random hacking since then on the Rakudo ng branch.

• Corrected dispatches on proto-objects, and along the way fixed a few things that were quite likely bugs in master.
• Got signature introspection back in place.
• Switched back on @ requiring the Positional role and & requiring the Callable role when binding parameters.
• Got parallel dispatch back in place, and we parse it a like STD now.
• Added back parsing of fatarrow, and its ability to be used to supply named parameters
• Added back submethods.
• Made .= parse and work again; we almost certainly need to re-visit the way I did it, though, since while I didn't find a better way yet, myself and Pm both agree the current way is not ideal.
• Put back Inf, -Inf, +Inf and NaN, which parse as literals.
• Brought .arity and .count back over from Rakudo master and simplified them a bit along the way, thanks to other refactors.
• Get placeholder variables parsing and mostly working again.

So, a whole grab back of little bits here and there. Things continue to come together nicely, but a source of some frustration is that our passing test count remains low. The issue here isn't that we aren't adding back lots and lots of things, but that the test files often require features aside from the things they're actually testing. This is part and parcel of having a Perl 6 test harness - indeed, one issue we have is that we didn't put back eval yet, so eval_dies_ok tests can't yet be run. But another issue is the temptation for test writers - and I'm just as guilty as anyone - to write tests using Perl 6 language features unrelated to the thing being tested themselves. For example, many of the tests for signature introspection rely on the parallel dispatch operator. The test file itself is beautiful and elegant, but unfortunately I'm still working towards putting back things we need - even though I have signature introspection back in place - because of other features it just assumes. So I expect we'll hit a point where we very suddenly get back loads of tests - the problem is that in the meantime, we don't have as much testing to tell us of regressions as would be ideal.

OK, time for me to fly to Latvia, for the first Baltic Perl Workshop, now! :-) Thanks to Vienna.pm for funding this work.

This is a report on what I did on my Rakudo Day on Friday, but didn't manage to finish writing up until now.

It was time to switch focus from building up metamodel primitives in the Rakudo refactor for a bit and dig into the work of bringing back a bunch of the Perl 6 setting - the built-in classes and functions that are implemented in Perl 6. In doing so, I knew I'd discover lots of other little bits that would need putting back, which would guide my efforts. I chose to work towards bringing back Any-num.pm first - various bits relating to numbers. Preparing to do that led me to fix up a whole raft of other bits.

• Needed to bring the way we parsed method declarations more in line with STD.pm.
• Needed to update the identifier rule to match STD.pm so we could have identifiers with hyphens in.
• Got basic support for when/default blocks in place; mathw++ had already done given blocks.

After that, I could also bring Int.pm back into the setting. That went pretty smoothly, though needed a few tweaks since subs have "my" scope by default in Perl 6 now, something that we never updated master to handle, but have got right from the start in the ng branch. Adding that in allowed me to get rid of a few of the "cheats" that we'd had in place beforehand.

Continuing on the theme of number handling, my next act was to try and bring back Rat and Complex. Rat went in with a bit more hassle. I discovered that I'd managed to make a mess of declaration order mapping between source and the PAST tree we built. Happily, once I understood the problem, the fix was only a few lines of patching. However, then I hit on something else weird: we seemed to be calling wrong multi-candidates.

I was sure I'd made a mistake in the code-gen...but no, it all looked right. Bug in the signature generation? Nope, those looked good too. It wasn't going to be a multi-dispatcher bug really, given that this had been brought in practically unchanged from master - or at least, without any notable changes. After a lot of searching I discovered that we really weren't entering the multi-dispatcher at all, but instead always jumping to the same candidate - but only in other code in the setting, not in stuff done in the REPL. Realizing it was only an issue with code in the same compilation unit was a big clue. Due to changes in the way we're doing code generation now - in order to not create so many bogus objects of the wrong type at startup that we only then have to re-bless - a nasty Parrot issue had shown up to crash the party. Worse, while I'd call it an optimization bug, pmichaud++ correctly pointed out that a lot of existing code was almost certainly going to be relying on things working this way - the optimization was providing a feature, pretty much. Thus my proposal to just disable it wasn't quite going to fly. This issue still somewhat remains "to solve", but I'm hopeful we'll get some solution of some sort today, as I suspect it's going to become a bit of a serious blocker otherwise. Blocker as in, we probably can't pass any of the multi dispatch tests before it's fixed, even though the dispatcher itself is likely fine. What a PITA.

After hunting that down, I only really had time to get Complex.pm back in. There was much commenting out still, but that will guide me on what I needed to hack on next. And happily, having put these bits back led colomon++ to hack away on them a bit over the weekend. Thanks to Vienna.pm for sponsoring my work on Rakudo.

So, a new week is here, and hopefully this will be a week of lots and lots of progress on the ng branch of Rakudo. We've made a pretty good start, and things are just about starting to hit the exciting stage now where we're starting - piece by piece - to bring back the Perl 6 setting. In there now we already have cases of operator and trait modifier definitions and well as some initial class declarations. Basically, that means that we've got a lot of the fundamentals in that we need for the rest of the setting. A lot of the bits we are still missing aren't actually that hard either. In fact, it's starting to feel like we might be getting to the point where this refactor is nearly "over the hump" and we can start winning back passing spectests quickly.

I continue to be happy with how things seem to be overall much cleaner now in the grammar and actions, as well as the general organization of the code base. My focus has been mostly on continuing to get things related to the object model and the type system back in place. Some of the highlights have been:

• Adding the augment scope modifier, which is for adding extra bits to a class definition. This is especially useful when we want to define most of a class in Perl 6, but need to have some of the low-level bits available in the stage 1 compiler, which we use to compile the setting. This hadn't used to be a scope declarator, but rather a compiler-handled trait; we've regressed on various tests for quite a while since the spec changed, but we'll be able to have those back when the branch lands.
• Getting multi-methods back in place again. This wasn't too much work at all, and I'm quite happy with the end result, which no doubt does some better error detection. We use these in the setting heavily, so they - along with augment - were another big roadblock removed.
• Getting sub traits back in place, which unblocked pmichaud++ to work on export things. I'll leave him to blog the details, but we've made some nice wins there, in terms of handling things better than we have before.
• Being able to pretty much copy-paste the error handling for bad type names in scoped declarators from STD.pm into Rakudo's grammar. STD is really pretty good at giving decent diagnostics when things go wrong, and with the new grammar engine we're being able to bring a bunch of these into Rakudo. "In "my" declaration, typename Monkey must be predeclared" is such a nicer error to get than the rather less informative "Malformed declaration", which doesn't even tell you what sort of declaration it was, let alone what was wrong with it.
• Adding back a couple of things that we were writing in PIR before but with Perl 6 implementations this time. Strangely enough, they're much nicer to look at. :-)

Thanks to Vienna.pm for sponsoring this work. I'm now done with catching up the Rakudo days I didn't do in September while wandering around Asia, so now it's into the ones I didn't do in October. Given how much there is to do, I'm glad I've had a nice store of them to draw on at this point! :-)

I've spent most of my waking hours today (and a bit of yesterday) starting to build things out on top of the metamodel foundations in Rakudo's ng branch. It's feeling much cleaner that what we had before, and I've not found anywhere yet where it's feeling like I'm having to stick magic into the actions that is specific to a certain type of package. In fact, compared to before, the package_def action method is really quite compact and clean. The whole process of constructing a class (or role or grammar) is just a series of method calls on the metaclass.

Here's a list of things that I've got done since the last blog post.

• Got classes inheriting from Any again by default, which took me on a detour into getting the add_parent method of the metaclass working. We can't quite do inheritance again yet though - that needs traits to be completed too.
• Added handling of the grammar package declarator back in. Now the "inherit from Grammar by default and not Any" stuff is just done by us having a metaclass for grammars - GrammarHOW - that subclasses the metaclass for classes - ClassHOW. It simply overrides the compose method (what we call when we're finished defining a package) and checks if there are any explicit parents, and if not adds Grammar as a parent, then delegates up to the superclass to do the rest of the work.
• Got us parsing trait modifiers and building AST for them. We have protoregexes now, so I could make it just pretty much just like STD. There were also some tweaks to synchronize with. Much of it was just brining code in from the existing actions and re-organizing it a bit.
• One I had those in place, I got attribute accessor generation done. I had wanted traits so I could straight away add the "is rw" variant too. Again, the accessor method is added thorugh the metaclass, and this time it got a proper signature (I suspect there was probably a bug in the previous implementation to be found - maybe even a ticket - over this).
• In doing the above, I realized that I needed to sort out handling of the invocant. Before, we were having Parrot handle it for us, which meant it was handled out of band with respect to the signature binder. In ng we just handle it like any other parameter, apart from we also make it available through "self" (or only make it available through self if the invocant is not specifically requested). This fixes the "can't put where clauses on the invocant" issue that came up a while back - now (in theory at least - didn't test it) you'll be able to write a bunch of multis that dispatch based on constraints on the invocant.
• I started preparing to put roles - parametric ones too - back in place, again built around the metamodel approach. I'm hoping things get cleaner again as a result of that.
• Helped Patrick with working out why compilation is so painfully slow. I timed all of the stages, and it turns out - at least on my box - that we're spending about 10/11ths of the time in the very final stage, turning POST (a tree representing the Parrot opcodes) to PIR. Since that just walks a tree and builds up a string, that's all rather strange. pmichaud++ is investigating.

Basically, my aim is to get us able to start bringing the Perl 6 setting back as soon as possible. Today Patrick did the infrastructural work for that, which was a great start. I was close to us having inheritance working by writing a trait modifier in there, but then realized that while lexical subs now work great (like, way cleaner than in master), package ones don't. Fail. Ah well, it's 3am and I need sleep. Thanks to Vienna.pm for sponsoring this chunk of work.

It's taken me pretty much a whole Rakudo day to figure out how to go about the package declarator support in the ng branch, but I think it's been worth it. The result is not only a design, but some initial code that starts to re-build our classes and modules support in terms of it.

There are a couple of big things to take away from this. The first is that we're going to have the vast majority of things in place for custom meta-classes when the ng branch merges. This is something we were rather more distant from in master, and something that people were hoping we'd get in place by Rakudo *, so I'm glad that we're going to be getting most of the way there on this. The really big deal here is that we're getting rid of a lot of stuff where things "just worked somehow" and re-building it all on top of calls to the meta-class. We're re-building how stuff happens with the new grammar and actions anyway, so it is a fitting time to get it in place.

So what are your options if you want to write a custom metaclass? First, you go and write the class - possibly subclassing one of the existing built-in metaclasses if you just want to do some small customization (for example, I expect GrammarHOW, the metaclass for grammar packages, will just be a very simple subclass of ClassHOW that at class composition time checks if we have a parent and if not makes Grammar the default parent rather than Any, then delegates to the super class). Second, you declare a sub-language and introduce a new package declarator to the grammar; if we want to support "controller" we'd add something a rule to our sub-language like:

token package_declarator:sym<controller> {
    :my $*PKGDECL := 'controller';
    <sym> <package_def>
}


Note that you're actually adding a new type of package to the language here that will be associated with your metaclass. So how do we make this association? There's a context variable %*HOW which contains the mapping from the value set in $*PKGDECL to the name of the metaclass to use. You'll notice this means that we could actually just want our controller keyword for documentation, and just map it straight through to ClassHOW, and not write a metaclass, or we can associate multiple package declarators with the same metaclass.

While this covers the common cases, there's a few times where we need to do some slightly different code generation in terms of different packages. There's some hooks for this too, and while those will not be spec, but rather Rakudo specific, it does mean that the overall implementation will get a lot cleaner, with the Actions.pm file that maps parse tree to abstract syntax tree now not having to know the details of different types of package.

Already it feels a lot cleaner with the few bits that are in place so far, and I think as we start to build out on top of this, it's going to lead to something that overall is a lot less magical, and a lot more maintainable and extensible, than what we had before. All of which make a pause for thought, rather than a charge into the code, very much worthwhile. Thanks to Vienna.pm for making it possible for me to take the time to do these things. :-)

I wanted to get a little rest and seeing friends in this weekend alongside helping move ng along - that is, the branch where we're re-building Rakudo atop of the new regex engine. Thus I took a Rakudo day and split it up over the two days, yesterday and today, so I could get a nice combination of work and play. Here's some of what I've been up to; I expect Patrick will blog later on today some of the outstanding work he has been doing too.

I guess the work we've done over this weekend could be divided into two overlapping tracks. Patrick has taken on a lot of the language fundamentals and had a key focus on us being able to run Test.pm again, which will open up the rest of the sepctest suite to us. It's currently looking fairly likely that we'll reach that milestone within the next 24 hours, which is amazing progress. I've also been doing some things that will help us on the way to that, but my focus has on getting some key infrastructure in again so we can soon start bringing parts of the Perl 6 setting back into play.

One of the things I had to get back in place was type name handling, so we can differentiate our types from our other names. That was a lot neater this time, since it could just go in much more STD-ish as an assertion - written in NQP rather than PIR (there's a lot of this now). After that, it was easy to get typed signatures parsing again. I fixed up our name parsing along the way a bit too - the NQP grammar did something simpler, whereas we needed something a bit more complex for Perl 6.

One of the big changes we've had in from the start - in order to bring ourselves in line with the spec - is that subs are lexically scoped rather than package scoped by default. After some pondering, I realized that we could handle lexical multis in a far, far nicer way than we had been doing in the past, so I got that implemented. In fact, I'm now expecting I can follow the same pattern for methods too. It's a lot, lot cleaner than what we were doing before, and additionally avoids a lot of start-up reblessing, which was one of our big startup costs before. We really are taking the opportunity to apply our past experience and the new grammar engine features to build the grammars actions up in a much neater way, and I'm really happy with the results so far.

One of the things Patrick has been working to get in place is the Parcel data structure. Along with that, we've been reshaping our approach to issues like flattening. I've mostly been involved with that at the level of discussing the approach with Patrick, and he's led the implementation charge on it. Today he hit on the need for a small tweak to the method dispatcher, which was familiar ground for me, so I got it patched up. A silly fail on my part later, it was all working rather nicely. Finally, I got some pieces of roles patched back in to the new build.

One thing I've been preparing for is getting our class and role declaration support back in place. I'm planning to approach it in a rather different way than we have been up until now, the motivation partly being neater and more maintainable code, but also getting us on track to handle custom meta-classes. I'm not saying we'll actually have those when the ng branch lands, but equally I'm aiming for us to be a whole bunch closer. The other big difference in approach here is that I'm aiming for us to be able to keep a lot of the package information trivially introspectable during the building up of the class or role, which will make some things much easier and better encapsulated. Further, we can also keep that information around for assisting a future optimizer/type checker. (As an aside, I've been arranging signature things in a similar way during the course of my signatures grant. I guess the general principle is that while PAST is pretty much sufficient for expressing the semantics of the program, sometimes it's desirable to keep some higher level information around for entities that don't have a specific type of PAST node dedicated to them - mostly because languages tend to care about them in very different ways, so they don't belong in PAST anyway.)

Thanks to Vienna.pm for funding my work at this important stage in Rakudo's development.