use Perl

The Pugs journal has moved from my use.perl journal  to http://pugs.blogs.com/.

I have imported all the old posts; comments are not yet imported.

I'm very grateful to have received the kind comments, and hope that the authors won't take offense of their removal.

I am $working my way out of this personal/financial hell, so I can arrive in .au this Friday in one piece. Judging from past experience, it should get better as soon as I'm not in .tw. ;-)

jhi gave me a heavy book, Unicode Demystified, which really made sense to me. I brought jhi's concerns to #parrot, mainly about the combinatorial explosive danger of introducing charsets other than ASCII/Latin1/Unicode, and the benefit of shifting away from UTF8 as internal encoding, by determining the optimal charset (i.e. 7/8/16/21 bits of Unicode) at PIR compilation time. Leo seems receptive, but more benchmark number is needed.

So I returned to .tw, abducted by Unicode Demystified, abused by jetlag (did lots of design work on paper as LCD made me doze off), as well as cleaning up the huge backlog of forgotten/repressed/misremembered commitments.

Then I was abducted by another fantastic book -- Code Complete 2nd Edition -- which convinced me that this whole ten-day yakshaving was actually worth it.

Happily, my anxiety level now is at its all-time low in ten years, and commitment to Perl6/Pugs at its all-time high in ten months, so that's definitely improvement. More details later. :-)

Liz and I had a SubEthaEdit session on coroutines today, with the brief notes merged to S17 (Liz is working on Englishify them at this moment).

Basically, a coroutine is an object that supports a .start method that initiates the lightweight thread behind it, and return a object that may be manipulated like any other threads. If you call it like a nullary function, it activates the thread until it hits yield, in which case it gives the control back to the caller. The return inside the thread will end itself immediately. Either way, if you attempt to call a finished thread, an exception will be thrown.

Then we get the extra magic of calling a Coroutine objects like a Code object. This will trigger &Coroutine::postcircumfix<( )>, which will take care of calling .start again after the thread has finished, and arrange it so the next call to the coroutine will resume the ongoing thread instead of starting a new one.

Finally, there is the question of calling an ongoing coroutine with parameters. I think rebinding the parameters makes sense (as you can inhibit it using is copy or explicit { my $x := $OUTER::x }), but iblech's idea of having yield returning the Arglist makes sense too. In any case this is userland code, so one can change the behaviour with a trait or a subclass of Coroutines.

On the PIL2 front, I looked at the three special forms (Assignment, Binding, Apply) and decided to unify them into method calls, so the user can override them all inside the same object model -- "Everything is an object."

The Apply form is easiest, as it's just method postcircumfix<( )> that takes an Arglist and gives back an Arglist. See the new S03 for more about the cool things you can do with Arglists.

Assignment is also manageable under the container model. It will be method infix:<=> on Array, Hash, Scalar objects. The List objects constructed with infix:<,> will also respond to this method, to implement the ($a, $b, $c) = (1, 2, 3) forms.

Binding almost feel like it should be a non-user-overridable special form, except we remembered the permanently-semi-official idea of Siglist objects, constructed with the :() notation. It turns out that if we specify binding as a method on the Siglist object, and that the compiler desugars LHS := RHS always as :(LHS).infix:<:=>(RHS), it will all Just Work. Under this regime, 3 := 4 will still be a compile-time error, as :(3) would make no sense -- the macro that turns sub (3) {...} into sub ($ where {$_ ~~ 3}) {...} would not apply to bare Siglist construction forms.

So, PIL2 is looking good with fewer nodes and tighter object integration now. Maybe we are not yet at io's zero keywords level, but it's getting dangerously close. Note that all of this is outside specced domain -- we'll need to work them back to Synopses and tests, and run them through p6l and @Larry.

Of course, to recover a respectible performance under this regime, we need to rely on optimizers, early binding, and/or type inference. But that's two milestones ahead, so let's get the dynamism part right first...

To that end, I fixed the Parrot codegen, removing deprecated opcodes and syntaxes. To my surprise and delight, I discovered that if I turn the parrotIsBrokenXXX flag to False, most tests in t/01-sanity/* still passes -- they used to break horribly, eaten alive by the rabid register alligator.

However, the new Parrot removed the ability to fake nested scopes with label-delimited inline closures, so some tests are now infinite-looping when compiled to Parrot. The correct (and easy) way to fix it would be using the new lexpad spec -- hopefully by Parrot 0.3.2 we will get both side implemented and work in harmony.

PGE got much more powerful with its ability to return anything, in addition to static Match objects. As a concrete demonstration, PGE now comes with a builtin p6rule rule that parses a Perl 6 rule string into a PGE::Rule object -- which will automagically appear in Pugs's $/ as a hash object. To wit:

pugs> ('1' ~~ /<p6rule>/)<p6rule>
Match.new(
   ok => bool::true,
   from => 0,
   to => 1,
   str => "1",
   sub_pos => (),
   sub_named =>
     { "expr" =>
         Match.new(
           ok => bool::true,
           from => 0,
           to => 1,
           str => "1",
           sub_pos => (),
           sub_named => {
               "type" => "term:",
               "value" => "1"
             }
         )
     }
)

Of course, it works for arbitrarily complex rules, not just a single character literal.

Note that we can't yet call the returned object's methods in Pugs, and there's no way to define rules using Perl 6 methods. This is because Pugs/Parrot doesn't really have a OO message protocol between them, and there are several complementary ways to solve this:

• Implement calls to/from PMCs in embedded parrot, in the same way we tackled SVs in embedded Perl 5. This is trivial, but won't work with external Parrot (which is the only option in Win32 right now).
• Compile Perl 6 OO code that adheres to the object model to PIR. This would require MM/PIL2 and lexpad, but we are going to do them anyway.
• Port PGE to Perl 6. This is I hope what's going to happen in the long run, and will instantly carry the Rule engine to JavaScript and Perl5 runtimes, so it would be a net win.
• Port PGE to Haskell, and write translators between it and Parsec. This will net us a truly rebindable grammar in the Haskell runtime, and allows us to translate Pugs.Parser piecemeal to Rules, so it's also somewhat attractive.

On a completely unrelated note, gaal is writing an article for TPR (The Perl Review) about Pugs -- feel free to chime in! (As usual, ask for a committer bit on freenode #perl6 if you hadn't got one.)

Last week's Shibuya.pm meeting features Catalyst, Ajax, Pugs and ICFP. Takesako-san did a nice presentation (PPT in Japanese) about Pugs, though some slides leaves me feeling excessively flattered. ;-)

But now, happy in the knowledge that Parrot 0.3.0 can fly with Pugs interop intact, I'll go back to MM/PIL2 hacking ere sleep. See you tomorrow!

• Chip at last delivered a sane and wonderful lexical pad spec for Parrot. I'm working on codegenning to it -- I wonder if Leo will beat me to it by implementing the spec.
• Patrick committed the long-anticipated PGE::OPTable precedence parser, and posted an update to P6C about it. This means we can finally start working on porting Parsec rules to Perl 6 rules, and bootstrap the previously-nonbootstrappable Parser component to Perl 6.
• I consulted with Stevan and iblech and decided that metamodel bootstrapping should be done only once, in the compiler; the compiler will serialize ::Object and ::Class as opaque objects. This alleviates the burden of runtime writers even more.
• I checked in code to make compile-time objects survive pugs -CPugs (and the trivial test-only pugs -BPugs), which makes pugscc applicable to most Perl 6 programs.
• To move beyond pugscc's native approach of serializing compiler state, I repo-copied PIL1 into PIL2, and started the long-overdue PIL renovation to support opaque objects, lexical hoisting, labels, pragmas, and other neat things that's long overdue. Sadly YARV can't help much here, as it doesn't seem to handle Ruby's BEGIN block.
• kane, liz and I did a bit SubEthaEdit on S22 draft to collect thoughts on Perl 6's module support and 6PAN issues. Sadly, during the lunch break, I suffered through serious carsick, and cannot resume the session with kane. My apologies. :-/
• Stevan polished the MetaModel bootstrap sequence to a point that the call graph is looking like S-expressions, so he went ahead and converted them into that. Now we just need a codegen to Perl6/Hs/PIR to finish the bootstrap of the metamodel part -- nothingmuch mentioned the codegen maybe written in attribute grammars (AGs), and that sounds like a good idea.

By the way, the main Pugs subversion is again down for the weekends, so try this fallback for latest version.

That's it for now, I think... More hacking awaits me. :)

Among the runtimes, probably only the Haskell one can implement them in full at this moment, although the others can simulate the behaviour using lightweight multiplexing.

I learned that Parrot's threading code (and design) are currently missing... So after Chip is done with the lexical pads, I'd like to work with him and Leo a bit, to make sure that we are on the same page at grokking the STM+Continuation model. Come to think about it, maybe Dan will suddenly decide to publish Cola's subversion repository and come up with some related stuff as well...

Stevan started working on the MetaModel Bootstrapping sequence, a shared call graph on runtime-provided kernel objects that will establish the Perl 6 object model. The good thing is that it is agnostic to runtimes -- I need only to implement the small amount of underlying intrinsics in Haskell, then we can get a object model for free. This approach is taken from PyPy, with some influence from Ruby/YARV as well. Hurray for symbiotic evolution! :-)

I checked in some fix to Parrot's overview.pod, removing a paragraph that has bothered me for a long time. Originally, it claimed that Parrot can make use of the vast amount of research literature on compiling to hardware CPUs, compared to the fewer papers on compiling to stack machines. The reality however is that most hardware papers doesn't apply to the infinite-register Parrot, and there just are not many papers on Itaniums (which resembles Parrot the most), compared to the huge amount of JVM and CLR papers.

I changed it to say that in non-JIT settings, register machines takes fewer operations to perform typical high-level language tasks, and the new scheme of "lexicals are just registers" extends this benefit to intermediate values held in variables as well. Now I feel much happier. For more detail, see this paper referred from Jonathan.

Liz notices that in our Perl 5 embedding examples, we always start with use perl5:DBI;, which is a pretty good indication of the number one module to port over from Perl 5. It turns out that the guts of DBI::PurePerl and stub DBDs are just a few hundred lines, and dduncan volunteered to take a look to port it over to Perl 6, possibly even with a port of DBD::mysqlPP using Pugs's existing sockets support. Woot!

That's it for today, I think. See you tomorrow!

Full journal instead of stubs tonight, as I'm in the car driving back to Echt, which gives me about two hours of journaling time.

This afternoon, after sketching out the PA02 draft with Wendy, we drove to Amsterdam for the Perl Monger's meeting. I could't do coding in the car, so I fell back t oreading PickAxe instead, finally finishing it (except the huge library reference parts) at the pre-meeting dinner at a nice Chinese restaurant.

I like the PickAxe book a lot. Its explanation of EigenClasses ("that something", in Matz's words) and MetaModel was excellent, and the ObjectSpace introspection API made a lot of sense too. However, the constant beating of the "better than Java" dead horse can get a bit tiring at times. :-)

Ruby's lack of container types also means it doesn't have the related set of Perlish ergonomic memes -- eg. sigils, autorefs, contexts, ties, inference, guards -- though that also makes it easier to reason about. However, seeing that Perl 6's dynamic semantics is basically "Ruby++", there's probably nothing precluding a Ruby frontend to Pugs/PIL. On the other hand, it may be easier to write a YARV->PIR translator.

The talks were reasonably successful, except I went massively overtime, finishing at almost midnight so people had to run to catch the last trains. Fortunately, by following suggestions from Wendy and Liz about slowing down the pace, the talk apparently worked better; the audience is visibly more receptive than it was at EuroOSCON. I still need to slow down a bit more, leave enough pause time for jokes, and cut most of the digressions, but all in all things seem to be improving.

The Pugs talk was received most warmly, so much so that I said I'd like to get a list of attendees and send a committer bit invitation to all of them. In particular, people seem to really like the idea of turning use pugs; into a CPAN module, that translates your inline Perl 6 code into Perl 5, preserving the original P6 code (but hide them from the Perl 5 parser), and insert the compiled Perl 5 code, with a SHA digest so those parts can be recompiled when the Perl 6 code changes. This technique resembles Acme::Bleach, Inline.pm and Inline::File, and has the effect of letting people write Perl 6 code that can be uploaded to CPAN to be used as pure-perl5 modules.

After the talk, Jouke asked whether Ponie will still be relevant, with Pugs targetting Perl 5 and Parrot simultaneously. The answer is a future-tense yes: the first generation of Perl 6 programs are going to be written with use perl5:DBI; in them, so they will run just fine when compiled to Perl 5. However, when Parrot proves to be a more reliable and efficient runtime for Perl 6, we will then want to compile those Perl 6 programs to Parrot. For Parrot to support use perl5:DBI, Ponie is absolutely neccessary. However, until we make the Perl5 and Parrot targetting robust enough, there would be little practical motivation to hack on Ponie,

Jouke also kindly offered his place to host me in my Erdosing next year; I'd like to work out declarative bindings that work both as GUIs and dynamic websites using the same piece of code, and Jouke's experience with OLE and WxWidgets would be helpful there.

The "Visual Basic Rocketh" talk was controversial as ever. It does feel strange to play advocate for a language with no free implementations, but I think the two main messages -- evolve or die and shared runtimes are good -- was received reasonably well, so I'm still glad I did the talk. I was also quite surprised that someone in the audience recognized the photo of Erik Meijer...

It was past 10am at that point, so I asked whether the mongers really want to hear the 45min "Learning Haskell" talk. Most people concurred, but I then made the mistake of going through every slides in my careful-explanation mode, so it took well over an hour. In retrospect, I should have dropped the nitty-gritty detail of technology demonstrations, and focus on getting the bare basics across instead.

Still, many interesting feedback ensued: Abigail caught a potential infinite loop bug in the partition-based qsort example, and I saw an "Aha" look on several people's face when I explained that print $ f x is the same as print (f x). I could also have went further and show that print . f $ g x is the same as print (f (g x)), but function composition would probably make the overtime even worse. The overall feeling is that this talk really should be turned into a 3-hour tutorial and splitted into two parts.

So, a very interesting (if a bit long) day, and a lot of fun. See you tomorrow!

• leo++ for finishing the variable-sized register frames
• pmichaud++ for getting shift/reduce parser to parse rules itself
• allison++ for announcing the availability of tree transformations (aka Attribute Grammar) and linking to compiler tools document. I love the upfront declaration of Rule number one: writing compilers is not hard., but would like to amend it with ...with the right tools. Read the comptools documents for the set of Right Tools coming to Parrot.

Liz and I keep finding serializable continuations very useful for S17, especially for intospection and development purposes (imagine "stepping back" in your debugger). It seems that the next JVM release may support Rhino-style serializable continuation, which would make it a first-class perl6 target.

Lots of nice feedback on S17. Still need to tackle coro, junctions and hyperops in the face of multiple core CPUs.

Stevan started populating Perl6-ObjectSpace with Real Code (tm).

iblech fixed Pugs parser and internals to deal with the syntax change in the new S06. The commit log provides a succint summary.

Looking forward to tomorrow's Amsterdam.pm meeting.

Note to self: fix the bug reported by xinming: class a { has $.a; method update { $.a; } };class b { has $.a; submethod BUILD { a.new( a => $.a ).update; }; }; b.new( a => 20 );

Wendy and I started annotating the illustrations with simple question/answer notes, captured in the rough sketch that will hopefully become an accessible PA02: Architecture. Pugs has long suffered from the steep initial learning curve; the Apocrypha series was planned to remedy that, but somehow I lost the perspective of explaining things in a concise fashion. To that end, the ObjSpace coding was delayed for a day, but I think it's worthwhile to apply the same Q/A treatment for PIL, Object Space, MetaModel and the rest of Pugs jargons. Feedback welcome!

Now I'm off to Amsterdam.pm meeting, which means lots of communication and less coding for today. Wish me luck!

Yesterday and this morning was a lot of fun, and also very productive.

I had a SubEthaEdit+Skype session with Stevan; we worked out a sketch of the Object Space and primitive operations that supports them. Back when the focus at 6.2.x was about flow control semantics, I designed PIL with Chip and Leo to simplify the possible semantics Perl 6 can take. Now for 6.28.x, the ObjSpace and PrimOps are the logical counterpart for the OO part of the language. We learned whatever we can from Python's PyPy and Ruby's YARV (cf. Online Compiler), as the goal is identical to theirs -- have a shared minimal interface that all runtimes ought to provide, to support a consistent, fast and portable metamodel on top of it.

I also worked with Liz in two design sessions, which produced some rough notes of S17: Concurrency. Many of Liz's Threads::* modules on CPAN are workarounds for Perl 5's principle of most surprise ithread model, and we spent much time making sure that those won't happen under Perl 6 semantics.

Aside from that, we played with the idea of is atomic, is critical and is throttled traits on Code objects, and they seem to capture many key abstractions found in everyday concurrency programs. The idea of providing an unified interface to multiple concurrency models -- lightweight multiplex, OS threads, and process forks -- is also alluring, so we toyed with more bizzare ideas like having $thread.detach promote the thread into a process so it can survive its parent's demise.

The feedback to both texts were so far very positive, although we really need to elaborate the text more. A new set of Visiolization diagrams won't hurt either, as it's about time to update the Cafepress merchandise designs.

Apart from Liz's amazing sense of "Right Thing"ness, Wendy has also been helpful during the design process; not suffering from overexposure to perl5 workarounds and perl6 mindset, she provides the much needed real-time usability feedback. I think I'll need to do a Type System design session with her soon, as a sanity-check for the "incremental staticness" idea of Perl6's -- it is rare that I meet someone with a higher internal clock speed and raw mental capacity compared to mine. :)

Last night eric256 extended the test matrix concept with a wonderfully informative Test Catalog that visualizes the link between tests, synopses, and smoke results.

Also, reading PickAxe and PyPy documentations makes me rethink if Perl 6 is really dynamic enough. For example, would defining &postcircumfix:<( )> for Hash classes do what __apply__ does for Python, namely having arbitrary hash serve as closures? Definitely p6l stuff, although I'll probably run those random thoughts off #perl6 before posting them.

Today I'm going to focus on coding up the lower layers of the object space, and see how much bootstrapping I can do with those bare primitives. It should be fun. Wish me luck!