Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Journal of luqui (5770)

Monday July 09, 2007
06:19 PM

I hate boilerplate

Friday February 23, 2007
11:37 PM

Dwimmery sanity

Friday January 19, 2007
05:46 AM

Perl 6 Rules: Elementary Compositionality + More Vocabulary

Sorry for the bare link: I thought it would be best if I put all my writing in my one blog (which is easily backupable) than scattered around in several.

Perl 6 Rules: Elementary Compositionality and More Contrived Vocabulary

Tuesday May 23, 2006
08:07 AM

The reason I left Perl 6

Sometime around January or February, I suddenly disappeared from the Perl 6 scene. Before then, I was actively discussing, proposing, hacking on pugs, and everything. In an instant, it all stopped. I've had some time to reflect on that, so I'll write it down because it is presently keeping me from sleeping.

I'll first say that it wasn't out of disgust for the language or its process. I like Perl, and I like the direction Perl 6 is going for the most part. The forces at work were much more subtle than that.

Before I begin, I'd like to say that I'm writing this from memory, not checking my records. The dates may be inaccurate, and my comments, particularly those about Damian, are probably emotionally biased in some respect. If I went back and looked at the details, the result would probably be even more mild than I describe here, but the "feel" of the situation is really what is important to this article, so that's how I'm going to keep it.

Designing Perl has interested me for so long because it has satisfied two fundamental desires I have had. First, I could exercise my mathematical creativity (something that is very difficult to do in a math program where you are working with traditional math which has been very well explored). Second, I have discovered that the last few years of my life have been about social artistry. I tried to start the Miki (music wiki) project and I love game design for this reason. The design of Perl, though I did not realize it at the time, is social artistry.

Gradually a tension in the design team arose between me and Damian. I usually credit Damian as the one who taught me to argue, because if my arguments were not rock solid, he would poke holes in them until they were nothing. This was wonderful for certain debates, but it soon became a problem. He and I probably have some different principles of language design, but I was never able to identify where we fundamentally differed. In any case, it came to a point where he would respond in opposition to almost every proposal I had. This isn't an attack on himhe definitely had good reasonsbut it had an effect on me.

I can demonstrate most clearly with the Junctions debate (this was the most accented, but the same thing happened subtly a couple more times). Perl was a work of art to me, and I saw the semantics of Junctions as a big ugly spot. I would propose a solution and be opposed (because the solution had some semantic holes; we don't want to take away functionality). So I would get out my paper and my whiteboard, working on a solution, mulling, exploring, for several weeks both actively and passively until I came up with what I considered more beautiful but still accomplishing the same semantic ideas, in some cases with a little more work (allow an asymptotic constant; they were in the same order of expressibility). Damian would find a yet smaller hole, we'd argue for a bit, then I would go back to the drawing board and start the process over again. When I was not opposed entirely, Larry would usually pick the bits of my proposal that enabled some feature and give a variant of my syntax to it. This didn't satisfy me at all, because my proposals were not about adding a particular feature to Perl 6, but about the beauty of the interplay of some features that previously seemed unrelated.

In the end, I did some of my best work during these minidebates. I had reached my limit on several proposals in November or December, and the holes that were found (not always, but usually, by Damian) seemed small, obscure, and without support (as if the word "useful" were a universally understood and accepted metric not needing support). I respect Perl for not ignoring programming's linguistic heritage, but the virtue of mathematical creativity was extinguished when most of the design team substituted that for attention to programming's mathematical heritage, which also must not be ignored! The virtue of social art was extinguished when the solutions that I had worked for weeks on and I considered truly beautiful lost to syntax bloat and "DWIM tables". It bascially came down to the rest of the team considering Perl much more pragmatically than I did, which may end up better for the end result anywayit is hard to tell.

Yet I persevered until school interfered. In January, I was taken by surprise when two of the three "easy math classes" I took turned out to be very challenging. In particular, the homework for my Set Theory class was usually about four proofs. These proofs were not simple; they required insight, understanding, and ingenuity. Each one usually took an hour or more and ended up being only three or four lines long. This exhausted my brain, but also fulfilled my desire for mathematical creativity. Now I have a masters thesis in the area to work on.

In the middle of February, I started a fortnightly musical jam night with a few of my friends. This group of inexperienced musicians made some vaguely tolerable music sometimes, nothing I would ever pay money for. Yet it became the most exhilarating, most enjoyable thing I had ever done. Very gradually we improved, recruited guest musicians every once in a while, and the more fun it became. Now that school is out again, in addition to these (which are now weekly) I have joined a band of quite competent musicians and am touring around some other jam groups having fun with them. I found fulfillment for my desire of social artistry, and it was musicin particular, improvisation.

Working on Perl 6 was wonderful; I may never again have the pleasure of working with so many polite, diplomatic, intelligent people on one project. Even the conditions that caused me to inadvertently leave were very mild. But it's not too likely that I'll be coming back as a language designer, simply because I've found other avenues for these desires to travel. I'm not the kind of person to multitask in my interests.

Thursday January 26, 2006
04:28 AM

Yesterday's Perl 6 Meeting Notes

Patrick:
        - customer project deadlines keep jumping and scopes keep creeping
        - the deadline is next Wednesday
        - I should be done then
        - did send Allison an e-mail about the operator precedence parser

Jesse:
        - you'll go into the Perl 6 parser then?

Patrick:
        - yes
        - maybe adding some features to PGE driven by parser demands

Larry:
        - also have a work deadline
        - but sneaked in a synopsis change from "till" to "ff"
        - threw some conjectural stuff in S12 to bless ^ as a shorthand for .meta

Luke:
        - I think it's important to keep the difference between a type and a
            package concrete
        - I thought ^ was for types
        - I'll see what I make of your changes and *then* whine

Jesse:
        - P5 to P5 is still about 95%?

Larry:
        - haven't done anything on that in the last week

Luke:
        - enjoying my processor arch class
        - might dive into Parrot to apply some of my knowledge in a while
        - otherwise working on an algorithm and paper to dispatch pure
            multimethods as fast as possible
        - can also incorporate compile-time information if it's available
        - getting those two things at once is pretty hard
        - the algorithm is still in progress
        - I can build a decision tree that dispatches to the right multi-method
            provably with as few checks as possible
        - you can rule out some candidates based on compile-time information
        - you can use a bit of the theory that falls out of that to do type
            inference
        - that's probably a couple of weeks away
        - it might be worth a senior thesis

Allison:
        - Punie almost does simple if statements
        - meeting with Chip in a bit
        - I'll work on operators next
        - it looks like Patrick's solution might fix my problem
        - have about five or six people reviewing AL 2.0
        - don't know if I'll get any comments
        - have one more week on it

Jesse:
        - I asked to have that list created
        - I haven't seen anything yet
        - Chip answered some of Leo's questions on Monday

Luke:
        - thinking about overriding stuff in classes
        - attributes that have a constant value through the lifetime of an object
            are different beasts from attributes that can change
        - I'm searching for a way to make that distinction without imposing
            horrible cognitive or finger blocks on people

Larry:
        - like our readonly versus contant distinction?
        - on ordinary variables
        - bound in a constant way
        - but not a constant
        - different iterations of this function may have a different value
        - can't optimize it away in that sense

Luke:
        - the fact that attributes aren't read-only inside of an object causes
            problems with inheritance
        - that distinction may be nice to have if it works cognitively

c:
        - is that an implementation detail not exposable?

Luke:
        - more about the intent of its use rather than its actual use

Thursday January 19, 2006
09:13 PM

Yesterday's Perl 6 Meeting Notes

Luke:
        - I started school today
        - I have five hours in a row of classes
        - started to learn Scala outright, as if I were to program in it
        - I read three papers about its cool object model
        - it seems that it has every cool idea I've already thought of
        - the idea that a role isn't just an object really seems like a good idea
            to me
        - they're not just for composing into classes
        - they're for composing into anything
        - a collection of stuff
        - I don't see any downsides except for the lack of strict subtyping
        - but there's a neat result: parameterized classes
        - they're the same as virtual classes
        - just make roles purely compositional
        - not subtyping things
        - eventually I'll write a document on my thoughts

c:
        - why do you lose subtyping?

Luke:
        - if you can compose anything, you can compose class names
        - you can compose those, you lose subtyping based on how that class is
            used in the rest of that role
        - if you use the virtual class for argument types, you have to have a
            contravariant relation

Larry:
        - busy with my day job
        - just keeping up with the mailing lists
        - I have another week or two of heavy duty work
        - eventually I'll get back to the translator
        - I have to prepare a talk for it for OSDC Israel
        - also thinking through how to make a lower impedence mismatch connection
            between my AST and some of the Pugs projects

Jesse:
        - is it worth throwing random ASTs at them?

Larry:
        - not yet

Jesse:
        - did they convince you to visit Japan?

Larry:
        - didn't take much convincing

Allison:
        - haven't touched Punie in about three days
        - I finished off comma lists since last week
        - still haven't heard from Patrick
        - have other things I need to fix too
        - I might start on functions before operators
        - besides control structures, that's about it

c:
        - don't forget regexes
        - they gave me trouble

Allison:
        - I might just implement a compatibility layer

Jesse:
        - how's the licensing going?

Allison:
        - I'll release something within two weeks, even if I don't have comments
            from everyone I asked to comment

Jesse:
        - sounds good

Damian:
        - working frantically to write a keynote for next week
        - involved a lot of photoshopping Peter Jackson
        - also negotiating with Japanese folks for the YAPC there
        - preparing for the talk has thrown up a lot of questions for the mailing
            list
        - I'm still curious about a special way to declare class methods
        - Larry seems to be keen not to tie them down
        - there are a lot of people who'd like the option

Larry:
        - it can always be the option

Damian:
        - I'm trying to show syntax, though
        - I could just have a blurry screen
        - maybe I'll just show it with some fake syntax

Luke:
        - I couldn't come up with a good single word for that
        - Rob Kinyon kind of has a point with the polymorphism there

Larry:
        - more of a cultural one, as far as I see it

Damian:
        - no doubt about that
        - there are plenty of people who want to leave Java without leaving the
            Java mindset

Larry:
        - it'd be interesting to see how much information the type inferencer
            gives us for free

Luke:
        - it would be wise not to rely on it
        - there hasn't been much research into inferencing these type models

Damian:
        - even inferrence doesn't mark them physically in the syntax
        - people like that

Luke:
        - it's like Haskell
        - you don't have to put explicit types on your functions
        - but people do -- for documentation and future-proofness

Damian:
        - and checking your logic of the system
        - you're showing what you believe about a system

Luke:
        - Haskell will catch you eventually
        - but it won't point you to the right place without prototypes

Damian:
        - I almost had a bad moment and suggested "state"

Luke:
        - I don't understand how partially undefinedness supports prototype-based
            OO
        - replying to Rob's thread might answer that

Larry:
        - I'll think about that

Jesse:
        - fighting with customers mostly this past week
        - went to the GPL v3 thing Monday and Tuesday
        - talking to potential Ponie contributors, but one fell through
        - two people are interested in hacking
        - they asked what there is to do
        - Jerry Gay is poking at getting a Windows box for regular Parrot builds

Thursday January 12, 2006
07:43 PM

Overrrrrriding

I'm toying with the idea of "composable modules" for Perl 6. It all started with the combinator library for Parse::Rule in the pugs distribution. I wanted to keep a uniform interface for combinators, but keep the options for a choice of evaluation strategy (CPS, Parrot JIT compiled, etc.) and choice of medium (utf8 string, array of objects, etc.).

The way I ended up doing it was to use stateless objects as modules. That is, to create combinators for a text medium with a CPS strategy, you would do:

class TextCPS {
    does Strategy::CPS;
    does Medium::Text;
    # implement one or two required combinators
}
my $c = TextCPS.new;
$c.quantify($c.any_char, :min(4));  # / .**{4...} /

These roles used virtual classes all over the place. For example, the Medium role, which holds medium-specific combinators, has to refer to a Pos class, which reperesents a medium-specific position in a stream. However, that Pos will change based on what specific medium implementation is in use, so the Pos class is virtual.

But since the combinator objects don't have any state, they're really more like modules: collections of functions. But we still want composable behavior.

But the thing that's been on my mind recently is the overriding problem. What are you, and what are you not, allowed to override? You can clearly override functions, as long as the new function's type is a subtype of the old function's type. That's what OOers have been doing for ages. It also makes sense to override constants, since they're just functions that take no arguments, as long as the subtyping relationship holds.

But what about variables? For variables that can be assigned to and read from, the type must be exactly the same, or more generally, isomorphic (A and B are isomorphic if A <: B and B <: A). Type isomorphisms are something that I've wanted in Perl for a while, but never really figured out how they would work.

There's a pattern here: if you read from it, new <: old; if you write to it, old <: new. But what about classes? A class is just a kind of constants. By the pattern, the overriding class must be a subclass of the overridden class. But I don't think that's quite accurate. For starters, it's wrong, because:

role Foo {
    class Bar {...}
    method foo(Bar --> Int) {...}
    method bar(Int --> Bar) {...}
}

Here, you can't override Bar with a subclass of itself, because then you lose Liskov substitutability on foo(). You can't override Bar with a superclass of itself, because then you lose Liskov substitutability on bar(). So if you do override Bar, it has to be an isomorphic class.

That's because when you override a constant, you require not that the new constant is <: to the old one, but the new constant's type is <: to the old one. So really we have to require that the overriding class's type is a subtype of the overridden class's type. As far as I recall, types of types are kinds, and all non-parameteric classes would all have the same kind, so that trivially holds. But clearly that's not the only constrant that is on these special constants.

The constraints come from the usage of the class within a scope, which is a little bit scary (mandatory inference kinds of things). We could be like Scala and allow you to specify the relationship that must hold, but that's requiring that the user know a little too much about type theory. The constraint might just be that the class doesn't cause any other subtype constraints to be violated in overriding. That is:

role Foo {
    class Bar {...}
    method foo(--> Bar) {...}
}

Our constraint is that for any subclass X of Foo, typeof(X::foo) <: typeof(Foo::foo). So if you override Bar with a supertype of Bar, then you break the constraint, so that isn't allowed.

Stepping back to the original question: what is a module? I'd say a module is a collection of virtual things that refer to each other. It is closed, so you can't add to the collection; you can only derive new collections based on it. The constraints are exactly Liskov substitutability constraints.

Actually, the constraints might be more liberal than Liskov. It might just be that a derived module still has to be well-typed. That is a characteristic of a code-reuse construct rather than a typing construct. That characteristic might be something to think about with roles, too.

Wednesday January 11, 2006
08:40 PM

Today's Perl 6 Meeting Notes

Damian:
        - things are looking up
        - I took a break, which helps
        - first week back at work from the near year
        - lots of mail
        - I'll be in Asia and Tokyo in March
        - otherwise enjoying the holidays
        - I've been tracking Larry's updates
        - intend to send more feedback soon
        - I'll be at a Linux conference in New Zealand the week after next
        - I'll have time then

Patrick:
        - lots of non-Perl busy-ness here
        - will pick up on PGE again tomorrow
        - another project moved up a deadline, so that was busy
        - should free up some time tonight

Jesse:
        - what's the next checkpoint there?

Patrick:
        - actually writing a parser for Perl 6 code
        - I have the various pieces in the constellation to bring them together
        - figuring out what to do with the parse trees comes after that
        - but we can generate parse trees
        - I'll also be on #parrot again

Allison:
        - working on extending Punie still
        - Patrick, I mailed a question about the operator precedence parser
        - working on comma-separated expressions
        - discovered an interesting thing
        - you can set up a PGE grammar to make it easier or harder to write tree
            transformations
        - the Perl 1 style of recursive rules makes trees harder
        - at least, harder than repetition within the rule

Luke:
        - what are you turning things into?

Allison:
        - some things need flat lists
        - some things don't, but flat lists are easier to iterate over

Patrick:
        - it can make a huge difference
        - it's nice to have both options with a nice syntax
        - is there a reason not to treat comma as an operator?

Allison:
        - not sure
        - I'll decide which way works best later
        - but I suspect repetition may be simpler

Patrick:
        - my plans for operator precedence allow list-associativity
        - so it may work easier

Larry:
        - Perl 1 had a lot of hard-wired things in its grammar
        - Perl 5 was more flexible

Allison:
        - definitely interesting and informative

Larry:
        - various bad luck
        - desperately trying to follow along on everything at once
        - not really responding
        - my day job is really busy for the next couple of weeks
        - haven't done anything on the translator
        - I'm letting that gurgle in the background

Jesse:
        - what's your goal for the translator?
        - 100% test suite?

Larry:
        - I don't know that it can ever get entirely there
        - but it's effectively there already
        - I don't know how much I'll have to warp it
        - I'm pretty confident I'm getting all of the information out
        - I don't know if I'm getting it all out in the most convenient form for
            the translator
        - it'd be good if someone could merge it back in to the Perl 5 mainline
            at some point
        - my impression is that Perl 5 is not mutating rapidly at this point
        - not a big rush

Luke:
        - what are its effects on parsing speeds?

Larry:
        - very little
        - only a few conditional tests if you don't want the extra information
        - a separate grammar file with extra goodies
        - just a wrapper around yyparse mostly
        - that was part of my original spec
        - do need some stuff in the tokenizer, but they're guarded by conditionals

Luke:
        - not much this week
        - thinking about generalizing the design of Parse::Rule into useful
            language constructs
        - I had to pull some dirty design tricks
        - talked a bit on p6l about composable modules
        - now thinking about building an object and type system out of them
        - haven't had much time to work on it
        - but it looks promising
        - my final project for cognitive science was on tree adjoining grammars
        - they seem appropriate for our metaoperators
        - instead of building a separate operator for each metaoperator-operator combination, you
            could set that up as an auxiliary tree in that theory
        - you can parse them top-down
        - as long as you have parameterized rules
        - I wonder if I can massage that into a grammar that actually works
        - does PGE do parameterized rules?

Patrick:
        - it only understands strings so far
        - subrule name, colon, space -- everything else is an argument
        - from Apocalypse 5
        - maybe never blessed into a Synopsis
        - official syntax is subrule( list )
        - PGE doesn't understand that yet

Jesse:
        - spent two weeks mostly not working
        - ended up writing some code, which was weird
        - heard from Chip on Monday; he's back now
        - have had some interest in Ponie pumpkining
        - is anyone blocking on anything external?

c:
        - what's the legal documents status?

Allison:
        - it's probably time for the public review

Luke:
        - Audrey is prodding us to come up with tagged unions
        - the theory.pod syntax isn't very nice
        - can't put my finger on why
        - I looked at OCaml's tag types
        - put a tag on a type and get a new type
        - then create unions of types and get a tagged union
        - but I've never seen them used in practice
        - I wonder how well they fit common uses

Larry:
        - the type becomes a value bit of the new tag
        - used as a discriminant at runtime, if not earlier?

Luke:
        - I think so
        - Perl has scalars where they are kind of deranged tagged unions with more
            than one tag at once
        - but ignoring that...

Larry:
        - I'm agreeable to the idea that unions should be discriminated
        - the C idea of unions is a complete botch
        - there has to be some way of telling them apart
        - if they're objects, they have their own built-in identity
        - but I'm not a great type theoretician

08:35 PM

Last Week's Perl 6 Meeting Notes

(Sorry, forgot to post these last week)

Luke:
        - until last Wednesday, I worked on PGE in Perl 6
        - then I worked on something completely unrelated
        - PGE in Perl 6 now, barring the operator precedence parser, does
            everything that the Parrot one does
        - except for parsing rules
        - but Audrey is doing that

c:
        - does it produce match objects like PGE does?

Luke:
        - yes
        - last week I thought about things
        - XQuery, tree matching
        - pondering a data comprehension module for Perl 5
        - whatever can make processing big, complex data structures quick and easy
        - working with Audrey on context and coercion
        - we have some disagreements about how that works
        - mostly we're on the same page
        - wrote a document describing pure MMD scheme
        - including return type dispatch
        - related to context
        - those are in the Pugs repository
        - I'll hack on them and then pass them along

Larry:
        - did about 18 refactorings of the Perl 5 to Perl 5 translator
        - ends up with a more adequately typed AST

Luke:
        - typed how?

Larry:
        - I'm not quite sure what I mean either
        - a lot of the stuff internally to Perl is implicit
        - there's no name for a particular node in the tree
        - they happen accidentally
        - we rearrange kids somehow and change a few flags and it applies
            differently
        - trying to name everything, or at least get close
        - they're types in that sense
        - it's not yet well-typed in the sense that I've looked at those names
            and tried to figure out what's common between them
        - it's nothing like PIL
        - it's just trying to give a name to everything so we can have a hope at
            translating at some point
        - basically there are more node types currently than there are operator
            ppcodes in Perl 5
        - there's now a Statement, which doesn't really exist
        - there's an opcode for transitioning into a statement in Perl 5
        - but nothing really contains a statement
        - that's a lot of the refactoring work
        - make a more ordered tree
        - now, of all the tests in the core modules, it translates more than 99.5%
            correctly
        - I haven't tried CPAN; I'd have to download it
        - I also haven't bothered to try to hook this up to bleadperl
        - I can probably delegate that eventually
        - most of these refactorings come about due to thinking about what I need
            to translate to Perl 6
        - making sure that the information is there
        - haven't actually tried to translate anything yet
        - eventually I'll get up the courage to translate $foo[] to @foo[] and go
            from there
        - otherwise, thinking about what people say on the mailing lists
        - admittedly not being very responsive to that

Allison:
        - planning to take time off around Christmas to do development
        - but I had my wisdom teeth out
        - started working on expanding the Punie grammar again
        - made it into sexpr part of the original perly.y
        - ran into the traditional left recursion problem
        - decided to use PGE's operator precedence parser instead

Luke:
        - with regard to left recursion, there's a simple transform you can do
            with closures to turn right-recursion data structures into ones that
            look like left recursion

Allison:
        - rather than doing left recursion, I flipped around
        - broke it into two pieces
        - the first element, and the rest of the expression

Luke:
        - x [\+ x]* ?

Allison:
        - expression matches
        - expression = expression
        - expression = expression + expression
        - instead, say expression = term + rexpression

Luke:
        - that's my biggest complaint about Parse::RecDescent
        - it has a nice leftop which does that
        - but then you get back a flat structure of all the matches you want
        - I want a left recursive structure
        - it's easier to parse things that look left-recursive
        - you only have to handle two arguments, not N
        - left-recursion transform might be nice in grammars, if it's generic...

Allison:
        - for now, all I need in Perl 1 is operator precedence
        - looks pretty cool so far
        - I'll try to finish it off by the end of the day
        - it'd be nice to have comments in the syntax of the independent grammar,
            but...

Luke:
        - my friend the crazy C++ programmer is trying to learn a new language
        - I invited him to learn Perl 6
        - I'm going to try to teach him Perl 6 without any past knowledge of
            dynamic languages

Wednesday December 21, 2005
06:49 PM

Today's Perl 6 Meeting Notes

c:
        - Stevan and Audrey are working on PIL2 minilanguage
        - they want to talk about how to optimize it
        - I'll corner Allison and have her show me how to hook it up to TGE

Allison:
        - probably won't have my Parrot day until Friday
        - I still plan on working next on supporting more operations in Punie
        - probably will do a whole pile at once
        - I need to genericize it not to parse on the name of the operation
        - "there's an identifier here"
        - then add a step to do semantic checking
        - check for only valid operations

c:
        - sounds like what PIL2 needs

Allison:
        - very co-useful

Jesse:
        - is Roberta back in town?

Allison:
        - we're supposed to talk to a Perl-friendly company soon
        - not sure if it'll work during the holidays
        - will send out a few more review copies

Luke:
        - I'm porting PGE to Perl 6
        - it's really fast
        - I can do the fundamental operations in an hour
        - haven't found the easiest way
        - will probably look at how PGE does it
        - using CPS, as it's easy for the easy stuff
        - the general quantified style isn't as easy that way
        - coming along a really well
        - Perl 6 is a nice language to write in, compared to PIR

Jesse:
        - is this as an exercise for fun?
        - is it for a chunk of Pugs?
        - does it eventually need to happen?

Luke:
        - all three
        - our best backend, which is JavaScript right now, doesn't even have rules
        - that needs to happen to get stuff out fast

Allison:
        - if the lowest level you guarantee to support is the HLL, you have to
            implement a lot of stuff in the HLL to support it everywhere

Jesse:
        - what's your status of outstanding design decisions?
        - it'd be nice to get a concrete list of missing design
        - what we need for 6.0
        - it seems like that'll become a list of blockers soon
        - we're getting runnable implementations that run