use Perl

The Perl 6 design team met by phone on 30 June 2010. Allison, Patrick, and chromatic attended.

Allison:

• working on Parrot packages for Debian experimental
• seems like a good idea to do that before the 2.6 supported release
• there was also a request for Rakudo packages
• not sure if I'm the best person to do it

Patrick:

• I'm sure we should package Rakudo Star

Allison:

• Debian had a packager for those, but I haven't looked at the packages
• this'd be an early run of what we'll do with Rakudo Star

Patrick:

• we're not quite ready for packaging that yet
• maybe a couple of weeks
• finished the List and Iterator types for the #30 release
• adjusted Rakudo's Associative and Positional roles
• much cleaner implementation now
• that'll require a few small spec changes
• redid Rakudo's container types
• more robust
• preparing for autovivification of hashes and arrays
• expect to finish those in the next couple of days
• there was no container model previously; the code was consequently crufty
• lots of cleanup of incorrect assumptions
• Rakudo lists are now properly lazy
• comment syntax fixed
• ROADMAP updated
• fixed the meaning of Nil; it's defined, not undefined
• added the sink prefix (?)
• fixed setting of $!
• started fixing bugs and closing tickets on Monday, did 15 or 20
• mostly already fixed in the previous couple of weeks
• looking at the implementation of the series operator
• spec is self-contradictory or ambiguous or both
• waiting for Larry's clarification
• fixed a bug in $*ARGFILES
• had a nice contribution of that implementation last week
• that behavior works on any set of files, not just those on the command line
• working on autoviv
• have some regex backtracking bugs to fix
• will work on closures after that
• put together three new YAPC presentations
• the Rakudo Star presentation will become a video cast or a blog post or both

c:

• worked on a slew of Parrot optimizations for Rakudo
• have a few more to go
• might have to create a Rakudo branch temporarily
• will try to help merge the new GC
• working on a metamodel for Parrot objects, informed by Perl 6 and Moose

This took longer than I expected, but the draft of the Modern Perl book is available for review. I'm especially interested in hearing from people who don't consider themselves expert Perl 5 programmers. The goal of the book is to explain how Perl 5 works (and how to write Perl 5 effectively) to help novices become adepts.

The Perl 6 design team met by phone on 16 June 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

• documented TOP (again), and explained how parsing is initiated and how it actually works
• series operator (...) now picks a monotonic function when using single characters as endpoints
• STD can now catch duplicates involving protos as well as onlys
• STD no longer advises removal of parens on spaceless sub() declaration
• mostly advised sorear and pmichaud
• Stefan is finishing the boostrap of the STD parser
• also working on adding a parallel NFA and DFA engine
• no, he doesn't want to generate all the states in advance
• it works faster lazily

Allison:

• working on chroot environments with something more secure than chroot
• relevant to building Parrot packages
• looking at some bugs for Will

Patrick:

• Rakudo developers decided not to make extra special effort to make a June release of Rakudo Star
• the calendar works against us
• the new date for the release is July 29
• we're I comfortable with hitting that target
• we won't be happy with the results of moving heaven and earth to release in June
• there are lots of advantages
• one disadvantage is not having Rakudo Star at YAPC::NA
• one big advantage is using the supported Parrot 2.6 release as the basis
• I'll write a post outlining the plan in the next couple of days
• otherwise working on lists and interators in Perl 6 and Rakudo
• after deciding to make iterators immutable, Larry and I realized that solves many problems
• everything works out as plain as day after that
• very happy with that design
• the incorrect assumptions of the old model were pervasive
• replacing the old pieces is taking a while, which is no surprise
• this approach feels right though
• the new branch does things no previous version could do
• slices work much better, for example
• metaoperators work properly
• map is lazy
• slurpy arguments in lists are lazy by default
• no weird binding or action at a distance problems
• plenty of changes to Associative and Positional roles
• those are now super clean and may be lazy
• more features work
• ~30 failing tests (not test files, just tests) now, ~500 last night
• most of the current failures are minor
• will try to merge the branch before the release
• replacing lots of ugly code with fewer lines of elegant code
• Jonathan and others have worked on lots of other pieces
• adding plenty of new features
• looking forward to tomorrow's release

c:

• editing the Rakudo book
• moving the Rakudo release date may let us have a printed book available about the same time
• depends on how much there is left to write

The Perl 6 design team met by phone on 09 June 2010. Larry, Allison, Patrick, and chromatic attended.

Larry:

• not much spec change this week
• figured out a syntax for a regex block to return more than one cursor
• based on gather/take
• in STD hacking, continued to assist Stefan O'Rear in getting STD bootstrapped via viv
• now that it's bootstrapped, we're refactoring things that make sense now
• we're now starting to move bits of Cursor code from Perl 5 into Perl 6
• refactoring the grammar for sanity of design
• started upgrading STD to normal Perl 6 syntax where it previously catered to gimme5's limitations
• for example, switched STD's old .<_from> and .<_pos> hash lookups to using .from and .pos accessors
• started the prep work for moving EXPR out of STD to make it generally available to any grammar wanting operator precedence
• in STD parsing, made Perl 5 $< detection have a longer token to avoid confusion with match variables
• STD no longer attempts two-terms detection on infix_circumfix_meta_operator
• STD now parses >>R~<< correctly, or at least dwimmily
• STD doesn't complain about P5isms in printf formats like "%{$count}s"
• STD was parsing /m and /s with the opposite semantics
• termish now localizes $*MULTINESS in its scope so that inner declarations aren't accidentally multified
• STD now carps about package Foo; as a Perl 5 construct

Allison:

• talked to Chris Shiflett, a PHP developer, on someone from the PHP community to sit on the Parrot board
• will be in the US for a few weeks

Patrick:

• working on list simplification
• had a couple of breakthrough ideas on Monday
• working on the implementation now
• worked out inversion lists for character class matching in regexes
• will make them faster, especially with long ranges of character classes
• fixed a half-dozen tickets in RT
• fixed Rakudo hash constructors
• fixed an intermittent bug with colon-pair signatures
• two possible parses exist in STD, but we removed an unneeded one in Rakudo
• fixed a bug with Parrot's exit opcode
• NQP and PAST needed an update not to cheat with PASM constants
• I fixed that too
• Vasily added multisub and multimethod support to NQP, that was a big plus
• fixed the ** quantifier in regexes to understand surrounding whitespace
• regex engine tried to match beyond the end of a string, so I added guards for that
• will work on lists furiously before the next release
• I don't think it'll take long
• closures are next, hope to have those in place by the weekend

c:

• released a new version of Pod::PseudoPod::LaTeX to support the various books in progress

The Perl 6 design team met by phone on 02 June 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

• mostly, I supported sorear in bootstrapping STD to use viv instead of gimme5
• his stage 2 and stage 3 now output identical Perl 5 versions of STD
• produces a huge amount of warnings
• appears to require Perl 5.12 at the moment
• working on both of those
• S03 refines hyper dwimminess to be more like APL, with modular semantics
• S02 refines Blobs to simply be immutable Bufs, with similar generic characteristics
• S02 now describes native blob types
• implemented post-declaration checks for BEGIN and use, since those can't wait for end of file
• STD no longer loses existing bindings when we go to a sublanguage
• STD now uses $*GOAL variable only as informative, never as a "stopper"
• instead, we create a <stopper> rule for $*GOAL if necessary
• can check for that only, instead of that or $*GOAL
• answering lots of questions on how STD and viv work besides that

Allison:

• did a lot of research on graph color algorithms for register usage algorithms
• will finish my finals on Monday

Will:

• trying to herd the discussion of dynop libraries
• a recent branch to close an old ticket broke a lot of assumptions
• some bugs have become more visible because of these changes
• hope to get that cleaned up this week

Allison:

• I liked your suggestion of bringing back the getstderr and related opcodes

Will:

• trying to resurrect Partcl
• stuck on a TT #389 closing issue
• not sure how to fix that, the way things are now

Patrick:

• working on the iterator and list design
• brainstorming the implementation
• will implement somethine one way or another this week
• people keep implementing workarounds for the current system
• they'll bite us eventually
• Moritz and I worked on making the regex engine returning real Perl 6 objects
• that mostly works
• exposes some places where lists don't work exactly right
• the workarounds there made me replan the list and iterator implementation
• answered some questions online
• Jonathan added a better backtrace algorithm for Rakudo
• reports Perl 6 source lines instead of PIR lines
• I'll review his code
• think I can borrow it for NQP for all HLLs
• Jonathan reports that it was a lot easier in NQP than PIR

c:

• trying to answer a few Parrot design questions
• looking at the continuation of design from Perl 1 - 4 to Perl 5 and Perl 6
• hope to have coding time soon

The Perl 6 design team met by phone on 26 May 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

• :() syntax is now always signature
• we now use foofix:[...] as the general op form instead of foofix:(...)
• refactored the sematics of :nth and :x
• :nth() now only ever takes a monotonically increasing list
• S03 now explains how "not-raising" works on != and ne
• it now basically matches the intuitions of an English speaker via HOP definition of negate metaop
• STD sometimes didn't require semi between statements
• statement modifiers are expression terminators but not valid statement terminators
• an unexpected statement modifier word like if could terminate one statement and start another
• fixed up backslashes in character classes to allow \s etc and reject \u etc
• STD was accidentally using the same lexpad for different multis
• Cursor now treats :() on name extension as a signature always, never as a categorical
• we shouldn't introduce the stopper for circumfix until we're in the circumfix, or we can't use the same char on both ends
• placeholder messages error messages are now much more informative and correct
• we now disallow use of placeholder after same variable has been used as a non-placeholder, even for an outer reference
• renamed add_macro (which it doesn't) to add_categorical (which it does)
• participating frequently in discussions on semantics both on irc and p6l
• working closely with sorear++ as he brings viv closer to bootstrapping, yay!
• soon can bootstrap past gimme5

Allison:

• worked on Pynie this week in my limited spare time
• one goal is to generate the parser directly from the Python grammar
• wrote a small, lightweight PEG parser which generates a match tree from the Python 3 grammar
• can generate a lexer directly
• right now it creates a parse tree
• looks similar to the match nodes of NQP-rx
• dumps out a tree to the PIR parser
• working on PaFo elections for next year, but trying to delegate those
• will have more time after June 7

Will:

• working on Perl 6 advent tests
• many more people are doing more work than me
• liasing with Rakudo folks for any important Parrot bugs before the Rakudo Star release
• my current direction there is "don't break anything"

Patrick:

• sorear added hash flattening to NQP
• lots of work on closures in PAST and NQP
• they properly clone
• fixes some lexical problems
• need to get that to work in Rakudo
• that's tougher; Rakudo has to wrap Parrot subs
• wrapper object needs cloning as well, along with its attributes
• we'll add a new PAST node type to help
• that node understands contexts
• essentially a way to add void context optimizations to your AST
• that solves many problems in Rakudo beyond closures
• added a setting into NQP along with its test suite
• not automatically loaded, but available
• contains standard hash and array methods
• Parrot's ops2c project uses those
• other people can update and enhance that setting as necessary
• NQP also has the ability to parse type names
• NQP doesn't do anything with them yet
• eventually they'll allow the use of multis
• cleaning up some NQP bugs regarding lexicals and package storage of subs
• Bruce Keeler enabled variable interpolations in regexes
• working on some refactorings to simplify that approach
• works in NQP and Rakudo now
• that's a feature we've never had before
• Rakudo's REPL now works better, thanks to sorear
• HLLCompiler now written more in NQP as part of that
• NQP now can do eval
• NQP remembers lexicals in interactive mode now
• adding that to Rakudo is more complex
• working on that
• pleased with the progress on #perl6

c:

• reviewing long term plans for GC and Lorito
• should have more time free soon

The Perl 6 design team met by phone on 19 May 2010. Larry, Will, and chromatic attended. Patrick added his notes later.

Larry:

• S03 makes more explicit that doctrine that ~~ topicalizes, and removes smartmatch table fossils that automatically fall out from that
• S05 renames 'accent' to 'mark' for better Unicode conformance
• :a and :aa changed to :m and :mm
• S05 disrequires retroactive semantics on :samecase and :samemark
• the method form must now explicitly add case or mark modifiers to the pattern
• regularized mm// to ms// to avoid confusion with new :m ignoremark option
• STD now does a bit better at diagnosing bogus ??!! constructs of various sorts
• STD now correctly adds operators to symbol tables as subs
• CORE.setting now has protos of all the operators so they can be recognized as subs too
• Cursor now canonicalize operator names in the symbol table
• btw, not quite like specced
• STD now reads user's mind on 'Str $toto' to intuit missing declarator
• STD now properly diagnoses a typename between routine declarator and sub name

Will:

• working on code for Carl Masak, trying to get his poker code example running on Rakudo
• both fun and frustrating
• some stuff doesn't quite work yet
• going through the Advent examples
• adding them to spectests
• make sure we won't regress on such public examples
• other people are helping with that now

c:

• will get back to editing the Rakudo book soon
• hope to have it in print by YAPC, but no guarantee

Patrick:

• fixed closures in NQP, as a precursor for fixing them in Rakudo
• worked with sorear on REPL in Rakudo and PCT in general
• ported the NQP "standard library" done by japhb++, bacek++, and many others into the nqp-rx repository and made it part of the standard build sequence for nqp and Parrot
• decided we need a new "context sensitive" node type in PAST, will be used to create proper closures and to handle sink context
• worked with bacek on adding better multimethod support to PAST and nqp-rx
• discovered a problem with lexical subs in NQP being automatically entered into the package namespace (and some existing code relying on this behavior)
• did some initial fixes to at least get things entered properly, but a complete fix may require a deprecation cycle
• plan to review others' patches this week
• plan to fix REPL, closures, and sink context in Rakudo (since those are currently large pain points)
• plan to work on loops and iterators after that

The Perl 6 design team met by phone on 12 May 2010. Larry, Allison, Patrick, and Will attended.

Larry:

• clarified usage of brackets around infixes
• added various 128-bit types to the spec; we might make them arbitrarily extensible via role
• at least LLVM could support this, even to non-powers-of-two sizes
• modernized the paleolithic grammatical category description in S02
• STD now uses double-quote rules for interpolating @foo[] into regex
• STD now gives better message on 1__3
• added the specced 128-bit types to CORE.setting
• added minmax function to CORE.setting
• implemented circumfix:«X Y» as grammar derivation
• currently only allows a >> inside
• now also recognizes foofix:("\x[face]") and foofix:("\c[YOUR CHARACTER HERE]") without actually evaluating
• playing with factoring yaml out of gimme5, since viv is not likely to go that route.
• mostly just answered a lot of questions on irc
• egged people on about concurrency issues

Patrick:

• thought on handling closures properly
• have a solution, just need some time to implement
• discussion on changes to CodeString
• work on compiler toolkit to avoid CodeString, using StringBuilder instead where possible, in PCT, NQP, and rakudo. Pretty easy, no downstream projects block on a deprecation issue
• after that, lists
• also been answering questions on interactive mode (REPL) for rakudo et al. (the issue with losing lexicals)

Allison:

• resolved the git conversation pretty well (for Parrot's repo migration)
• worked on a pure PEG parser (following the paper), straight PIR, single day; now self-parsing. Interesting project, is lightweight. currently has memoization, but that might not be right for us because of backtracking. With some more effort, could probably handle EBNF form (useful for python)
• could be setup for developer status for Debian which will improve our packaging status for Debian and Ubuntu

Will:

• Parrot CodeString performance improvements
• we're definitely faster in branch, but some feedback from pmichaud should help us clean up the API a bit as well, look for those to hit trunk in the next few days
• Parrot makefile deps cleanup

The Perl 6 design team met by phone on 05 May 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

• various spec updates, some major
• removed p5=> description because it's not supported in core
• deleted self:sort construct because self isn't a real syntactic category
• explained Perl patterns in terms of PEGs, and spec'ed tiebreaking rules explicitly
• last but not least, finally purveyed the long-threatened revamp of proto to keep routine and method semantics similar
• they all now work much more like the multiple dispatch semantics currently used by STD, where we always call the proto first
• the proto is then always in charge of the actual multiple dispatch; it can of course delegate that
• and the default for a null body corresponds closely to current semantics
• in hacking news, the lexer generator mislaid any alternative that was a bare . pattern, so cursor_fate never called its alternative, oops
• took me a long time to run that one down, because it resulted in a horrendous backtrack causing mysterious misplaced errors
• revamped character class parsing to be more helpful and correct
• STD now check a normal regex bracket's innards for old-school character class, and warns if found
• added a .looks_like_cclass method to Cursor to detect most accidental uses of P5 ranges
• some valid P6 brackets will complain, but the workarounds are easy
• just put whitespace on both ends is one way
• removed a few of these old-school-ish character classes from STD
• changed :tr language to :cc language since character classes share it
• (translation pays more attention to ordering, but the language is the same)
• turned out parsing character classes discovered issues in STD; various character classes needed to backslash # that would otherwise be a comment
• to that end, we now allow \# in character classes instead of misparsing as unspace
• if we find an invalid - in a regex, we now presume we're in an old-school character class and fail with a sorry instead of a panic to give the character class code a shot at it
• STD now uses ~ syntax for regex brackets to set $*GOAL correctly
• cleaned up recursive panic detection; it was possible to get both false positives and negatives before
• STD shouldn't use 'note' to emit a panic inside a suppose because that leaks the message that should be trapped
• STD now suppresses duplicate sorry messages more correctly
• sorry no longer uses panic in a supposition, but dies directly to throw the exception to the suppose's try block
• STD now allows subscripts on regex variables so $x[0] isn't taken as a character class; still needs speccing

Patrick:

• can we make them consistent?

Larry:

• historically S05 has allowed bare arrays to mean interpolation

Patrick:

• we've never had a working implementation of that

Larry:

• a bare @ would be illegal

Patrick:

• it's currently illegal

Larry:

• you'd have to backslash it to match part of an email address
• it's not like the @ alternations are a big deal one way or another
• that'd be a little more consistent
• I forced it to think of the sigil as $ than what it really is

Patrick:

• after seeing how Jonathan et all did interpolation for quoted strings, I thought we should do the same thing in regexes

Larry:

• STD now has a partial fix to prevent leakage of ::T from role signatures
• unfortunately, the current fix will lose signatures of file-scoped generic roles
• this probably has to do with not knowing whether we're really going to want a new pad; unfortunately we'd have to look ahead to know that currently
• various other minor tweaks and bug fixes in STD and Cursor

Patrick:

• mostly responding to messages and reports
• should be able to get back to coding full-time and online for the next week
• plan to resolve the list and closure issues with NQP and Rakudo
• will answer other questions and try to keep other people productive
• planning for the Rakudo Star release on June

Allison:

• busy with the last week of classes
• spent most of it writing a little language with PCT
• it was easy to use and easy to swap the stages of PCT
• I remembered what Patrick did with LOLCODE
• also had a discussion of source code control systems
• next week should be more productive
• need to work more closely with Debian packagers to get packages into Debian

Will:

• cleaning out as many deprecations in Parrot as possible
• trying to improve the speed of CodeString after the immutable STRINGs merge
• bundling lots of little concats helps
• hope to merge in an optimization branch for that by the weekend
• want to make that faster or less memory intensive
• may require the use of a new StringBuilder for Parrot
• hopefully will result in a faster Rakudo build

Patrick:

• I've never seen CodeString take a long time
• unless you run into memory problems

• * discussion of the StringBuilder PMC *

c:

• still working on optimizations, particularly CodeString
• looking at more PBC and PBC-building optimizations
• PBC size went down dramatically and startup improved for Rakudo
• should have that much faster for the 2.4 release
• will poke at GC tasks starting next week

The Perl 6 design team met by phone on 28 April 2010. Larry, Allison, Jerry, Will, and chromatic attended.

Larry:

• caught up on a week's worth of backlog
• made a few spec tweaks
• discussed them with other people
• trying to make error messages more awesome in STD
• working on the ability to parse the insides of character classes
• STD doesn't like parsing itself recursively there
• need to iron out a few things
• enum names can now be variables

Allison:

• Debian packages ready to ship to Debian sponsors
• putting together a list of GC tasks
• cleaned out the existing page, have the big things listed
• trying to decide which tasks to do first
• doing a lot of reading and research
• my little language project is due on Monday
• HLLCompiler was enormously useful
• will start working on the GC stuff next week
• should also start a fresh pass through the ticket queue
• added a workaround for the final remaining TT #389 bug
• Jonathan had a test case

Will:

• tried to focus on getting Rakudo blockers removed

c:

• spent some time getting Rakudo to work with trunk
• will need a Rakudo guts hacker for the last part
• worked on the compact_string revamp branch with Vasily
• merged now
• that makes trunk about 12% faster than the 2.3.0 release
• will work on a few Rakudo profiles once it works with trunk again
• expect at least a 5% performance improvement there
• have some other ideas, but won't do them without profiling first
• came up with a scheme to reduce PBC size by coalescing strings
• Peter Lobsinger is exploring that