use Perl

This took longer than I expected, but the draft of the Modern Perl book is available for review. I'm especially interested in hearing from people who don't consider themselves expert Perl 5 programmers. The goal of the book is to explain how Perl 5 works (and how to write Perl 5 effectively) to help novices become adepts.

My progress and communication about the Test::Builder2 grant has been nothing short of appalling. There is a sort of herky-jerky progress where I figure out a design problem, push the code forward, then remember a use-case that throws a wrench in the whole design and the whole thing comes to a screeching halt again.

At the QA hackathon we elegantly solved the problem of things like die-on-fail and Test::NoWarnings but then ran afoul of things like Test::Warn and Test::Exception which runs tests inside of tests but those aren't actually part of the test stack.

Confused? I'll post more about it another time. Point is, TB2 continues to move forward, its just that there's long periods of rumination between sprints of development. And I get distracted by other projects. At this rate I'll be collecting Social Security before I collect the second half of the grant. I really want TB2 to happen, but something decisive has to be done. I work best with hard deadlines, so the plan is to clear a month for working mostly on TB2. A lot of the wibbling is trying to come up with the most elegant solution, but I usually have a less than elegant way to solve it and move forward. If its between an elegant TB2 that doesn't exist and a less elegant TB2 that does, well, go with the one that exists. With the way my schedule is looking, that will probably be mid-August to mid-September. If that doesn't produce an alpha, then I'll kill the grant.

That doesn't mean TPF gets nothing for your money. Chunks of TB2 can be harvested to improve TB1. Specifically, the TB2 formatting and history objects. The TB2 formatter makes the guts of TB1 cleaner, and it also allows it to produce something other than TAP. Used together, history and formatter allows non-Test::Builder based test frameworks to work together with Test::Builder providing even more flexibility. This is already done in the TB2 branch.

Looking at the grant deliverables, most of it is done:

* Split up global shared Test resources into individual objects
    * The test counter
    * The output filehandles
    * The plan
 
* Allow hooks for global beginning and end of test functions.
    * Ensure multiple hooks "stack"
    * die on fail
    * debug on fail
 
* Hooks for global beginning and end of test actions
    * Example: A safer Test::NoWarnings
    * Example: Don't cleanup temp files on failure so they can be debugged
 
* Allow for test output other than TAP
 
* Allow another Test::Builder-like module to work in the same process
  as Test::Builder (for example, sharing the counter).
 
* Rewrite Test::Builder in terms of Test::Builder2.

Here's what's not complete:

* Split up localizable behaviors into objects
 
* Allow individual test modules to locally override Test::Builder2 behaviors
 
* Allow test modules to globally override Test::Builder2 behaviors
    * How the plan works

Since I'm not writing to the letter of the law, there's more than that to be done before release, but the project does move.

I spent most of YAPC::NA mildly sick, sleep deprived and writing talks. Each of those things alone isn't so bad, but put all together meant I had time and energy enough to do my talks, discuss with people after, and that's about it. As a result, I was kind of dead in the head most of the time and didn't do a whole lot of interaction with people. I didn't feel like I got the most out of the one opportunity a year I get to hang out with huge gobs of Perl folk.

One of the things which I wanted to do at YAPC was get gitpan restarted. It can run right now, but the code is a mess and needs to be babied. It needs a rewrite. That rewrite was supposed to happen at YAPC but see above. I'm doing that now, using MooseX::Declare, perl5i and Path::Class just to mess around with them seriously. Also log4perl, which I'm finally learning a decade late. Its fun, far more pleasant than knocking it together without, once you learn to cope with Moose's idiosyncrasies. Better to learn the quirks of one complete system than nine incomplete ones.

That's what's absorbing my time right now. After that I want to add subroutine signatures to perl5i and the über file and directory objects. They were supposed to be in, at least as a prototype, by YAPC but that didn't work out. Using MooseX::Declare, Path::Class and perl5i together has me drooling for them.

Will Coleda, representing the TPF, found me at YAPC and mercifully did not break my legs. We hashed out a plan to make a last stab at Test::Builder2 before calling the grant done. That's not going to happen until August, I'll post about that later.

Oh, and I have a talk to do at OSCON about how the world is going to end in 2038 assuming 2012 doesn't claim the prize first. And a two part Git tutorial for Drupal programmers that I'm developing into a commercial class. And two paid clients to keep happy.

One thing I *don't* have to worry about is MakeMaker. Gird your loins, MakeMaker has provisionally been handed off to Matt Trout. Maybe I need to worry about it more...

The Perl 6 design team met by phone on 16 June 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

• documented TOP (again), and explained how parsing is initiated and how it actually works
• series operator (...) now picks a monotonic function when using single characters as endpoints
• STD can now catch duplicates involving protos as well as onlys
• STD no longer advises removal of parens on spaceless sub() declaration
• mostly advised sorear and pmichaud
• Stefan is finishing the boostrap of the STD parser
• also working on adding a parallel NFA and DFA engine
• no, he doesn't want to generate all the states in advance
• it works faster lazily

Allison:

• working on chroot environments with something more secure than chroot
• relevant to building Parrot packages
• looking at some bugs for Will

Patrick:

• Rakudo developers decided not to make extra special effort to make a June release of Rakudo Star
• the calendar works against us
• the new date for the release is July 29
• we're I comfortable with hitting that target
• we won't be happy with the results of moving heaven and earth to release in June
• there are lots of advantages
• one disadvantage is not having Rakudo Star at YAPC::NA
• one big advantage is using the supported Parrot 2.6 release as the basis
• I'll write a post outlining the plan in the next couple of days
• otherwise working on lists and interators in Perl 6 and Rakudo
• after deciding to make iterators immutable, Larry and I realized that solves many problems
• everything works out as plain as day after that
• very happy with that design
• the incorrect assumptions of the old model were pervasive
• replacing the old pieces is taking a while, which is no surprise
• this approach feels right though
• the new branch does things no previous version could do
• slices work much better, for example
• metaoperators work properly
• map is lazy
• slurpy arguments in lists are lazy by default
• no weird binding or action at a distance problems
• plenty of changes to Associative and Positional roles
• those are now super clean and may be lazy
• more features work
• ~30 failing tests (not test files, just tests) now, ~500 last night
• most of the current failures are minor
• will try to merge the branch before the release
• replacing lots of ugly code with fewer lines of elegant code
• Jonathan and others have worked on lots of other pieces
• adding plenty of new features
• looking forward to tomorrow's release

c:

• editing the Rakudo book
• moving the Rakudo release date may let us have a printed book available about the same time
• depends on how much there is left to write

The Perl 6 design team met by phone on 09 June 2010. Larry, Allison, Patrick, and chromatic attended.

Larry:

• not much spec change this week
• figured out a syntax for a regex block to return more than one cursor
• based on gather/take
• in STD hacking, continued to assist Stefan O'Rear in getting STD bootstrapped via viv
• now that it's bootstrapped, we're refactoring things that make sense now
• we're now starting to move bits of Cursor code from Perl 5 into Perl 6
• refactoring the grammar for sanity of design
• started upgrading STD to normal Perl 6 syntax where it previously catered to gimme5's limitations
• for example, switched STD's old .<_from> and .<_pos> hash lookups to using .from and .pos accessors
• started the prep work for moving EXPR out of STD to make it generally available to any grammar wanting operator precedence
• in STD parsing, made Perl 5 $< detection have a longer token to avoid confusion with match variables
• STD no longer attempts two-terms detection on infix_circumfix_meta_operator
• STD now parses >>R~<< correctly, or at least dwimmily
• STD doesn't complain about P5isms in printf formats like "%{$count}s"
• STD was parsing /m and /s with the opposite semantics
• termish now localizes $*MULTINESS in its scope so that inner declarations aren't accidentally multified
• STD now carps about package Foo; as a Perl 5 construct

Allison:

• talked to Chris Shiflett, a PHP developer, on someone from the PHP community to sit on the Parrot board
• will be in the US for a few weeks

Patrick:

• working on list simplification
• had a couple of breakthrough ideas on Monday
• working on the implementation now
• worked out inversion lists for character class matching in regexes
• will make them faster, especially with long ranges of character classes
• fixed a half-dozen tickets in RT
• fixed Rakudo hash constructors
• fixed an intermittent bug with colon-pair signatures
• two possible parses exist in STD, but we removed an unneeded one in Rakudo
• fixed a bug with Parrot's exit opcode
• NQP and PAST needed an update not to cheat with PASM constants
• I fixed that too
• Vasily added multisub and multimethod support to NQP, that was a big plus
• fixed the ** quantifier in regexes to understand surrounding whitespace
• regex engine tried to match beyond the end of a string, so I added guards for that
• will work on lists furiously before the next release
• I don't think it'll take long
• closures are next, hope to have those in place by the weekend

c:

• released a new version of Pod::PseudoPod::LaTeX to support the various books in progress

The Perl 6 design team met by phone on 02 June 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

• mostly, I supported sorear in bootstrapping STD to use viv instead of gimme5
• his stage 2 and stage 3 now output identical Perl 5 versions of STD
• produces a huge amount of warnings
• appears to require Perl 5.12 at the moment
• working on both of those
• S03 refines hyper dwimminess to be more like APL, with modular semantics
• S02 refines Blobs to simply be immutable Bufs, with similar generic characteristics
• S02 now describes native blob types
• implemented post-declaration checks for BEGIN and use, since those can't wait for end of file
• STD no longer loses existing bindings when we go to a sublanguage
• STD now uses $*GOAL variable only as informative, never as a "stopper"
• instead, we create a <stopper> rule for $*GOAL if necessary
• can check for that only, instead of that or $*GOAL
• answering lots of questions on how STD and viv work besides that

Allison:

• did a lot of research on graph color algorithms for register usage algorithms
• will finish my finals on Monday

Will:

• trying to herd the discussion of dynop libraries
• a recent branch to close an old ticket broke a lot of assumptions
• some bugs have become more visible because of these changes
• hope to get that cleaned up this week

Allison:

• I liked your suggestion of bringing back the getstderr and related opcodes

Will:

• trying to resurrect Partcl
• stuck on a TT #389 closing issue
• not sure how to fix that, the way things are now

Patrick:

• working on the iterator and list design
• brainstorming the implementation
• will implement somethine one way or another this week
• people keep implementing workarounds for the current system
• they'll bite us eventually
• Moritz and I worked on making the regex engine returning real Perl 6 objects
• that mostly works
• exposes some places where lists don't work exactly right
• the workarounds there made me replan the list and iterator implementation
• answered some questions online
• Jonathan added a better backtrace algorithm for Rakudo
• reports Perl 6 source lines instead of PIR lines
• I'll review his code
• think I can borrow it for NQP for all HLLs
• Jonathan reports that it was a lot easier in NQP than PIR

c:

• trying to answer a few Parrot design questions
• looking at the continuation of design from Perl 1 - 4 to Perl 5 and Perl 6
• hope to have coding time soon

The Perl 6 design team met by phone on 26 May 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

• :() syntax is now always signature
• we now use foofix:[...] as the general op form instead of foofix:(...)
• refactored the sematics of :nth and :x
• :nth() now only ever takes a monotonically increasing list
• S03 now explains how "not-raising" works on != and ne
• it now basically matches the intuitions of an English speaker via HOP definition of negate metaop
• STD sometimes didn't require semi between statements
• statement modifiers are expression terminators but not valid statement terminators
• an unexpected statement modifier word like if could terminate one statement and start another
• fixed up backslashes in character classes to allow \s etc and reject \u etc
• STD was accidentally using the same lexpad for different multis
• Cursor now treats :() on name extension as a signature always, never as a categorical
• we shouldn't introduce the stopper for circumfix until we're in the circumfix, or we can't use the same char on both ends
• placeholder messages error messages are now much more informative and correct
• we now disallow use of placeholder after same variable has been used as a non-placeholder, even for an outer reference
• renamed add_macro (which it doesn't) to add_categorical (which it does)
• participating frequently in discussions on semantics both on irc and p6l
• working closely with sorear++ as he brings viv closer to bootstrapping, yay!
• soon can bootstrap past gimme5

Allison:

• worked on Pynie this week in my limited spare time
• one goal is to generate the parser directly from the Python grammar
• wrote a small, lightweight PEG parser which generates a match tree from the Python 3 grammar
• can generate a lexer directly
• right now it creates a parse tree
• looks similar to the match nodes of NQP-rx
• dumps out a tree to the PIR parser
• working on PaFo elections for next year, but trying to delegate those
• will have more time after June 7

Will:

• working on Perl 6 advent tests
• many more people are doing more work than me
• liasing with Rakudo folks for any important Parrot bugs before the Rakudo Star release
• my current direction there is "don't break anything"

Patrick:

• sorear added hash flattening to NQP
• lots of work on closures in PAST and NQP
• they properly clone
• fixes some lexical problems
• need to get that to work in Rakudo
• that's tougher; Rakudo has to wrap Parrot subs
• wrapper object needs cloning as well, along with its attributes
• we'll add a new PAST node type to help
• that node understands contexts
• essentially a way to add void context optimizations to your AST
• that solves many problems in Rakudo beyond closures
• added a setting into NQP along with its test suite
• not automatically loaded, but available
• contains standard hash and array methods
• Parrot's ops2c project uses those
• other people can update and enhance that setting as necessary
• NQP also has the ability to parse type names
• NQP doesn't do anything with them yet
• eventually they'll allow the use of multis
• cleaning up some NQP bugs regarding lexicals and package storage of subs
• Bruce Keeler enabled variable interpolations in regexes
• working on some refactorings to simplify that approach
• works in NQP and Rakudo now
• that's a feature we've never had before
• Rakudo's REPL now works better, thanks to sorear
• HLLCompiler now written more in NQP as part of that
• NQP now can do eval
• NQP remembers lexicals in interactive mode now
• adding that to Rakudo is more complex
• working on that
• pleased with the progress on #perl6

c:

• reviewing long term plans for GC and Lorito
• should have more time free soon

The Perl 6 design team met by phone on 19 May 2010. Larry, Will, and chromatic attended. Patrick added his notes later.

Larry:

• S03 makes more explicit that doctrine that ~~ topicalizes, and removes smartmatch table fossils that automatically fall out from that
• S05 renames 'accent' to 'mark' for better Unicode conformance
• :a and :aa changed to :m and :mm
• S05 disrequires retroactive semantics on :samecase and :samemark
• the method form must now explicitly add case or mark modifiers to the pattern
• regularized mm// to ms// to avoid confusion with new :m ignoremark option
• STD now does a bit better at diagnosing bogus ??!! constructs of various sorts
• STD now correctly adds operators to symbol tables as subs
• CORE.setting now has protos of all the operators so they can be recognized as subs too
• Cursor now canonicalize operator names in the symbol table
• btw, not quite like specced
• STD now reads user's mind on 'Str $toto' to intuit missing declarator
• STD now properly diagnoses a typename between routine declarator and sub name

Will:

• working on code for Carl Masak, trying to get his poker code example running on Rakudo
• both fun and frustrating
• some stuff doesn't quite work yet
• going through the Advent examples
• adding them to spectests
• make sure we won't regress on such public examples
• other people are helping with that now

c:

• will get back to editing the Rakudo book soon
• hope to have it in print by YAPC, but no guarantee

Patrick:

• fixed closures in NQP, as a precursor for fixing them in Rakudo
• worked with sorear on REPL in Rakudo and PCT in general
• ported the NQP "standard library" done by japhb++, bacek++, and many others into the nqp-rx repository and made it part of the standard build sequence for nqp and Parrot
• decided we need a new "context sensitive" node type in PAST, will be used to create proper closures and to handle sink context
• worked with bacek on adding better multimethod support to PAST and nqp-rx
• discovered a problem with lexical subs in NQP being automatically entered into the package namespace (and some existing code relying on this behavior)
• did some initial fixes to at least get things entered properly, but a complete fix may require a deprecation cycle
• plan to review others' patches this week
• plan to fix REPL, closures, and sink context in Rakudo (since those are currently large pain points)
• plan to work on loops and iterators after that

The Perl 6 design team met by phone on 12 May 2010. Larry, Allison, Patrick, and Will attended.

Larry:

• clarified usage of brackets around infixes
• added various 128-bit types to the spec; we might make them arbitrarily extensible via role
• at least LLVM could support this, even to non-powers-of-two sizes
• modernized the paleolithic grammatical category description in S02
• STD now uses double-quote rules for interpolating @foo[] into regex
• STD now gives better message on 1__3
• added the specced 128-bit types to CORE.setting
• added minmax function to CORE.setting
• implemented circumfix:«X Y» as grammar derivation
• currently only allows a >> inside
• now also recognizes foofix:("\x[face]") and foofix:("\c[YOUR CHARACTER HERE]") without actually evaluating
• playing with factoring yaml out of gimme5, since viv is not likely to go that route.
• mostly just answered a lot of questions on irc
• egged people on about concurrency issues

Patrick:

• thought on handling closures properly
• have a solution, just need some time to implement
• discussion on changes to CodeString
• work on compiler toolkit to avoid CodeString, using StringBuilder instead where possible, in PCT, NQP, and rakudo. Pretty easy, no downstream projects block on a deprecation issue
• after that, lists
• also been answering questions on interactive mode (REPL) for rakudo et al. (the issue with losing lexicals)

Allison:

• resolved the git conversation pretty well (for Parrot's repo migration)
• worked on a pure PEG parser (following the paper), straight PIR, single day; now self-parsing. Interesting project, is lightweight. currently has memoization, but that might not be right for us because of backtracking. With some more effort, could probably handle EBNF form (useful for python)
• could be setup for developer status for Debian which will improve our packaging status for Debian and Ubuntu

Will:

• Parrot CodeString performance improvements
• we're definitely faster in branch, but some feedback from pmichaud should help us clean up the API a bit as well, look for those to hit trunk in the next few days
• Parrot makefile deps cleanup

If I wanted to write a Strawberry program to run as a daemon and periodically pop up alerts on my machine, and I had no Windows Perl GUI programming experience other than an abortive look at WxPerl years before anybody ever thought of Strawberry, what module would I be looking for?