Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

pmichaud (6013)

pmichaud
  (email not shown publicly)
http://www.pmichaud.com/

Patrick Michaud is the pumpking for the Rakudo Perl 6 compiler. He holds a Ph.D. in Computer Science and was formerly a Professor of Computer Science at Texas A&M University-Corpus Christi. He is currently a software developer and consultant focused on open source development and applications, including Perl, PmWiki, and Linux.

Journal of pmichaud (6013)

Tuesday December 23, 2008
01:35 AM

Perl 6 Scripting Games

Are you interested in playing with Perl 6 and Rakudo Perl but can't figure out what to do? Here's an idea that came up during Jon Allen's talk "The Camel and the Snake" at YAPC::EU in Copenhagen.

For the past few years Microsoft has sponsored an annual Scripting Games competition, where they publish problems of varying difficulty levels to be solved using Perl 5, VBScript, Python, or other languages. I think it might be very interesting and useful to see people develop and publish Perl 6 solutions to those problems.

So, here's my idea: If you're interested in learning more about Perl 6, select one or more problems from the Scripting Games website, develop solution(s) for them in Perl 6, and then publish your solutions somewhere along with a description of what you like, don't like, learned, didn't learn, etc. about Perl 6 and Rakudo Perl.

One of the things we've observed from our experience with the November Wiki and other similar projects is that having "running code examples" and "real problems" is one of the best drivers for compiler and language development. I'm thinking that having people craft solutions to the scripting problems might do more of the same for Rakudo, while also sparking discussion and reflection on Perl 6 itself.

So, where to start? Start by obtaining and building a copy of Rakudo Perl, write and test your solutions to one or more problems, and post the results and your experiences somewhere for others to see. You can post to use.perl.org, your private blog, the perl6-users mailing list, or anywhere else you find convenient. The point isn't to develop a centralized repository of solutions (although we can do that), but rather to use the problems as a way to spread discussion, feedback, and experience with Perl 6.

I should also make it plain that people are very likely to run into some of Rakudo Perl's "rough edges" -- places where we don't yet implement key features or where they don't work exactly as they're supposed to. But to the designers and implementors that's part of the point -- we need to know where those rough edges are. Overall, I'm hoping that with the recent improvements to Rakudo Perl there won't be so many rough edges as to make the effort more disappointing than enjoyable. And there are lots of people eager to answer questions and help out on the perl6-users mailing list, IRC #perl6 (irc.freenode.net), and other common Perl forums. It's all about learning and improving what we have with Perl 6.

I look forward to seeing your questions and answers.

Happy holidays,

Pm

Monday December 22, 2008
03:26 AM

Two days. 562 new passing tests.

For some unknown reason I woke up at 3am on Saturday morning and couldn't get back to sleep. So, I decided I might as well use the time for some productive Rakudo hacking.

I decided to start by reviewing the spectest suite a bit. In the past couple of days I've noticed that although we've been increasing the number of spectests that Rakudo passes, we didn't seem to be shrinking the number of tests that are in the spectest suite but don't attempt to run (the grey area in the graphs). So, I decided to review the test files that we don't run, instead of the ones we do. I also reviewed the RT queue for patches to apply.

By 23:59 on Sunday, we had 526 more passing tests than we started with on Saturday. In fact, in the past two weeks Rakudo has increased its count of passing spectests by over 1,000. We're now at 5,790 total passing tests.

Here's some of what was accomplished this weekend:

  • recognize the difference between array and hash subscripting.
  • align with a change to S02 that now uses \c for decimal character encoding
  • added the .kv method to Pair, to enable the pair.t spectests
  • recognize more unicode codepoints (though still not all) as alphabetic
  • enable the various unicode bracketing characters for quotes
  • added the unicode versions of the infix hyperops we added last week
  • added the p5=> "fat comma" operator
  • cleaned up the readline method on IO
  • recognize radix conversions in strings
  • add +Inf, -Inf, and NaN processing
  • add some workarounds so that Complex numbers work again (they broke with mmd)
  • implement the use of the "whatever star" in list and array slices

Getting the "whatever star" to work was a nice surprise -- thunking the arguments to postcircumfix:<[ ]> turned out to be quite a bit easier than I had expected. Rakudo can now do things like @a[*-10 .. *-1] to get the last ten elements of an array.

Over the weekend Cory Spencer submitted several incredibly useful patches to fix array initialization in my() lists and properly refactor many of the list builtins. He has two more patches in the queue that I plan to review and likely apply tomorrow morning.

Overall, I'm very satisfied with the progress Rakudo contributors have made over the past couple of weeks. In addition to getting new features to work, there's also been a lot of internal refactoring and cleanup that will make things more efficient and help with the next phases of development. So, although I'm not sure that we'll be able to repeat the past couple of weeks, things still look very good for ongoing improvements to Rakudo.

Don't forget: we have a Rakudo Perl Twitter feed if you want to be notified of all of the new updates to Rakudo Perl.

Pm

Friday December 12, 2008
05:27 PM

Rakudo now passing over 5,000 spectests

I'm pleased to announce that as of yesterday Rakudo Perl is now passing over 5,000 tests in the "official test suite" (spectests). You can see the current state of things in the graph:

http://www.pmichaud.com/perl6/rakudo-tests-2008-12-12.png

Earlier in the week Adam Kennedy mentioned that it seems a bit sneaky to talk simply about the growth in pass rate without giving a basis for the number (1). This is a very reasonable question that comes up from time to time and in July I wrote a longish answer explaining why (2). The short answer is that the official test suite is itself a rapidly moving target so the percentages don't really say all that much.

The graph above tells the story -- but I've never really explained in a posting how we read this. The top of the grey area indicates the total number of tests in the spectest suite -- as of today (00:00 CST) there are 9,356 tests in the official test suite. If that number seems low to you, well, you're right. That's why were frequently asking for people to help flesh out the test suite -- more on this in a bit.

The top of the green area indicates the number of spectests that Rakudo is currently passing -- 5,004 as of today. So Rakudo is passing a little over half (53.5%) of the available spectests.

Within the "official test suite" there is a subset of test files that we designate as the "Rakudo regression tests". We maintain this list of subset files because there's little point in running Rakudo on a test file where we expect all of the tests to fail -- it just makes the testing take longer. So the top of the yellow area is the number of tests currently in the "Rakudo regression list" -- around 6,895 as of today. The blue area is thus the tests we can parse but not pass ("todo"), and the yellow area is those tests we can't parse but are part of the regression suite. Red of course is used to indicate failing tests, and we hope we don't see much of that. The tall red lines you see in the graph are places where the Parrot or Rakudo build was broken as of 00:00 CST on that day (and thus we were failing all tests).

So this graph indicates:

  • the growth and rate of growth of the spectest suite (top of grey area)
  • the growth and rate of growth in Rakudo's passing tests (top of green area)
  • the ratio of Rakudo's passing tests to the overall suite (top of green versus top of grey), and
  • the number of tests included in the Rakudo regression suite (top of yellow area).

You can of course determine other things from the graph -- these are the things I tend to be interested in.

The graph also makes it clear why knowing the pass percentage rate by itself can be misleading. Toward the end of June we were passing 25% of the tests (1080/4311) and today we're passing 50.3% of the tests. However, today's 50% is from a much bigger testing suite than what we were using in June.

There's also an issue of "what are the spectests actually covering?" For some insight to that, here are the current test results on a per-synopsis basis:

  Synopsis           Spectests         Rakudo
  ------------------   ---------      -------------
  S02 - bits              1858         814  (43.8%)
  S03 - operators         1946        1156  (59.4%)
  S04 - control            654         287  (43.9%)
  S05 - regex             1536         848  (55.2%)
  S06 - routines           525         204  (38.9%)
  S09 - data                62          16  (25.8%)
  S10 - packages            49           0  ( 0.0%)
  S11 - modules             53          30  (56.6%)
  S12 - objects            591         289  (48.9%)
  S13 - overloading         51           0  ( 0.0%)
  S16 - io                 238          27  (11.3%)
  S17 - concurrency         28           0  ( 0.0%)
  S29 - functions         1750        1309  (74.8%)
  integrated tests          15          24  (60.0%)

  total                   9356        5004  (53.5%)

Here I've listed only those synopses for which we currently have tests -- the other sections don't have any spectests yet. As you can see, there are many areas of the Perl 6 specification that are undertested, and thus adding tests for those areas means our base "spectest" value will continue to grow.

So, ultimately my primary measurement of progress continues to be "how many tests are we passing", although through the graphs I do keep an eye on the percentage of spectests we're passing. For long term planning, I think that a really good milestone will be when we're passing ~15,000 tests; then I think we'll be able to say that we have good coverage of the Perl 6 language spec. (15,000 is just a best guess, the actual number may turn out to be significantly more or less than that. Only time and work will tell.)

Of course, for those who are primarily interested in playing or using Perl 6, as opposed to the question of "when will it be released", the primary metric is whether Rakudo implements the features needed for their primary application(s). We're gaining ground on that every day, and now that we have some of the key core features in place (list assignment, slices, working lexical vars, etc.) I expect to see a jump in our momentum. And our experience with the November wiki and mod_parrot has been that real applications find the most useful bugs to fix and push us along quickest towards running code. To me, "running code" is the real measure of success or failure here.

As you can see, we still need test developers and reviewers. New tests need to be written for under-tested areas of the spec, and we're still in the process of migrating tests from the old pugs test directory into the "official spectest" location. Generally this just a little more than moving a file -- we also want to review the tests for accuracy and put "fudge" markers in place so Rakudo's test harnesses know when to skip/todo individual tests. Fortunately, Rakudo now has far fewer instances of "we can't test that because some other unrelated feature is missing". There's also a growing pool of people who can help shepherd newcomers along through the test building process. It's not at all hard, it just takes some orientation.

Anyway, I'm really glad we've reached the 5k milestone for Rakudo, and I'm looking forward to rapidly reaching the next.

Pm

P.S.: For those who are interested in doing their own number crunching, the daily progress statistics are at http://svn.perl.org/parrot/trunk/languages/perl6/docs/spectest-progress.csv and the test summaries and graphs can be generated using the scripts in Rakudo's tools/ subdirectory. Enjoy!

12:34 PM

It's sort of like...

Earlier in the week, fw wrote a journal post about his First Perl6 program. I was quite excited to see this, but then read a comment from educated_foo that pointed out that the Perl 5 version is actually shorter to write.

That didn't sit too well with me, because we really ought to make common things simpler, not harder. What really bugged me (about Perl 6, not about fw's post) was the following for sorting a hash by values:

for %words.pairs.sort: { $^b.value <=> $^a.value } -> $pair {
    say $pair
}

Sorting hashes by value is a common operation, and although I can shorten the above a little bit

.say for %words.sort: { $^b.value <=> $^a.value };

it's still a bit long for my taste. That { $^b.value <=> $^a.value }
just bugs me.

Then Moritz Lenz made a comment on #perl6 that perhaps sort should do something different with arity-1 blocks, and I then had an epiphany that led to the following pattern for sorting hashes by value:

%hash.sort: { .value }

I like this so much, I've gone ahead and implemented it in Rakudo, even though it's not officially part of the spec. (I'm hoping it'll be adopted as such.) So now in Rakudo we have the following:

> my %hash = { c=>5, b=>7, a=>-4, e=>9, d=>0 };
> .say for %hash.sort: { .value }
a       -4
d       0
c       5
b       7
e       9
 
> .say for %hash.sort: { .key }
a       -4
b       7
c       5
d       0
e       9

That seems much nicer. The general principal is that if the comparison argument to "sort" takes less than two arguments, then it's used to generate the values to be compared instead of the result of a comparison.

Of course, we aren't limited to just keys or values -- any operation we want to perform on the items being sorted will work. To sort by the magnitude of the values of the hash:

> .say for %hash.sort: { .value.abs }
d       0
a       -4
c       5
b       7
e       9

And of course this generalizes to more than just hashes; if @dogs contains a list of Dog objects we want to sort:

@dogs.sort: { .name }      # sort dogs by name
@dogs.sort: { .age }       # sort dogs by age

This works because the ".name" and ".age" methods are invoked on each of the objects in the list to determine the value to use for that object in the sort.

Or for a simplistic case-insensitive sort:

> my @a = <Fruit CHERRY danish Apple berry BaNaNa apricot>;
> .say for @a.sort: { .lc }
Apple
apricot
BaNaNa
berry
CHERRY
danish
Fruit

Besides clarity, another big advantage of

@a.sort: { .lc }

over

@a.sort: { $^a.lc leg $^b.lc }

is that in the first version we compute .lc on each element only once for the entire sort, whereas in the second version it's computed once for each comparison. And since we're typically doing O(n^2) comparisons, the first version can save a lot of method or function calls.

So, that was my fun for the morning. Implementing this behavior turned out to be really easy -- in fact, I wrote the algorithm first as Perl 6 before translating it into PIR:

multi method sort(@values: &by) {
    ...
    if &by.arity < 2 {
        my @v     = @values.map(&by);
        my @slice = (0..^@v).sort: { @v[$^a] cmp @v[$^b] };
        return @values[ @slice ];
    }
    ...
}

The code just uses &by to compute the values (@v) to be used in the sort, does a sort on the indexes based on a comparison of those values, and uses the resulting sorted index list to return the (sorted) slice of the original list. Note that the items in the result list are themselves unchanged from the original list -- they simply have a new sequence according to the &by criteria.

Since then PerlJam++ has suggested that we should do similar things for min and max:

my $longest_string = @strings.max( { .chars } );

This would use .chars as the criteria for determining the longest string but returns the longest string itself.

Perl 6 is very cool.

Pm

P.S.: By the way, this means that the solution to the problem in fw's original post becomes:

my %words;
$*IN.slurp.comb.map: { %words{$_}++ };
.say for reverse %words.sort: { .values };

Not all of the above works in Rakudo yet (I think .comb is not yet implemented), but at least it's getting closer to what we'd really like to see here.

Thursday December 11, 2008
12:45 AM

Out of the office

I'm out on business travel today and tomorrow (Wed/Thu), so I'm a bit limited on writing a detailed post of the day's developments. I should be back to longer posts by Friday.

For today I think I'll just advertise that we have a Twitter feed set up at http://twitter.com/rakudoperl where I've been posting update headlines on Rakudo progress as they occur. The various Rakudo developers will continue to make the longer blog posts on use.perl, rakudo.org, and other locations; I figure this is just another easy (and fun) way to provide even more views and insight into how Rakudo is progressing.

Wednesday December 10, 2008
01:14 AM

"It's beginning to look a lot like Christmas..."

A lot has happened with Rakudo Perl over the past few days. The biggest news is that Rakudo is now supporting list assignment and list slices. (Hash slices will show up in a day or two.) Now that we have those features working, I can finally start to say that Rakudo Perl is starting to feel to me like, well, Perl.

So, while we're still a good distance from an true Perl 6 release (a.k.a. "Christmas"), today we at least seem to have a tree with some shiny ornaments on it and even a few presents under the tree. :-)

Getting slices and assignment to work required a fair bit of refactoring of the base classes and operations, and I also did a lot of code cleanup which really needed to be done anyway. While cleaning up assignment code I also fixed up the assignment metaoperators (things like +=, *=, etc.) so that most of them are automatically generated instead of written by hand. Then for fun I went ahead and added some basic Perl 6 reduction operators.

I'm sure some are asking "What in the world are 'reduction operators'?" Well, they are another of the many shiny new presents Perl 6 is bringing us. A reduction operator is indicated by square brackets, and it turns an infix operator into a list operator. For example, while infix:<+> adds only two operands, the [+] operator will add together all of the elements of a list. Similarly, [*] means "multiply all of the elements of the list", and [<=] returns true if the elements of a list are numerically sorted.

    $sum = [+] @x;                      # sum all elements of @x
    $smallest = [min] $a, $b, $c, $d;   # minimum of $a, $b, $c or $d
    $issorted = [<=] @x;                # true if @x is numerically sorted
    $fact = [*] 1..$n;                  # $n factorial

Today Jonathan extended some of the work I did on reduction operators to add more of them, and then added many of the infix hyperoperators (e.g., >>+<<) and cross operators. See Jonathan's post for more details about those.

All of us working with Parrot and Rakudo are excited at the progress being made with Rakudo -- as of tonight we're closing in on 4,900 passing spectests, and I'm hoping we can make it to 5,000 by Tuesday's Parrot release. It really means I need to find a day to review the existing tests and RT tickets for things we're really passing or could be passing with just a little bit of effort.

I'll also be working to update our milestone and roadmap documents, and try to present a much clearer picture of where things presently stand for Rakudo.

On a Parrot note, this past weekend I also made some improvements to PGE and Parrot that give us a ~20% improvement in parsing and code generation speed.

Pm

Thursday December 04, 2008
03:18 PM

Rakudo now supports inline PIR

I've just added the capability to write inline PIR directly in Perl 6 subroutines in Rakudo Perl. This is done using the :PIR adverb on the 'q' quote operator, thus one can write:

        q:PIR { say 'hello' }

and whatever is in the bracketing characters will be embedded "inline" with the surrounding Perl 6 code. The special %r placeholder can be used in the PIR to specify the return value of the PIR:

        my $a = q:PIR { %r = box '123' };
        say $a; # "123\n"

Within the PIR it's safe to use any of the PIR registers $P0..$P9, $S0..$S9, $I0..$I9, and $N0..$N9. Register numbers higher than 9 run the risk of conflicting with automatically-generated registers generated by PCT. One can also declare local symbols, but may need to be careful not to conflict with any local symbols generated by PCT (those are somewhat rare, however).

Of course, we don't expect this to ever be a fixture of "true" Perl 6 programs -- this is just here to make it easier to write Perl 6 builtins for Rakudo (i.e., the "Prelude") and perhaps help others who are bootstrapping modules on Parrot. Ultimately the :PIR flag will only be available via an appropriate 'use PIR' statement or the like.

Also, the quote sequence will eventually become 'Q:PIR' with an uppercase 'Q', but a parser issue constrains us to stick with the lowercase 'q' for now.

More coming soon!

Pm

Sunday November 30, 2008
11:51 PM

.subst now honors :g(lobal) flag

Today I added in support for the :global/:g flag in the .subst method (S05); now we can do things like:

        > my $i = 0; say "There's more than one way".subst(/e/, { ++$i }, :global);
        Th1r2's mor3 than on4 way

Previously the .subst method would only work on the first match in a string. Rakudo still doesn't recognize the case where the :global flag is part of the regex, but at least we now have at least one concise mechanism for performing global search/replace on a string.

Within the closure argument one can use $/, $0, $1, etc. to refer to the matched portions of the string in the replacement. Getting that to work meant that we needed to fix the binding of $/ within closures -- it turns out the mechanism we were trying to use previously (!OUTER) can't really work in Parrot. As a bonus, fixing this simplified actions.pm slightly, and removed several internal subroutine calls that were being performed at the start of each closure.

Of course, the pattern and replacement arguments to .subst can be simple strings, but any variable substitutions in the replacement string then refer to the outer environment and not the values from the matched string.

Now we just need to verify and add a few more tests for this. :-)

Pm

Friday November 28, 2008
02:42 PM

infix: now works on Ranges in Rakudo

Moritz found a bug in the infix:<Z> operator -- it wouldn't properly zip over a range:

    09:04 <moritz_> rakudo: say (<a b c> Z 1..10).perl
    09:05 <p6eval> rakudo 33300: OUTPUT[elements() not implemented in class 'Range'&#9252;current
                                  instr.: 'infix:Z' pc 4039 (src/gen_builtins.pir:2555)&#9252;]

Normally we would expect this to return a list of lists:

        (("a", 1), ("b", 2), ("c", 3))

The problem here was that infix:<Z> was trying to determine the size of each of its elements by using the 'elements' opcode and then get the individual elements by using a subscript. This failed for two reasons: (1) Ranges don't support postcircumfix:<[ ]>, and (2) Rakudo's implementation of Range didn't work for the 'elements' opcode.

The real approach, as with most things dealing with lists, is to use iterators. I changed the algorithm that it creates an iterator for each argument to infix:<Z>, and then walks those iterators in parallel to build the elements of the final result list. When any of the argument iterators is empty, we're done building the list for infix:<Z>.

As a general rule, I suspect most list operators and builtin functions should not be counting the number of elements or using subscripts to access individual elements of a List. There are likely a few more functions like this in Rakudo that we should be replacing.

Pm
Tuesday November 25, 2008
03:10 PM

Lexicals have arrived

Earlier today I finally got lexicals to work properly in Parrot (and Rakudo). This means that recursion now works properly, as do a whole host of other things. It resulted in closing at least eight tickets in the Rakudo and Parrot RT trackers, and we're still looking to see how many spectests it enables.

We're still missing list interpolation, list assignment, and a few other basic things, but lexicals support was a big blocker for many things we've wanted to do. I'm glad it's finally done.

Many thanks to (irc nicks) jonathan, infinoid, masak, moritz, particle, bacek, chromatic, rgrjr, Tene, tewk, allison, and many others who helped with debugging and testing.

Pm