geoffrey's Journal http://use.perl.org/~geoffrey/journal/ geoffrey's use Perl Journal en-us use Perl; is Copyright 1998-2006, Chris Nandor. Stories, comments, journals, and other submissions posted on use Perl; are Copyright their respective owners. 2012-01-25T02:42:06+00:00 pudge pudge@perl.org Technology hourly 1 1970-01-01T00:00+00:00 geoffrey's Journal http://use.perl.org/images/topics/useperl.gif http://use.perl.org/~geoffrey/journal/ Parrot Plumage: A Month Goes By http://use.perl.org/~geoffrey/journal/39875?from=rss <p> It's been over a month since <a href="http://use.perl.org/~geoffrey/journal/39713">my last entry</a>. Life pretty much had me swamped for the last five weeks, but nevertheless I did get some Plumage work done. </p><p> It would be a daunting task to try to go back and remember all the details of the last five weeks, so I've decided to take <a href="http://use.perl.org/~pmichaud/journal/39834">a hint from pmichaud</a> and just include my <code>#parrotsketch</code> reports (edited a bit here): </p><blockquote><div><p> <tt>DONE IN PLUMAGE:<br>* Handle fetching over old working dir, including when changing repo types (as partcl did)<br>* Add rx() function for compiling regex strings, until nqp-rx is ready for use<br>* Likewise all_matches()<br>* Various small cleanups<br>* Fix gitoriousparser plugin for dalek-plugins<br>MAD PROPZ:<br>* darbelo++&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# help with cleanups<br>* dukeleto++&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # plumage source tree reorg<br>* dukeleto++ and Tene++&nbsp; &nbsp; &nbsp;# plumage test harness and test suite<br>* dukeleto++ and Infinoid++ # gitoriousparser work enabling a parrot-plumage dalek feed<br>EOR<br> <br>DONE IN PLUMAGE:<br>* Dependency handling, including remembering installed projects (not the final paradise, but Good Enough For Now)<br>* Automatically sudo if parrot_bin directory not writable by user<br>* New 'projects' (list known projects) and 'showdeps' (show resolution for all dependencies) commands<br>* Plumage metadata dir can now be overridden by conf file (for testing support)<br>* Several new functions in Glue.pir and Util.nqp<br>* Lots more docs for Glue.pir and Util.nqp<br>* More tasks broken out of my head into TASKS<br>MAD PROPZ:<br>* dukeleto++ # Testing, testing, testing; factor out Util.nqp from main program<br>* darbelo++&nbsp; # Matrixy metadata (and making it buildable against installed Parrot)<br>EOR<br> <br>DONE:<br>* Improve validation of metadata<br>* Refactoring and function documentation<br>* Much improved Makefile (with automatic Makefile rebuilding)<br>WIP:<br>* import_proto.p6 (Import proto-managed projects into Plumage metadata)<br>* Analyzing discussion surrounding major CPAN META spec upgrade (which is in design phase)<br>MAD PROPZ:<br>* darbelo++ # Plumage's NQP configure brought to other projects<br>* Austin++&nbsp; # Makefile education<br>BLOCKERS:<br>* import_proto.p6 blocking on proto's installed-modules branch<br>EOR<br> <br>DONE:<br>* Talked at length with Plobsing++ re: current NCI problems<br>* Brain dumped to https://trac.parrot.org/parrot/wiki/NCITasklist<br>WIP:<br>* Converting Plumage to make use of new NQP-rx features<br>* Pushing the envelope of what NQP-rx has<br>* Exchanging feature requests with pmichaud++ via http://wiki.github.com/perl6/nqp-rx/plumage-requests<br>* Moving Glue.pir functionality to Util.nqp where possible<br>* Further expanding Util.nqp to cover more common functionality<br>* Cleaning up and expanding Plumage's test suite<br>NEXT UP:<br>* More of everything in WIP section<br>BLOCKERS:<br>* Several local Plumage branches blocked waiting for various NQP-rx features<br>EOR</tt></p></div> </blockquote><p> So there you have it -- a month of Plumage work, in shorthand. I also finally got around to setting up Perl-specific microblogging accounts on <a href="https://twitter.com/japhb">Twitter</a> and <a href="http://identi.ca/japhb">identi.ca</a>; I'm <code>japhb</code> on both of them just as I am in <code>#parrot</code> on <code>irc.parrot.org</code>. As always, don't hesitate to drop by and ping me. If you'd like to join the Plumage effort, check out the code in the <a href="http://gitorious.org/parrot-plumage/parrot-plumage">Parrot Plumage repository</a>; read the <code>README</code> for the general overview, then come to <code>#parrot</code> to get your questions answered! </p> geoffrey 2009-11-11T05:53:52+00:00 parrot Parrot Plumage "Day" 8: Getting a bit smarter http://use.perl.org/~geoffrey/journal/39713?from=rss <p> <a href="http://use.perl.org/~geoffrey/journal/39697">When last we left off</a>, Plumage had just managed to install its first project, thus moving from the "infant" to "toddler" phase of development. </p><p> In preparation for reaching "preschooler" status, I spent time this week removing some hackishness and making Plumage generally smarter about basic operations. Some of the changes were very simple, such as always printing out system commands to be spawned before actually executing them, or pushing some boilerplate code down into utility functions rather than forcing every calling site to go through the same contortions. </p><p> Some changes were infrastructural, including basic utilities like <code>map()</code>, <code>exists()</code>, and <code>does()</code> and workarounds for NQP limitations. The latter category included new functions such as <code>as_array()</code>, <code>call_flattened()</code>, and <code>try()</code>, which Tene++ and I added to <code>Glue.pir</code> to make calling convention mismatches a bit less painful. </p><p> Other changes were larger, such as adding command line option handling, allowing the user to ignore failures in various build stages, and handling exceptions thrown by Parrot ops and low-level Parrot libraries. Handling (possibly several) config files (including merging the config information across them) and moving all project builds to a separate location -- defaulting to <code>~/.parrot/plumage/build/</code> -- also fell into the "larger changes" pile. </p><p> Still others amounted to cleanup and general code maintenance, such as compiling <code>Glue.pir</code> to <code>Glue.pbc</code> during the <code>plumage</code> build so that all libraries could be directly loaded as pure bytecode at runtime, finding less hackish ways of working around NQP, and cleaning up documentation and comments throughout. </p><p> Finally, darbelo++ added a new project (partcl) to the metadata collection; I hope to add a few more in the next week or two. </p><p> All in all, no major new milestones reached -- but the code feels a bit more solid now, behavior is "more correct" in several places, and the user experience is definitely better. Not a bad week's work. </p><p> As always, you can check out the code at the <a href="http://gitorious.org/parrot-plumage/parrot-plumage">Parrot Plumage repository</a>, and don't hesitate to ping me on IRC -- I'm <code>japhb</code> on <code>#parrot</code> at <code>irc.parrot.org</code>. If you'd like to join the effort, read the <code>README</code> for the general overview, then come to <code>#parrot</code> to get your questions answered! </p> geoffrey 2009-10-05T08:25:54+00:00 parrot Parrot Plumage Day 7: First Working Install http://use.perl.org/~geoffrey/journal/39697?from=rss <p> After posting for <a href="http://use.perl.org/~geoffrey/journal/39682">Day 6</a>, Tene and I realized that <a href="http://irclog.perlgeek.de/parrot/2009-09-28#i_1546254">the <code>instructions</code> section of the metadata spec needed to change</a> -- the original "design" didn't survive actual implementation. While I made the necessary <a href="https://trac.parrot.org/parrot/wiki/ModuleEcosystem#MetadataProposal">spec changes</a>, Tene++ worked on several iterations of an HTTP fetcher in pure PIR. </p><p> Today we made heady progress. Two days ago, we only had basic fetching half-working. Today Plumage was able to coax its first project all the way from fetch through configure, build, test, and install! </p><p> darbelo++ wins the project prize as his decnum-dynpmcs project (support for decimal arithmetic in Parrot) was the first successful Plumage install. Close and Blizkost didn't quite make it -- Close has failing tests (which Plumage currently refuses to ignore), and Blizkost wouldn't build on my machine, though it's not clear if this was a problem with Plumage or with Blizkost itself. </p><p> There's definitely still a huge amount of work left to be done, but today marks a major milestone nonetheless. Plumage is doing a very important piece of the job it was designed for, and I'm feeling pretty proud of the progress so far. </p><p> As always, you can check out the code at the <a href="http://gitorious.org/parrot-plumage/parrot-plumage">Parrot Plumage repository</a>, and don't hesitate to ping me on IRC -- I'm <code>japhb</code> on <code>#parrot</code> at <code>irc.parrot.org</code>. If you'd like to join the effort, read the <code>README</code> for the general overview, then come to <code>#parrot</code> to get your questions answered! </p> geoffrey 2009-09-30T04:25:30+00:00 parrot Parrot Plumage Day 6: New committers! http://use.perl.org/~geoffrey/journal/39682?from=rss <p> "Day" 6 turned out to actually be hours here and there from most of the week. After Tene's initial <code>Makefile</code> patch initiated <a href="http://use.perl.org/~geoffrey/journal/39663">Day 5</a>, <a href="http://irclog.perlgeek.de/parrot/2009-09-22#i_1528183">Tene and darbelo began to pester me for details</a> on the design and current capabilities. Once he realized it wasn't as far along as he needed, <a href="http://irclog.perlgeek.de/parrot/2009-09-22#i_1528222">Tene switched his pestering to tasks</a> that I could break out and hand off to parallelize the work a bit. </p><p> While I was thinking about that (and answering a blizzard of other questions from the <code>#parrot</code> crowd), I realized I needed to make the source tree more friendly for other committers. I added a CONTRIBUTING section to the <code>README</code>, moved the old probe tests (TimToady++ for that name) to their own directory, and made a couple other minor changes. </p><p> With that, I added Tene++ and darbelo++ as the first new committers to the <a href="http://gitorious.org/parrot-plumage/parrot-plumage">parrot-plumage repository</a>. W00t! </p><p> It didn't take long before <a href="http://irclog.perlgeek.de/parrot/2009-09-22#i_1528987">darbelo noticed a licensing issue</a>. After some discussion, the members of <code>#parrot</code> decided he was probably correct. I was assigning the copyright for Parrot Plumage to the Parrot Foundation, but not requiring new committers to have signed Contributor Licensing Agreements ("CLAs") sent in to PaFo. It happened that Tene and darbelo were both Parrot committers, which meant they already had signed CLAs -- so I simply changed the <code>README</code> to reflect the CLA requirement, and we neatly escaped the problem. (Note that for now we <em>don't</em> require Parrot Plumage committers to be full Parrot committers, merely to have a signed CLA sent in.) </p><p> Once I finally got some mental space, I went back to tanking on the tasks that I could hand off to other developers, and codified the first few into the <code> <a href="http://gitorious.org/parrot-plumage/parrot-plumage/blobs/master/TASKS">TASKS</a> </code> file. darbelo++ and then Tene++ began picking off tasks and completing them over the next couple days, while I refocused on my day job. </p><p> Tene's work culminated in adding the <code>fetch</code> command, with the ability to handle both Subversion and Git repositories. This worked like a charm with our sample Git-based project, <a href="http://github.com/jnthn/blizkost">Blizkost</a>. Unfortunately, we discovered our first hiccup when our sample Subversion-based project, <a href="http://code.google.com/p/close/">Close</a>, would only partially fetch -- it appears to have submodules that require special authentication. Ah well, no rest for the wicked. </p><p> With that problem added to <code>TASKS</code>, and the previous commits reviewed and lightly edited, "Day" 6 drew to a close. </p><p> As always, you can check out the code at the <a href="http://gitorious.org/parrot-plumage/parrot-plumage">Parrot Plumage repository</a>, and don't hesitate to ping me on IRC -- I'm <code>japhb</code> on <code>#parrot</code> at <code>irc.parrot.org</code>. If you'd like to join the effort, read the <code>README</code> for the general overview, then come to <code>#parrot</code> to get your questions answered!</p> geoffrey 2009-09-28T01:04:29+00:00 parrot Parrot Plumage Day 5: Configure.nqp and a 'proper' Makefile http://use.perl.org/~geoffrey/journal/39663?from=rss <p> Shortly after <a href="http://use.perl.org/~geoffrey/journal/39622">Day 4</a> day job deadlines loomed, and I expected Day 5 to be delayed at least a week. Thanks to some late nights this morning's status meeting at the day job went well, so Tene++'s ping on <code>#parrot</code> caught me with a few spare tuits and the strong desire to take a break from the tasks of the last few days. </p><p> Tene had pasted a patch to the <code>Makefile</code> for Plumage, essentially suggesting that it be a bit less hard-coded and instead use the <code>parrot_config</code> binary to fill in some of the variables. Unfortunately, his patch used backticks (<code>`...`</code>) to capture the output of <code>parrot_config</code>, which is not portable across <code>make</code> implementations. In fact, it appears that every <code>make</code> does this differently -- GNU Make uses the <code>$(shell<nobr> <wbr></nobr>...)</code> construction, BSD Make uses the special <code>!=</code> assignment operator, and so on. </p><p> Asking around, it soon became clear that this morass of incompatible syntax for capturing shell output is one of the reasons everybody just uses a Configure script of some sort. In fact, a simple Configure script does nothing more than replace markers in a makefile template (typically called <code>Makefile.in</code>) with the proper text, and write out the completed <code>Makefile</code> to be fed to <code>make</code>. </p><p> That sounded like exactly the kind of simple substitution that I needed for the Plumage <code>Makefile</code>, but I couldn't just use one of the zillion existing <code>Configure</code> scripts, because it needs to be written in NQP for much the same reasons that Plumage itself does. </p><p> Well, I clearly couldn't leave <em>that</em> situation lie, so a couple hours later, <code>Configure.nqp</code> was born. After implementing <code>slurp()</code> and <code>spew()</code> functions in the <code>Glue.pir</code> "builtins" library, reading the template and writing the output again were trivial, but the interesting task turned out to be implementing <code>subst()</code> to do the text substitutions. </p><p> Parrot has for quite some time now shipped with PGE, the Parrot Grammar Engine. Among other things, PGE has built-in support for the Perl 5 and Perl 6 grammar/regex languages. With some gracious help from pmichaud++ and a good bit of spelunking in the Rakudo <code>src/builtins/</code> directory, I pieced together the necessary bits to instantiate the Perl 6 variant, use it to compile the regex argument to <code>subst()</code>, and iterate over the matches performing the substitutions. </p><p> At first I implemented purely static string replacement; the same new string would be used to replace every match point in the original. This method would however be horribly slow for the task at hand -- I'd have to run the substitution across the entire makefile text for each of the hundreds of config values that <code>parrot_config</code> knows about. </p><p> Taking a hint again from the Rakudo implementation of <code>subst()</code>, I also made it possible to supply a sub instead of a plain string as the replacement argument. This sub gets called with each match object in turn, and returns the appropriate string to use for the replacement. This makes the relevant code from <code>Configure.nqp</code> remarkably clean: </p><blockquote><div><p> <tt>my $replaced<nobr> <wbr></nobr>:= subst($unconfigured, '\@&lt;ident&gt;\@', replacement);<br> <br>sub replacement ($match) {<br>&nbsp; &nbsp; my $key&nbsp; &nbsp;<nobr> <wbr></nobr>:= $match&lt;ident&gt;;<br>&nbsp; &nbsp; my $config<nobr> <wbr></nobr>:= %VM&lt;config&gt;{$key} || '';<br> <br>&nbsp; &nbsp; return $config;<br>}</tt></p></div> </blockquote><p> (Yes, <code>replacement()</code> could easily be written as a one-liner. It seemed a bit clearer this way.) </p><p> The plain string form of <code>subst()</code> still gets used, to fix the slash orientation for Windows systems: </p><blockquote><div><p> <tt>if ($OS eq 'MSWin32') {<br>&nbsp; &nbsp; $replaced<nobr> <wbr></nobr>:= subst($replaced, '/', '\\');<br>}</tt></p></div> </blockquote><p> A few miscellaneous cleanups and an update to <code>README</code> later, and the <code>Makefile</code> cleanup task (which I expected to remain near the bottom of the task list for a fair bit, nibbling at the edge of my conciousness) is Just Done. That's a good feeling. </p><p> As always, you can check out the code at the <a href="http://gitorious.org/parrot-plumage/parrot-plumage">Parrot Plumage repository</a>, and don't hesitate to ping me on IRC -- I'm <code>japhb</code> on <code>#parrot</code> at <code>irc.parrot.org</code>. </p> geoffrey 2009-09-22T07:03:22+00:00 parrot Parrot Plumage Day 4: First Bones of the Skeleton http://use.perl.org/~geoffrey/journal/39622?from=rss <p> <a href="http://use.perl.org/~geoffrey/journal/39516">Day 1</a> and <a href="http://use.perl.org/~geoffrey/journal/39598">Day 2/3</a> got me (mostly) prepared to begin the actual coding for the <a href="http://gitorious.org/parrot-plumage/parrot-plumage">Parrot Plumage</a> prototype. There were just two things left to figure out: how to access command-line arguments, and how to initialize complex data structures. </p><p> The first problem turned out to be relatively simple to solve with a couple more additions to the glue library to initialize <code>@ARGS</code> and <code>$PROGRAM_NAME</code> from the <code>IGLOBALS_ARGV_LIST</code> field of the Parrot interpreter globals. </p><p> The second problem (initializing complex data structures) required a couple silly hacks, but I got a decent workaround in the end. Allow me to explain<nobr> <wbr></nobr>.... </p><p> NQP natively can index into hashes, arrays, and complex arrangements of same with relative ease. Want to access an entry in a hash of arrays of hashes, with some fixed keys and some varying keys? No problem. Something like this will do: </p><blockquote><div><p> <tt>$age<nobr> <wbr></nobr>:= %pets{$type}[$num]&lt;age&gt;;</tt></p></div> </blockquote><p> The problem is, there's no short and simple way to <em>initialize</em> the <code>%pets</code> hash. NQP doesn't have syntax for hash or array literals, which makes it rather painful to instantiate complex structures. Oh, I suppose you could write a whole bunch of these: </p><blockquote><div><p> <tt>%pets&lt;feline&gt;[0]&lt;age&gt;<nobr> <wbr></nobr>:= 5;<br>%pets&lt;feline&gt;[1]&lt;age&gt;<nobr> <wbr></nobr>:= 6;<br>%pets&lt;canine&gt;[0]&lt;age&gt;<nobr> <wbr></nobr>:= 4;<br>...</tt></p></div> </blockquote><p> but as you can imagine that gets old rather quickly -- and doing the initialization in raw PIR would be even more painful. Luckily, during my previous hack day I'd brought the JSON parser up to current standards, so I used it for a little trick. I defined the data structure I wanted as a JSON string, parsed it into a real data structure using the <code>data_json</code> language that now ships with Parrot, and then ran a fixup routine on the few bits that didn't translate easily. </p><p> That last bit might require a little explanation. I could only create data structures this way containing data types the JSON language knows how to represent: numbers, strings, arrays, hashes, booleans, etc. However, there's no way to represent a code reference, and one of the applications I had in mind was a function table for mapping user commands, such as <code>version</code> and <code>install</code> to the subroutines that would implement them. </p><p> My solution was to replace the code references with the <em>names</em> of the subroutines I wanted in the JSON text, and then after parsing the JSON run a peculiar bit of inline PIR to replace the names with the real code references. The fixup function in question looks like this: </p><blockquote><div><p> <tt>sub fixup_commands ($commands) {<br>&nbsp; &nbsp; Q:PIR{<br>&nbsp; &nbsp; &nbsp; &nbsp; $P0 = find_lex '$commands'<br>&nbsp; &nbsp; &nbsp; &nbsp; $P1 = iter $P0<br>&nbsp; &nbsp; &nbsp; fixup_loop:<br>&nbsp; &nbsp; &nbsp; &nbsp; unless $P1 goto fixup_loop_end<br>&nbsp; &nbsp; &nbsp; &nbsp; $S0 = shift $P1<br>&nbsp; &nbsp; &nbsp; &nbsp; $P2 = $P1[$S0]<br>&nbsp; &nbsp; &nbsp; &nbsp; $S1 = $P2['action']<br>&nbsp; &nbsp; &nbsp; &nbsp; $P3 = get_hll_global $S1<br>&nbsp; &nbsp; &nbsp; &nbsp; $P2['action'] = $P3<br>&nbsp; &nbsp; &nbsp; &nbsp; goto fixup_loop<br>&nbsp; &nbsp; &nbsp; fixup_loop_end:<br>&nbsp; &nbsp; };<br> <br>&nbsp; &nbsp; return $commands;<br>}</tt></p></div> </blockquote><p> The short explanation is that the loop in the above PIR block iterates over each entry in the <code>$commands</code> hash, and for each <code>action</code> key replaces the matching string value (the subroutine name) with the subroutine itself, looked up using the <code>get_hll_global</code> op. And it works! </p><p> These problems out of the way, I could finally start writing a prototype of the <code>plumage</code> command line tool. Time eventually ran out on my hacking day, but I managed to implement basic versions of <code>usage</code>, <code>version</code>, and <code>info</code> commands, and a command parser and dispatcher that could be charitably described as "beyond minimalist". </p><p> While the first two commands just provide info about <code>plumage</code> itself, the last one is the first command that actually provides real functionality. In particular, <code>info</code> simply loads the metadata for a given project and prints the details for the user. </p><p> In order to have some metadata to display with my brand new <code>info</code> command, I took a shot at filling in the fields from the <a href="https://trac.parrot.org/parrot/wiki/ModuleEcosystem#MetadataProposal">Metadata Proposal</a> manually for a single project. In this case, I chose <a href="http://github.com/jnthn/blizkost">Blizkost</a> because it's useful, cool, and non-trivial without being insane. </p><p> Unsurprisingly, I came across a couple underspecified bits in the metadata proposal, but things mostly seemed to make sense. <a href="http://gitorious.org/parrot-plumage/parrot-plumage/blobs/master/metadata/blizkost.json">The result</a> is a tad wordy, but it is after all intended to be generated mostly automatically. </p><p> That done, I found myself out of time for this hack day, but things are looking up for implementing an ultra-simple version of the <code>install</code> command next time. Until then, I invite you to <a href="http://gitorious.org/parrot-plumage/parrot-plumage">browse the repository</a> or just come by #parrot on irc.parrot.org and ping me (<code>japhb</code>). See you there! </p> geoffrey 2009-09-14T07:51:51+00:00 parrot Parrot Plumage Day 2/3: On the Shaving of Yaks http://use.perl.org/~geoffrey/journal/39598?from=rss <p> After spending <a href="http://use.perl.org/~geoffrey/journal/39516">Day 1</a> mostly exploring the boundaries of NQP, I was hoping to put the pedal to the metal and start the Parrot Plumage implementation in earnest. I ended up with more bald yaks instead. </p><p> During the first day I discovered that I needed two features added to NQP to make progress on the ecosystem tools: the ability to do cross-language <code>eval</code> (a prime raison d'&ecirc;tre for Parrot), and the ability to declare object attributes directly for proper OO (NQP having made do so far with implicit or PIR-coded attribute definitions). Attribute declaration was not ready yet, but Tene++ had produced a (surprisingly simple) implementation of cross-language <code>eval</code>, so I decided to push on in that direction. </p><p> The first thing I wanted to do was parse JSON data using the existing JSON parser that ships with Parrot. Unfortunately, it had been some time since the JSON parser had been updated, and it was still conforming to an older compiler API. At the raw PIR level, the old API looks like this: </p><blockquote><div><p><nobr> <wbr></nobr><tt>.local pmc json, data<br>load_bytecode&nbsp; 'compilers/json/JSON.pbc'<br>json = compreg 'JSON'<br>data = json(text)</tt></p></div> </blockquote><p> The old API is still fully functional for doing work with just one language (plus PIR, which is always available), but it doesn't support working with multiple high-level languages in the same program. Thankfully, the new API involves only minor changes: </p><blockquote><div><p><nobr> <wbr></nobr><tt>.local pmc json, code, data<br>load_language&nbsp; 'data_json'<br>json = compreg 'data_json'<br>code = json.'compile'(text)<br>data = code()</tt></p></div> </blockquote><p> Essentially, the new API makes just two changes. First, the compiler is loaded using the <code>load_language</code> op, rather than the more generic <code>load_bytecode</code> op. Second, rather than the compiler being a simple subroutine called directly on the source text to produce a final result, a compiler is now an object with a <code>compile</code> method that converts source text into a subroutine representing the "program". Since JSON is a non-executable language, this subroutine merely creates and returns the data structure representing the JSON text -- so the last step is to call the subroutine to get that data structure. </p><p> Updating the JSON parser to the new API would have been relatively simple, but for one problem -- the new API requires the compiler have a lowercase name. Those of us with some experience dealing with cross-platform development will immediately blanch upon discovering such a requirement and realizing (as in this case) that the existing JSON compiler not only used an uppercase name internally, but some of the files that implemented it were uppercased in source control, while others weren't. Oops. </p><p> After some discussion on <code>#parrot</code>, we decided the least havok for our users meant copying the compiler to a new name and deprecating the old one. Since we needed a new name, and "compilers" for data-only languages are somewhat special, we came up with the informal convention of prefixing the data format's name, in lowercase, with <code>data_</code>. Thus <code>data_json</code> was born. </p><p> Some hacking later, I discovered a few namespacing issues in the original JSON compiler; fixing those allowed <code>data_json</code> to finally work from high-level languages such as NQP, as well as from multiple namespaces in PIR. Leaving some final details (such as converting any existing tests) for another day -- or another enterprising coder, hint hint -- I went on to the next task. </p><p> Using the ecosystem tools should be as easy as possible for the user. Rather than this: </p><blockquote><div><p> <tt>parrot<nobr> <wbr></nobr>/path/to/NQP/compiler/nqp.pbc plumage.nqp install foo</tt></p></div> </blockquote><p> I'd much rather have this: </p><blockquote><div><p> <tt>plumage install foo</tt></p></div> </blockquote><p> Thus the next task was to figure out how to produce a proper executable from the <code>plumage.nqp</code> source. Some generous cargo culting, a (sortof) proper <code>Makefile</code>, and a <a href="http://gitorious.org/parrot-plumage/parrot-plumage/commit/6a904f5754bf434df99ae239fe009228e582e1ce#plumage.nqp">judicious hack</a> later, I could produce a working <code>plumage</code> executable for a trivial bit of NQP. </p><p> Sadly, by that point, my available time for the hacking session had run out, so we'll see what the next session brings. If you'd like to help, discuss interactions with other projects, or even just ask questions, come by <code>#parrot</code> on <code>irc.perl.org</code> and ping me (<code>japhb</code>). See you there! </p> geoffrey 2009-09-08T17:16:05+00:00 parrot Parrot Plumage Day 1: Much Ado About NQP http://use.perl.org/~geoffrey/journal/39516?from=rss <p> The time has clearly come to start work on Parrot's module ecosystem -- the module installer, search tool, dependency checker, and so on. We've been discussing various pieces on #parrot, #perl6, and parrot-dev for a few weeks, and a couple weeks ago we reached rough consensus. At that point, I collected all the comments and emails and wrote up a <a href="https://trac.parrot.org/parrot/wiki/ModuleEcosystem">draft design document</a>. I guess you could call that 'Day 0'. </p><p> Today I managed to commit a few hours to getting started actually <em>implementing</em> some of that design. It turned out that didn't happen, but I did manage to set up a <a href="http://gitorious.org/parrot-plumage/parrot-plumage">project repository for the prototype</a> and spent the rest of the time exploring NQP. </p><p> NQP stands for "Not Quite Perl 6", a (considerably) simplified variant of Perl 6 that lends itself well to optimization and clean implementation. It is one of the standard languages shipped with Parrot and generally used as a tool for creating more complex languages (such as, unsurprisingly, the Rakudo implementation of Perl 6). </p><p> Because it is shipped with Parrot, we can be confident that it will always be there, no matter what other languages the user may or may not have installed. And because it is a subset of Perl 6, it's easy to use (though a tad wordy in a few places that full Perl 6 provides heavier syntactic sugar). Thus I decided to implement the module tools using NQP. </p><p> Unfortunately, NQP is a little <em>too</em> simplified right now. It includes all the necessary features to implement parser actions for a compiler (the most common use for it), but this will be the first project using NQP to write fully independent command line tools. These tools are even more demanding in that they will need to work pretty extensively with platform dependencies -- from system configuration to process spawning. </p><p> Because there isn't any existing NQP code that does the kinds of things this project will need, I spent most of today exploring the boundaries of what existed in NQP already, what was missing, and how much could be filled in by borrowing magic from Rakudo. At this point, I can pull in environment variables, Parrot configuration, and OS information; and I can shell out to external commands, optionally capturing the output. That all went fairly smoothly, if slowly. </p><p> Unfortunately, I had more problems with writing OO code in NQP. It turns out that most of the OO code written in NQP creates classes implicitly or through hand-coded PIR. Even though there are <code>class</code> and <code>method</code> declarators, there appears to be no way in pure NQP to declare object attributes -- nor an obvious way to fake them using hand-coded <code>new</code>, <code>BUILD</code>, or <code>bless</code>. </p><p> Of course, I can always fall back to hand-coded PIR, even faking my own declarators, but there was some agreement that this is something NQP really should provide out of the box; I'll be talking with pmichaud++ about this in the coming days. </p><p> The project goal is to have at least the basic toolchain implemented and working smoothly for Parrot 2.0, in keeping with the official 2.0 vision: "Production Users". In order to have things running smoothly by then we need to get the prototype up and running soon, so that we can iterate several times before Parrot 2.0 is released. If you'd like to help, come by #parrot on irc.perl.org and ping me (<code>japhb</code>). See you there! </p> geoffrey 2009-08-24T06:30:12+00:00 parrot Limits of automated testing http://use.perl.org/~geoffrey/journal/24747?from=rss <p> Recently I was doing debugging-via-email of some graphics code and it turned out that the problem became perfectly obvious with a single screenshot of the demo scene. I realized that I have a habit of designing test scenes for my graphics code that make rendering errors pop out to the human eye -- most any problem becomes obvious in a second or two. This is most definitely a Good Thing. </p><p> Of course, many readers will point out that I should have a strong automated test library to catch these sorts of issues; the user should just be able to run <code>make test</code> and get a nice test failure describing the problem. That's great advice, and I'd love to give the users a nice automated test suite . . . but <em>how?</em> </p><p> The problem here is that OpenGL is guaranteed to produce consistent output from run to run, but only for the same hardware, drivers, resolution, color depth, and so on. Change any of these, and the output is allowed to change quite drastically. Yes, there are some bounds to how far afield the output can stray -- but these bounds are, to put it lightly, pretty loose. </p><p> So how do you automate the testing, when OpenGL drivers are often buggy and sometimes vary widely from vendor to vendor and system to system? You can't keep canonical images in the testing library to compare against because the output from the developer's system is unlikely to match pixel-by-pixel on basically anyone else's system. You can't even save the output from the first test run, and compare against it in succeeding builds; if the user happens to upgrade their video drivers, or alter the desktop resolution or color depth, or make some other completely reasonable change to their system, the pixels are quite likely to be different. </p><p> I <em>could</em> spend a great deal of effort writing a fuzzy image matcher that determined whether two images matched within the available OpenGL guarantees -- and submit the result as a dissertation on computer vision. It's certainly an interesting subject for research, and I wouldn't be surprised to hear that SGI did exactly this to create their conformance testing library, though I've never seen it -- but it has little to do with the graphics work in front of me. I'd rather spend my time actually producing working rendering code. </p><p> This brings me back to my original method -- design test scenes that can take advantage of the human visual system by making real rendering errors pop out of the noise of absolute pixel value differences. An incorrect pixel here or there would be difficult for the eye to detect (unless the color was WAY off), but I'm looking more for "Well, clearly something's wrong with texturing -- the big wooden box is rendered as a plain black cube." </p><p> Yes, a human needs to get involved. But with proper test design any problems should be obvious -- so obvious that the user may react even before being able to verbalize the issue, because reacting quickly to seeing something <em>wrong</em> is a survival trait. In many cases, this process can actually be <em>faster</em> than running a full automated test suite for a complex system. Granted, it won't help autobuilders with automated test runs on the latest source revision, but it does have its place. </p><p> None of this is specific to graphics code, of course; that's just the area I've played the most with. These testing issues will come up anywhere you need to test interactions with the real world, where APIs often don't guarantee perfect reproduction on all systems, in all environments. How do you know, for instance, that the user can actually hear the music you are trying to play? </p><p> In the last few years I've come across quite a few books and articles espousing automated testing, and going into great detail about how to write automated tests and test harnesses in every development environment and programming language under the sun. What seems to be missing is a decent treatise on producing good user-oriented tests. Maybe I'm just looking in the wrong places . . . . </p> geoffrey 2005-05-18T02:23:22+00:00 journal Win32 SDL_perl 1.20.3 released http://use.perl.org/~geoffrey/journal/24638?from=rss <p> Wayne Keenan is back working on the SDL_perl port for Win32, and I am hosting the binary builds for him. This <a href="http://www.broadwell.org/graphics/#downloads">release</a> brings the Win32 port up to 1.20.3 (matching most of the free *nixen), which means all of the examples from my <a href="http://www.broadwell.org/graphics/#articles">Building a 3D Engine in Perl articles</a> should now work perfectly. </p><p> Full PIGGE support and other fixes beyond 1.20.3 are in progress. </p> geoffrey 2005-05-11T20:21:43+00:00 journal PIGGE 0.1.2 released http://use.perl.org/~geoffrey/journal/24621?from=rss <p> Following the smashing success of yesterday's show, <a href="http://www.broadwell.org/graphics/pigge/">PIGGE</a> has now been held over for a second straight day, now in a spanking new version: 0.1.2! Faster, better, spiffier, and now with a <a href="http://www.broadwell.org/graphics/pigge/changelog">changelog</a>! </p><p> Changes copied below for your viewing pleasure: </p><ul> <li>Improve and refactor font and HUD support</li><li>Improve and refactor font test routine (the default HUD)</li><li>Autogenerate toggle_show_foo commands for all render.show.foo config items</li><li>Add ported toggleable HUD to challsall-blackhole port</li><li>Add ported HUD to chalsall-cube port</li><li>Skip rendering for invisible WorldObjects</li><li>Implement toggle_show_axes for chalsall-blackhole port</li><li>Mark toggle_lighting command as working instead of broken for chalsall-cube port</li><li>Fix some incorrect and/or incomplete statistics in the trends doc</li><li>Fix a previously untriggered braino bug in Callgraph.pm resulting in bad code output for the refactored draw_hud routine</li></ul> geoffrey 2005-05-11T01:00:54+00:00 journal Mike check and PIGGE first release http://use.perl.org/~geoffrey/journal/24601?from=rss <p> Testing, testing . . . </p><p> Check, check, check. </p><p> Is this thing on? Ah, good. Hello there! Good to see you all out there. We've got a marvelous show for you tonight and . . . <em> <small>What? What do you mean they cancelled? Bloody hell. Well what else have we got? Is that it? OK, we're running with it.</small> </em> </p><p> Right, OK folks, we've got a new show for you! It involves farm animals and explosions and . . . <em> <small>What was that again?</small> </em> . . . and dancing boxes! You'll laugh, you'll cry, you'll think I'm insane to have this on my stage! And you'll be <strong>right</strong>. </p><p> One and all, I present to you: <a href="http://www.broadwell.org/graphics/pigge/">PIGGE</a> version 0.1.1! </p><p> <em> <small>That's an awful low version number . . . first time on stage? You handed me an act that's never been on stage before?!? Oh man, I have <strong>got</strong> to get out of this business . . . I'll be ruined for sure, I can tell already . . .<nobr> <wbr></nobr>.</small> </em> </p> geoffrey 2005-05-09T18:35:47+00:00 journal