bart's Friends' Journals http://use.perl.org/~bart/journal/friends/ bart's Friends' use Perl Journals en-us use Perl; is Copyright 1998-2006, Chris Nandor. Stories, comments, journals, and other submissions posted on use Perl; are Copyright their respective owners. 2012-01-25T02:07:08+00:00 pudge pudge@perl.org Technology hourly 1 1970-01-01T00:00+00:00 bart's Friends' Journals http://use.perl.org/images/topics/useperl.gif http://use.perl.org/~bart/journal/friends/ DBD::SQLite 1.31 releasing next week and may break your code http://use.perl.org/~Alias/journal/40526?from=rss <p>After 6 or 7 months (mainly waiting around for the next "recommended upgrade instruction" from the SQLite project) the latest DBD::SQLite release should occur next week.</p><p>You can get the 1.30_06 release candidate from the CPAN, or from the following URL if your mirror hasn't synced yet.</p><p><a href="http://svn.ali.as/cpan/releases/DBD-SQLite-1.30_06.tar.gz">http://svn.ali.as/cpan/releases/DBD-SQLite-1.30_06.tar.gz</a></p><p>Apart from the normal batch of SQLite upgrades (from 3.6.22 to 3.7.2), bug fixes, and minor enhancements, this release has two changes that may break your code.</p><p>These changes have been in the dev releases for some time, but you may want to take the opportunity to test more intensively if you use either of the following features.</p><p>1. BLOB columns with UTF8 content</p><blockquote><div><p> <tt>- Resolved #54271: Inserting a string with utf-8 flag on<br>&nbsp; corrupts BLOB data; now BLOB data is always stored as bytes<br> &nbsp; (without the utf-8 flag) even if it has the flag set (ISHIGAKI)</tt></p></div> </blockquote><p>2. FTS3 queries</p><blockquote><div><p> <tt>- Added support for FTS3 tokenizers written in Perl. Added tests<br>&nbsp; and documentation on how to use FTS3. Changed compilation flag<br>&nbsp; to use the recommanded -DSQLITE_ENABLE_FTS3_PARENTHESIS<br>&nbsp; *** MAY POSSIBLY BREAK OLD APPLICATIONS THAT ALREADY USED FTS3 ***<br>&nbsp; (DAMI)</tt></p></div> </blockquote><p>If you are currently using FTS3, please see <a href="http://search.cpan.org/perldoc?DBD::SQLite::FTS3Transitional">DBD::SQLite::FTS3Transitional</a> which contains a helper function for automatically upgrading old FTS3 queries to the new syntax.</p> Alias 2010-09-09T02:11:54+00:00 journal Should Module::Install move to explicit plugin declaration? http://use.perl.org/~Alias/journal/40523?from=rss <p>Module::Install has been through a long period of gradual stability over the last year, without any really dramatic improvements to the grammar or APIs.</p><p>With the more urgent "it doesn't work with blah" stuff mostly solved now, one of the big remaining issues is around error clarity and excessive magic.</p><p>For example, some random author that is trying to checkout a Catalyst project needs:</p><p>1. To have Module::Install installed.<br>2. To have Module::Install::Catalyst installed.</p><p>In the case of the former, you get the semi-cryptic but at least standard "Can't find inc/Module/Install.pm in @INC" message, so the error is resolvable.</p><p>But in the latter case, you're likely to get something like "Unknown command 'catalyst_ignore'", with no real obvious resolution mechanism.</p><p>I think this idea of automatic plugin discovery is starting to hit it's limits in terms of clarity.</p><p>And so I'd like to do something counter to my natural instincts here, and make M:I more verbose.</p><p>I'm thinking of something like the following for explicitly declaring the use of a non-core Module::Install extension.</p><blockquote><div><p> <tt>use inc::Module::Install qw{ Catalyst XSUtil };</tt></p></div> </blockquote><p>This would both allow M:I to error with a much more meaningful error when you don't have a plugin, and also prevent the loading of unused plugins which should prevent accidental plugin collisions (some of which I've seen occurring in the CPAN Testers machines).</p><p>Thoughts?</p> Alias 2010-09-06T02:26:06+00:00 journal Amiga Ethernet http://use.perl.org/~scrottie/journal/40515?from=rss <p>Yesterday, I got my X-Surf 3cc Ethernet card I broke down and ordered for my Amiga 3000. There's some backstory about serial consoles, Sparcs, and the cluster, but it's not important. The 3000 was also packaged as a Unix machine, running a pretty standard port of SysV. It was the first Amiga standard with an MMU and SCSI. It'll also kick out 1280x resolution graphics at 2bpp. Commodore sold an Ethernet board for it along with Unix on tape.</p><p>The X-Surf is really an ISA card, probably NE2000, mounted in a little carrier. There are confusingly few pins attached and the logic on the carrier amounts to a few small 7400 series chips and one slightly larger chip that also couldn't possibly have enough logic on it to do what it does. And then just to convince you that your nuts, it adds an IDE port that alone has more lines than the one little adapter chip does. The Amiga really is a machine for psychopaths, by psychopaths. Everyone sits around all of the time trying to out psycho everyone else. Just take a look at the demo scene for the thing. Amiga virtually defined the demo scene.</p><p>I have/had Amiga OS 3.9 on the thing. 3.9 is post-Commodore death. Someone bought the rights and sold them and someone bought them and sold them and so on until a sue happy band of self righteous ruffians managed to convince the remaining user base buying the rights at garage sale prices entitled them to be king of the squalid kingdom so that they could go around lynching anyone else trying to do anything for the Amiga. Anyway, OS 3.9 is pretty recent as far as Amiga stuff goes, even though it's ten years old. Most people stopped at 3.1. 3.9 only came out on CD-ROM. The 3000 doesn't have a bay but it does have SCSI, so the CD-ROM, when needed, gets hung off the side with the case open. I could also set up an enclosure and plug it into the back. I could also probably buy one of those.</p><p>X-Surf's stuff did not want to install.</p><p>X-Surf actually had an installer, which is impressive. AmigaOS 3.x has a scripting language for installers and an interpreter for that. This installer gave you the choice of two TCP stacks. AmigaOS 3.9 comes with a TCP stack but you can still swap it out. It's a bit Windows 3.1-like in that regard. The options are GENESiS/AmiTCP and Miami. GENESiS, the AmiTCP configurerer and dialer that cames with AmiTCP, was shipped in a version requiring libraries not included in AmigaOS3.9 so it wouldn't run. AmiTCP would, and AmiTCP was on the HD, though buried a bit. Miami is shareware/crippleware. It required the same library, MagicUI, that I didn't have.</p><p>I spent hours sorting out what required what and what I did and didn't have and how these various packages worked and fit together. That's ignoring the device driver for the ethernet card which is straight forward. The Amiga has a directory for libraries (which end in<nobr> <wbr></nobr>.library; the Unix terseness is missing from AmigaOS even though a lot of the feel is there). AmigaOS3.9 also won't read iso9660 filesystem CDs. Perhaps some BoingBag update fixes that but the BoingBag updates themselves are large<nobr> <wbr></nobr>.lha archives. I'm avoiding plugging the serial line into a Unix machine and speaking kermit or zmodem or something to transfer stuff. I've been down that road. Eventually I burned AmigaSYS4, a version of AmigaOS3.9 with lots of add-ons and the various BoingBag updates on it, stick it in the Amiga, and was able to steal MUI off of it and get both TCP stacks running.</p><p>Amiga programmers love to do ports of Unix software and add GUIs. They've been doing this for ages. They've had gcc since the early ages of gcc, and I ran the Amylaar MUD driver on AmigaOS 1.3 to do development locally, also in the dark ages. Kicking around on aminet.net from the Amiga, I see PHP, MySQL, Apache, bittorrent, Python, bind9, samba, VNC, and all sorts of stuff. No one ports just the client. If they port the client, they port the server, too. In the case of AmiTCP, the suite of utilities you'd expect are there, such as host, finger, traceroute, and so on, but to configure TCP/IP, you run a little GUI program and it asks you questions. It took Linux ages to get to this point and Amiga was doing it long before. One of the extras on the Extras disc, even as far back as 1.3, was a version of emacs with drop down menus.</p><p>Completely unsurprisingly, the 16mhz 68030 processor running AWeb (which does some JavaScript) is vastly faster than firefox on my 1.2ghz Centrino Linux machine. Amiga programmers do not write slow software. It's entirely against their nature. Threading is fantastic. It'll do downloads, render several jpgs in the page, update the page layout as HTML comes across, and never lose snappy UI responsiveness. On firefox, I yank on the scrollbar only to have it ignore me and snap back, or else the scroll bar doesn't move at all, or the whole thing just goes away for a few heart sinking seconds, making me wonder if it just crashed.</p><p>My ambition is to get a desk in a shared office space going and stick this baby there with an updated video card that does high res, high bit depth graphics. If I'm willing to start replacing and upgrading chips on the motherboard, I can take the thing up to a gig of RAM, too, and NetBSD supports it if I ever decide I want to see how firefox runs on a 16mhz processor. What I'm really hoping for is someone to take the latest Coldfire chips from Motorola's spin off, Freescale, and do an 800mhz accelerator card for the Amiga 2000/3000/4000. That would RULE.</p><p>-scott</p> scrottie 2010-08-25T20:09:48+00:00 journal Speaking at Microsoft TechEd - Any issues you want raised? http://use.perl.org/~Alias/journal/40508?from=rss <p>Next week I will at Microsoft's TechEd Australia event, courtesy of Microsoft Australia and Microsoft Open Source Labs.</p><p>More specifically, I'll be attending the Open Source mini-conf and discussion day on Tuesday, and presenting in the Community Presentations to Microsoft session on the current state of Perl and Windows on Wednesday.</p><p>Likely topics will include a review of the first year of the CPAN Testing Lab and a second-generation based on their Cloud Services, free code signing certificates for open source developers, and what issues are slowing us down or blocking progress.</p><p>So consider this your opportunity to raise any outstanding issues you have with Microsoft and Perl. What problems are you still seeing, what would like fixed or changed, and what is on your want-to-have list?</p><p>I'll try to address as many of your issues as possible in the time I have available with them (which is actually pretty substantial).</p> Alias 2010-08-20T03:20:46+00:00 journal Cute caps http://use.perl.org/~jdavidb/journal/40507?from=rss <p>I'm doing some quick code generation (the output is Java), and I found myself writing the below routine. I like it because of the names I picked for the variables. Not exactly self-documenting (although it is when you think about it), but this is throwaway. You can probably tell what the code is doing and why I named the variables as I did, and you might be entertained.</p><blockquote><div><p> <tt>sub uc_prop<br>{<br>&nbsp; my($prop) = @_;<br>&nbsp; my $p = substr($prop, 0, 1);<br>&nbsp; my $P = uc($p);<br>&nbsp; my $rop = substr($prop, 1);<br>&nbsp; return "$P$rop";<br>}</tt></p></div> </blockquote> jdavidb 2010-08-19T21:55:41+00:00 journal Consistent GUIs; Or, Using WPF for Good and Not Evil http://use.perl.org/~Mark+Leighton+Fisher/journal/40505?from=rss <p> <a href="http://www.rollthunder.com/SoftwareThatDoesntSuck/WpfForGoodAndNotEvil.htm">Using WPF for Good and Not Evil</a> is a nice little write-up on how we, as developers, need to consider why and how we might change the user interface of programs developed in WPF. My take on it is that "Just because you can do something does not mean you SHOULD do something."</p><p> <i>(Ob.Perl: Perlesque should let you program directly in WPF by using the<nobr> <wbr></nobr>.NET libraries.)</i> </p> Mark Leighton Fisher 2010-08-19T16:43:59+00:00 others Bundles Packages Builds Releases Tests http://use.perl.org/~jozef/journal/40504?from=rss <p>When I showed Benjamin (a college of mine) my <a href="http://news.perlfoundation.org/2010/08/2010q3-grant-proposal-cpan-to.html">TPF 2010Q3 grant proposal</a>, we slipped to a discussion about deploying Perl software. Nice (and recent) list of different approaches can be found <a href="http://www.modernperlbooks.com/mt/2010/08/on-deployment.html">@modernperlbooks.com</a>. To sum it up =&gt; TIMTOWTDI. Which is good, but none of those is perfect. The Perl+CPAN world is way too complex.</p><p>During out discussion with Benjamin I proposed an idea of shipping the application altogether with the OS. Insane? The base Debian system is ~190MB, all the rest is needed for the application. Then deployment will be a matter of running this system on a virtual machine, somewhere in the cloud or in a simple chroot. (btw any Linux distribution can have any other Linux distribution working in a chroot) The files will never clash, all the "machines" would be dedicated. No fear of putting files <a href="http://jozef.kutej.net/2010/06/where-to-put-files.html">where they belong to</a>.</p><p> <small>(<a href="http://jozef.kutej.net/2010/08/bundles-packages-builds-releases-tests.html">crossposted</a>)</small> </p> jozef 2010-08-19T15:09:12+00:00 journal How I spent my day today (or, slowass.net pops a hole) http://use.perl.org/~scrottie/journal/40501?from=rss <p>1. Ran backups<br>2. Verified integrity of ssh on my local system versus last backup; changed local passwords<br>3. Verified integrity of my linode chpass with md5sum versus previous backup<br>4. Locked accounts; fixed changes to shell for system programs, removed additional accounts, changed passwords<br>5. Killed root processes and shells; accounted for all of the shells and processes in ps<br>6. Compared md5sums of everything in ps, login shells, rsync, inetd, su, vmlinuz, ps and various things between previous backup and current<br>7. compared nmap to netstat -lnp; accounted for netstat -lnp entries<br>8. Ran find to find setuid/setgid programs; verified no additional ones exist; ran md5sum against existing ones<br>9. Replace sshd, ssh and their config files and host keys; restarted sshd; relogged and changed passwords<br>10. Upgrade sshd<br>11. Killed<nobr> <wbr></nobr>.ssh directories<br>12. Temporarily took some services down until I can decide if I trust/replace them (squid, cron, sendmail)<br>13. diff -r'd between the two backups; read through the output to account for all changes to the system (new files and changed files) (several notable)<br>14. Ran find to find world writable files; ran find to find device files in the wilds of the filesystem</p> scrottie 2010-08-17T05:30:44+00:00 journal Class::XSAccessor now even faster'er http://use.perl.org/~Alias/journal/40497?from=rss <p>The new 1.07 release of <a href="http://search.cpan.org/perldoc?Class::XSAccessor">Class::XSAccessor</a> mentions the use a new somewhat-evil technique for making the code even faster than it was previously.</p><p>But how much faster is it?</p><p>The following are being run on a fairly typical corporate Windows XP machine, with Strawberry Perl 5.10.1 and thread support.</p><p>First, some benchmarks using the previous 1.05 release (two runs)</p><blockquote><div><p> <tt>Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true...<br>accessor_get:&nbsp; 1 wallclock secs ( 2.51 usr +&nbsp; 0.00 sys =&nbsp; 2.51 CPU) @ 3976143.14/s (n=10000000)<br>accessor_set:&nbsp; 2 wallclock secs ( 3.09 usr +&nbsp; 0.00 sys =&nbsp; 3.09 CPU) @ 3233107.02/s (n=10000000)<br>constructor: 16 wallclock secs (15.67 usr +&nbsp; 0.00 sys = 15.67 CPU) @ 638080.65/s (n=10000000)<br>&nbsp; &nbsp; &nbsp;false:&nbsp; 2 wallclock secs ( 1.91 usr +&nbsp; 0.00 sys =&nbsp; 1.91 CPU) @ 5243838.49/s (n=10000000)<br>&nbsp; &nbsp; getter:&nbsp; 1 wallclock secs ( 2.34 usr +&nbsp; 0.00 sys =&nbsp; 2.34 CPU) @ 4266211.60/s (n=10000000)<br> &nbsp; predicate:&nbsp; 1 wallclock secs ( 2.38 usr +&nbsp; 0.00 sys =&nbsp; 2.38 CPU) @ 4210526.32/s (n=10000000)<br>&nbsp; &nbsp; setter:&nbsp; 2 wallclock secs ( 3.27 usr +&nbsp; 0.00 sys =&nbsp; 3.27 CPU) @ 3061849.36/s (n=10000000)<br>&nbsp; &nbsp; &nbsp; true:&nbsp; 1 wallclock secs ( 1.80 usr +&nbsp; 0.00 sys =&nbsp; 1.80 CPU) @ 5564830.27/s (n=10000000)<br> &nbsp; <br>Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true...<br>accessor_get:&nbsp; 3 wallclock secs ( 2.51 usr +&nbsp; 0.00 sys =&nbsp; 2.51 CPU) @ 3976143.14/s (n=10000000)<br>accessor_set:&nbsp; 3 wallclock secs ( 3.14 usr +&nbsp; 0.00 sys =&nbsp; 3.14 CPU) @ 3183699.46/s (n=10000000)<br>constructor: 15 wallclock secs (15.73 usr +&nbsp; 0.00 sys = 15.73 CPU) @ 635566.29/s (n=10000000)<br>&nbsp; &nbsp; &nbsp;false:&nbsp; 2 wallclock secs ( 1.86 usr +&nbsp; 0.00 sys =&nbsp; 1.86 CPU) @ 5379236.15/s (n=10000000)<br>&nbsp; &nbsp; getter:&nbsp; 3 wallclock secs ( 2.50 usr +&nbsp; 0.00 sys =&nbsp; 2.50 CPU) @ 4000000.00/s (n=10000000)<br> &nbsp; predicate:&nbsp; 3 wallclock secs ( 2.47 usr +&nbsp; 0.00 sys =&nbsp; 2.47 CPU) @ 4050222.76/s (n=10000000)<br>&nbsp; &nbsp; setter:&nbsp; 4 wallclock secs ( 3.13 usr +&nbsp; 0.00 sys =&nbsp; 3.13 CPU) @ 3200000.00/s (n=10000000)<br>&nbsp; &nbsp; &nbsp; true:&nbsp; 2 wallclock secs ( 1.98 usr +&nbsp; 0.00 sys =&nbsp; 1.98 CPU) @ 5037783.38/s (n=10000000)</tt></p></div> </blockquote><p>And now again with the new 1.07 release.</p><blockquote><div><p> <tt>Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true...<br>accessor_get:&nbsp; 2 wallclock secs ( 1.75 usr +&nbsp; 0.00 sys =&nbsp; 1.75 CPU) @ 5711022.27/s (n=10000000)<br>accessor_set:&nbsp; 1 wallclock secs ( 2.69 usr +&nbsp; 0.00 sys =&nbsp; 2.69 CPU) @ 3721622.63/s (n=10000000)<br>constructor: 15 wallclock secs (15.62 usr +&nbsp; 0.00 sys = 15.62 CPU) @ 640000.00/s (n=10000000)<br>&nbsp; &nbsp; &nbsp;false:&nbsp; 1 wallclock secs ( 1.28 usr +&nbsp; 0.00 sys =&nbsp; 1.28 CPU) @ 7806401.25/s (n=10000000)<br>&nbsp; &nbsp; getter:&nbsp; 1 wallclock secs ( 1.56 usr +&nbsp; 0.00 sys =&nbsp; 1.56 CPU) @ 6397952.66/s (n=10000000)<br> &nbsp; predicate:&nbsp; 2 wallclock secs ( 1.92 usr +&nbsp; 0.00 sys =&nbsp; 1.92 CPU) @ 5205622.07/s (n=10000000)<br>&nbsp; &nbsp; setter:&nbsp; 3 wallclock secs ( 2.50 usr +&nbsp; 0.00 sys =&nbsp; 2.50 CPU) @ 4000000.00/s (n=10000000)<br>&nbsp; &nbsp; &nbsp; true:&nbsp; 2 wallclock secs ( 1.55 usr +&nbsp; 0.00 sys =&nbsp; 1.55 CPU) @ 6464124.11/s (n=10000000)<br> &nbsp; <br>Benchmark: timing 10000000 iterations of accessor_get, accessor_set, constructor, false, getter, predicate, setter, true...<br>accessor_get:&nbsp; 2 wallclock secs ( 1.78 usr +&nbsp; 0.00 sys =&nbsp; 1.78 CPU) @ 5614823.13/s (n=10000000)<br>accessor_set:&nbsp; 3 wallclock secs ( 2.63 usr +&nbsp; 0.00 sys =&nbsp; 2.63 CPU) @ 3809523.81/s (n=10000000)<br>constructor: 16 wallclock secs (15.69 usr +&nbsp; 0.00 sys = 15.69 CPU) @ 637429.88/s (n=10000000)<br>&nbsp; &nbsp; &nbsp;false:&nbsp; 2 wallclock secs ( 1.22 usr +&nbsp; 0.00 sys =&nbsp; 1.22 CPU) @ 8203445.45/s (n=10000000)<br>&nbsp; &nbsp; getter:&nbsp; 2 wallclock secs ( 1.53 usr +&nbsp; 0.00 sys =&nbsp; 1.53 CPU) @ 6535947.71/s (n=10000000)<br> &nbsp; predicate:&nbsp; 2 wallclock secs ( 1.78 usr +&nbsp; 0.00 sys =&nbsp; 1.78 CPU) @ 5614823.13/s (n=10000000)<br>&nbsp; &nbsp; setter:&nbsp; 2 wallclock secs ( 2.56 usr +&nbsp; 0.00 sys =&nbsp; 2.56 CPU) @ 3903200.62/s (n=10000000)<br>&nbsp; &nbsp; &nbsp; true:&nbsp; 2 wallclock secs ( 1.48 usr +&nbsp; 0.00 sys =&nbsp; 1.48 CPU) @ 6738544.47/s (n=10000000)</tt></p></div> </blockquote><p>The numbers are pretty impressive.</p><p>The 'accessor', 'setter', 'predicate' and 'true' methods are about 25% faster, while 'getter' is a whopping 60% faster and (curiously) 'false' is about 50% faster as well.</p><p>Constructors are really the only thing that hasn't changed.</p><p>Impressive work, even if the code is a bit risky.</p> Alias 2010-08-16T03:04:05+00:00 journal Why does Object::Tiny only support getters http://use.perl.org/~Alias/journal/40496?from=rss <p><a href="http://perlalchemy.blogspot.com/2010/08/objecttinyrw-and-moosexnonmoose.html">http://perlalchemy.blogspot.com/2010/08/objecttinyrw-and-moosexnonmoose.html</a></p><p>Zbigniew Lukasiak tries out <a href="http://search.cpan.org/peldoc?Object::Tiny">Object::Tiny</a> and wonders why it is that I didn't allow for the creation of setters when it is only a one line change.</p><p>Like most<nobr> <wbr></nobr>::Tiny modules the reason is a bit complex and the result of compromises.</p><p>Object::Tiny began as an attempt to create a lighter, faster, version of Class::Accessor. A way to bulk-generate the accessor code I had to type over and over again.</p><p>However, where I differ is a strong preference for light and elegant API design.</p><p>And so I decided to implement mine with as little implementation code as possible, and as little API code as possible.</p><p>Once you have decided to go down the simplicity path, there's a couple of standard techniques you often end up using.</p><p>The first and most important is state reduction.</p><p>In their introduction to Erlang, the founders of that language describe state as one of the main sources of failures in programs. And so anything that removes state, at the very least unnecessary state, is a positive. Especially if the state reduction also results in code reduction, and a reduction in computation.</p><p>So take the following example, where we create an object with some attributes and then run some code that will use those object attributes..</p><blockquote><div><p> <tt>my $object = Class-&gt;new;<br>$object-&gt;foo(1);<br>$object-&gt;bar(2);<br>$object-&gt;do_something;</tt></p></div> </blockquote><p>This is a use case that we see fairly often, but it's really quite horrible code. It is really only the object-oriented equivalent of something like the following.</p><blockquote><div><p> <tt>our $Object::foo = 1;<br>our $Object::bar = 2;<br>do_something('Object');</tt></p></div> </blockquote><p>It is especially bad code if the following code would throw an exception.</p><blockquote><div><p> <tt>my $object = Class-&gt;new;<br>$object-&gt;do_something;</tt></p></div> </blockquote><p>If this blows up, then you are REALLY doing something wrong, because you have allowed the creation of completely invalid objects. Now anybody taking one of these objects as a parameters needs to do with following.</p><blockquote><div><p> <tt>sub foo {<br>&nbsp; &nbsp; my $object = shift;<br>&nbsp; &nbsp; unless (<br>&nbsp; &nbsp; &nbsp; &nbsp; $object-&gt;isa('Class')<br>&nbsp; &nbsp; &nbsp; &nbsp; and<br>&nbsp; &nbsp; &nbsp; &nbsp; defined $object-&gt;foo<br>&nbsp; &nbsp; &nbsp; &nbsp; and<br>&nbsp; &nbsp; &nbsp; &nbsp; $object-&gt;foo &gt; 0<br>&nbsp; &nbsp; &nbsp; &nbsp; and<br>&nbsp; &nbsp; &nbsp; &nbsp; defined $object-&gt;bar<br>&nbsp; &nbsp; &nbsp; &nbsp; and<br>&nbsp; &nbsp; &nbsp; &nbsp; $object-&gt;bar &gt; 2<br>&nbsp; &nbsp; ) {<br>&nbsp; &nbsp; &nbsp; &nbsp; die "Invalid object";<br>&nbsp; &nbsp; }<br>}</tt></p></div> </blockquote><p>If you are going to create an object for something, you HAVE to be sure that the objects are trustworthy.</p><p>And so you should never allow objects to exist that are invalid. EVERY object should be a valid object.</p><p>At the absolute minimum objects should be able to default every attribute to something reasonable and unlikely to cause problems.</p><p>But this still results in excess and wasteful work, because the object has to transition through two or more states.</p><p>You start with an object with parameters and defaults, and you validate them. And then you change on of the attributes immediately, validating it AGAIN. In the mean time, your object exists in a state that it will never actually be used in.</p><p>And so everywhere you possibly can, you should be setting attributes in the constructor rather than afterwards.</p><blockquote><div><p> <tt>my $object = Class-&gt;new(<br>&nbsp; &nbsp; foo =&gt; 1,<br>&nbsp; &nbsp; bar =&gt; 2,<br>);<br>$object-&gt;do_something;</tt></p></div> </blockquote><p>Less state, less complexity, less CPU, and less bugs.</p><p>If we accept this model of pushing all the configuration into the object up front to reduce state, then why change the object arbitrarily?</p><p>In fact, anything that you ARE going to change should be done under very controlled conditions.</p><p>It should require a dedicated method to apply the change, it should require validation, and work. It shouldn't be trivial, and it shouldn't be automatic.</p><p>If I had my way, Moose would set is =&gt; 'ro' by default, to make people think before they go about simply allowing stuff to change.</p><p>It also happens to let you shrink down the API markedly.</p><p>There are three potential use cases available when implementing accessors. Everything readonly, everything readwrite, or mixed.</p><p>With Object::Tiny, I was aiming for the smallest possible code.</p><p>Implementing either all-readonly or all-readwrite can be done with the following.</p><blockquote><div><p> <tt>use Class qw{<br>&nbsp; &nbsp; foo<br>&nbsp; &nbsp; bar<br>};</tt></p></div> </blockquote><p>By contrast, if we want to allow mixed readonly and readwrite, we would need some way of distinguishing. Something like the following.</p><blockquote><div><p> <tt>use Class {<br>&nbsp; &nbsp; readonly =&gt; [ 'foo' ],<br>&nbsp; &nbsp; readwrite =&gt; [ 'bar' ],<br>};<br> &nbsp; <br>use Class [ qw{<br>&nbsp; &nbsp; foo<br>} ], [ {<br>&nbsp; &nbsp; bar<br>} ];<br> &nbsp; <br>use Class {<br>&nbsp; &nbsp; foo =&gt; 'ro',<br>&nbsp; &nbsp; bar =&gt; 'rw',<br>};</tt></p></div> </blockquote><p>No matter how you try, there's always an inherent additional element of complexity that results from the split between them.</p><p>And so the decision to go with all-readonly in Object::Tiny is a combination of these two issues.</p><p>If went with all-readwrite, I'm practically encouraging bad behaviour and more bugs. If I went with mixed accessors, the API would remain relative complex.</p><p>In the end, the best way to achieve both API simplicity and code safety is to only provide read-only accessors, and anything more complex should require both though and effort.</p> Alias 2010-08-15T01:39:50+00:00 journal Packaging Perl with Wix http://use.perl.org/~ddick/journal/40491?from=rss <p>Just got home after giving a talk on <a href="http://perl.net.au/wiki/Melbourne_Perl_Mongers/Meeting_History_2010_08">packaging Perl applications for win32 platforms using WiX.</a> </p> ddick 2010-08-11T12:36:32+00:00 journal CPAN Testers 2.0: The death of via email is wrong. http://use.perl.org/~jk2addict/journal/40488?from=rss <p>I finally got around to installing perlbrew, 4 flavors of perl and updating some modules for recent perls. I've been sending a lot of email test reports lately.</p><p>Last night I got the email from bitbucket about CPAN Testers 2.0 and the death of sending reports via email with wiki links/instructions on how to setup<nobr> <wbr></nobr>.cpanreports/config.ini for a metabase id/transport via HTTP.</p><p>This is all well and good, but it feels wrong to me. I use CPAN::Mini. The point of which is partly to have CPAN when you're not online. The same reason I use git. With SMTP emails I could just queue up test reports in the local postifx and they would get delivered later when I'm online.</p><p>Now, that's is no more. I'm forced to be online to send HTTP reports. Seems like a bad idea. NOw, the only thing I can do is toggle off reporting when I'm offline, which really means I'll hate doing that and just turn off reporting all together.</p><p>Why did reports via email have to go away?</p> jk2addict 2010-08-10T02:49:45+00:00 journal Ctypes update: pretty objects http://use.perl.org/~doubi/journal/40485?from=rss <p>Since my last couple of posts, things have been done: </p><ul> <li>Complete revamping of call logic</li><li>Finished objects for basic C Types (int, double etc.)</li><li>A fairly sound cast() function for Ctypes</li><li>Array objects</li></ul><p> Despite this, I don't think I'm going to get everything implemented for the GSoC deadline<nobr> <wbr></nobr>:-/ Still a long list of features I want to add. Particularly notable by their absence are Pointers, Structures, Unions, paramflags and support for special Win32 objects (COM stuff, HRESULT and friends).</p><p>Crap... when you list it like that, sounds pretty dire, don't it?</p><p>Don't beat me up just yet though. I'll definitely get Pointers done in the next 48 hours. Not sure how tricky Structs and Unions will be but they'll definitely be done by the deadline too. <i>Anyway</i>, instead of wasting more time worrying I want to talk a bit more about my favourite topic...</p><p> <b>Type object API revisited</b> </p><p>In response to my <a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">full post on the matter</a> [0], <a href="http://plasmasturm.org/">Aristotle</a> [1] pointed out that if tie'd behaviour was the objective we could have the rather nice</p><blockquote><div><p> <tt>c_int my $intobj = 5;</tt></p></div> </blockquote><p> This is cool, but I was scundered by the fact that you couldn't then call methods on your $intobj. Boo-hoo.</p><p>The solution was embarrassingly simple, actually. I didn't so much think of it as came across it naturally when implementing Array types. To make an $arrayobject act like an array, I overloaded <code>'@{}'</code> to return the tied member of its internal hash, so you use it as an array and as an object like so:</p><blockquote><div><p> <tt>use Ctypes;<br>my $array = Array( 1, 2, 3, 4, 5 );<br># makes array of smallest necessary type, c_short<br> <br>print $array;&nbsp; &nbsp; # Ctypes::Type::Array=HASH(0x8b2d550)<br> <br>print @$array;&nbsp; &nbsp;# 12345<br> <br>my $int1 = c_int( $$array[0] );<br># $int1-&gt;val = 3<br> <br>my $int2 = c_int( $array-&gt;[1] );<br># $int2-&gt;val = 2<br> <br>my $oops = c_int( $array[2] );<br># $oops = 0, $arr[x] is undef<br> <br>print "Top index of \$array is $#$array\n";<br># Top index of $array is 4<br> <br>print $array-&gt;type-&gt;typecode;&nbsp; &nbsp;# 's' for short</tt></p></div> </blockquote><p>Some might call sigil soup on that, but I like it. It gives you the best of both tied variables and objects. When using the object to do regular array stuff, you can think of '$array', including the dollar, as the var's identifier, so you put another sigil in front depending on what you want from it (@$array for contents as a list, $$array[x] for returning individual values, etc). For object stuff, just momentarily remember that '$array' is an object in its own right and call methods on it.</p><p>The embarrassing thing is, I basically overlooked the fact that <code>'${}'</code> is also overloadable. So we could easily assign to simple Type objects like <code>$$int = 10</code>, which is an improvement of 2 characters on the current shortest option, <code>$int-&gt;(10)</code>. Well, not counting the spaces...</p><p>But it's less about the character count than about looking at a piece of code and immediately seeing that the value 10 is being assigned to something, which is the semantic that operation is representing, rather than it appearing that some method is being called with 10 as an argument. I don't think the hyper-sigilism is too bad in exchange for that. It'll make Ctypes code 'distinctive' to look at<nobr> <wbr></nobr>:-)</p><p>As always, comments are strongly encouraged; on my pointer to array problem, the Type object API, or the project in general. Go on, take a shot...</p><p> [0] <a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.htm<nobr>l<wbr></nobr> </a> <br> [1] Aristotle: <a href="http://plasmasturm.org/">http://plasmasturm.org</a></p> doubi 2010-08-09T03:24:22+00:00 journal My reply to Removing database abstraction http://use.perl.org/~scrottie/journal/40482?from=rss <p>My longwinded response to http://blogs.perl.org/users/carey_tilden/2010/08/removing-database-abstraction.<nobr>h<wbr></nobr> tml :</p><p>Don't get trapped in the mindset of "I have to use X because if I don't, I'm doing it wrong". Quite often, if you don't use X, it's entirely too easy to do it wrong if you don't know what you're doing. You probably don't want to re-implement CGI parameter parsing, for example. But that's not the same thing as saying that you should always use CGI because it's a solved problem so never do anything else. Nothing is a solved problem. mod_perl is a valid contender to CGI and now Plack is a valid contender to mod_perl. FastCGI was and is a valid contender to mod_perl and servers like nginx do awesome thing. Yet, tirelessly, the fans of one explain that the competing ideas are somehow not valid.</p><p>Sorry, I'm trying to do proof by analogy here. It isn't valid except to the degree that it is. I'll get to databases in a minute.</p><p>Quick recap: there are lots of valid approaches; using an alternative is not the same as re-inventing the wheel.</p><p>Furthermore, the heaviest technology is seldom the long term winner. Witness the return to lighter HTTP pipelines. For ages, Apache boasted being a bit faster than IIS, in response to which I could only wonder why Apache was so slow.</p><p>Okay, back to databases. DBIx::Class to a relational database is a valid option. It's also very heavy. It alo doesn't really let you scale your web app out unless the database in question is DB2, Oracle, or one of a few of those that runs on a cluster with a lot of processors rather than just one computer. Otherwise you've just added a new bottleneck. DBIx::Class makes it harder to do real relational work -- subqueries, having, or anything hairy. At the very least, you have to create a class file with a lot of boilerplate, reference that file from other files that you made or generated, and stuff the same SQL into there. Abstracting the querying away in simple cases makes it easier to query the database without thinking about it. This leads you to query the database without thinking about it. That's a double edged sword. In some cases, that's fantastic.</p><p>Lego blocks make it easy to build things but you seldom buy home appliances built out of Legos. Even more so for Duplo blocks. Some times easy tools are in order; some times, low level engineering with an RPN HP calculator is absolutely in order.</p><p>Okay, I'll get back to databases in a minute here, but I want to talk about something outrageous for a moment -- not using a relational database at all.</p><p>I wrote and use Acme::State for simple command line and daemonized Web apps. It works well with Continuity and should work with the various Coro based Plack servers for the reason that the application stays entirely in one process. All it does is restore state of variables on startup and save them on exit or when explicitly requested. It kind of goes overboard on restoring state and does a good enough job that it breaks lots of modules if not confined to your own namespace, hence the Acme:: designation.</p><p>Similarly, people have used Data::Dumper or Storable directly (Acme::State uses Storable under the hood) to serialize datastructures on startup and exit. In AnyEvent frameworks, it's easy to set a timer that, on expiration, saves a snapshot. Marc Lehmann, the man who created the excellent Coro module, has patches to Storable to make it reenterant and incremental, so that the process (which might also be servicing network requests for some protocol) doesn't get starved for CPU while a large dump is made. His Perl/Coro based multiplayer RPG is based on this idea. With hundreds of users issuing commands a few times a second, this is the only realistic option. If you tried to create this level of performance with a database, you'd find that you had to have the entire working set in RAM not once but several times over in copies in parallel database slaves. That's silly.</p><p>You can be very high tech and not use a database. If you're not actually using the relational capabilities (normalized tables, joining tables, filtering and aggregating, etc), then a relational database is a slow and verbose to use (even with DBIx::Class) replacement for dbmopen() (perldoc -f dbmopen, but use the module instead). You're not gaining performance, elegance or scalability, in most cases. People use databases automatically and mindlessly now days to the point where they feel they have to, and by virtue of having to use a database, they have to ease the pain with an ORM.</p><p>Anytime someone says "always", be skeptical. You're probably talking to someone who doesn't understand or doesn't care about the tradeoffs involved.</p><p>Okay, back to databases. Right now, it's trendy to create domain specific languages. The Ruby guys are doing it and the Perl guys have been doing it for ages. Forth was created around the idea -- the Forth parser is written in Forth and is extensible. Perl 5.12 lets you hijack syntax parsing with Perl in a very Forth-ish style. Devel::Declare was used to create proper argument lists for functions inside of Perl. There's a MooseX for it and a standalone one, Method::Signatures. That idea got moved into core. XS::APItest::KeywordRPN is a demo. Besides that, regular expressions and POD are two other entire syntaxes that exist in<nobr> <wbr></nobr>.pl and<nobr> <wbr></nobr>.pm files. It's hypocritical to say that it's somehow "bad" to mix languages. It's true that you don't want your front end graphic designer editing your<nobr> <wbr></nobr>.pl files if he/she doesn't know Perl. If your designer does know Perl, and the code is small and doesn't need to be factored apart yet, what's the harm? It's possible to write extremely expressive code mixing SQL and Perl. Lots of people have written a lot of little wrappers. Here's one I sometimes use:</p><p>http://slowass.net/~scott/tmp/query.pm.txt</p><p>Finally, part of Ruby's appeal -- or any new language's appeal -- is lightness. It's easy to keep adding cruft, indirection, and abstraction and not realize that you're slowly boiling yourself to death in it until you go to a new language and get away from it for a while. The Ruby guys, like the Python guys before them, have done a good job of building up simple but versatile APIs that combine well with each other and keep the language in charge rather than any monolithic "framework". Well, except for Rails, but maybe that was a counter example that motivated better behavior. Look at Scrapi (and Web::Scraper in Perl) for an example.</p><p>Too much abstraction not only makes your code slow but it makes it hard to change development direction in the future when something cooler, faster, lighter and more flexible comes out. Just as the whole Perl universe spent ten years mired down in and entrenched in mod_perl, so is there a danger that DBIx::Class and Moose will outlive their welcome. POE, which was once a golden child, has already outlived its welcome as Coro and AnyEvent stuff has taken off. Now there are a lot of Perl programs broken up into tiny non-blocking chunks that are five times as long as they could be, and the effort to move them away from POE is just too great. The utility of a package should be weighed against the commitment you have to make to it. The commitment you have to make to it is simply how much you have to alter your programming style. With Moose as with POE, this degree is huge. DBIx::Class is more reasonable. Still, it's a cost, and things have costs.</p><p>Thank you for your article.</p><p>Regards,<br>-scott</p> scrottie 2010-08-06T21:53:04+00:00 journal Trailer Theory - Reinvented for Ignite Sydney as Economics http://use.perl.org/~Alias/journal/40480?from=rss <p>Back in 2005 in <a href="http://use.perl.org/~Alias/journal/23820">only my fifth use.perl post ever</a>, I outlined an idea I had been developing for a couple of years that I called "Trailer Theory".</p><p>A few years ago on my Portable Perl world hack'a'tour, I took with me a lightning talk version of the concept. It was a pretty crude talk but was received, it seemed, fairly well by the development community.</p><p>Since that trip, and inspired by the unexpected conversion of my "Perl is unparsable" claims in the PPI docs into a formal mathematical proof (complete with "Kennedy's Lemma") I've been wondering if this "Trailer Theory" idea could really be developed as a proper scientific proof, and if so what would that look like.</p><p>A couple of months ago I presented a new version of the talk at Ignite Sydney, speaking to an mixed audience of Twitterati, social media, advertising and journalist types.</p><p>I've rebuilt the talk from scratch and tried to outline the same idea, but in the form of a kind of layman's Economics Proof.</p><p>I hope you enjoy the result.</p><p><a href="http://igniteshow.com/videos/using-economics-make-movies-suck-less">Adam Kennedy - Using Economics to make movies suck less</a></p><p><i><br>Notes for other speakers:</i></p><p><i>1. Ignite advances slides every 15 seconds, no clickers allowed. This turns out to take a shitload of practice to get right.</i></p><p><i>2. When someone says to you "Here's your mark, you need to stay on this line to be in the fixed spot" it helps to pay attention. FAIL<nobr> <wbr></nobr>:)<br></i></p> Alias 2010-08-04T05:40:53+00:00 journal Protesting the Moose with sugary alternatives http://use.perl.org/~Alias/journal/40477?from=rss <p>At work, we've been experimenting with Moose because in our enormous and complex codebase we think we can probably benefit a lot from the extra rigour that it brings.</p><p>Before I continue, let me note that ours is a typical mod_perl enterprise setup with 6gig of memory per machine, so any memory consumed before the Apache fork is essentially free.</p><p>So none of the issues people (including me) have with startup code and memory consumption apply in this case, and I won't be addressing performance issues in this post.</p><p>The consensus of the half-dozen people at work is how Moose tries to look like a declarative extension, but doesn't actually act like it.</p><p>The following is what Moose seems to have been aiming for.</p><blockquote><div><p> <tt>package Foo;<br> &nbsp; <br>use Moose;<br> &nbsp; <br>extends 'Bar';<br>with&nbsp; &nbsp; 'Role1';<br>with&nbsp; &nbsp; 'Role2';<br> &nbsp; <br>has this =&gt; (<br>&nbsp; &nbsp; is&nbsp; =&gt; 'ro',<br>&nbsp; &nbsp; isa =&gt; 'Str',<br>);<br> &nbsp; <br>has that =&gt; (<br>&nbsp; &nbsp; is&nbsp; =&gt; 'ro',<br>&nbsp; &nbsp; isa =&gt; 'Str',<br>);<br> &nbsp; <br>1;</tt></p></div> </blockquote><p>Unfortunately, this is what we've had to do instead.</p><blockquote><div><p> <tt>package Foo;<br> &nbsp; <br>use Moose;<br> &nbsp; <br>BEGIN { # When we use Catalyst<br>&nbsp; &nbsp; extends 'Bar';<br>}<br> &nbsp; <br>has this =&gt; (<br>&nbsp; &nbsp; is&nbsp; =&gt; 'ro',<br>&nbsp; &nbsp; isa =&gt; 'Str',<br>);<br> &nbsp; <br>has that =&gt; (<br>&nbsp; &nbsp; is&nbsp; =&gt; 'ro',<br>&nbsp; &nbsp; isa =&gt; 'Str',<br>);<br> &nbsp; <br>with&nbsp; &nbsp; 'Role1';<br>with&nbsp; &nbsp; 'Role2';<br>no Moose;<br>__PACKAGE-&gt;meta-&gt;make_immutable;</tt></p></div> </blockquote><p>This "real Moose" code totally spoils the dream of what we felt like we were going to get when we started to play with it.</p><p>Most of our current options for fixing this amount to either.</p><p>a) Add this extra dependency that will unscrew one of the other of the problems (namespace::autoclean)</p><p>b) Use this SECOND heavy sugar layer on top of the FIRST sugar layer, on top of Class::MOP.</p><p>Is fixing the syntax or writing light weight sugar really so hard?</p><p>As a kind of protest, I tried it for myself and managed to create <a href="http://search.cpan.org/perldoc?MooseX::Atom">MooseX::Atom</a>.</p><p>This still has some flaws, but the current equivalent of the above would just be this.</p><blockquote><div><p> <tt>package Foo;<br> &nbsp; <br>use MooseX::Atom [<br>&nbsp; &nbsp; extends =&gt; 'Bar',<br>&nbsp; &nbsp; with&nbsp; &nbsp; =&gt; 'Role1',<br>&nbsp; &nbsp; with&nbsp; &nbsp; =&gt; 'Role2',<br>&nbsp; &nbsp; has&nbsp; &nbsp; &nbsp;=&gt; [<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;this =&gt; (<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;is&nbsp; =&gt; 'ro',<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;isa =&gt; 'Str',<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;),<br>&nbsp; &nbsp; ],<br>&nbsp; &nbsp; has&nbsp; &nbsp; &nbsp;=&gt; [<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;that =&gt; (<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;is&nbsp; =&gt; 'ro',<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;isa =&gt; 'Str',<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;),<br>&nbsp; &nbsp; ]<br>];<br> &nbsp; <br>1;</tt></p></div> </blockquote><p>You can do the same thing for roles with <a href="http://search.cpan.org/perldoc?MooseX::Role::Atom">MooseX::Role::Atom</a>.</p><p>Now clearly, this might have some issues. It's the work of an hour and not a whole lot of thought.</p><p>But it's still light and clean, with all the class spec in one place up the top where people are used to seeing the declarative stuff in Perl modules.</p><p>Perhaps something like this might be a little better...</p><blockquote><div><p> <tt>package Foo;<br> &nbsp; <br>use MooseX::Hash {<br>&nbsp; &nbsp; extends =&gt; 'Bar',<br>&nbsp; &nbsp; with&nbsp; &nbsp; =&gt; [ 'Role1', 'Role2' ],<br>&nbsp; &nbsp; default =&gt; { is =&gt; 'ro' },<br>&nbsp; &nbsp; has&nbsp; &nbsp; &nbsp;=&gt; {<br>&nbsp; &nbsp; &nbsp; &nbsp; this =&gt; { isa =&gt; 'Str' },<br>&nbsp; &nbsp; &nbsp; &nbsp; that =&gt; { isa =&gt; 'Str' },<br>&nbsp; &nbsp; },<br>);<br> &nbsp; <br>1;</tt></p></div> </blockquote> Alias 2010-08-03T05:09:02+00:00 journal XML + XPath + GUI = Xacobeo http://use.perl.org/~jozef/journal/40476?from=rss <p> <a href="http://search.cpan.org/perldoc?Xacobeo">Xacobeo</a> a Perl GUI to visualize XML and perform XPath queries just entered Debian unstable - <a href="http://packages.debian.org/sid/xacobeo">http://packages.debian.org/sid/xacobeo</a>.</p> jozef 2010-08-02T14:07:42+00:00 journal DEFCON, day 1 http://use.perl.org/~scrottie/journal/40473?from=rss <p>Basically no mischief or craziness. Having DEFCON at a casino did to it exactly what I would have expected. No money pots to eat the cockroach, no naked fire jugglers, no getting thrown in the pool, no parties by the pool.</p><p>Bros outnumber the blackshirts now. They're talking loudly and proudly about how little they know and care.</p><p>Kilts are representing, too. The freaks are here. There's a Japanese gang dressed in kimonos. Some other Japanese guys walked by talking vigorously amonst themselves, laughing and pointing. Punks with combs, raisers and hairspray are in the contest area/lounge dispensing mohawks. They have their own official area. Strange hats abound. One kid has a fez. There are BDUs and lab coats. Lots of colored hair.</p><p>Aha! Finally spotted someone I knew -- Kevin, a friend of Ernie's, who also worked in the gaming industry, but on a different side of it.</p><p>People are sitting next to me reading long hex strings from the "background" of the talks description book.</p><p>They ran out of badges as usual. My flight got delayed two hours which seriously cut into my time here.</p><p>"My only crime is that of outsmarting you"... shirts have slogans.</p><p>There's a lot less interesting in WiFi and a lot more in smartphones. The common area is woefully inadequate.</p><p>UnixSurplus is here again and people are packing in to see old Unix hardware. Some people are going to be coming home with O2s.</p><p>More later, perhaps.</p><p>-scott</p> scrottie 2010-07-30T19:43:53+00:00 journal Stupid Lucene Tricks: Document Frequencies and NOT http://use.perl.org/~Mark+Leighton+Fisher/journal/40471?from=rss <ol> <li>You can get the document frequency of a term (i.e. how many documents have that term) through <b>Lucene.Index.IndexReader.DocFreq(t As Term) As Integer</b>.</li><li>You can get the <b>IndexReader</b> for a <b>Lucene.Search.IndexSearcher</b> through <b>IndexSearcher.GetIndexReader()</b>.</li><li>If you want to display the document frequencies for the individual keywords of a search, and a piece is a NOT phrase (like <i>-antibiotic</i> in <i>antimicrobial -antibiotic</i>), you cannot use <b>DocFreq()</b> directly. In that case, the document frequency can be computed as:<blockquote><div><p> <tt>&nbsp; &nbsp; &nbsp; DOCFREQ = count of all documents - DocFreq(TERM_NO_NOT)</tt></p></div> </blockquote><p>as in:</p><blockquote><div><p> <tt>&nbsp; &nbsp; &nbsp; DOCFREQ = 60227 - DocFreq(New Term("all", "antibiotic"))</tt></p></div> </blockquote><p> where the NOT piece was <i>-antibiotic</i> and <b>all</b> is the Lucene document field in question.</p></li></ol><p> <i>(Ob. Perl: Although PLucene is now 5 years out of date, <a href="http://code.google.com/p/csmeta/">Perlesque</a> should eventually let you get at Lucene.NET via a strongly-typed Perl 6.)</i> </p> Mark Leighton Fisher 2010-07-29T16:13:30+00:00 others Help? http://use.perl.org/~Alias/journal/40470?from=rss <p>(Before I begin, I should clarify I did not write this code, I'm just trying to maintain it)</p><p>The following error is the first thing spat out by make test for the Padre sync server, located at <a href="http://svn.perlide.org/padre/trunk/Madre-Sync">http://svn.perlide.org/padre/trunk/Madre-Sync</a>.</p><p>Wasn't the move of Catalyst to Moose going to make things easier?</p><p>Can someone explain how you debug this?</p><p>I get the basics, I can see the "Can't locate Madre/Sync/Schema.pm". But that file should be, I think, automatically generated. And I don't really get how to dig down the 75 caller levels from the start to the end to work out where the actual functionality is failing...</p><blockquote><div><p> <tt>not ok 1 - use Catalyst::Test;<br> &nbsp; <br>#&nbsp; &nbsp;Failed test 'use Catalyst::Test;'<br>#&nbsp; &nbsp;at t\01app.t line 7.<br>#&nbsp; &nbsp; &nbsp;Tried to use 'Catalyst::Test'.<br>#&nbsp; &nbsp; &nbsp;Error:&nbsp; Couldn't load class (Madre::Sync) because: Couldn't instantiate component "Madre::Sync::Model::padreDB", "Can't locate Madre/Sync/Schema.pm i<br>n @INC (@INC contains: blib\lib blib\arch C:/strawberry/perl/lib C:/strawberry/perl/site/lib C:\strawberry\perl\vendor\lib<nobr> <wbr></nobr>.). at C:\strawberry\perl\vendor<br>\lib/Class/MOP.pm line 132<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::load_first_existing_class('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::load_class('Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Catalyst/Model/DBIC/Schema/Types.pm line 21<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Model::DBIC::Schema::Types::__ANON__[C:\strawberry\perl\vendor\lib/C<nobr>a<wbr></nobr> talyst/Model/DBIC/Schema/Types.pm:21]('Madre::Sync::Schema') called<br>at C:\strawberry\perl\vendor\lib/Moose/Meta/TypeCoercion.pm line 63<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::TypeCoercion::__ANON__[C:\strawberry\perl\vendor\lib/Moose/Meta/T<nobr>y<wbr></nobr> peCoercion.pm:67]('Madre::Sync::Schema') called at C:\strawberry\per<br>l\vendor\lib/Moose/Meta/TypeCoercion.pm line 97<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::TypeCoercion::coerce('Moose::Meta::TypeCoercion=HASH(0x4a2b444)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/Moose<br>/Meta/TypeConstraint.pm line 90<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::TypeConstraint::coerce('Moose::Meta::TypeConstraint=HASH(0x4a2985<nobr>4<wbr></nobr> )', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\lib/M<br>ooseX/Types/TypeDecorator.pm line 206<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval {...} called at C:\strawberry\perl\vendor\lib/MooseX/Types/TypeDecorator.pm line 205<br>#&nbsp; &nbsp; &nbsp; &nbsp;MooseX::Types::TypeDecorator::AUTOLOAD('MooseX::Types::TypeDecorator=HASH(0x4a<nobr>3<wbr></nobr> 1424)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\l<br>ib/Moose/Meta/Attribute.pm line 743<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x4b46<nobr>7<wbr></nobr> fc)', 'Madre::Sync::Schema', 'Madre::Sync::Model::padreDB=HASH(0x4db<br>7c64)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Attribute.pm line 398<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(<nobr>0<wbr></nobr> x4b467fc)', 'Moose::Meta::Instance=HASH(0x4db7ed4)', 'Madre::Sync::M<br>odel::padreDB=HASH(0x4db7c64)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Class.pm line 567<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/C<br>lass.pm line 540<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Class.pm<br>line 256<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Object.pm lin<br>e 25<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Object::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at generated method (unknown origin) line 4<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Model::DBIC::Schema::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at C:\strawberry\perl\vendor\lib/MooseX/<br>Traits/Pluggable.pm line 131<br>#&nbsp; &nbsp; &nbsp; &nbsp;MooseX::Traits::Pluggable::new_with_traits('Madre::Sync::Model::padreDB', 'Madre::Sync') called at C:\strawberry\perl\vendor\lib/CatalystX/Componen<br>t/Traits.pm line 146<br>#&nbsp; &nbsp; &nbsp; &nbsp;CatalystX::Component::Traits::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Cl<br>ass/MOP/Method/Wrapped.pm line 48<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/<nobr>M<wbr></nobr> ethod/Wrapped.pm:49]('Madre::Sync::Model::padreDB', 'Madre::Sync', '<br>HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 89<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Model::DBIC::Schema::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/C<br>atalyst.pm line 2502<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2502<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::setup_component('Madre::Sync', 'Madre::Sync::Model::padreDB') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2416<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 54<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/<nobr>M<wbr></nobr> ethod/Wrapped.pm:64]('Madre::Sync') called at C:\strawberry\perl\ven<br>dor\lib/Class/MOP/Method/Wrapped.pm line 89<br>#&nbsp; &nbsp; &nbsp; &nbsp;Madre::Sync::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 1142<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::setup('Madre::Sync') called at blib\lib/Madre/Sync.pm line 62<br>#&nbsp; &nbsp; &nbsp; &nbsp;require Madre/Sync.pm called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 114<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:118]() called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 74<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval {...} called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 67<br>#&nbsp; &nbsp; &nbsp; &nbsp;Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('C<nobr>a<wbr></nobr> talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a<br>t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call<br>ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S<br>ub/Exporter.pm line 742<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Ca<nobr>t<wbr></nobr> alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per<br>l\vendor\lib/Catalyst/Test.pm line 112<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2<br>#&nbsp; &nbsp; &nbsp; &nbsp;main::BEGIN() called at blib\lib/Madre/Sync.pm line 0<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval {...} called at blib\lib/Madre/Sync.pm line 0<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval 'package main;<br># use Catalyst::Test @{$args[0]};<br># 1;<br>#<br>#<nobr> <wbr></nobr>;' called at C:/strawberry/perl/lib/Test/More.pm line 858<br>#&nbsp; &nbsp; &nbsp; &nbsp;Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p<br>m line 833<br>#&nbsp; &nbsp; &nbsp; &nbsp;Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7<br>#&nbsp; at C:\strawberry\perl\vendor\lib/MooseX/Types/TypeDecorator.pm line 208<br>#&nbsp; &nbsp; &nbsp; &nbsp;MooseX::Types::TypeDecorator::AUTOLOAD('MooseX::Types::TypeDecorator=HASH(0x4a<nobr>3<wbr></nobr> 1424)', 'Madre::Sync::Schema') called at C:\strawberry\perl\vendor\l<br>ib/Moose/Meta/Attribute.pm line 743<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x4b46<nobr>7<wbr></nobr> fc)', 'Madre::Sync::Schema', 'Madre::Sync::Model::padreDB=HASH(0x4db<br>7c64)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Attribute.pm line 398<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(<nobr>0<wbr></nobr> x4b467fc)', 'Moose::Meta::Instance=HASH(0x4db7ed4)', 'Madre::Sync::M<br>odel::padreDB=HASH(0x4db7c64)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Class.pm line 567<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Class/MOP/C<br>lass.pm line 540<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Meta/Class.pm<br>line 256<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x4942ffc)', 'HASH(0x4db53ac)') called at C:\strawberry\perl\vendor\lib/Moose/Object.pm lin<br>e 25<br>#&nbsp; &nbsp; &nbsp; &nbsp;Moose::Object::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at generated method (unknown origin) line 4<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Model::DBIC::Schema::new('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4b404ec)') called at C:\strawberry\perl\vendor\lib/MooseX/<br>Traits/Pluggable.pm line 131<br>#&nbsp; &nbsp; &nbsp; &nbsp;MooseX::Traits::Pluggable::new_with_traits('Madre::Sync::Model::padreDB', 'Madre::Sync') called at C:\strawberry\perl\vendor\lib/CatalystX/Componen<br>t/Traits.pm line 146<br>#&nbsp; &nbsp; &nbsp; &nbsp;CatalystX::Component::Traits::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Cl<br>ass/MOP/Method/Wrapped.pm line 48<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/<nobr>M<wbr></nobr> ethod/Wrapped.pm:49]('Madre::Sync::Model::padreDB', 'Madre::Sync', '<br>HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 89<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Model::DBIC::Schema::COMPONENT('Madre::Sync::Model::padreDB', 'Madre::Sync', 'HASH(0x4c6b93c)') called at C:\strawberry\perl\vendor\lib/C<br>atalyst.pm line 2502<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2502<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::setup_component('Madre::Sync', 'Madre::Sync::Model::padreDB') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 2416<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP/Method/Wrapped.pm line 54<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::Method::Wrapped::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP/<nobr>M<wbr></nobr> ethod/Wrapped.pm:64]('Madre::Sync') called at C:\strawberry\perl\ven<br>dor\lib/Class/MOP/Method/Wrapped.pm line 89<br>#&nbsp; &nbsp; &nbsp; &nbsp;Madre::Sync::setup_components('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst.pm line 1142<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::setup('Madre::Sync') called at blib\lib/Madre/Sync.pm line 62<br>#&nbsp; &nbsp; &nbsp; &nbsp;require Madre/Sync.pm called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 114<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:118]() called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 74<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval {...} called at C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 67<br>#&nbsp; &nbsp; &nbsp; &nbsp;Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('C<nobr>a<wbr></nobr> talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a<br>t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call<br>ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S<br>ub/Exporter.pm line 742<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Ca<nobr>t<wbr></nobr> alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per<br>l\vendor\lib/Catalyst/Test.pm line 112<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2<br>#&nbsp; &nbsp; &nbsp; &nbsp;main::BEGIN() called at blib\lib/Madre/Sync.pm line 0<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval {...} called at blib\lib/Madre/Sync.pm line 0<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval 'package main;<br># use Catalyst::Test @{$args[0]};<br># 1;<br>#<br>#<nobr> <wbr></nobr>;' called at C:/strawberry/perl/lib/Test/More.pm line 858<br>#&nbsp; &nbsp; &nbsp; &nbsp;Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p<br>m line 833<br>#&nbsp; &nbsp; &nbsp; &nbsp;Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7"Compilation failed in require at C:\strawberry\perl\vendor\lib/Class<br>/MOP.pm line 114.<br>#&nbsp; at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 121<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::__ANON__[C:\strawberry\perl\vendor\lib/Class/MOP.pm:125]('Couldn\'<nobr>t<wbr></nobr> instantiate component "Madre::Sync::Model::padreDB"...') called at<br>C:\strawberry\perl\vendor\lib/Try/Tiny.pm line 98<br>#&nbsp; &nbsp; &nbsp; &nbsp;Try::Tiny::try('CODE(0x36d759c)', 'Try::Tiny::Catch=REF(0x33a9584)') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 125<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::load_first_existing_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Class/MOP.pm line 137<br>#&nbsp; &nbsp; &nbsp; &nbsp;Class::MOP::load_class('Madre::Sync') called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 24<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Test::__ANON__[C:\strawberry\perl\vendor\lib/Catalyst/Test.pm:93]('C<nobr>a<wbr></nobr> talyst::Test', 'all', 'HASH(0x36d768c)', 'HASH(0x25de3a4)') called a<br>t C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 493<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::_expand_group('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d4994)', 'HASH(0x25de3a4)', 'HASH(0x36d755c)', 'HASH(0x25de334)') call<br>ed at C:\strawberry\perl\vendor\lib/Sub/Exporter.pm line 424<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::_expand_groups('Catalyst::Test', 'HASH(0x36d4ce4)', 'ARRAY(0x36d723c)', 'HASH(0x25de3a4)') called at C:\strawberry\perl\vendor\lib/S<br>ub/Exporter.pm line 742<br>#&nbsp; &nbsp; &nbsp; &nbsp;Sub::Exporter::__ANON__[C:\strawberry\perl\vendor\lib/Sub/Exporter.pm:756]('Ca<nobr>t<wbr></nobr> alyst::Test', '-all', 'HASH(0x36cc8d4)') called at C:\strawberry\per<br>l\vendor\lib/Catalyst/Test.pm line 112<br>#&nbsp; &nbsp; &nbsp; &nbsp;Catalyst::Test::import('Catalyst::Test', 'Madre::Sync') called at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2<br>#&nbsp; &nbsp; &nbsp; &nbsp;main::BEGIN() called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 2<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval {...} called at C:\strawberry\perl\vendor\lib/Catalyst/Test.pm line 2<br>#&nbsp; &nbsp; &nbsp; &nbsp;eval 'package main;<br># use Catalyst::Test @{$args[0]};<br># 1;<br>#<br>#<nobr> <wbr></nobr>;' called at C:/strawberry/perl/lib/Test/More.pm line 858<br>#&nbsp; &nbsp; &nbsp; &nbsp;Test::More::_eval('package main;\x{a}use Catalyst::Test @{$args[0]};\x{a}1;\x{a}', 'ARRAY(0x2308474)') called at C:/strawberry/perl/lib/Test/More.p<br>m line 833<br>#&nbsp; &nbsp; &nbsp; &nbsp;Test::More::use_ok('Catalyst::Test', 'Madre::Sync') called at t\01app.t line 7<br># BEGIN failed--compilation aborted at (eval 11)[C:/strawberry/perl/lib/Test/More.pm:858] line 2.</tt></p></div> </blockquote> Alias 2010-07-29T13:23:44+00:00 journal You just upload your application, ... http://use.perl.org/~jozef/journal/40465?from=rss <p>It is amazing what happened to the Python thanks to just one company. The killer feature of that company is that they can handle and manage millions of servers and even give them for a public usage =&gt; "..., there are no servers to maintain: You just upload your application, and it's ready to serve your users.". I would like this to be true one day for Perl too!</p><p>I really do. In coming months (years?) I'll try to plug together &#171; MiniCPANInject + SmokeTesting + ContinuousIntegration + Virtualization + Debian + Monitoring &#187; to create an architecture for Debian-Perl hosting. Actually who cares about OS used, when the uploaded code just works, right?... A hosting that is aware that one server is not enough for everyone and will offer an architecture for development, user-acceptance, staging and production environments to start with a project from $day == 1.</p><p> <small>(<a href="http://jozef.kutej.net/2010/07/you-just-upload-your-application.html">crossposted</a>)</small> </p> jozef 2010-07-28T17:00:46+00:00 journal Desperate Perl; or A Tale of Two Languages http://use.perl.org/~Mark+Leighton+Fisher/journal/40463?from=rss <p>Piers Cawley's <a href="http://www.bofh.org.uk/2010/07/25/a-tale-of-two-languages">A tale of two languages</a> (if you haven't already seen it) speaks to the public perception that Perl remains a desperation language ("Desperate Perl") suited only for gluing things together when nothing else will do.</p><p>Meanwhile elsewhere in the real world, there is plenty (possibly a majority IMHO) of maintainable, understandable, well-written, efficient Perl code ("Large Scale Perl" as described by Piers). Worth a read.</p><p> <i>(Although I like the name "Desperate Perl" a lot, I think that the names "Scripting Perl" and "Programming Perl" also describe these separate Perl programming styles in a less-emotional fashion (which is occasionally useful.))</i> </p> Mark Leighton Fisher 2010-07-27T17:25:47+00:00 links Strawberry Perl install rolled back http://use.perl.org/~jdavidb/journal/40462?from=rss <p>Strawberry Perl 5.12.0 was almost completely installed when suddenly it flashed some message I didn't see into the install wizard and the progress bars started moving backward! I have never seen anything like it. I realized the progress bar caption had been changed to simply "Rolling Back Action" and watched as at least three anonymous "actions" were rolled back, progress bar by progress bar. Then the install wizard simply told me "Strawberry Perl Setup Wizard ended prematurely Strawberry Perl Setup Wizard ended prematurely because of an error. Your system has not been modified. To install this program at a later time, run Setup Wizard again. Click the Finish button to exit the Setup Wizard."</p><p>I wish it would tell me what the error was so I might have some hope of correcting it.</p> jdavidb 2010-07-26T20:34:33+00:00 journal For people running Perl conferences http://use.perl.org/~Alias/journal/40461?from=rss <p>Leo Lapworth comments in <a href="http://blogs.perl.org/users/leo_lapworth/2010/07/for-speakers-at-perl-conferences.html">"For speakers at Perl conferences"</a> that you should record your own talks at conferences, because organisers cannot be trusted to release the videos they take of you.</p><p>I wholeheartedly agree. I've spoken at numerous conferences, including several large ones, and only in about 10-20% of cases have the videos of me EVER appeared.</p><p>I'm amazed that conference organisers can put in so much effort into recording (dozens of tapes, multiple cameras, multiple operators) and then never produce anything as a result.</p><p>The only time I'm aware of that a full talk of mine has appeared online was at a Linux.conf.au, who had a dedicated video team of eight people.</p><p>Where is all this footage, and why does it never get processed? Why even bother recording it? If you don't have time, send me the raw tape of my own talk and I will do it myself if needed.</p><p>YAPC::NA? You listening?</p> Alias 2010-07-25T12:55:04+00:00 journal A reminder - Padre Second Birthday Hackathon this weekend http://use.perl.org/~Alias/journal/40460?from=rss <p>This is just a quick reminder that this weekend is the Padre's second birthday party and hackathon.</p><p>If you are a Padre user, please drop in and say hello to the team. We'd love to hear how you are using Padre and where your main needs are for the next year.</p><p>If you are interested in trying out Padre for the first time, or trying your hand at improving it for the first time, it's a great time to get started because we'll have plenty of people around to provide guidance and advice.</p><p>Some of the plans for this weekend are to bring all the plugins up to date with the latest versions of the plugin API, to start the merge of the ConfigSync branch, and if we can, to start on the Madre server that will serve as the ConfigSync and Telemetry server for Padre.</p><p>I look forward to seeing you there!</p> Alias 2010-07-23T03:53:28+00:00 journal The mod_perl debugger you've always wanted http://use.perl.org/~jjore/journal/40458?from=rss <p>"The mod_perl debugger you've always wanted" is on my new blog at: <a href="http://bit.ly/agzWSU">http://bit.ly/agzWSU</a></p> jjore 2010-07-22T17:53:07+00:00 journal Profiling your website while live on a production cluster http://use.perl.org/~Alias/journal/40457?from=rss <p>As I mentioned in my last post, by day I wrangle a large web application that occasionally verges on being too complex to for mere humans to understand.</p><p>Curiously, because it is private and the income rate is high (we probably average something like $5 in gross turnover per page view) we don't have to deal with a lot of servers to deliver it, by internet standards anyway.</p><p>But the application is still way too big to profile easily by normal methods, and certainly on production it's way too heavy, even if we applied targeted profiling using something like <a href="http://search.cpan.org/perldoc?Aspect::Library::NYTProf">Aspect::Library::NYTProf</a>.</p><p>Between the web servers, transaction server, database, search engine and cache server, we are probably only dealing with 12 servers and 30 CPUs. Of course, these servers are all horribly expensive, because they are all server-virtualised, network-virtualised, doubly redundant (high-availability + disaster-recovery) and heavily monitored with high end support contracts.</p><p>One of our most sensitive issues is database load.</p><p>We have a ton of database tables (about 200) and lots of medium sized queries running across them. One of our main failure modes is that some deep change to code boosts the quantity of some significant query, which stresses the database enough to cause contention and lock-storms, leading to site failure.</p><p>Complicating things, big parts of some pages are embedded in other pages. So attributing load and lag to one part of the website, or to Oracle, is tricky and hard to benchmark in advance (although we do load test the main load paths to catch the worst cases).</p><p>For a long time, we've had a mechanism for zoned profiling the production site, so we can allocate wallclock costs to different general areas of the site.</p><p>But it is fragile and unreliable, requiring perfect maintenance and every developer to remember to write this kind of thing everywhere.</p><blockquote><div><p> <tt># Embed a foo widget in the current page<br>$perf-&gt;push_timing('foo');<br>foo($bar);<br>$perf-&gt;pop_timing;</tt></p></div> </blockquote><p>Since you don't know this profiling system exists unless you've seen it somewhere else in the code before, and it's hard to care about something that is orthogonal to the problem you are actuall solving, this mechanism has degraded over time. While we still get some pretty cacti graphs showing load breakdown, they are highly unreliable and you can never be entirely sure if the load attribution is correct.</p><p>This kind of performance monitoring as a "cross-cutting concern" is a textbook use case for Aspect-Oriented Programming, and so in our Christmas 2009 code freeze window I set about trying to rewrite the push/pop_timing profiler using <a href="http://search.cpan.org/perldoc?Aspect">Aspect.pm</a>.</p><p>Unfortunately, Aspect.pm turned out to be a bit too slow and naive for my needs. But after a 6 month delay to do a 90% rewrite of Aspect.pm, I now finally have something sophisticated enough (and fast enough) to meet my needs.</p><p>So today I'm releasing the shiny new <a href="http://search.cpan.org/perldoc?Aspect::Library::ZoneTimer">Aspect::Library::ZoneTimer</a>, which will serve as the main plank of our new production profiling system.</p><p>The idea behind ZoneTimer is to define each performance zone as a pointcut. The aspect will track the movement of your code across each zone boundaries and build a running total of the exclusive time spent in each performance zone.</p><p>When your program exits the top-most performance zone, the totals will be sent to a handler function.</p><p>A typical use of the ZoneTimer aspect looks something like this</p><blockquote><div><p> <tt>use Aspect;<br> &nbsp; <br>aspect ZoneTimer =&gt; (<br>&nbsp; &nbsp; zones =&gt; {<br>&nbsp; &nbsp; &nbsp; &nbsp; main&nbsp; &nbsp; &nbsp; =&gt; call 'MyProgram::main' &amp; highest,<br>&nbsp; &nbsp; &nbsp; &nbsp; search&nbsp; &nbsp; =&gt; call 'MyProgram::search',<br>&nbsp; &nbsp; &nbsp; &nbsp; widgets&nbsp; &nbsp;=&gt; call qr/^MyProgram::widget_.*/,<br>&nbsp; &nbsp; &nbsp; &nbsp; templates =&gt; call 'MyProgram::render',<br>&nbsp; &nbsp; &nbsp; &nbsp; dbi&nbsp; &nbsp; &nbsp; &nbsp;=&gt; call qr/^DB[DI]::.*?\b(?:prepare|execute|fetch.*)$/,<br>&nbsp; &nbsp; &nbsp; &nbsp; system&nbsp; &nbsp; =&gt; (<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; call qr/^IPC::System::Simple::(?:run|runx|capture)/<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; call 'IPC::Run3::run3'<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; |<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; call qr/^Capture::Tiny::(?:capture|tee).*/<br>&nbsp; &nbsp; &nbsp; &nbsp; )<br>&nbsp; &nbsp; },<br>&nbsp; &nbsp; handler =&gt; sub {<br>&nbsp; &nbsp; &nbsp; &nbsp; my $top&nbsp; &nbsp; &nbsp; &nbsp;= shift; # "main"<br>&nbsp; &nbsp; &nbsp; &nbsp; my $start&nbsp; &nbsp; &nbsp;= shift; # [ 1279763446, 796875 ]<br>&nbsp; &nbsp; &nbsp; &nbsp; my $stop&nbsp; &nbsp; &nbsp; = shift; # [ 1279763473, 163153 ]<br>&nbsp; &nbsp; &nbsp; &nbsp; my $exclusive = shift; # { main =&gt; 23123412, dbi =&gt; 3231231 }<br> &nbsp; <br>&nbsp; &nbsp; &nbsp; &nbsp; print "Profiling from zone $top\n";<br>&nbsp; &nbsp; &nbsp; &nbsp; print "Started recording at " . scalar(localtime $start) . "\n";<br>&nbsp; &nbsp; &nbsp; &nbsp; print "Stopped recording at " . scalar(localtime $stop)&nbsp; . "\n";<br>&nbsp; &nbsp; &nbsp; &nbsp; foreach my $zone ( sort keys %$exclusive ) {<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; print "Spent $exclusive-&gt;{$zone} microseconds in zone $zone\n";<br>&nbsp; &nbsp; &nbsp; &nbsp; }<br>&nbsp; &nbsp; },<br>);</tt></p></div> </blockquote><p>This example breaks out the cost of a typical small web application into a general zone, a special zone for the search page, and then splits the costs of generating widgets, rendering the HTML template, waiting for the database, and making calls to the underlying system.</p><p>Start and stop times are returned as a two element array exactly as returned by C, and the exclusive zone totals are returned as integer microseconds (all math is done in these integer microseconds to prevent floating point corruption).</p><p>The use of Aspect allows us to easily mix special cases via the pointcuts, such as the use of "highest" which makes sure that "main" only matches the first time it is seen, and any widget which that does a re-entry into main still has that cost attributed to the widget. We've also hooked into multiple system call modules to measure system call cost, because we know different modules our program consumes will use different methods for interacting with the system.</p><p>While the handler I've shown here will just print out a summary of the call, in our environment at work the profile report handler will format the exclusive times into a special log message and then send it via the super-quick and non-blocking <a href="http://search.cpan.org/perldoc?Log::Syslog::Fast">Log::Syslog::Fast</a> module to the UDP localhost syslog service, where it is mirrored to disk for debugging use, and then forwarded on to our main company-wide syslog server.</p><p>On our setup, we can then use an excellent commercial log analysis product called <a href="http://www.splunk.com/">Splunk</a> (limited free version also available if you want to try it out) to do our tracking and trending of the performance for the entire application across the entire cluster.</p><p>The nicest thing about the Aspect system is that it scales the performance cost and complexity risk directly to the complexity of the use case you are using it for.</p><p>If it turns out that the Aspect hooks are too intrusive and causing a performance drain, or accidentally causing some weird edge case behaviour, you can simply turn off that the Aspect and restart the servers and the performance penalty just vanishes.</p><p>Maintenance of the profiler zones is really easy, because they are all listed clearly in one place.</p><p>Depending on your use case, you could even define the performance zones in your website configuration, and then adjust the profiler zone boundaries directly on production (although I'm not sure I'd want to do that in our specific case, since we're fairly paranoid about untested code going into production).</p><p>This ZoneTimer is just the first of several Aspect-based tools I plan to release over the next few months. If you get a chance to try out any of these new toys, I'd love to hear feedback on how well they work for you.</p><p>In addition to the value of the ZoneTimer aspect itself, one other thing that some people might find of interest is just how little code it took to get this all working.</p><p>The entire Aspect::Library::ZoneTimer was implemented in about 80 lines of code, which you can see at <a href="http://cpansearch.perl.org/src/ADAMK/Aspect-Library-Timer-0.04/lib/Aspect/Library/ZoneTimer.pm">http://cpansearch.perl.org/src/ADAMK/Aspect-Library-Timer-0.04/lib/Aspect/Libra<nobr>r<wbr></nobr> y/ZoneTimer.pm</a></p><p>This is small enough that you could just clone the module and tweak it for your own use case, if you wanted to creat a new and tricky customised profiler.</p> Alias 2010-07-22T02:27:33+00:00 journal Following convention across language boundaries http://use.perl.org/~doubi/journal/40455?from=rss <p>I've been meaning to write this post for a while. I'd like your opinion on it.</p><p>In <a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">my last post</a> [0] I touched on the question of following the Python ctypes API in the context of Type objects. The issue of how closely to stick to the Python API has come up a few times. There just bits and pieces I'm not wild keen on. The whole <a href="http://docs.python.org/library/ctypes.html#callback-functions"> <code>WINFUNCTYPE / CFUNCTYPE</code> function prototype factory thing</a> [1] for callbacks is a good example (in our implementation creating callbacks is as simple as <code> my $cb = Ctypes::Callback-&gt;new(\&amp;perl_func, &lt;returntype&gt;, &lt;argtypes&gt;);</code>).</p><p> <b>Who is the audience?</b> </p><p>The argument <em>for</em> of course is the assumption that some of the first and most important users of Perl's Ctypes module will be C library authors or porters who already have a Python binding and will be interested in doing the same thing for Perl. Obviously if the Perl Ctypes implementation followed Python's 100% then the friction for those users would be low as possible and we might get more library bindings written sooner.</p><p>I think that's a worthy objective. I'm just not completely decided on whether the best ordering of preference is, <em>"Copy the Python API exactly and improve and add features where possible,"</em> or, <em>"Write the best module possible, and follow the Python conventions where you can and it makes sense."</em> </p><p>My personal preference is for the latter. I've always seen Perl authors as the main clients of the module, which I think makes sense, especially over the long term. So I'd preference doing things <em>better</em> to following Python's conventions to a fault.</p><p> <b>Example: Functions and prototypes</b> </p><p>A good example of an API change is the fact that Python doesn't have a simple way to set Function object properties using constructor arguments. A feature of ctypes is the ability to use auto-load and use functions like so:</p><blockquote><div><p> <tt>&gt;&gt;&gt; print hex(windll.kernel32.GetModuleHandleA(None)) # doctest: +WINDOWS<br>0x1d000000</tt></p></div> </blockquote><p> This is fine for simpler functions, but for more complex examples you might have to specify argument types and/or return type. The <em>simple</em> way to do this is in <a href="http://docs.python.org/library/ctypes.html#specifying-the-required-argument-types-function-prototypes">three separate steps</a> [2]:</p><blockquote><div><p> <tt>&gt;&gt;&gt; strchr = libc.strchr<br>&gt;&gt;&gt; strchr.restype = c_char_p<br>&gt;&gt;&gt; strchr.argtypes = [c_char_p, c_char]</tt></p></div> </blockquote><p> The <em>other</em> way is to use a <code>[WIN|C|PY]FUNCTYPE</code> factory function appropriate to (the C calling convention of) your system, which returns a prototype <em>class</em> which in turn must be instantiated in one of <em>four</em> different ways to get the actual function object you want to use. A cynical reader of the ctypes docs would also point out that the whole mechanism is sequestered into the Reference document, left out of the Tutorial part altogether (apart from the part on callbacks, because there's no other way to make them).</p><p>In Perl's Ctypes on the other hand, we have sensible, somewhat clever constructors for Functions, so we can combine the three lines above into one statement, and not have to worry about generating new bespoke classes/packages:</p><blockquote><div><p> <tt>my $strchr = CDLL-&gt;libc-&gt;strchr({ restype =&gt; [ 'c_int' ],<br>&nbsp; argtypes =&gt; ['c_char_p', 'c_char'] });</tt></p></div> </blockquote><p> or even</p><blockquote><div><p> <tt>my $result = CDLL-&gt;libc-&gt;strchr({ restype =&gt; [ 'c_int' ],<br>&nbsp; argtypes =&gt; [qw(c_char_p c_char)] })-&gt;("abcdef", "d");&nbsp; &nbsp; # "def"</tt></p></div> </blockquote><p>I think this is a good example of Doing It Better. In the Perl module, you're of course still able to specify Function properties individually if you want with <code>$strchr-&gt;argtypes()</code>. And we'll almost certainly replicate the <code>*FUNCTYPE</code> shenanigans later too, if only to appease porters.</p><p> <b>Out with the old?</b> </p><p>That's fine where different interfaces can live alongside each other, but in the case of the fundamental behaviour of Type objects, what I consider an improvement in behaviour would represent a divergence from the Python way of doing things (<a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">see previous post</a> [0]).</p><p>In these instances, which philosophy should win out? <em>"Copy the Python API exactly and improve and add features where possible,"</em> or, <em>"Write the best module possible, and follow the Python conventions where you can and it makes sense?"</em> I think the answer needs to come logically from the expected client&#232;le of Perl's Ctypes module. Maybe there are other factors to think about as well. I've stated my leanings, but I'm very keen to hear more opinions on the matter.</p><p>[0] <a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.htm<nobr>l<wbr></nobr> </a> <br> [1] <a href="http://docs.python.org/library/ctypes.html#callback-functions">http://docs.python.org/library/ctypes.html#callback-functions</a> <br> [2] <a href="http://docs.python.org/library/ctypes.html#specifying-the-required-argument-types-function-prototypes">http://docs.python.org/library/ctypes.html#specifying-the-required-argument-typ<nobr>e<wbr></nobr> s-function-prototypes</a> </p> doubi 2010-07-20T18:10:18+00:00 journal Thoughts on a Ctypes::Type object API http://use.perl.org/~doubi/journal/40454?from=rss <p>For the past few days I've been considering and experimenting with the design of simple Ctypes::Type objects. These are objects which, funnily enough, represent C data types for manipulation in Perl.</p><p> <b>The reference implementation</b> </p><p>Looking at the Python ctypes module, there were various things I didn't like. <a href="http://docs.python.org/library/ctypes.html#fundamental-data-types">Python's simple types</a> [0] can be summarized thusly:</p><blockquote><div><p> <tt>&gt;&gt;&gt; i = c_int(42)<br>&gt;&gt;&gt; print i<br>c_long(42)<br>&gt;&gt;&gt; print i.value<br>42<br>&gt;&gt;&gt; i.value = -99<br>&gt;&gt;&gt; print i.value<br>-99<br>&gt;&gt;&gt;</tt></p></div> </blockquote><p> Having to specify <code>i.value</code> seemed cumbersome for an object which essentially represents just that value and some rules for what kinds of values it can contain. So I started trying various things with <code>tie</code>'ing and <code>overload</code>ing. Indeed, I was about to start a <a href="http://gitorious.org/perl-ctypes/perl-ctypes">fourth project branch on types</a> [1] before reigning in and having another think about fundamental behaviour.</p><p> <b>Metaphor clash</b> </p><p>The <a href="http://perldoc.perl.org/overload.html#Metaphor-clash">'Metaphor Clash' section</a> [2] of perldoc <code>overload</code> made clear the difficulty I was having with my thinking. If you want to be able to say simply,</p><blockquote><div><p> <tt>$int_obj = 42;&nbsp; &nbsp; &nbsp; &nbsp; # assign new value to Ctypes::Type::c_int object</tt></p></div> </blockquote><p> and not have $int_obj smushed and replaced by a simple IV, you're forced to do something like,</p><blockquote><div><p> <tt>$other_obj = $int_obj-&gt;copy();</tt></p></div> </blockquote><p> on those occasions when that's the outcome you really want. That might not be regular for objects in Perl, but I think that this is more suitable in this domain, where as I mentioned before, objects are just there to represent special kinds of values. Particularly, it would let you do this:</p><blockquote><div><p> <tt>$c_int = $c_long;&nbsp; &nbsp; &nbsp; &nbsp; # Put long value into integer type, with<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# appropriate checking on STORE</tt></p></div> </blockquote><p> rather than having to say</p><blockquote><div><p> <tt>$c_int-&gt;val = $c_long-&gt;val;</tt></p></div> </blockquote><p> <b>Normal usage</b> </p><p>I think that once you've instantiated a Ctypes::Type object, what you're going to be doing 90% of the time is assigning values to it or assigning its value to other things. The times when you want to squash the object with a different object or value, instead of just creating a new one, will be rare, and there can be special methods for doing those things.</p><p>What do you think? Is this a reasonable generalisation to make? How would <em>you</em> like Type objects to work?</p><p> <b>Other issues</b> </p><p>There are a couple of other issues tangential to this which I'd like to flag up while we're at it... </p><ul> <li>The latter behaviour, while IMO preferable, will I think be harder to implement. I haven't yet found a way using only Perl to hide the process of <code>tie</code>'ing a variable from the user. This is horrible:<blockquote><div><p> <tt>my $int = 5;&nbsp; &nbsp; # regular scalar<br>c_int(\$int);&nbsp; &nbsp; # make it a Ctypes::Type</tt></p></div> </blockquote><p> but it's one of the only ways to enable the assignment of new values to the object without using an accessor every time (the other way is returning a reference from <code>c_int()</code> and requiring users to say <code>$$int</code> all the time). I think the desired effect would be possible, but <code>c_int()</code> will have to be written in XS. That's fine, but I want to sound out opinions on whether I'm making the right choices on functionality first.</p> </li><li>The latter behaviour, while IMO better than that in Python's ctypes, may represent a significant difference from the de facto 'conventional' way of doing things. This raises a bunch of issues to be addressed in a subsequent post.</li></ul><p>Please do let me know what you think of the above proposals. I'm highly suggestible.</p><p> [0] <a href="http://docs.python.org/library/ctypes.html#fundamental-data-types">http://docs.python.org/library/ctypes.html#fundamental-data-types</a> <br> [1] <a href="http://gitorious.org/perl-ctypes/perl-ctypes">http://gitorious.org/perl-ctypes/perl-ctypes</a> <br> [2] <a href="http://perldoc.perl.org/overload.html#Metaphor-clash">http://perldoc.perl.org/overload.html#Metaphor-clash</a> </p> doubi 2010-07-20T18:04:33+00:00 journal How we deploy massive Perl applications at work http://use.perl.org/~Alias/journal/40453?from=rss <p>Every now and then, we hear people talking about mechanisms for doing Perl in a commercial environment and how they deal with packaging and dependencies.</p><p>This is mine.</p><p>At Corporate Express, our main Perl application is a 250,000 line non-public monster of a website that has over 100,000 physical users and turns over about a billion dollars. It implements huge amounts of complex business functionality, and has layer upon layer of security and reliability functions in it because we supply to multinationals, governments and the military (only the stuff that doesn't blow up of course). Our<nobr> <wbr></nobr>.pm file count is around 750, and our test suite runs for about 4 hours (and is only around one third complete).</p><p>Lest you suspect that 200,000 lines is wasted in re-implementing stuff, the main Build.PL script has around 110 DIRECT dependencies, and somewhere in the 300-500 range of recursive dependencies. Loading the main codebase into memory takes around 150-200meg of RAM.</p><p>When I joined the team, the build system was horribly out of date. The application was stuck on an old version of RedHat due to go out of support, and as a Tier 1 application we are absolutely forbidden from using unsupported platform.</p><p>So I took on the task of upgrading both the operating system and the build system for the project. And it's a build system with a history.</p><p>Once upon a time, long ago, the project went through a period where the development team was exceptionally strong and high skilled. And so of course, they created a roll-your-own build system called "VBuild".</p><p>They built their own Perl, and along with it they also built their own Apache, mod_perl, and half a dozen other things needed by the project. This is similar to many suggestions I hear from high-skilled people today, that at a certain point it's better just to build your own Perl. VBuild this created in the pre-commercial Linux era, so it's not an entirely unreasonable decision for that time period.</p><p>Unfortunately, a few years later the quality of the team dropped off and VBuild turned into a maintenance nightmare because it required a high-skill person to maintain it.</p><p>At the time, the Tier 1 "Must be supported" policy was coming into effect, and after the problems with custom-compiling they decided to go with the completely opposite approach of using only vendor provided packages, in a system called "UnVBuild".</p><p>Since their platform of choice was RedHat, this had become troublesome even before I arrived. Worse, in the change from RHEL 4 to RHEL 5, some of the vendor packages for things like XSLT support were dropped entirely leaving us in a bind.</p><p>My first instinct was to return to the build everything approach, but the stories (and commit commentary) from that time period reinforced the idea that complete custom build was a bad idea. Office supplies is hardly a sexy industry, and the ability to entice good developers into it is a quite legitimate risk.</p><p>So in the end, I went with an approach we ended up nicknaming "HalfBuild". The concept behind it is "Vendor where possible, build where needed".</p><p>We use a fairly reasonable chunk of vendor packages under this model. Perl itself, the Oracle client, XML::LibXML and a variety of other things where our version needs are modest and RHEL5 contains it. We also use a ton of C libraries from RHEL5 that are consumed by the CPAN modules, like all the image libraries needed by Imager, some PDF and Postscript libraries, and so on.</p><p>One RPM "platform-deps" meta-package defines the full list of these system dependencies, and that RPM is maintained exclusively by server operations so that we as developers are cryptographically unable to add major non-Perl dependencies without consulting them first.</p><p>On top of this is one enormous "platform-cpan" RPM package built by the dev team that contains every single CPAN dependency needed by all of our Perl projects.</p><p>This package lives in its own home at<nobr> <wbr></nobr>/opt/cpan and is built using a process rather similar to the core parts of Strawberry Perl. With PREFIX pointing into<nobr> <wbr></nobr>/opt/cpan, we first build a hand-crafted list of distribution tarballs to upgrade RHEL5 to a modern install of CPAN.pm (without overwriting any normal system files).</p><p>We then boot up the CPAN client from<nobr> <wbr></nobr>/opt/cpan, and tell it to install all the rest of the dependencies from a private internal CPAN mirror which is rsynced by hand specifically for each new CPAN build. This ensures that the build process is deterministic, and that we can fix bugs in the build process if we need to without being forced to upgrade the modules themselves).</p><p>The CPAN client grinds away installing for an hour, and then we're left with our "CPAN Layer", which we can include in our application with some simple changes to @INC at the beginning of our bootstrapping module.</p><p>The<nobr> <wbr></nobr>/opt/cpan directory for our project currently weighs in at about 110meg and contains 2,335<nobr> <wbr></nobr>.pm files.</p><p>Updating<nobr> <wbr></nobr>/opt/cpan is still something of an exercise even under this model because of potential upgrade problems (Moose forbidding +attributes in roles hit us on our most recent upgrade) but the whole process is fully automated from end to end, and can be maintained by a medium-skill Perl hacker, rather than needing an uberhacker.</p><p>Over the last 2 years, we've upgraded it around once every 6 months and usually because we needed to add five or ten more dependencies. We tend to add these new dependencies as early as we can, when work that needs them is confirmed but unscheduled.</p><p>We also resort to the occasional hand-copied or inlined pure-Perl<nobr> <wbr></nobr>.pm file in emergencies, but this is temporary and we only do so about once a year when caught unprepared (most recently Text::Unidecode for some "emergency ascii'fication" of data where Unicode had accidentally slipped in).</p><p>While not ideal, we've been quite happy with the<nobr> <wbr></nobr>/opt/cpan approach so far.</p><p>It means we only have to maintain 5 RPM packages rather than 500, and updating it takes one or two man-days per 6 months, if there aren't any API changes in the upgrade.</p><p>And most importantly it provides us with much better bus sensitivity, which is hugely important in applications with working lives measured in decades.</p> Alias 2010-07-20T02:25:21+00:00 journal