All the Perl that's Practical to Extract and Report
use Perl Log In
Shlomi Fish (918)
shlomif@iglu.org.il
http://www.shlomifish.org/
AOL IM: ShlomiFish (Add Buddy, Send Message)
Yahoo! ID: shlomif2 (Add User, Send Message)
Jabber: ShlomiFish@jabber.org
I'm a hacker of Perl, C, Shell, and occasionally other languages. Perl is my favourite language by far. I'm a member of the Israeli Perl Mongers, and contribute to and advocate open-source technologies. Technorati Profile [technorati.com]
Journal of Shlomi Fish (918)
Reflections on Syndicating my Fortune Cookies as Atom
Another technical entry, on my fortune cookies' FortuneXML work again. When I last left you, I was able to transform the XML sources of the fortune cookies to XHTML and to plaintext, and ran into a nasty stack smash that prevented me from uploading the module to the CPAN.
Then, I decided that now that each fortune had its own unique URL, and that people didn't have the time to regularly see what has changed, it was high time to get a web-feed syndication for it. I decided to use XML-Feed directly, and to generate an Atom feed. (So far I only did RSS using XML-RSS).
It seemed logical that I'd simply go over all the individual id="" attributes in the XML files I maintained, and see which ones were added. Then I'll generate a feed containing the most recent 20 or so. However, the XML does not contain the dates where they were added. So I decided that for simplicity I'll record the dates all the IDs were added into a YAML file, which in turn will be updated there. (A database of some sort may be better eventually.)
Now, how to find the most recent 20 dates from the YAML and the XMLs? One option would be to collect all the dates and sort them, but that seemed wasteful to me. Recalling my Computer Science Data Structures studies I remembered that I could use a priority-queue for that, and remembered there was a module for that on the CPAN. A search for "Priority Queue" or "PQ" yieleded nothing, but I then searched for Heap and found Heap.pm, which implements several Heaps that can be used as a priority queue.
Heap is a very impressive module. I remember that back when I studied at the Technion, it used to be the only usable sourceware (and open-source) implementation I found of Fibonacci heaps. I decided to use the Binary heap this time, though.
I had to write some wrapper code to get it working. I decided against writing automated tests before I wrote the code (= "Test-Driven Development") and just write the code and get it to working, because I found that I didn't exactly know what it would generate. Anyway, I now have the script working.
I encountered some problems in trying to find how to render just one fortune element and not the entire file. After not finding anything about it in the XML::LibXSLT documentation, I opted to tweak the XSLT stylesheet and create an optional parameter for it to act upon.
Eventually, I got an Atom feed. Firefox displays it fine, but it doesn't validate. Part of the problem is that I needed to trim down the feed entries' content from the "<html>" and other such tags. But another issue is that XML-Feed does not support all the required Atomisms.
I've noticed both XML-Feed and XML-atom have suffered from neglect (including the lack of support for those Atomisms), and decided to adopt them. I set up a repository for XML-Feed on BerliOS and started to get into it. First thing I did was get rid of ExtUtils::AutoInstall, and I have also given repository access to another person who is interested in revamping them. Thanks to BingOS for helping me with some Module::Install-ism.
As it turns out, the XML-LibXSLT bug was actually a Cooker-specific bug, which seems like it is caused from problematic compilation flags.
So world-domination through Unix-like fortune cookies is still making progress. I'll probably release XML-Grammar-Fortune in its current form soon since the smash stack I got is highly system-dependent.
Perl and Solitaire Games
This entry will be a technical entry, but on a relatively lighter side: Solitaire games. Solitaire games (also known as "patience" games) are any of several kinds of single-player games played using one or more decks of cards. Despite common belief, the Windows-game called "Solitaire" is actually Klondike, which is only one variant of Solitaire. Freecell is another famous variant of Solitaire.
So anyway, I spent some of the past two weeks working on Games-Solitaire-Verify, which is a CPAN module to verify automated solutions of solitaire games. So far only Freecell is supported, but there's a lot of modularity in place for other variants. The intention of this module is to be used as part of the automated test suite of Freecell Solver, which is a solver for several variants of Solitaire, which lacks a comprehensive test suite.
I should note that I implemented the very first version of Freecell Solver in Perl, but it was horribly slow. Of course, I did some very illogical things speed-wise, like (1) putting all the states into an unsorted array and using linear search to scan it - and (2) constantly deserialising states into objects and data strcutures of individual cards, columns, and boards, and doing the comparison the hard way. So it was dog slow. I ended up re-implementing it as (saner) C code which turned out to be considerably faster than the Perl version. Moreover, my next C version was faster by a factor of about 100 (not because I was smarter, but I was less dumb), and since then I incorporated many other speed optimisations. (There are also several C and Assembly-hosted solvers that are much faster than mine.)
In any case, someone contacted me about the solver on Facebook, and after talking to him on Yahoo Messenger, we decided that he, I and another guy will talk on Freenode on the newly started #solitaire. There, I was quickly quickly joined by tybalt89, who is a master golfer, and who decided to write his own solver for Klondike in Perl. After a while he finished it and so I took a look at the source code.
From my impression it seems like the kind of code a golfer would write: strings instead of objects (with regular expressions and other text processing to manipulate them), a call to shuffle() from List::Util with a random seed, no use warnings, lots of obscure two-letter variable names, a lot of trailing "or" clauses, and it also has $hearts, $clubs, etc. instead of a hash. I talked about it with another master golfer and he said that "golfing rots the brain", and he said a program he gave as a solution to someone's problem had many golfish paradigms as well.
tybalt89 reported that his program is relatively fast, but still fails on many deals.
Meanwhile, I finished writing Games-Solitaire-Verify and released it on the CPAN. Then I decided to test if it could detect than invalid solution was indeed invalid. And it turned out an invalid solution was still reported as valid (!) making the reported verdict useless. What I found out was that the complete solution analysis silently stopped after the first move or two.
I fixed the bug and released a better version (with more tests). Then I
decided to extract a few more classes
(Games::Solitaire::Verify::Freecells and
Games::Solitaire::Verify::Foundations) out of the
relatively monolithic
I still need to adapt the module to verify solutions of other variants except Freecell, and that's what I'm planning to work on next. But it was good enough to add some system tests to Freecell Solver (using Perl, Test::More and TAP). What I hope is that tests such as this will allow me to revamp , refactor and enhance Freecell Solver with more confidence that I didn't break anything. I'm especially looking forward to the conversion from GNU Autoconf to CMake, which depends on the test suite to make sure it doesn't break anything.
Finally, I also found out that my C-based Freecell Solver program, did not always output things the right way, without trailing whitespace, etc. Since my primary motivation with Games-Solitaire-Verify was to verify the solutions of Freecell Solver, and because I wanted to compare to the precise output, I needed to emulate these mis-behaviours in my Perl code as well. I don't like it very much, but I guess I'd rather emulate Freecell Solver to test it, than fix it and risk breaking something. Nevertheless, my intention is to provide flags for a saner output for both the C and Perl programs in the future.
Happy Solitaire!
Freenode Channel for Catalyst - ##catalyst
Since I'd like to start working on a new Catalyst-based project, and since I cannot and would rather not chat on irc.perl.org, and since #catalyst (with a single sharp-sign) on Freenode exists only to redirect people to irc.perl.org, and they don't tolerate any other discussion there (which is probably against Freenode policy, but that's life), then I've started a new channel.
Introducing ##catalyst on Freenode - the unofficial and completely non-hostile Catalyst channel. We're already 5 people and one super-intelligent bot there. If you're interested in Catalyst, then please consider joining it too.
Dealing with Approval Addiction (and Implied Stress Periods)
Well, despite the fact that I hardly publicised my last essay about the "Closed Books", it has been chromatic'd. Rumours are that all the bloggers whose blog posts/essays were deprecated on chromatic's blog are now rich, famous and the object of the affection of many attractive members of the appropriate sex. Memo to self: prepare a limited edition T-shirt: "My blog post was chromatic'd. I pwn you as a blogger."
Seriously now, while the publicity was probably good for me, I was indeed a bit overwhelmed with what chromatic said, and felt down. It's not because I believed I was wrong, but because I respected chromatic, because he's a good programmer, a good author, a good editor (with whom I collaborated on several articles on O'Reilly-Net), an interesting blogger, and a good guy. (And yes - these are all compliments). I'm not sure he's really a friend of mine, but I certainly respect him.
Now Paul Graham says in "What you'll wish you'd known" (footnote 4) that:
The second biggest regret was caring so much about unimportant things. And especially about what other people thought of them.
I think what they really mean, in the latter case, is caring what random people thought of them. Adults care just as much what other people think, but they get to be more selective about the other people.
I have about thirty friends whose opinions I care about, and the opinion of the rest of the world barely affects me. The problem in high school is that your peers are chosen for you by accidents of age and geography, rather than by you based on respect for their judgement.
Now in my case, I may be somewhat immature because I tend to sometimes care about what many people think of me, rather than just my close friends (offline or online). Now being down (or "depressed") because someone disapproves of you, is perfectly natural and normal. However, since I have Bipolar disorder, it threw me into the so-called "hypomania" (= "below-mania"), which is not healthy, and disrupts my functionality.
Hypomanias are a variation of "Clinical Depressions" or "Clinical anxieties". Dealing with the latter is described in the highly recommended book "Feeling Good", which I believe is a necessary read even for non-depressive people, in order to understand how people think, and as a preventitive measure.
I don't accuse chromatic of making me hypomanic. I've received my share of past criticism in the past, and will receive again. I also was often criticised for insulting people myself, due to the fact I tend to be tactless. How you deal with criticism is ultimately the responsibility of the receiving end, as even for me, most criticism will not affect me.
In any case, Feeling Good mentions four general "addictions" that can make one clinically depressed (or Hypomanic):
- Approval Addiction.
- Productivity Addiction - you care about your work, how productive you are, how much you achieve, etc. (Much more common among men.)
- Love addiction - you want to be loved a lot. More common among women.
- Perfectionism - you want to be perfect in everything you do.
Now based on reading the descriptions in the book, I believe I have been having Approval Addiction and Productivity Addiction, and neither of the other two.
I used to get into clinicial depressions and anxieties (due to approval, etc.) which are even worse to deal with than hypomanias are. And during Manias, which I also had, but must avoid at all costs from now on, I lose most control of myself.
I started writing an essay titled "Dealing with Hypomanias". It is not professional Psychological advice, as I am not a qualified therapist, but it is given as a way to dispense my advice from the Point-of-View of a Bipolar person, who's learned a little about it. It's still very rough on the edges, as it started from a plain-text email, but any constructive comments would be welcome.
While I may be a bit immature due to my approval addiction, I'm still mature enough to learn from my mistakes. At one time, a prominent member of the Linux-IL mailing list said there that one should not take the advice I've given in the "End of Info-Tech Slavery" (an essay I still mostly stand by, but am about to update with a sequel/correction), and rather take the advice in my links. I was offended from it. Then I understood he was over-generalising and not criticising any of the points I raised in my essay, or explaining why he disliked them, and realised that he was very intelligent, but nevertheless what people call an "idiot". So when he said he doesn't think my Random Tweakers idea for a startup have commercial value (again without properly explaining why or without asking me how I intend to make money), I didn't take what he said to heart.
So now I think chromatic will "suffer" a similar fate in his criticisms against me. No, chromatic is certainly not an idiot, at least not in most regards. But now I know better than to take what he says to heart. I'll listen to what he says and read it at my free time, and often find merit in what he says, but he's still someone I know better than to be affected by him.
I'm not disabling comments here, because I have a policy against it, due to the fact that I believe a blog post without comments is anti-social and defeats the point. But please be gentle, civil, rational and logical. I'm still a bit hypomanic now, though it's getting better, and don't need any more grief. (However, I don't guarantee I'll read what you said immediately, or reply to it.)
I feel this was probably the most off-topic use.perl.org journal post I ever posted, but I've seen much more off-topic blog posts on use.perl.org, and it is relevant to open-source software, as I'm certainly not the only FOSS geek who is either Bipolar or much less Unipolar/Depressive.
And in case you wanted to suggest that: yes, I am taking medication - it doesn't absolutely prevent the hypomanias, but it may help (not sure about that). And I am seeing a Therapist, who gave me a lot of good advice. I should do the so-called "Cognitive Exercises" more-often, but I'm usually carried away with chatting or working on code or text to do them, but they do help a lot.
Here's hoping we can deal with our frustrations more easily, because people are not perfect and the world is not perfect, but they are still pretty darned good.
New Essay about Open-sourcing Books
A conversation on Ask Bjørn Hansen's blog (famous for being an administrator of perl.org and involved in other Perl-related projects) prompted me to write this essay, which talks about why it is unwise and harmful not to make one's books publically available online. Perl is mentioned there and so is Ruby and git and other technologies.
Comment here or on the announcement of the essay on my homepage's blog, where no registration is required for commenting.
XML-Grammar-Fortune
One of my many computerised passions is to collect quotes in UNIX-like fortune format. Throughout the years, I have formed a moderately large collection of them in several files. As time went on, I noticed a few problems. First of all, they were all in large plaintext files, and pointing someone to a quote involved giving a link to the fortune, and saying "search for Foobar". Moreover, since they were just chunks of text, they couldn't hold any meta-data.
At one time, I heard of someone who created an XML grammar to describe Unix fortunes, but a Google search was no help in finding that. And I also have the grand "Fortunes Mania" vision for a community site that for collecting and sorting quotes. This vision was very intimidating, but recently, I decided to take a small baby step by defining a grammar for fortunes as XMLs. So I present to you the XML-Grammar-Fortune distribution.
I've taken quite a lot of time to think about what I wanted there. One thing I concluded was that there are several different types of fortune cookies: run-of-the-mill quotes, IRC conversations, excerpts from screenplays, structured plaintext, HTML, etc. Therefore, the XML grammar should be able to have several different types of sub-nodes, which each corresponds to a certain class of fortune cookies
Until now I've used DTDs for defining my XML schemas, but for XML-Grammar-Fortune, I decided to learn Relax NG, which I was told was easier than the W3C XML Schemas. I was very impressed from Relax NG - it's easy, it's fun, and it's powerful. One problem I've encountered was that, when validating a document using it, XML::LibXML (version perl-XML-LibXML-1.66-1mdv2008.1), does not give the line number where the validation error has occured. To overcome such problems, one needs to look at the diffs or bisect the document.
Anyway, I defined a Relax NG Schema for the documents, and made sure that some basic examples will validate (test-driven-development-style). Then I worked on an XSLT stylesheet to convert them to XHTML.
When I started, I only had one fortune type - <raw>, which is a gigantic <pre> block with some meta-data. I gradually implemented more fortune types: irc, quote and screenplay, whose RNG and XSLT were based on XML-Grammar-Screenplay, with a lot of ugly copying-and-pasting.
I gradually converted more and more fortunes to have a richer XML semantics. The XML grammar requires an id for each fortune, and also allows specifying a title-element, and some fields in the <info> tag, like "author" or "work". For example all the "Friends" fortunes were converted to XML by first normalising the screenplay and then using a script I wrote to convert them to XML.
So I had all the fortunes as XMLs, but now the plaintext versions went out of sync. So I coded a Perl module to convert them from XML to plaintext.
I should note that due to a problem with XML-LibXSLT and perl-5.10.0, I didn't upload it to CPAN yet, because I do not want to receive so many failure reports.
On a different note: my former co-worker has read "Perl for Perl Newbies" in order to learn Perl, liked it a lot, and told me I should add more to it. That also feels good.
Nominating szabgab for the White Camel Award
I alread sent it to jose-at-pm.org and advocacy-at-perl.org, but here it is just in case:
I'd like to nominate Gábor Szabó (the Hugarian-Israeli Perl programmer, not the Jazz musician) for his contributions to the Israeli and Global Perl Communities. See:
Gabor Szabo is responsible for the fact that the Israeli Perl community, and that of other dynamic languages has took off. So far he:
- Set up an alternate mailing list, which proved to be more active - perl@perl.org.il.
- Set up a tradition of monthly Israeli Perl meetings, which have turned out to be popular.
- Organised 3 YAPC::Israel conferences - YAPC::Israel::2003, YAPC::Israel::2004 and YAPC::Israel::2005 and an OSDC::Israel conference - OSDC::Israel::2006.
- He helped organise the Hungarian Perl Workshops in Hungary, and a Hungarian YAPC.
- He set up CPAN Forum - as a way to get help and ask questions with individual CPAN distributions. Also allows users to tag their modules using delicious-like tags.
- His commmercial venture - http://www.pti.co.il/ - has been responsible for training material and sessions for Perl and other FOSS technologies, as well as general FOSS development and support. Some of its material has been made available online free-of-charge.
- He has written many weblog entries, comments, emails and other material about communal and technical issues.
Aside from all that, he has done numerous contributions of code, and documentation.
Next OSDClub Tel Aviv Meetings
On Tuesday, 3-June-2008 (tomorrow or today depending how you look on it), on 18:30, the OSDClub of Tel Aviv, which is a joint venture of the Tel Aviv Linux club, and Perl-Israel (and some other FOSS clubs who want to join the fun), will hold a social meeting. It will take place at the Café of Tel Aviv University near the junction of Einstein and Haim Levanon Streets. This social was scheduled because Zvi is in Israel. (If you don't know who he is, then come to meet him.)
Much later, on Sunday, 15-June-2008, OSDClub Tel Aviv will meet to share some Vim/gvim (= the text editor) Tips and Tricks. We'll meet in Schreiber (Math & CS) building in Tel Aviv University. We have more meaty presentations planned for July, but we hope these two meetings will be a a useful start after a long neglect.
No outgoing email to @perl.org and @pm.org
Hi all! Due to a technical problem, I have been unable to send any outgoing email to the @perl.org and @pm.org domains from either my home email address or my gmail.com email address. This is while outgoing mail to other places is mostly working, and most people have no problem sending email to the perl.org and pm.org domains. I receive bounces on any mailing list addresses, or @perl.org forwards.
This is the reason why I've been unusually quiet there lately, so don't worry. I can still be reached in many ways, and have many ideas for things I'd like to post on my blogs and on my homepage.
Anyway, cheers, and sorry for the inconvenience.
Perl vs. Ruby on Two Idioms
I've been meaning to blog about it for a few weeks, so here goes. In a Ruby-Israel thread someone asked how to convert an array to a hash that contains all of its keys as members (and "true" as a value). They came up with:
a = [1, 2, 3, 'other']
h = a.inject({}) {|h, v| h[v] = true; h }
In Perl, it's:
@a = (1,2,3, 'other');
my %h = (map { $_ => 1 } @a);
Much cleaner. I also though of a way to do a flat concatenation of an array of array references. Like [[1,2,3],[4,5,6],[7,["One", "Two",],9]] into [1,2,3,4,5,6,7,["One", "Two",],9]. In Ruby it is:
[[1,2],[3,4],[5,6,["Hello","There"],7]].inject([]) { |a,e| a+e }
While in Perl it is:
(map { @$_ } ([1,2],[3,4],[5,6,["Hello","There",],7]))
Both of these are caused by the fact that lists are not references in Perl, and that one can initialise arrays and hashes from them. This is while in Ruby or Python (and most Lisps) they are references. That or Ruby lacks more primitives.
--Mark Hahn, <hahn@neurocog.lrdc.pitt.edu>
