use Perl

Someone -- male -- noticed and loudly proclaimed having noticed that a "sexist" word was used on the workshop's IRC channel. Of course, they didn't say what or who said it. I'm concerned it might have been me. I said that a certain talk "sucked donkey dong. and not in a good way". Perhaps I said other things.

Since the Ruby conf fiasco (link below), we've been a bit trigger happy to identify, point out, and fix anything possibly offensive to... well, I guess, anyone who could be offended.

http://theworkinggeek.com/2009/06/dirty-presentations-xkcd-and-the-perils-of-140 -cha.html

Commenting on that link (which commented on the Ruby presentation): It seems like they're saying that because it's stereotypical of adolescent geek sexuality, it's bad. Yes, there are bad (especially of what's underdeveloped) aspects in geek sexuality, but not everything in geek sexuality is automatically bad.

Grown men can't be juvenile; nether can geeks. The other day, I threw a small beanbag embroidered with the world "douchelord" at a woman (a well humored one -- she's married to a sometimes adolescent college roommate I had and she's the one who bought the beanbag) who was going off the plot intricacies of the _three_ Twilight books, which she read. I was being subjected to female adolescent sexuality and it was painful. Thank heavens (I believe in empty heavens free of deities and free of undead humans) that, as adults, we're able to communicate with the opposite gender. Or not.

I don't claim to offer wisdom here. Perhaps I should claim to offer the point of view of the average sexist idiot.

As it stands, (and I was told this -- I didn't count) two women attended the workshop which (ditto) had about 140 people at it. Should our concern really to be try to make it even more "professional" in hopes that the extremely few extremely tiny omissions are what's keeping the floodgate of women closed? And if it's so important that gender balance be struck in gatherings, why aren't we going to pottery class and book club?

Often I suggest on IRC and in person, through off color jokes, that I like sheep. I am essentially from Minnesota, after all. Sheep are not offended by this, I'm pretty sure -- the jokes, I mean. But humans often are, are often in the same way and to the same degree as any other remark I make, and as far as I know, there's no goal to try to make Frozen Perl more friendly to sheep.

I suppose my goal is, as has always been, is to offend people. It's my non-outgrown geek adolescent sexuality behind the wheel. I realize that it's possible to make inside jokes where the only people who would be offended don't get the joke, but that doesn't serve my goal. I suppose I want to tweak the normals. I hang out with freaks. I adore them. Lack of freaks makes me anxious. Frozen Perl makes me anxious. The phrase "white bread" comes to mind. The one mohawk went a long way to making me feel at home. Minnesota does pretty well on the freak scale -- better than Phoenix. Perhaps -- I can't say -- that this feeling out of place gives me some shared empathy with the primary demographic that we're so eager to make feel comfortable.

I guess when it boils down to it, on one hand, we're a barren wasteland of human sexuality trying to make people of different sexes and sexualities feel comfortable, and on the other hand, even though we've trained ourselves to react with horror and disapproval at anything resembling a gender issue, geeks are generally among the most non-judgmental and open-minded.

If a certain -- and that standard is set to a very low, non-demanding level -- if a certain level of color can't be established at a con, my bored brain might make it it's goal to get thrown out. I might fancy myself the title of the first person to get thrown out of a Perl con.

Hmm. "Uncomfortable and boring"... I've heard that phrase before.

-scott

Why is PHP so much easier for newbies?

Why does Java have the best IDE tools?

Why is Ruby prettier than Perl?

Why does Perl have the best package repository?

As I've played through Mass Effect 2 over the last few weeks, I see some interesting parallels.

In the Mass Effect universe, human technology is bootstrapped by the discovery of an ancient abandoned alien observation outpost on Mars, and the further discovery that the dwarf planet Charon is really an abandoned but active interstellar jump gate covered in ice.

Other similar species have done the same, resulting in a galactic community of around a dozen civilisations all based around the same basic technological underpinnings.

Despite these civilisations believing a recently (50,000 years) extinct civilisation built the gates, it turns out the technology is perhaps millions of years old.

Every 50,000 years, the synthetic AI race that built them returns from hiding in intergalactic space to wipe out all of the existing advanced species based on "their" technology, and reset the galaxy for the next set of civilisations to rise.

In a conversation between the game's protagonist and one of these old AIs, we are lambasted by the AI for taking the shortcut on their technology. The jump gates and other technology is left in place intentionally, so that each new generation of civilisations take a controlled and predictable development path, making it easier to destroy them.

The AI posits that it is the overcoming of adversity on your own that drives true technological advancement, and that easy routes make you (technologically) weak.

I think you can see something similar in the development of the different programming languages.

Java is long and wordy, taking a long time to type. The need to work around this limitation resulted in the proliferation of powerful IDEs, resulting in the annual 20 million line of code Eclipse release train.

PHP as a web language would have been stillborn if it didn't deal competently and quickly with the need to easily deploy code, the result of which is that you can effortlessly just change .html to .php, add a hello world tag, and upload via FTP as normal (something Perl still can't do well).

Python's need to gain mindshare against an entrenched Perl led to a huge focus on being easy to learn, to a simplification of the language, and to hugely popular things such as the PyGame library and game competitions.

Faced with the lack of truly great package repository, and with a web-heavy community, Ruby became the "prettiest" language. Creating an elegant website is both expected and required if you are going to gain mindshare for an idea.

And Perl's messy syntax and difficulties in the area of maintaining large codebases, combined with a pragmatic sysadmin-heavy community, resulted in an unmatched packaging system that allowed code to be maintained in small pieces, with enormous volumes of support infrastructure around it.

The ease of publishing and trend to smaller package that the CPAN allowed conversely means that the Perl community has never really had the need for pretty and elaborate websites, and the smaller package size means that we lack the giant headline libraries that make the payoff from website work better.

Our bias towards a pragmatic tech-savvy sysadmin userbase means we haven't really provided anything like the focus on learnability that has driven Python's gradual dominance in the mindshare of the young. It takes a certain rigour in your prioritisation to intentionally remove and dumb down existing powerful features so that the language is easier to learn.

Even for Strawberry, which focuses on the userbase with the lowest traditional knowledge, we intentionally have the smallest and most maintainable website possible and we don't even have the kind of introductory screencasts that we really really need (which should be easy but which I never seem to find the time to do).

If you throw a bunch of Perl coders against some PHP coders in a website competition, it is not unexpected that when both sides play to their strengths you will see something like http://geo2gov.com.au/html?location=e.g.+1+Oxford+Street from the Perl coders and something like http://www.hackdays.com/knowwhereyoulive/postcodes/view/2000 from the PHP coders.

The former required a massive amount of data extraction, transformation, aggregation, a gigabyte-sized PostGIS database, and deployment via a Linux virtual appliance to Amazon EC2 to allow for strategic load-shedding.

The latter required the ability to turn data into presentable and understandable information for real humans, and to make it pretty enough that they WANT to look at it.

Driving true technological progress, then, may often be about identifying weaknesses that are hard to solve but aren't completely impossible (and don't have any crippling long-term conceptual flaws at an economic or project-management level).

The three best projects I have driven - PPI, Strawberry, and (in part) Padre - all share this property. All three of these represent hard but not impossible problems, and require an awareness about which issues are intractable and which issues merely exist because there's been no need to solve them any better.

Padre in particular has suffered greatly from issues with Wx quality and threading. But given the low takeup of both threading and Wx it was reasonable to move forward on the basis that these would be fixed once there was something depending on them, and driving a need to fix them.

All of our early problems are gone now, and there is continued pressure to find ways to improve our use of (and the efficiency of) Perl's native ithreads.

Similarly, the creation of Strawberry required a lengthy year-long process of fixing Win32 bugs in all kinds of toolchain and low level modules, because we'd never had a proper working developer feedback loop before.

Similarly, Perl's current push for marketing and blogging and websites is directly resulting from Python's success in mindshare capture.

So my question for you to ponder this week is the following:

What can you see that Perl as a whole struggles to do well, for which a good solution is not impossible, and is only being held back by smaller problems which would go away if there was a working candidate solution put in place that needed those small problem solved.

I really liked this talk - This Is Not Your Father's Von Neumann Machine by Cliff Click*

It's well worth watching. It has really nicely presented explanations of:

1. How and why a modern CPU executes out of order, and why cache misses are king.
2. How memory architecture means that two cores can cause writes to be read out of order, and the implications (your code goes boom^WSEGV).

* Brian Goetz was co-author of the talk, and co-presented it on its first outing, but this presentation of it just has Cliff Click.

My Frozen Perl Lightning Talk was on the topic of large legacy codebases, sort of. Specifically, it was on how to continue to grow them without having to look at them. A more serious talk might follow on this one -- one that involves useful code visualization and inspection. It's a topic that's been on my mind for a long time and one that my friends and I often discuss.

I monkey patched Test::Harness to accept floating point values for test results rather than merely "ok" and "not ok" and then used that as a fitness test for a genetic algorithm combined with a Markvo chains chatter box trained from Perl sourcecode. It kept alive and to bread permutations of the Markov-chains generated code that did better on the tests.

It was part political commentary -- "Perl programmers pioneered building massive unmaintainable codebases and it's Perl programmers who will take it to the next level!". Also with my monkey patching and the genetic algorithm's mucking about in the Markov Chains' package, I was showing off _hackish_ styled code. In what I consider good Perl spirit, interesting modules were wired together in interesting ways.

The first thing people heard during the day was that you shouldn't use anonymous subroutines (!!) or vi or emacs but instead use an IDE (!!). This was from a presentation done in Keynote with Comic Sans. The last thing they heard during the day was "experiment, have fun, explore".

I'm tired of the message at these sorts of things being "Perl is dangerous and has lots of sharp pointy bits that'll put your eye out... we don't want Perl to get any more of a bad rep so stay on the trail and keep your arms in the bus". Fuck that. There are precious few "at your own risk" languages. Assembly is one of them. C++ can be. Ruby hackers indulge playfulness in its place.

I keep telling people that if Perl turns its back on hackery, it'll lose that ecosystem *and* it still won't break into Java's and Python's. It'll wind up without a user base at all. Computer companies, languages, automobiles -- any product at all -- that's unclear on its identity winds up with no user base. Perl programmers need to know that Perl came out of sed, awk, BASIC-PLUS, and a pile of other languages, and that the sharp edges exist for a reason, and a lot of parts of Perl don't make sense in the context of yet-another-C-like-language.

Also, FYI... I've been writing a lot on http://scrottie.livejournal.com lately, including stuff that I should probably put or copy here. And there's me on Twitter too.

Oh yeah... the code is at http://slowass.net/~scott/tmp/Test-Float-0.1.tar.gz and in http://slowass.net/~scott/tmp/Test-Float-0.1/

-scott

Although Sikhs carrying the Kirpan are exempt from prosecution under the offensive weapons act

I didn't know about that one. I knew that a practising member of the Sikh faith is allowed to wear a turban instead of a helmet when riding a motorcycle. I wonder how many other such exemptions are carefully written into UK law. I need a list - Wikipedia you fail me! :-(

There were a few things that caught my attention in Facebook's presentation on HipHop, their PHP to C++ converter. It sounds like it relies on static analysis of the entire program's source, hence why they can't support eval, create_function etc. (22m25s in). I suspect that that sort of restriction would be, um, "interesting", in a general CPAN using environment, as a lot of modules build on various low level code that encapsulates eval, such as the traditional way h2xs did constants via AUTOLOAD. Also, as it's different runtime from Zend, so extensions need to be ported to it (19m in).

However, the most interesting part was a an early slide about memory usage, at 6m20. Transcribed:

150MB

for ($i = 0; $i < 1000000; $i++ ) {
      $a[] = $i;
}

700MB

for ($i = 0; $i < 5000000; $i++ ) {
      $a[] = $i;
}

(700M - 150M) / 4,000,000 = 144 BYTES

Does PHP really consume 144 bytes per integer value? Is that on a 32 bit or 64 bit machine?

For comparison, here is Perl:

$ perl -le 'for ($i = 0; $i < 1000000; $i++ ) { push @a, $i; }; print `cat /proc/$$/statm` * 4 / 1024'
22.4765625
$ ./perl -le 'for ($i = 0; $i < 5000000; $i++ ) { push @a, $i; }; print `cat /proc/$$/statm` * 4 / 1024'
118.44140625

which works out at 25.155 bytes per integer value, or under 20% of their figure for PHP. The odd number of bytes will be the malloc overhead spread across all the structures allocated from the same arena.

I have no idea what the usage of Python or Ruby are like, but there's a comment in the Unladen Swallow wiki:

Here at Red Hat we use Python for a lot of things. What we've observed is that execution performance is not the main issue (although it improving it would be greatly appreciated), rather it's the memory footprint which is the problem we most often encounter. If anything can be done to reduce the massive amount of memory Python uses it would be a huge win. I would encourage you to consider memory usage as just as important a goal as execution speed if you're going to tackle optimizing CPython.

I upgraded the firmware on my Netgear router today and it wouldn't let me use the LAN IP I usually use for it, 10.0.1.1, because it thinks my ISP uses that subnet, because I set the router to read from my own internal DNS. Took me awhile to figure out why it thought what it did, because it didn't occur to me that it would care what DNS addresses I gave it.

Cross-posted on <pudge/*>.

Ever looked back at something you've worked on and thought: "Gee, it's too bad that project didn't get to it's ultimate goal, but, I've learned a lot from it." I have one of those projects. Such is the world of technology, your toolset is constantly evolving and shape-shifting; even "The Next Big Thing" can become obsolete. We move on to the next "Next Big Thing."

One such area in the Web Developer's toolset is "Search." I'm sure we can all relate to the experience of our first textbox with some behind-the-scenes code doing SQL "SELECT ... LIKE" statements. Perhaps at first it was raw DBI calls; maybe moving on to an abstraction layer (ORM and whatnot) shortly thereafter. Here's where things get interesting.

What happens when this is no longer Good Enough(tm). Google being essentially ubiquitous, people expect to plunk some words in the box and magically get what they want out the other side. I put in "Cat Hat" -- why didn't it give me "Cat in the Hat"? Okay, no problem. We can do some field and query normalization; removing stopwords, add term parsing ... wait, wait wait. There has to be prior art for this.

In 2004, the options are somewhat limited as far as Free/Open Source search software goes. Especially in Perl land. Swish-e looks pretty neat. We actually did some prototyping with it. It was definitely a step ahead of plain old SQL. Plucene came on the scene. Unfortunately, it's poor performance was a bit of a non-starter for us. The fact that it was modeled after the Lucene Java library, however, caught my eye.

I wanted to harness their project and its community, and bring it into our little Perl world. Luckily for me, someone else had already started down that road. The Lucene Web Service was a project by Robert Kaye, sponsored by CD Baby, which allowed users to talk to Lucene via an XML-based web service. After using it for a while, we developed some patches for bug fixes and enhancements. Because of our momentum with the project, we were eventually given total control over its development.

We attempted to strengthen the project by hooking into some existing standards. We leveraged the Atom Publishing Protocol as an analogy for dealing with indexes and documents. Search results were returned as an OpenSearch document. A document's field-value pairs were listed in the XOXO microformat. Creating a client for this setup meant a bunch of glue between the existing components (XML::Atom::Client and WWW::OpenSearch).

Almost in parallel, the Solr project emerged. Similar idea, much more support behind it. In the end, our idea never got very far, and Solr has turned out to be a fabulous product -- which we now use.

To this end, the Lucene WebService website will (finally) be shutting down in about a week's time. I've moved the pertinent code and wiki data to github in case anyone wants it. I still think it has some niche applications, but without some serious revamping of the java code, it will likely just rot.

At least it's a project that has led me to bigger and better things.

On Tuesday, February 23rd, Jeff Thalhammer will speak on why you should create CPAN distros, even for proprietary code. He has worked on worked on several projects where all the private code was organized into CPAN-style distros, and then injected into a local copy of CPAN. They then used the CPAN tool chain to manage the entire build, test, and release process.

Jeff Thalhammer's CPAN page:
http://search.cpan.org/~thaljef/

Announcement posted via App::PM::Announce

RSVP at Meetup - http://www.meetup.com/San-Francisco-Perl-Mongers/calendar/12509158/

San_Francisco.pm (SF.pm) started off 2009 with a bang! Fred Moyer took on the daunting role of President, and Joe Brenner stepped up to the unforgiving role of Speakers Co-Chair. Both leaders relieved X-President Quinn Weaver, who sheperded SF.pm for 6 years leading up to 2009.

A Meetup web portal was created at http://www.meetup.com/San-Francisco-Perl-Mongers/ which served to facilitate organizing meetings. Six Apart generously donated a conference space for monthly meetings on the 4th Tuesday of the month (as has been a tradition for the last 10 years). Matt Lanier, the founder of SF.pm, liasoned with Six Apart to obtain this arrangement. We started off the year with a 35 person meeting on how not to use memcached - http://www.meetup.com/San-Francisco-Perl-Mongers/calendar/9432356/

With 15 official events in 2009 (http://www.meetup.com/San-Francisco-Perl-Mongers/calendar/past_list/) SF.pm had a banner year. We had a booth at OSCON, several celebrity speakers, lightning talks, and a growing membership as the year progressed. Tools developed in our forge such as App::PM::Announce allowed us to get out the message that SF Bay Area Perl was alive and on the move.

Red Hot Penguin Consulting LLC took care of the standard pizza fare, and Julian Cash Photography helped with soliciting food donations for the group from members (the monthly food/drink bill averaged $150 in 2009). SF.pm is still actively looking for a food and drink sponsor, so if you want to get your message out to SF.pm, the best way to our hearts is through our stomachs.

San Francisco Perl Mongers plans to continue steady growth throughout 2010 by focusing on what it did right in 2009. Regular meetings, growing membership, new and exciting talks, and some extra-perl-icular presences will help to solidify our presence as a stable and active technology group leader, and the pre-eminent Perl Mongers group in terms of regular events and attendance.