Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

richardc (1662)

richardc
  (email not shown publicly)
http://unixbeard.net/~richardc/

Journal of richardc (1662)

Monday March 08, 2004
12:55 AM

evolving - Paging Win32 and/or Device::SerialPort users.

[ #17800 ]

So pleasantly enough Device::Ericsson::AccessoryMenu turns out to have quite the life of its own, so much so that the latest release (0.8) is entirely the work of Arne Georg Gleditsch who sent me a patch last month which I shamefully only just got round to looking at.

Anyway in yesterdays mail I received a query about bluetooth devices going out of range, which is in part the disconnection/reconnection stuff mentioned in the TODO section of the pod.

The problem in play here is that when your bluetooth device goes away it's presented to the machine as 'your modem has been disconnected from your serial port' and I can't work out how to get that from the Device::SerialPort API (actually, I can't figure out very much at all from the documentation of Device::SerialPort and Win32::SerialPort, but that's a different matter).

One way around this whole mess is to avoid Device::SerialPort altogether, and handle the serial port directly in a posix stylee, the downside to which is but that means losing the plug compatibility with Win32::SerialPort and so Win32 compatibility in general.

I'm somewhat reluctant to lose an OS just to make my life easier (though if we were talking about AIX then that's quite a different matter...), but I don't really know that there are Win32 users to lose, and so on to the appeal.

If you're using Device::Ericsson::AccessoryMenu on Win32 please drop me a note let me know. Also if you know how to get Device::SerialPort to 'fess up about the device going away, that'd be super too.

Monday - 2004-03-08 05:52

Saturday February 21, 2004
07:47 PM

evolving - Sweet sweet release

[ #17548 ]

So I just finished cleaning up my zeroconf stuff enough to get it released, you should see them arrive on your local CPAN mirror any time soon.

After a grand renaming I shoved everything into the Net::Rendezvous::Publish namespace, since after experimentation I found that Net::Rendezvous is a pure-perl browser-only implementation, but that I liked the api and implementation enough to try and usefully extend that space.

(I did try and evaluate Net::MDNS::Server and Net::MDNS::Client but their embdedded mdnsd just didn't build for me on Panther)

So about that terminology then:

Zeroconf

Zero Configuration Networking or Zeroconf to its friends, is an IETF working group who have issued specifications on various things. Together those specifications are called Zeroconf. Wasn't that nice and clear of them.

One of the Zeroconf standards is mDNS, but there are also a bunch of standards that define automatic allocation of IP addresses and the like.

---

ZeroConf

How I keep mistyping Zeroconf. I say it as two words too. It's all a trap laid out for me I swear.

---

mDNS

Multicast DNS. A fun abuse of the DNS wire standards to add multicasting of service information. This is where the publishing and browsing of services we all know and love comes from.

---

Rendezvous

What Apple calls their adoption of the Zeroconf protocols. It's actually just a subset which contains Dynamic Configuration of Link-Local IPv4 Addresses, Multicast DNS, and DNS-SD.

- or -

It's just what everyone calls mDNS over link-local.

Anyway I'm glad to say that's 3 modules cleared from my yak pile, I can start on finishing Hook::Queue tomorrow, and everything will be just super.

Sunday - 2004-02-22 00:46

08:57 AM

evolving - Back with the new style

[ #17540 ]

First, apologies to you people reading via rss or use.perl, the syndicating view seems to really make article-style things look like ass. It's really much better looking in the original klingon, or the html/css thing that it transforms into.

Now some more of that old fashioned 'I tweaked my blog engine' guff.

I've moved in thinking from this being a ongoing diary to more distinct entries, just like a real blog. That means now the linkable pages are entries rather than days, and I'm putting a timestamp thingy at the bottom as a =head3 block so I can still figure out the day on which I wrote it. I might even teach the RSS template about that, if I feel the urge.

There, wasn't that suitably dull.

Saturday 2004-02-21 13:49

Friday February 20, 2004
06:57 PM

evolving - Friday 20th February, 2003 - Mac kool-aid

[ #17533 ]

When I joined Fotango, I asked my new overlords if I could get a Mac laptop as my work machine. Being good masters, or as part of their open source slave programme - the distinction isn't clear, they equipped me with a shiny powerbook (and after some supplier issues a working one.)

The upshot of this is that for the last couple of months I've been migrating into it, and embracing stuff like iCal and Rendezvous.

Here's a run-down of the kool-aid I've supped to date.

---

Fink

I've always treated the system shipped perl with great suspicion, mostly because it's often old, the modules I want aren't packaged for it, and if I manually install/upgrade modules they get trampled by system upgrades. As a result I generally install my own into /usr/local/perl5.xx and tend that while letting the system take care of itself.

Well with the laptop I went with fink, because compiling binary modules against fink-managed libraries has bit me before, the system perl is a 5.8.1-era thingie (a late 5.8.1 RC at least) so new enough, and it's easy enough to write package descriptions so the modules I want can be made available as packages.

It's going well so far, I got a talk out of it, and more package descriptions than you'd normally shake a stick at (caveat: I rarely shake sticks, which may skew the sample).

---

iTunes/iPod

Actually, I drank of the iTunes kool-aid with the release of the Windows version, but I have imported all my albums into the new machine. With X-Tunes it's a dream.

New to the mix is a 15Gb iPod. Given that no new technology acquisition goes unpuninshed I ran into a typical iPod problem - my music collection is bigger than it is.

iTunes has smart playlists, so I can say 'give me 15Gb of music', but the drawback is that I can say 'of albums' or 'of songs', and 'randomly', but that either gets me the first 15Gb of albums every time, or incomplete albums.

The solution of course was to write a script that uses File::Find::Rule and Mac::Glue to populate a playlist with 15Gb of randomly selected albums. Because that's a bit slow it also uses Term::StatusBar so you can see how far through you are.

---

Mail.app

I read a lot of email. OK, I skim a lot of email. I used to deal with all of it using mutt on penfold, and then I switched to using Mail.app's imap mode.

That was all good, apart from Mail.app's mail filtering rules don't want to filter into imap hosted folders. That's not so much of a problem for my on penfold because everything goes through procmail and Perl in a long-tended configuration (scripts and .procmailrc in the usual places.) At the office however, the imap server is a magic multi-domain virtual-user setup, so running procmail just isn't an option.

imapfilter to the rescue! Though it's got a slightly quirky configuration file (and what doesn't), I managed to bend it to my will and now everything is all well and filtered in the office too.

---

Rendezvous

Or is that Zeroconf? Or should that be mDNS? Or is that just a part of the zeroconf standard? I've a dread feeling that I'll have to put together another list like I did for the vCalendar debacle.

Anyway, it's really cool to watch things show up in Rendezvous Browser, especially if you're using modules you wrote to do it. More services should advertise themselves.

While I was busy debugging Net::ZeroConf::Backend::Howl George Chlipapa released Net::Rendezvous, which I must install and play with at some point, as it could well completely obsolete my 3 dists before I even got round to releasing them.

---

iCal

Apart from the adventure of parsing ics, it's been really rather painless.

On my todo list is to set up a private DAV share on unixbeard.net so that we can put household events like "I'm going to FOSDEM, feed the fish" in it, as we still seem not to have bought a paper 2004 calendar for the kitchen.

06:55 PM

evolving - Friday 20th February, 2003 - Fun with Calendars

[ #17532 ]

A couple of months ago, that pesky Paul Mison (if I get to be the "wily" Richard Clamp for Pod::Coverage I reserve the right to hand out one pesky to someone) planted an idea in my head, which lead me through something of a world of hurt...

First I'll tell you what the idea is, and you can tell me what the pain was.

Ready?

It would be cool to turn a page on our intranet into ics so we can
  add it to iCal.

Okay, so now take a moment to yourselves to figure out where the pain came from.

Done?

10 points if you guessed Net::ICal.

Now I know it's disclaimed as being alpha code, and that I shouldn't be surprised if it's not too good, but the CPAN dist doesn't even pass its own tests, and the CVS version didn't even get that far.

I even sat down for a few afternoons and worked on a fork of Net::ICal CVS which got the tests passing, and it still wasn't fit for duty.

There's a whole other paragraph which goes here and questions the parentage and sanity of Class::MethodMapper, but there's only so many times you can say rude words. Let's just say it's not suitable for something you want to parse, validate, and follow the flow of control without big sheets of paper and crayons. I like crayons, but I like clear flow of control even more.

Thus cheesed off, I put the idea aside for a few weeks, expecting that when I came back to it I'd just cave and write an alternative to the whole ball of wax.

That few weeks ended this week, when Nik pointed out to me that Text::vFile should be a suitable module to base a vCalendar parser on.

an aside - iCalendars, vCalendars, ics, vFile, rfc2445, make your mind up!

So why do I keep calling things ics one moment, vCalendars the other, and iCal the next? Well it's partly due to my brain being a swiss cheesed mess, but also because someone is out to get me.

rfc2445 / iCalendar

rfc 2445

     Internet Calendaring and Scheduling Core Object Specification
                               (iCalendar)

This baby is the specification for iCalendar-formatted files.

Weighing in at 148 pages it covers pretty much anything you need to do for online interactive calendars (maybe).

---

ics

The common extension for iCalendar files when stored on disk.

---

vFile

It turns out that the object/property/parameter encoding used by rfc 2445 is the same as the one used by rfc 2426 vCard MIME Directory Profile, apart from 2445 encoded documents tend to have deeply nested objects.

We follow Text::vFiles example in calling these documents vFile formatted.

---

vCalendar

Okay, this one's probably made up.

It comes about because iCalendar is a kind of vFile.

I think.

Look just stop looking at me like that.

---

iCal

An application from Apple Computer, Ltd. Parses, generates, and generally speaks rfc 2445.

Sometimes I call it iCal.app, in order to distinguish it from the standard in the same way that use Mail.app to refer to Apple Mail.

And thus overloading occured, in my branes at least.

Sadly, Text::vFile doesn't handle nested objects - which you really need for iCalendars - and clones a little too much (ie. at all) from Class::MethodMapper in its api and thinking.

At about the time we discovered this my impatience kicked in. "Surely it can't be that hard to parse vFile to a simple data structure" I said to myself, and half an hour of coding later it turned out it wasn't. Text::vFile::asData sprang into the world.

A couple more hours tagteaming with Nik to find the corner cases and fix them and we both managed to scratch our itches. Nik's handy what's going on script to summarise upcoming events, and my wiki to ical script for the intranet, both done fairly easily with the assistance of DateTime. Version 0.01 hit the CPAN last night, with 0.02 to follow this weekend.

06:50 PM

evolving - Friday 20th February, 2003 - mmm: every home should have one

[ #17531 ]

One of the most tedious things in writing a new module is just spinning up the distribution structure and the pod boilerplate to go in it. Of course, there are a whole bunch of ways to skin that cat, from the fugly h2xs, through ExtUtils::ModuleMaker, to that home grown that you've got sitting in ~/bin.

I'm using a home grown one too, but only in the sense that I'm using mmm, which my housemate Mark Fowler wrote.

It's got two big selling points for me. Firstly it creates a very sane module structure, which is an absolute requirement. Mainly though it's that it pulls all the boilerplate in using the Template Toolkit. This makes it trivial for me to just customise the sub-templates for when I'm not Mark, or I don't like one of his defaults.

If you've ever had an itch to use a module generator I'd suggest you take mmm for a spin - it offically Doesn't Suck.

06:48 PM

evolving - Monday 17th November, 2003

[ #17530 ]
Overly and repeatedly dumb.

So today I figured it'd be a good enough time to try and speedup Timesink some. For those not keenly stalking me or my pet projects Timesink is my web-based RSS aggregator which I wrote a while back on migrating away from a dying mac and NetNewsWire.

Anyway, there are a couple of parts of it which are kinda slow, one is the "what's unseen" calculation in the web frontend, and another is the scraper.

Now the unseen code looks like this:

package Timesink::DBI::Subscriber;

sub unseen {
      my $self = shift;
      my $feed = shift;

     my ($sub) = Timesink::DBI::Subscription->search({ feed => $feed,
                                                        subscriber => $self });
      my %seen = map { $_->item => 1 } $sub->seen;
      return grep { !$seen{ $_ } } $feed->items;
  }

That is; given the subscriber find out how many items in a given feed are unseen by that subscriber. That you get the actual objects is somewhat a side effect as the web inferface only really cares about the final count.

Okay I think, after determining that this is the slow spot with Devel::Profiler, time to rewrite that as a quick SQL query.

Small flaw in that plan, my live instance runs on mysql 4.0.16, and won't get upgraded till 4.1 until debian unstable does that for me, so I'm stuck with painful rewriting to emulate it. After about half an hour of that, I eventually admited defeat and added this comment:

+# XXX I be the slowest routine in Christendom.  a sub-select would
  +# probably help, if mysql 4.0 wasn't lame.

So moving on to the second step. Work on the speed of the scraper.

Now the scraper itself is fairly quick, it's only really waiting on upstream servers handing out RSS documents to parse, so if I could just parallelise the downloading that it's going to take less time, even if it's not really quicker.

Now I could see two ways around that, something finicky with LWP::Parallel or the brute force forking of Proc::Queue.

Given that I didn't want to rewrite LWP::Simple's mirror routine I decided to plump for forking, just so long as I remembered to disconnect the dbh I'd be fine, or so I reasoned. So this was my first stab:

Before:

for my $feed (@feeds) {
     my $rss = $self->get_rss( $feed ) or next;
     $self->scrape_feed( $feed, $rss );
  }

After:

for my $feed (@feeds) {
      my $pid = fork;
      die "couldn't fork $!" unless defined $pid;
      if ($pid == 0) {
          my $rss = $self->get_rss( $feed ) or next;
          $self->scrape_feed( $feed, $rss );
          exit;
      }
  }
  1 while wait != -1; # reap the kids

Spot the deliberate mistake? Well even if you did, I didn't for a time. Then Rafael asked me why I was grabbing his RSS 30 times a minute.

So I scratched my head, and eventually saw my mistake. Back in the old single process model if get_rss didn't return new rss that was your clue to check the *next* rss feed. Once I'd moved that into a multi-process model the job of the child is not to try again, but to exit gracefully. The fix was as simple as:

-         my $rss = $self->get_rss( $feed ) or next;
  +         my $rss = $self->get_rss( $feed ) or exit;

Case solved I thought, and went off to watch teevee.

Of course it doesn't end there. But there's bonus points if you guess my next (and hopefully final mistake of the evening).

Yes that's right, I'd forgotten to install the fixed version of the module, so come the next time the script ran it picked up the old DoS-happy version of the module and looped all over again. D'oh.

---

Nothing for months, and then two modules come along in one day.

IO::Automatic and Parse::Debian::Packages pretty much sprung to my fingers unbidden today, the latter a side effect of adding debian support to Leon's cool new Module::Packaged module, the former is a TT-like trick extracted from some code I found myself banging on.

Enjoy.

Monday November 17, 2003
08:10 PM

evolving - Monday 17th November, 2003

[ #15825 ]
Overly and repeatedly dumb.

So today I figured it'd be a good enough time to try and speedup Timesink some. For those not keenly stalking me or my pet projects Timesink is my web-based RSS aggregator which I wrote a while back on migrating away from a dying mac and NetNewsWire.

Anyway, there are a couple of parts of it which are kinda slow, one is the "what's unseen" calculation in the web frontend, and another is the scraper.

Now the unseen code looks like this:

package Timesink::DBI::Subscriber;

sub unseen {
      my $self = shift;
      my $feed = shift;
      my ($sub) = Timesink::DBI::Subscription->search({ feed => $feed,
                                                        subscriber => $self });
      my %seen = map { $_->item => 1 } $sub->seen;
      return grep { !$seen{ $_ } } $feed->items;
  }

That is; given the subscriber find out how many items in a given feed are unseen by that subscriber. That you get the actual objects is somewhat a side effect as the web inferface only really cares about the final count.

Okay I think, after determining that this is the slow spot with Devel::Profiler, time to rewrite that as a quick SQL query.

Small flaw in that plan, my live instance runs on mysql 4.0.16, and won't get upgraded till 4.1 until debian unstable does that for me, so I'm stuck with painful rewriting to emulate it. After about half an hour of that, I eventually admited defeat and added this comment:

  +# XXX I be the slowest routine in Christendom.  a sub-select would
  +# probably help, if mysql 4.0 wasn't lame.

So moving on to the second step. Work on the speed of the scraper.

Now the scraper itself is fairly quick, it's only really waiting on upstream servers handing out RSS documents to parse, so if I could just parallelise the downloading that it's going to take less time, even if it's not really quicker.

Now I could see two ways around that, something finicky with LWP::Parallel or the brute force forking of Proc::Queue.

Given that I didn't want to rewrite LWP::Simple's mirror routine I decided to plump for forking, just so long as I remembered to disconnect the dbh I'd be fine, or so I reasoned. So this was my first stab:

Before:

  for my $feed (@feeds) {
     my $rss = $self->get_rss( $feed ) or next;
     $self->scrape_feed( $feed, $rss );
  }

After:

  for my $feed (@feeds) {
      my $pid = fork;
      die "couldn't fork $!" unless defined $pid;
      if ($pid == 0) {
          my $rss = $self->get_rss( $feed ) or next;
          $self->scrape_feed( $feed, $rss );
          exit;
      }
  }
  1 while wait != -1; # reap the kids

Spot the deliberate mistake? Well even if you did, I didn't for a time. Then Rafael asked me why I was grabbing his RSS 30 times a minute.

So I scratched my head, and eventually saw my mistake. Back in the old single process model if get_rss didn't return new rss that was your clue to check the *next* rss feed. Once I'd moved that into a multi-process model the job of the child is not to try again, but to exit gracefully. The fix was as simple as:

  -         my $rss = $self->get_rss( $feed ) or next;
  +         my $rss = $self->get_rss( $feed ) or exit;

Case solved I thought, and went off to watch teevee.

Of course it doesn't end there. But there's bonus points if you guess my next (and hopefully final mistake of the evening).

Yes that's right, I'd forgotten to install the fixed version of the module, so come the next time the script ran it picked up the old DoS-happy version of the module and looped all over again. D'oh.

---

Nothing for months, and then two modules come along in one day.

IO::Automatic and Parse::Debian::Packages pretty much sprung to my fingers unbidden today, the latter a side effect of adding debian support to Leon's cool new Module::Packaged module, the former is a TT-like trick extracted from some code I found myself banging on.

Enjoy.

Wednesday October 29, 2003
04:51 PM

evolving - Tuesday 28th October, 2003

[ #15460 ]
Found Art

21:03 <@hex> hmm. http://www.brunching.com/cybimages/R/cyb-RICHARDC.gif
  21:03 <@hex> "I have patches! Die, worthless flesh-being!"

Friday September 19, 2003
07:54 PM

evolving - Friday 19th September, 2003

[ #14796 ]

It be talk like a pirate day today, but I need more grog to pull that off.

---

Released a 0.05 of Devel::LeakTrace. Minimal bugfixes, nothing to shout about really. The next version, now that's going to be cool.

---

I had another of those itches today with trying to build ponie on my laptop, so now I have a sketch for Devel::System a little module to intercept calls to the system function so that you can see how things are being invoked without cracking the script open.

It'll probably grow a callback interface and the ability to dry-run, assuming the chaos monkey is kind with his dispensation of tuits.