Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

tmtm (2563)

  (email not shown publicly)

Journal of tmtm (2563)

Wednesday August 03, 2005
06:01 AM

Class::DBI and Litigation

One rumour that I do feel unfortunately compelled to address is the concept that there is any legal ambiguity or action surrounding Class::DBI.

I am unaware of any legal issues relating to the code or use of it, and have never been party to any action, or any threatened action, in respect of it, either as plaintiff or as defendant.

As a director of several UK companies I have been, and currently am, involved in litigaton in relation to other completely unrelated matters, but none of this has ever been connected to open source code or projects in any way.

Sometimes 2 + 2 really does just equal 4.

Monday August 01, 2005
07:34 PM

Today on Gossip Central

I'm faced with quite a dilemma at the moment. There seems to be lots of misinformation going around, apparently mixed with a heady dose of outright lies.

On the one hand, I'd like to try to clear it all up (even though, really, I know that the world doesn't work that way, and that I wouldn't actually be able to).

On the other hand, though, to even try to do so would involve violating two principles I believe in:

1) Don't publicise disputes whilst you're trying to resolve them.

2) Don't publish private emails.

Even though some people seem to not believe in these principles, or at least don't follow them, for now I'm going to stick by them.

The complete disregard for number 2 in many quarters makes it difficult for me to even chat to people privately about a lot of these things at the moment.

So, for now at least, it looks like everyone will just have to choose whether or not to believe the gossip. And I'll just have to continue to be surprised at some of the people who do.

Thursday December 27, 2001
05:19 PM

So far but not quite near enough.

Spent most of today hacking on Class::DBI.

Tatsuhiko Miyagawa has pulled out the 'trigger' code that I wrote to allow Class::DBI::mysql::FullTextSearch to happen, so I was able to pull that all back out again and replace it with the single line:
  use Class::Trigger

Then I came up with a way to add a pre-insert trigger, which meant I was able to add the first version of constraints, and will now provide a much better way of allowing default values that can't be specified at the database (e.g. MySQL doesn't allow CURDATE() as a default, and you currently have to do much madness to implement this in Class::DBI).

I also had one of those "blinding flash of the obvious" moments on Christmas Eve, when I realised that, although it's quite unorthodox in database terms, Class::DBI really needs an ON SELECT trigger. So I added that, and now it makes it ludicrously simple to add code to, for example, convert all 'DATE' fields to Date::Simple objects as they get retrieved from the database.

I just need to finish off the new Cookbook, and I can release this version.

But before I could do that I got much too waylaid by a nasty memory leak, somewhere within the Class::DBI -> Ima::DBI -> DBI chain. My process to rebuild the Music Database kept getting bigger and bigger and bigger. It took about 2 hours to even find out where it was happening, and I still haven't worked out why.

There's some sort of nasty collision somewhere in the cached $sth code, but trying to get my head around all those closures, inheritable class-data lookups, and barely documented features of DBI just proved too much for today.

And tomorrow I really need to finish off some client work ...

Sunday December 16, 2001
03:38 PM

Untainting dates

One of my personal bugbears is web forms which make you jump through lots of hoops to enter dates - particularly the ones which make you guess what those hoops are.

So, for a client job recently which needed a date field, I wrote a 'date' plugin for CGI::Untaint which used Date::Manip to parse the date in whichever format you entered it ("2001-12-12", "12th December", "last Monday", "third tuesday in March", etc.) and then feed it to Date::Simple, so we'd end up with a standard format (with a nicely overloaded stringification for easy entry straight into a database - through Class::DBI, of course). [with credit to Simon Burns for the original idea].

The client loved it. Everything seemed great. I uploaded the module to CPAN.

Then I installed it on one my own sites, and everything fell over. Because ironically, the resulting value wasn't taint safe!

It turns out that Date::Manip isn't taint safe at all, doing lots of shell-type things to work out your timezone.

This puzzled me for quite some time. And then yesterday I had a brainwave. Recalling Damian's diary entry on locally replacing subs, I threw in a:
  local *Date::Manip::Date_TimeZone = sub { 'GMT' };
and everything was rosy again! (We only need dates, not times, so this shouldn't even cause any problems anywhere...)

Which got me to wondering about how to ensure that all my test suites work in a taint clean manner. Although make test can handle the -T option on the shebang line, running the test manually with perl -Ilib gives us the old Too late for "-T" option error.

I really need to read up on all the deep magic I can do with ExtUtils::MakeMaker to see if there's some nice way of adding something in there ...



All I need to do, of course, is perl -TIlib.

Of course, I still want to read up more on the magic of MakeMaker ...

Saturday December 15, 2001
07:03 PM

The Great Class::DBI Clean Up

The great Class::DBI clean up continues.

Today I pulled about 150 lines out of the main module into a support module. This was pretty hairy as it was the code that handled all the 'relationships' between classes - which is pretty important stuff for a module that handles database mappings!

I originally attempted to do this via mixins - basically just exporting all the methods I needed. But some of the methods over in Class::DBI proper needed to play with all the relationship data too, so I ended up exporting almost every method, including supposedly 'private' ones, which is always a worrying sign.

Thankfully the hooks I'd added when I was writing the Class::DBI::mysql::FullTextSearch module proved their worth. So I was able to take code, such as that for cascading delete, which needed to follow the relationship path, and implemented it trivially in the helper module just registering a delete hook.

Unfortunately, the other major point in Class::DBI where standard behaviour changes if there are relationships is create(). If you try to create with a field set to a related object rather than just its id
  CD->create({ title => $title, artist => $artist })
instead of
  CD->create({ title => $title, artist => $artist->id })
then it needs to DWIM. And we have no pre_create hook as there is no object at that point we can play with :(

For now, I've just implemented a _tidy_creation_data method which does nothing in Class::DBI proper, but which gets overridden when you set up a relationship to one that downgrades embedded objects to their ids.

I don't like this though, and I need to find a better way ...

Tuesday December 11, 2001
03:18 PM

"The approximate delay for your decision is about 2.5 years"

This morning I went to hear Tom Gilb speak. When I first heard he was coming to Belfast, I was quite excited; but then I heard he was speaking on software inspection, which sounded quite dull. It turned out to very interesting, although (or perhaps because?) he only managed to make it through less than 20% of his slides!

After pointing out the "so obvious, that no-one ever gets it" fact that inspections differ from reviews in that inspections are checking against a well defined 'standard', Tom proceded to give his three rules for inspecting a requirements document:

  1. Unambiguous to the intended readership
  2. Clear enough to test
  3. No design specifications mixed in

I've been immersed in the whole XP / Rapid Development culture for so long that I'd almost forgotten that there were people who still used 800 page requirements documents to write their design specs etc. But it's interesting to see the way that XP has taken a lot of the work that people have done in the Big Process Methodology field and totally shifted the goalposts.

Clear enough to test is a classic example of this. XP pretty much says that requirements *cannot* be untestable, as the requirements get expressed as tests! If something passes all the tests, then it's finished. If it's missing something, then write a test that exposes it.

As a developer, I've found this to be a great mindset to code under. I'm a complete convert to test-first programming. However, I haven't yet managed to convert clients to the mindset for acceptance testing. It's generally hard enough to get people to tell you what they think they want, without them having to come up with repeatable tests that 'prove' that the system works. I still like the idea though - maybe I just need to come up with a way of presenting it to a client that makes it sound good!

In other news, it seems my Test vs Overload problem boiled down to the fact that when overloading you need to either provide a sensible method to call for each overloadable thing, or provide a 'fallback'. Adding a fallback made it all work.

But I also learned in the process that perl treats
  print scalar(my($foo) = "bar");
very differently than
  print scalar(my $foo = "bar");

So Test::More's ok function, which is prototyped $;$, coercing scalar context, treats these differently too...

I've also had problems recently with Template Toolkit and Class::DBI not playing well together. For the most part, they're a wonderful combination - pass a single Class::DBI object representing a row of your database into your template and you have methods representing each of the columns, as well as more complex structures representing your relationships:
  [% FOREACH track = cd.tracks %]
    - [% track.title %] by [% %]
  [% END %]

One of TT's (many) wonderful features, is that it just DWIMs on the '.' character, giving you a hash element, or array element, or call a method on an object or whatever it needs to do (the above TT code wouldn't change at all if I had a hash of hashrefs instead of an object containing other objects etc..). However, this means that when you pass it a one element list, it will treat it as a long scalar.

In normal circumstances you can call .list on this to force TT to treat it like a list, and access .first, or suchlike. But, only if it's unblessed. If it's blessed it'll try to call the first() method instead (and fail if you don't have such).

Or should I say, 'it used to try', as a couple of posts to the TT list about this, and Hey Presto, Andy has patched everything to just DWIM.

I really love the world where you can leave the office at night with code that doesn't work, and return in the morning to discover it can, without your code even having been touched in the process :)

Monday December 10, 2001
01:30 PM

Phone Bills and overloading

Today I got my password for BT's "View My Bill" service. After spending a while playing around with the fact that you can enter names against phone bills and then view the bill with names instead of numbers, I decided I really liked this feature ...

... except for all the problems.

Particularly the way you can enter multiple numbers against the same person or company, except it won't recognise that fact on your bill, so you have to create a different entry in your address book for every number.

Which would be bad enough, except they also restrict you to 75 entries.

So, I decided it would be fun to download this, dump it to a local database, and then write a simple front-end to let me do it myself. Which would also go back as far as I wanted and not just my last 3 bills.

And they conveniently let you download CSV format, so I don't even need to do all the screen-scraping stuff that I started to do before I thought to check! *doh*

So, I put together a simple module to parse that, and decided it would be fun to have an interface that overloaded the iterator:

my $bill = Data::BT::PhoneBill->new($filename);

while (my $call = $bill->next_call) {
 print $call->date, $call->time, $call->destination,
  $call->number, $call->duration, $call->cost;

So I wrote my tests, and tried the code. And everything collapsed. In quite serious fashion:

Operation `!': no method found, argument in overloaded package Data::BT::PhoneBill at /usr/local/lib/perl5/site_perl/5.6.1/Test/ line 255.
WHOA! Somehow you got a different number of results than tests ran!
This should never happen! Please contact the author immediately!
END failed--call queue aborted.

After a lot of puzzlement I eventually tracked everything down to a small failing example. It seems Test::Builder doesn't work very well with overloading:

#!/usr/bin/perl -w
use strict;
package Eek;
use overload '' => \&get_next;
sub new { bless {} }
sub get_next { 1 }

package main;
use Test::More tests => 1;
ok my $foo = Eek->new, "We got an Eek";

So, playing with all this means I never got to write my database stuff, and I'll probably have to do some "real work" for the rest of the week.

Oh well.

Sunday December 09, 2001
05:10 PM

The perils of efficiency

Today I spent a long time banging my head off Class::DBI, and managed to chop another 200 or so lines out of it, in the never-ending clean-up quest.

Some of it was simple, like discovering that its 'rollback' method looked for the values you'd changed, and reloaded those from the database. So instead I made it just lose all memory of those, and let its normal 'lazy loading' mechanism fetch them the next time you asked for them.

Other bits were hairier, but deeply satisfying, like removing its support for pseudo-hashes. I don't think anyone actually ever tried to use this, and if they did they're much too sick. Getting rid of this also allowed me to lose two modules from its multiple-inheritance tree. (Not that it ever actually needed them there, due to their (accidental?) mix-in behaviour.)

But far and away the scariest bit was untangling a twisted maze of parallel class data structures. When you set up a Class::DBI class you tell it information about your database table: not just how to connect to it, but what the columns are, and how to group them. Then, when you call a method corresponding to one of the columns it works out what column groups it's in, and what other columns are in those groups, and fetches all those at the same time, on the assumption that if you've grouped them together well, you'll probably want to get these values soon anyway, and we may as well save a trip to the database.

So, for efficiency, we had two different data structures - one mapping columns to groups and one mapping groups to columns. In class data. With a lot of little support methods to look after all the fancy cases.

But, like most things that get implemented for efficiency's sake, it was fast becoming unmaintainable. So, I ripped it all apart to only store the data once, and just look it up both ways around.

The result? Code that's much easier to maintain and extend, and a performance hit that seems to be less than 0.3%.

And having done all that, I think I can see a few ways in which the higher level performance of this can be tweaked, probably resulting in a nice net gain, as well as cleaner code.

Another victory for the Rules of Optimisation.

Saturday December 08, 2001
06:12 AM

it's like shedding your skin

When BlackStar happened, it all seemed to happen too fast. It took over about four and half years of my life, and I can't really differentiate very much of it. (Other than the 'this is where it stops being fun' turning points...)

So, now that I've moved on, and have a new set of plans for Total World Domination(tm), a few people have suggested I keep a journal of it.

So, I guess I can do some of that here...

One thing I'm really keen on resurrecting is The Music Database. This was one of the things I worked on before BlackStar - basically a web front end to the CDDB project (the thing that auto-fills your playlist info when you put a CD in). The grandiose version was to use that as a basis for constructing the music version of IMDB, but before we got anywhere near there we had a big old fight with CDDB, just as they were turning commercial and deciding that all the user-submitted data actually belonged to them.

So now, five years on, CDDB have fallen into much disfavour, and freedb has picked up the mantle of keeping the whole thing open. And it makes me think this is doable in again.

So, I've downloaded the latest database, talked to the freedb guys about ways forward, knocked up a simple module for parsing their files, and starting playing...