Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

gambuzino (4517)

gambuzino
  (email not shown publicly)
http://nunonunes.org/
Yahoo! ID: gambuzinix (Add User, Send Message)
Jabber: gambuzino@jabber.org

Journal of gambuzino (4517)

Sunday August 07, 2005
12:50 PM

Searching the Flickr API and iterators

I've been slacking off on the Flickr-Tools project, mainly because it serves all my purposes as it is right now.

But then someone pinged me regarding the photos method in the Flickr::Person class (the one that will return all the photos for the user) and that was the incentive I needed to finally get to writing that method.

So now I remember why I left it half-way through...

The thing is that this method must use the flickr.photos.search method to get it's info and it could return a lot of info.

So flickr.photos.search has a mechanism for managing the potentially huge volume of information --it forces you to break up your request into "pages" of results-- which, of course, is the right way to do it.

But how do I map that onto my Flickr::Person class in a nice way? I surely don't want to make the user have to control things like results per page and current page and so on, that would defeat the purpose of the whole thing, but I sure as hell can't just go and fetch everything from Flickr because for a user with many photos I would blow up the computer's memory fast!

So after thinking about it for something like 10 seconds I knew that the right way to do it would be to use an iterator. Instead of returning an array of Flickr::Photo objects I should return a Flickr::Photogroup object (or somesuch) and it should have a few simple methods like next, previous and total. You know, a classic iterator. When it ran out of results it should then invoque the Flickr::API::Photos::search method to get the next batch of photos.

OK, that is a bit more work than I bargained for but it is the right way to do it.

But then all of the other methods which I've already written and which return groups of things (mostly photos) should also return this kind of object. And each instance should know how to get more results when it needs it, so it should be rather smart about what it is representing.

This means two things:

  1. I have to rewrite the interface for methods which are already out there (major turn-off)
  2. I have a lot more work ahead of me in order to do it

So I will make this re-write and, unfortunately, I probably won't be able to have it done by tomorrow as I told the poor soul who is expecting a new version of Flickr-Tools with the new method and, probably, no interface changes.

Should have though of that before, now shouldn't I? Duh!

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/08/searching_the_f.html)

Friday June 17, 2005
12:41 PM

New version of Flickr-Tools

Since I know I'll be off-line and unable to do much development on the next few weeks, I'm trying to get as much features as I can in there right now, so even small enhancements like the ones I did today are going to be pushed out to CPAN as a new version.

Release early, release often, they say...

For version 0.02 I cleaned up some documentation bugs (yes, I do take some care with the documentation) and added a couple of methods which where no brainers to implement but I hadn't had the time to put in there.

There is at least one other method which would be really useful (get the list of all the photosets a given photo belongs to) but I'm waiting to see if the good folks at the Flickr API developers team can give us that method so as not to force me to do a lot of API calls just for this one. I do hope they find the time, but with all the buzz surrounding the new auth specs proposal I'm not that confident they will. First things first, I guess...

Anyway, let's see how much more I can do before going away on vacation at the end of next week.

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/06/new_version_of.html)

Thursday June 16, 2005
12:01 PM

Flickr-Tools is up on CPAN

Well, that's just it, Flickr-Tools is up on CPAN and should be made available any time now.

I've decided that I just can't sit on it any longer. It is far from complete but I've just looked at a big thread on the Flickr developer's mailing list with the promising title of "Big changes to the Flickr API" and which I'm guessing has something to do with the authentication methods or somesuch.

So, given that I'm rather pressed for time as of late (contrary to what I'd expected, sigh...) I'll just post it as-is, limited and all, and start work on version 0.02 ASAP.

As it is, it already allows people to get information on persons, photos and photosets and to manipulate high-level...ish objects with this info.

With the modules structure I've put in place it is really easy to add support for more core API methods so now it is mostly a question of filling it up with what's missing.

Oh and I have to read the thread carefully to see how the authentication stuff will be handled (if that is what the thread is really about).

Getting this out the door is basically a way to get some testing for free on other platforms and to give anyone out there curious enough about this a chance to see my main design ideas for this distribution, so suggestions are always welcome. Help in the form of code is also greatly appreciated, but I'm not holding my breath... ;-)

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/06/flickr-tools_is.html)

Wednesday May 25, 2005
03:28 PM

And we have photos... Almost.

Following yesterday's post and in the interest of getting something out as soon as possible, I've made some headway into the Flickr::Photo class today.

Of course this was after some cleanups and slight changes to the Flickr::Person class.

I can now instantiate a photo object, see (part of) it's info and get an object representing it's owner, all auto-magically. Rocks!

In the mean time, it is a good thing I decided to leave the write part of the read/write equation for later as I noticed today (two days late!) that there was an email on the Flickr developers list informing that the 'setters' (the write methods) where changed and that they now all have to be done via POST instead of GET.

While this makes perfect sense, I'm pretty sure it will warrant some changes in the Flickr::API and until Cal makes those changes I'll keep the write part on the back-burner. (As if it weren't already there...) ;-)

The hardest part of making something like Flickr-Tools is that you just know that the second you put it up on CPAN there will be a ton of changes to the object's API which will become immediately clear.

Oh well, release often, release early, warn your users, listen to the community feed-back and all that jazz.

I really wish I could make the first release during the coming weekend... I'm not sure I'll be able to do it though, as real life is getting in the way. But I'll surely try.

(Did I just say "real life"? Hum... It seems my moo/mud-user personality is trying to take over when I'm not looking! )

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/05/and_we_have_pho.html)

Tuesday May 24, 2005
07:12 PM

We have people!

I just couldn't resist and after having finished the first version of the Flickr::API::* classes I just had to try and implement one of the high-level modules to test it out.

So I now have Flickr::Person full implemented (no tests just yet, though, I went way about this one the impatient way).

It took about 30 minutes to do it and works like a charm! It was extremely easy to write, it is lazy in getting data and, once it actually has to get any data, it caches it for future reference.

It's great when things 'just work' like this. Quite a rush!

Well, but now I've got to be a good boy and go to bed, tomorrow I have a chiropractor's appointment really early and I will pay for staying up late working on this.

Can't wait to get the other three modules finished. This version of the modules is read-only (you can only get information from Flickr, and not all available information at that) but once I get it out the door and into CPAN I'll start implementing the write part of it. And filling in the gaps on the read methods, of course.

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/05/we_have_people.html)

05:03 PM

Mock objects

Mock objects essentially rule.

With them I can trick my modules into asking my mock object for the data they would receive from a call to a third object (of which I know nothing about, except for the output it returns for a given input) and they are none the wiser about it. This is incredibly easy to do, and I'm not even sure I'm using it in the best and most efficient possible way!

So I've basically put aside the problems of testing with live data, with all that entails about having to have a live connection to the Internet, hopping that the Flickr API is responding, having to setup test data and so on.

Some people did make the point that this way, should the Flickr API change in the future, my module will still pass all the tests and then bomb out on the poor unsuspecting users. Which, of course, is absolutely true, but it is not what module testing is about. Module testing is about making sure that the code runs as expected on the platform it is being installed on. The API may change in the future, but this is something I have to find out by myself and make the necessary changes to the modules and release them again.

So now I've got a bunch of modules which take the results from the Flickr::API calls and turn them into a saner structure for a few selected calls (no time to do them all so far, but I've got the infrastructure set up so now it is just a question of filling up the gaps). All the calls I wanted to include in the first release of the distribution are done. Now to get the higher-level modules working with these and abstracting the User, Photo and Photoset concepts and off they go to CPAN.

This is actually fun!

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/05/mock_objects.html)

Friday May 20, 2005
06:09 AM

Something new everyday

The responses to my post yesterday about testing with Flickr, both at use Perl; and at nowhereland were great.

I made the post rather in a hurry and right after I've come across the issue so I really hadn't given it much thought.

Had I thought about it, I would surely have gotten to the solution that some people pointed to, which is that I should just setup an account for testing or even ask the Flickr crew to set one up for the community to use. And that would have been acceptable, even if I weren't totally happy with it. It would always feel like a kludgy solution which could require tweaking if ever someone made changes to the user data or it's photos (which people would be able to do given the info on the module's testing scripts). It would also mean that you'd have to have connectivity when you where installing the module (or force-install it which I'm always loath to do).

In the end it is a good thing that I did make the post in such a rush manner because another solution came up which has a lot of promise.

Two people pointed me to the concept of "mock objects" which, as Melo puts it, "represents an ideal Flickr, which always gives me the response I need". This seems just perfect for this situation.

So now I have a new goal: learn how to use Test::MockObject and integrate it into the Flickr-Tools development process.

Who would have though it? It actually pays off to be lazy! Oh wait, all geeks know that... Right...

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/05/something_new_e.html)

Thursday May 19, 2005
02:28 PM

On testing modules

I've been getting some good progress on the Flickr-Tools distro, mainly on the lower-level modules (Flickr::API::Photos, Flickr::API::People, etc). At this point I can already rewrite the pet project which got me into writing this group of Perl classes in the first place, using only these classes. Admitedly this is not much to boast about, but it feels nice anyway... :-).

On the way I've met some dead-ends and had to track back a few times, but nothing too alarming.

Only now I've come across a real stumper.

While developing the modules I've taken the "build your tests before writing any code" approach which is all fine and dandy while I'm working on it, but I just don't know how I'm going to release the modules with the tests as they are now.

The problem is that testing anything useful at all requires not only the usage of a Flickr API key but also some rather precise information on a person's info, a few test pictures, photosets, and so on.

Of course I'm using my own information for the tests, but I'm not really happy with releasing a module into the wild with all that information on it!

I know I could just capture the result of the original Flickr API response for each instance I needed it, change some values around and test my modules against that result but that is just evading the problem and a whole lot of extra work.

I just don't know how I'm going to tackle this one...

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/05/on_testing_modu.html)

Friday May 13, 2005
04:32 PM

More thoughts on Flickr API

After some more thought (nothing like a bad day to get your brain working in the background...) I came to this conclusions about the classes and namespaces that make sense for the Perl Flickr API.

First I think there really is room for the two sets of modules/classes I discussed: a lower level one which simply models the published Flickr API directly on Perl structures of hashes and arrays, and a higher level one which models the "logical" level of users, photos, photosets, etc.

Then I seem to have reached a reasonable naming structure for them. My current plan goes along these lines:

The lower-level API-mapping classes

These should go into the Flickr::API namespace and should be named following the Flickr API's naming convention, namely there should be the following classes:

  • Flickr::API::People
  • Flickr::API::Photos
  • Flickr::API::Photosets
  • Flickr::API::...

Each of these classes should then implement the functions that are published on the API, for instance Flickr::API::Photos::getInfo(), Flickr::API::Photos::search() and so on.

The higher-level concepts-mapping classes

Then there are the classes which map concepts such as a user or a photo. As I described in the previous post, these objects should rely on the previous ones to manage the communication with Flickr for them and should implement a representation of all the information that Flickr has on each item.

The way I see it now, the namespace for these objects should be the base Flickr namespace itself and there should exist the following classes:

  • Flickr::Person
  • Flickr::Photo
  • Flickr::Photoset
  • Flickr::...

I've already started writing some code, mainly the tests (aren't I a good little programmer?) and even though I will be away for the weekend on a small family gathering I'll try and have something done by monday when I'll have Internet access again and can try out my code on live data.

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/05/more_thoughts_o.html)

06:37 AM

Thoughts on a new Flickr API for Perl

(This entry is cross-posted from my personal weblog at http://nowhereland.nunonunes.org/archives/2005/05/a_better_flickr.html)

Sooo... Last night Tuxa went to a farewell dinner she and her friends threw for a colleague and I was left home alone without the usual many urgent things to do, so I took a stab at the Flickr modules I'm starting to write.

And now I'm trying to decide which is the better approach to take.

One possible approach would be to create a Flickr object which has a method for each of the Flickr API calls, such as $flickr->photos_getInfo or $flickr->people_findByEmail and so on.
The object should then return a hash structure that closely resembles the documented result for that call, so for example the $flickr->photos_getInfo method (described here) should return something along the lines of the following:

{
  id => "2733",
  secret => "123456",
  server => "12",
  isfavorite => "0",
  license => "3",
  rotation => "90",
  owner => {
             nsid => "12037949754@N01",
             username => "Bees",
             realname => "Cal Henderson",
             location => "Bedford, UK",
           },
  title => "orford_castle_taster",
  description => "hello!",
  visibility => {
                  ispublic => "1",
                  isfriend => "0",
                  isfamily => "0",
                },
  dates => {
             posted => "1100897479",
             taken => "2004-11-19 12:51:19",
             takengranularity="0",
           },
  permissions => {
                   permcomment="3",
                   permaddmeta="2",
                 },
  editability => {
                   cancomment="1",
                   canaddmeta="1",
                 },
  comments => 1,
  notes => [
             {
               id => "313",
               author => "12037949754@N01",
               authorname => "Bees",
               x => "10",
               y="10",
               w="50",
               h="50",
               content => "foo",
             },
           ],
  tags => [
            {
              id => "1234",
              author => "12037949754@N01"
              raw => "woo yay",
              content => "wooyay",
            },
            {
              id => "1235",
              author => "12037949754@N01",
              raw="hoopla",
              content => "hoopla",
            },
          ],
}

This kind of approach appeals to me because it gives me access to just the plain data, in a easy way, without all the XML-parser-generated-cruft, in a much more "perlish" way.

But while I consider this an improvement over the current Flickr::API module (which is great, by the way, and will be used as the foundation for all of this), it may still be too low-level for many people to use.

So there is another possible approach (OK, there are many other possible approaches but I'm focusing one on in particular) ;-) which is to create a set of higher-level objects which represent an entity like a "person" or a "photo" or a "photoset".

Then the user would simply ask the object to instantiate, say, the Flickr user whose email is something@somesuch (using flickr.people.findByEmail on the background) or the photo whose id is someid (using flickr.photos.getInfo). Then, if the user wanted some more information on the photo, for example, she could ask for the lens aperture used ($photo->apperture()) and the object would fetch the EXIF information for the photo in question (using flickr.photos.getExif), load it into the $photo object and return the information.
A major point to make here is that the object would fetch information in a "lazy" manner, so as to limit the number of interactions it has to have with the Flickr server.

My immediate needs are more than adequately met by the first approach to the modules and, therefore, I guess I'll surely implement them right now, but the second approach could be very interesting indeed...
But then, now that I think about it in a more structured way, the second way depends on the first one to process all the results from the raw API and convert them into useful data, so I may well end up implementing it too in the future...

What I haven't figured out just yet is which name-space to use.
Should I go for Flickr::API::Methods (as in Flickr::API::Methods::people_findByEmail() or Flickr::API::Methods::photos_getInfo())?
Or maybe go for Flickr::Tools...
Or I could go with Flickr::Photos (Flickr::Photos::getInfo() for instance), Flickr::People (Flickr::People::findByUsername()). Actually I think I like this last version best...

As for the higher-level modules I'm not entirely convinced they should even be in the Flickr namespace at all... But then I don't have to decide on it just yet so let's see what my mind is aimed at when I get to it.

If anyone reading this has any kind of comment, remark, idea, suggestion or tough about the subject I really would appreciate the feedback.