Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

2shortplanks (968)

  (email not shown publicly)
AOL IM: trelane2sp (Add Buddy, Send Message)
Yahoo! ID: trelane2sp (Add User, Send Message)

Mark Fowler has never been the same since he was elected leader of the London Perl Mongers. The strain manifests itself mainly in releasing various [] modules [] to CPAN, giving talks [], and use of the Trelane nick on for endless procrastination. Doctors are still seeking a cure.
Monday February 25, 2002
01:03 PM

The New World Of Pipelining

[ #3110 ]
So, at lunch today a few of us from the #openframe channel decided to meet up and have lunch to discuss Pipelining and All That. Dim Sum rocks my world. Justin, if you're reading this, thanks for letting me take a long lunch!

Anyway, between eating lots of stuff many aspects of pipelining were discussed. What's all about then? What's the difference between a segment (aka a slot) and a pipeline? In our various models how are we going to link them together? How to we dispatch? What is the store and what can it hold?

muttley talked about Tanker. james talked about openframe and acme complained about getting old (happy birthday tomorrow acme) and blech demanded more pastries.

pdcawley talked about problems with multiple entry points. It turns out that in some applications you simply need to enter in at a different point in the pipeline, or you need some other form of mechanism that floats around with the session (which is at a higher level than the current store) and allows for dynamic dispatch. This is needed for situations where you have multiple communication channels (e.g. multiple web servers, different irc channels, etc.) I'm sure he'll write a better journal entry discussing this.

Of particular interest to me is the complexity of a store in a pipeline model (the store is the name for the "thing" that flows through your pipeline from segment to segment, carrying the data with it.) james neatly sidesteps the issue by having an abstract store. I, being an awkward so and so actually want to know how this is going to work ;-)

The essential problem as I see it is that you should never throw data away. I always want to be able to access my data later on down the pipeline. This idea is simple, every time you do something you add more data, not take data away. In the standard ->get() and ->set() store this means setting a new key for each item in the pipeline. Once your nice segment has done something to "foo" then it shoves it in "bar". OpenFrame apparently currently does this kind of action by having one of each object type in the store, so that when you move from an Apache::Request to your broken up request (having extracted the data you want) you have it under another key.

The essential problem is this; When I insert a segment into an existing pipeline I want to do two things:

  • Preserve the existing data so that should something later on down the pipeline need it, it's accessible
  • Transparently alter data as it passes through so that later segments can use this segment's data being naive of it's presence

Of course, you can see that these ideas are mutually exclusive. In order to transparently replace data (so that you can later act on it and are none the wiser that a segment in the middle has altered it) you have to do just that - and then you can't access the original data anymore, because you've replaced it! Arrrgh.

My proposed solution to this was that whenever ->set() is called the module stores it under a key and the name of the segment that altered it. This way you can either request the generic version with just the key (in which case you'll get the latest version back) or you can request with the data stored under a key the named segment left. Of course, this isn't a perfect solution (if you do this too much you might want to insert something transparently in the pipeline between those elements, and now you need a third key to disambiguate the data, and so on and so on.) But it's a relatively simple solution that might be good enough.

Of course, if someone can come up with a better solution, let me know...

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.
  • Is how do Pipeline/Tanker's differ from SAX pipelines/machines. I mean apart fromt the fact that James doesn't grok SAX pipelines :-)
    • I'm only really interacting at the annoying question level atm (I am the thorn in other people's side's.)..I'm not that near the code, so you might want to ask muttley or james, but...

      Tanker (from what I understand) is a wrapper framework that wraps around pipelines to add concurency to them and other framework support issues. It's pretty much pipeline agnostic, so you (in theory, it's all 'in theory' code) wrap it round any pipeline - even existing SAX ones. is just a simple pipeline concep

      • Kip Hampton's articles on are the best place to start. Just try and read them with an open mind, thinking all the time: "This stuff has applications outside of XML". Honest guv.

        Unfortunately it's hard to think about SAX pipelines while taking them out of the context of XML, but really they are highly generic, and a very interesting solution space for a wide class of problems.

        It's certainly fun to see people starting to really rave about Pipelining solutions, when this really was one of the major b
        • Of course it was the basis for axkit all those years ago. I still remember the talk you gave at Don't think people won't try and steal ideas from you too...

          I'm sure you'll agree that it's probably not a good idea to shoehorn Axkit into being an IRC bot for example though ;-)