Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Matts (1087)

Matts
  (email not shown publicly)
I work for MessageLabs [messagelabs.com] in Toronto, ON, Canada. I write spam filters, MTA software, high performance network software, string matching algorithms, and other cool stuff mostly in Perl and C.

Journal of Matts (1087)

Wednesday August 30, 2006
12:42 PM

Non-blocking IMAP

[ #30808 ]

In thinking about things to ship with AxKit2 I thought it would be cool to have a webmail that doesn't suck written in Perl. You know - drag and drop with AJAX, HTML editor, decent speed, etc.

Of course all the perl IMAP libraries are blocking - they don't drop in well to a non-blocking framework. So I set out to write a non-blocking library for IMAP.

Turns out it was easier than I hoped. Net::IMAP::Simple (which is probably "Simple" because it doesn't support much of IMAP, but I'll probably figure that out in a few weeks) is mostly really well written, and was fairly easy to extend. I had to re-write quite a few of the methods, though some of the changes can probably be incorporated into the base class as they make for a nice improvement to the design.

Result is I can now do this:

use Net::IMAP::Simple::NB;
 
Danga::Socket->AddTimer(0, sub {
    # Create the object
    my $imap = Net::IMAP::Simple::NB->new('felony:143') ||
       die "Unable to connect to IMAP: $Net::IMAP::Simple::errstr\n";
 
    print "IMAP: $imap\n";
 
    $imap->login('matt','xxxxxxx', sub {
        my $login_ok = shift;
        if ($login_ok) {
            print "Login OK\n";
            $imap->select("INBOX", sub {
                my $nm = shift;
                print "Got $nm Messages\n";
                my $i = 1;
                my $sub;
                $sub = sub {
                    my $headers = shift;
                    print grep { /^Subject:/ } @$headers;
                    $i++;
                    $imap->top($i, $sub) unless $i == $nm;
                };
                $imap->top($i, $sub);
            });
        }
    });
});
 
Danga::Socket->EventLoop;

Where the "sub" callbacks are called when the Net::IMAP::Simple methods return with their responses, and in the meantime the event loop is not blocked.

One of the problems with building IMAP based webmail is you don't want to be re-establishing connections to the IMAP server on every hit. AxKit2 can get around that problem by just keeping the connection open and hooking the request up to the right connection using a cookie/session-id.

Lots of work still to do, but now it's just a small matter of programming.


List all Journal entries
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Non-blocking IMAP 21 Comments More | Login | Reply /

 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • that use of sub { } closures to handle event-driven programming rocks so hard -- I used it a lot in SA 3.2 in the Mail::SpamAssassin::AsyncLoop class, too.

    I'd hate to have to go back to the old model of event-driven programming, with objects to hold all state between events, etc.
    • Closures are such an incredibly powerful programming paradigm, especially for this kind of app - it's hard to believe that some languages get by without them.

      What's really cool is that closures in perl just do what you expect them to do. The hard thing to do is explain closures and make them look like they might not be what's expected - because when you look at the syntax you just think "yep, that's normal".
  • Don't those kinds of closures leak (anonymous recursive subs, that is), unless maybe you use maybe Scalar::Util::weaken or one of those combinator [perl.org] -type thingies (or some Devel::* module I think or something else)? Or do you really even care in this instance? :-)
    • I don't think they leak - there were some bugs with eval blocks and anonymous subs, but I think all those leaks have been closed now.

      This design is used a lot by IO::AIO (which is used in some people's production apps) and thus if it leaked we would find out about it darn fast.
      • I thought it happened whenever you had the my $sub; $sub = sub { ...$sub... }; construct, and that whenever that code got called, the reference count for $sub would never go to zero. Is what you're doing any different than this (leaks in 5.8), or did this sort of thing recently get fixed?:

        my $foo = sub {
          my $sub;
          $sub = sub {
            $sub;
          };
        };

        my $i;
        print "Start: ";
        <STDIN>;
        while (++$i) {
          my $f = $foo->();
          unless ( $i % 100_000 ) {
            print "$i: ";

        • That does indeed seem to leak, but it's fixed by adding weaken($sub) inside the anonymous sub, so that's no big deal to me. Thanks for the heads up though.
          • I know weaken() was one of the options, but I'm trying to become a functional weenie, so I'm starting to like the combinator approach (this example ripped and tweaked from the Moose source):

            sub U {
                my $f = shift;
                sub { $f->($f, @_) };
            }

            sub Y {
                my $f = shift;
                U(sub { my $h = shift; sub { $f->(U($h)->())->(@_) } })->();
            }

            # Now we need a recursive anonymous function
            my $fact = Y(sub {
              my $f = shift;
              sub {
                my $n = shift;

            • Yeah, it's just a shame that code is so cryptic. No wonder perl gets a bad rap (though I guess it's copied from some lamda calculus stuff). You'd think they could use more than single character variables and function names for it all.

              • I agree that the Y() is cryptic (and you can't blame perl too much), but the U() is not that hard to understand, and if they're tucked away in a library (as they are in Moose::Autobox albeit in an OO form), then you don't have to look at them, and they're pretty simple to use. In other words, "don't look behind the curtain, just use the Y-combinator" :-)

                If you have freedom in what args your recursive sub accepts (which I assume you don't in the above instance), then you can let the first arg be the sub its

              • It’s the construct that’s cryptic, not the Perl code. It’s no easier to understand in Lisp, although the core concept is actually very simple. The basic idea is that you make an anonymous function recursive by making it take a reference to itself as a parameter, so that it can call that to recurse on itself.

                The actual implementation requires doing a series of code sit-ups that makes the fully built combinator very hard to follow. I think I’ll have to rewrite it for myself in Pe

                • I think decently named params would make it understandable.

                  • I would usually agree but in this case I don’t see how. Using $func or $closure in place of $f wouldn’t do anything to clarify any of what’s going on, and I can’t think of any good name of $h. The functions are no easier to name more sensibly. It would help the legibility of the Y combinator to break it down a bit and assign subexpressions to suitably named variables, but the difference wouldn’t be anything to write home about.

                    These are simply near-unreadable constructs.

                    • So here's where I'm coming from: The Y combinator is useless in the code I showed, because $sub isn't called explicitly - it's a callback. So having the Y combinator re-written to be more readable, with better variable names, and comments, might allow me to write a version specifically for my code that doesn't require the use of weaken().
                    • In practice these things just don’t need to be readable once you get one working;

                      That sums up my frustrations with higher order mathematics pretty effectively. I strongly suspect that the Haskell compiler rejects identifiers with more than one vowel or over eight characters, but I can't prove it, for example.

                    • I suppose. I can’t read the Y combinator either, y’know. I can derive it when I start from scratch, though, and for as long as that model is in my head, I can reason about it.

                      I just can’t build it inductively from reading the source/expression. I have to deduce it every time from scratch.

                      It may be a matter of practice.

            • So here's a version that doesn't leak, that uses this approach, and isn't so darn cryptic:

              # wrap an anonymous function into a function that takes $func as the first
              # parameter so we can make use of it from within the call.
              sub wrap_func {
                  my $func = shift;
                  return sub { $func->($func, @_) };
              }
               
              ...
              # sometime later...
               
              $imap->message_info($i, wrap_func(sub {
                  my $sub = shift;
                  my $flags = shift;
                  my $headers = shift;
                  p

              • Well yes, you get away without the Y combinator here. wrap_func is a rather nondescript name, though, don’t you agree? A better one might be something like curry_self or apply_self_maker or some such, although it’s probably not obvious to someone without basic knowledge of functional programming why these names are in fact better. All of them sound similarly meaningless, though the better ones might even be a little scary. And to someone with functional programming knowledge, U would be just a

                • Your "fix" doesn't work.

                  wrap_func() took a function, and returned a new function that took the given $func as the first argument. Therefore, in my callback $sub is not "wrapped", so needs to be re-wrapped.

                  • Oh d’uh. I got confused between properties of the Y and U combinators.

                    • I'm shocked. How could you confuse the two!

                    • Heh. I’d read about the Y combinator before but I only started studying it in anger yesterday because I realized I wouldn’t actually understand it on any deeper level if I didn’t sit down to derive it from first principles for myself.

                      My grasp is strengthening by the hour, but you can see I’m not quite there yet. :-)

Stories, comments, journals, and other submissions on use Perl; are Copyright 1998-2006, their respective owners.