Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

kappa (294)

kappa
  (email not shown publicly)
http://kapranoff.ru/
Perl Hacker since 1998. Lives in Moscow, Russia.

Journal of kappa (294)

Thursday March 11, 2010
05:45 PM

Job queues

[ #40240 ]
I need a persistent reliable distributed job queue with a good Perl interface.

TheSchwartz would be great but it looks dead. Brad Fitzpatrick is in the valhalla of Google, last release was in 2008 and cpantesters report more failures than successes.

gearman is not reliable. It won't retry jobs on another worker nor workers can return failed jobs.

Resque is too ruby-specific.

beanstalkd looks almost fine, but it does not distinguish between job types. You cannot easily register one worker for sending emails and another worker for converting images.

Should I start looking into various Java-based monstrosities like ActiveMQ?


List all Journal entries
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Job queues 8 Comments More | Login | Reply /

 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.

  • TheSchwartz isn't dead, it's just so stable. We use heavily on our production sites such as typepad.com every day. We own the maintenance bit on that module so we can ship if there are critical bugs etc. which we haven't heard any yet.

    Gearman could be made reliable if you use Brian's C Gearman server, while you can still use the Gearman perl binding http://gearman.org/ [gearman.org]

    • TheSchwartz: CPAN Testers PASS (2) FAIL (136)

      Patch was available in 2008: https://rt.cpan.org/Public/Bug/Display.html?id=38570 [cpan.org]

      • Yes, that's a shame and I will try to upload the new version soon.

        But as you can see from the patches, that's not a code fix, just a bad test. Not something considered as dead / unused.

      • It was fixed on the SVN repo more than an year ago but no CPAN release has been made, which was now done a few minutes ago.

    • Thanks for information!

      C Gearman server does not retry jobs if a worker returns failure.

      But it looks like we can trick it to retry by dropping connection from worker.

      • > C Gearman server does not retry jobs if a worker returns failure.

        Because returning a failure is not an error :)

  • I have no experience but I've considered it a few times. If I remember correctly it speaks HTTP/JSON like CouchDB so accessing via perl (or anything) is easy.

    --
    benh~
  • If you need separate workers, do look at "tubes" in Beanstalk. I have several workers each watching a separate tube. For example email jobs go into the email tube and image conversion jobs into the imagecon tube. Another solution is to have a general purpose worker started and pass the script that you need to run as a serialized data structure. The worker can then fork and run whatever script is passed to it - email sender or an image converter.
Stories, comments, journals, and other submissions on use Perl; are Copyright 1998-2006, their respective owners.