Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

iburrell (4155)

iburrell
  ianburrellNO@SPAMgmail.com
http://znark.com/
AOL IM: imburrell (Add Buddy, Send Message)
Jabber: ianburrell@gmail.com

Journal of iburrell (4155)

Saturday June 02, 2007
06:57 PM

Reverse proxy mirror

[ #33410 ]
Does anybody know of software that could be used to build a mirroring reverse proxy? I am looking for something to cache and mirror files for distributions and other archives of mostly static files.

To clients, it looks like a copy of the archive. If a file does not exist, or is too old, it is fetched from the master servers and saved to disk. The files can be populated from CDs or mirrored through rsync to get complete copies. Ideally, it would load balance from multiple mirror servers.

It is possible to set something up with Apache2, mod_proxy, and mod_disk_cache but that does not allow load balancing between servers and stores files in an opaque cache.

Would Perlbal be usable for this? Could it be extended? I think a mod_perl module could be written.


List all Journal entries
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Reverse proxy mirror 4 Comments More | Login | Reply /

 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • Just use Apache with Squid (or another balanceable proxy). mod_proxy's ProxyRemote directive lets you pick another proxy to use, and Squid can be load balanced quite well.
    • Can Squid store the files on disk in a structure that mirrors the master servers? Part of what I am looking for is a way to preload the cache with files from the CD or DVD. I also want to be able to sync parts of the mirror with rsync or wget.
      • Why would you worry about the storage structure? As long as everything appears to be the same through http, does the physical layout on disk matter?

        No, Squid won't use a similar structure, but you can preload anyway, by just requesting the relevant URIs.

        If you're not loading from a normal internet site, though, you may have to set up a tiny http server for this purpose yourself. Or perhaps another proxy :)

        • Not really. Squid’s directory structure doesn’t look like what it has cached, but it’s pretty easy to hack some Perl to bulk-feed a directory structure plus given base URI into the Squid cache dirs.

Stories, comments, journals, and other submissions on use Perl; are Copyright 1998-2006, their respective owners.