My BackPAN indexer (YAPC::EU 2008 slides) made it's first complete pass through my MiniCPAN yesterday:
The total size of BackPAN is about 100,000 distributions, so I think this means that I could index all of BackPAN in less than a week.
Right now I output everything as YAML, one YAML file per distribution. The data organization is sloppy and sometimes redundant because I haven't paid attention to it. You can get the tarball of all 16,000 files. Take a look to see if there might be anything else you'd want the indexer to record about a distribution. If you're interested in making some sort of CPAN service, let me do the work of cataloging the information you need.
If you want to play with this, get MyCPAN::Indexer from CPAN Search, or if you want to play with everything, checkout the sources from Github. You probably can't install in from CPAN since it depends on a couple of modules which only have developer releases right now.
The thing you'd want to play with is examples/backpan_indexer.pl. It's a little messy right now because I bolted on a Tk interface (see video one or two) that lives in examples/tk.pl and a dispatcher that lives in examples/steak.pl. My next step is to make those pluggable modules so you can note in the configuration file which interface and dispatcher you want, and as long as they have the right interface, they'll do whatever they do.
After a little bit more work on the indexing stuff, the next step is to take all of those YAML files and distill them into something that is easier to search, then hook up some sort of search interface to them. I'll probably first write a command-line tool (although with wonderful MVCness). I want to feed the index any file in @INC and get a report:
$ cpan_index `perldoc -l Foo`
Foo.pm's fingerprint found in Foo-Bar-0.05.tgz
Author: Joe Snuffy (SNUFFY@cpan.org)
Release date: Nov 11, 1998, 23:59:59
Latest version on CPAN: Foo-Bar-0.06.tgz
Joe Snuffy (SNUFFY@cpan.org) (first come)
Joe Cool (CAMEL@cpan.org) (co-maintainer)
Also came with:
!!!Bar.pm, installed version 0.08 (does not match Bar.pm from Foo-Bar-0.05.tgz)
ABC.pm, installed version 0.05 (matches ABC.pm in Foo-Bar-0.05.tgz)
Baz.pm from Baz-0.67.tgz
Quux.pm from Quux-0.01.tgz
CPAN Testers Matrix:
0.01 Dec 31, 1969, 23:59:59 SNUFFY (BackPAN)
0.02 Jan 31, 1995, 23:59:59 SNUFFY (BackPAN)
0.03 Jun 6, 1996, 23:59:59 SNUFFY (BackPAN)
0.04 Oct 31, 1997, 23:59:59 SNUFFY (BackPAN)
****0.05 Nov 11, 1998, 23:59:59 SNUFFY (CPAN)
0.06 Sep 5, 2008, 23:59:59 CAMEL (CPAN)