From: JamesBritt <james@...>
Date: 2002-11-09T05:00:55+09:00
Subject: Re: CPAN Style installer

> From: tom [mailto:tom@u2i.com]
> Sent: Friday, November 08, 2002 11:31 AM
> To: ruby-talk ML
> Subject: Re: CPAN Style installer
>
>
> On Sat, 9 Nov 2002,  JamesBritt wrote:
>
> > I wrote a knock-off script using REXML to slurp in the RAA XML and
> then download
> > every file listed.  Yes, it was slow, but parsing the entire file
> did complete
> > in a few minutes.  This on an AMD K6 400 MHz with (I think) 256K RAM.
>
> Fair enough, but I would like as close to instant as I can get - a few
> seconds at most.

BTW, the entire RAA XML file is around 14K lines.  Maybe I missed it in your
original post, but why do you want to read the entire thing, and how often would
you want to do it?

You can pull back specific RAA info using the RAA XML-RPC interface.  (And I
have some almost-ready-for-release code that converts the returned data
structure into RSS/RDF.)

>
> > BTW, many of the RAA entries do not have an actual download URL, but rather
> > point to another HTML page where some user intervention is needed.  This was
> > almost always the case for anything hosted on Sourceforge.
>
> I filter out all files without a tar.gz or tgz extension so this problem
> is somewhat minimized.

Some packages are simple *.rb files, though.  Rimport, for example.

>
> > And, unsurprisingly, many URLs just didn't work.  But I did grab 551 out of
> > roughly 700 files, which came to 25 MB when tarred and zipped. The uber-sumo
> > package.
>
> As I was messing around with the xml file, I ran a script to find those
> links which didn't work - I have a list of those if anyone is interested.

I neglected to track such stuff, though I may rewrite the code.  I needed some
quick info for somebody who is considering  hosting a mirror of all RAA
packages.  They wanted to know how many apps there were, and just how much disk
space might be needed.

James

>
> -Tom
>
>
>