From: James Moore <jamesthepiper@...>
Date: 2007-07-24T00:00:13+09:00
Subject: Re: Processing a huge xml file

On 7/23/07, Tim Perrett <freestyle_kayaker@hotmail.com> wrote:
> I was wondering what advice anyone could possibly hand me about
> processing a huge XML (in fact its an XSD file)

Something's going wrong.  20k lines is a pretty small XML file; we're
sucking in files that are larger than that (50meg or so - a little
less than a million lines long) many times a day using the Ruby libxml
bindings and not seeing a similar issue.  It's possible that your
average line length is _much_ longer than ours, of course.  Our normal
process size is about 400m, but a big chunk of that is the processing
we're doing on the data; I want to say that the size after loading in
the xml is in the 200m range, but I haven't looked at that for a
while.

Are you doing stream processing?  We never tried to load the whole
document at once, so there may be an issue doing that.

 - James Moore