From: Robert Klemme <shortcutter@...>
Date: 2007-07-23T23:19:43+09:00
Subject: Re: Processing a huge xml file

2007/7/23, Tim Perrett <freestyle_kayaker@hotmail.com>:
> Hey guys
>
> I was wondering what advice anyone could possibly hand me about
> processing a huge XML (in fact its an XSD file)
>
> Overall, its about 20,000 lines of XML to load. Even on my macbook pro
> with 2GB of RAM, libxml-ruby eats it up extremely quickly (and about 2.8
> GB of virtual memory). This is obviously unacceptable, but I am not sure
> that a work around exists?
>
> I wanted to load in the schema in order to validate the messages and xml
> I was generating. Has anyone any ideas on a potential work around?

The generic answer would be, use a XML stream parser (as opposed to a
DOM parser). Even if you directly fill up a model that contains the
whole document it's likely less resource intensive than a DOM.  Of
course it's optimal (resource wise) if you can do your validation on
the fly (i.e. while stream parsing).

Kind regards

robert