From: Caleb Clausen <vikkous@...>
Date: 2010-05-01T02:57:50+09:00
Subject: Re: Inverse of stream parser

On 4/30/10, Brian Candler <b.candler@pobox.com> wrote:
> I plan to parse a huge XML document (too big to fit into RAM) using a
> stream parser. I can divide the stream into logical chunks which can be
> processed individually. If a particular chunk fails, I want to append it
> to an output XML file, which will contain all the failed chunks, and can
> be patched up and retried.
>
> To do this, I want to be able to regenerate the XML of the failed chunk,
> preferably identical to how it was seen.
>
> The options I can think of are:
>
> 1. A stream parser which gives me the raw XML alongside each parsed
> item; I can concatenate the raw XML into a string.
>
> 2. A stream parser which gives me the byte pos of the current node, so I
> can seek back within the file to fetch the XML again
>
> 3. A stream parser which gives me events to identify the different parts
> of XML, together with an inverse process to which I can replay the
> events and get the XML back again.
>
> Playing with REXML StreamListener, I can get a series of method calls
> like start_tag(...) and end_tag(...), and I can collect these into an
> array; is there existing code which would let me squirt that array and
> recreate the XML? Any other options I should be looking at?

From my experience, REXML is far too wimpy to deal with data on this
scale. (Among other things, it was too slow.) I suggest using the
'stream parser' (a misnomer, this is really a lexer) in libxml
instead. I don't know for sure if it can reconstruct the original text
the way you want, but that should be possible.

I think the class you'd want is LibXML::XML::SaxParser. See
http://libxml.rubyforge.org/.