From: John W Higgins <wishdev@...>
Date: 2010-05-01T03:30:29+09:00
Subject: Re: Inverse of stream parser

--005045015b64633bcc0485786f83
Content-Type: text/plain; charset=ISO-8859-1

Morning,

On Fri, Apr 30, 2010 at 9:26 AM, Brian Candler <b.candler@pobox.com> wrote:

> I plan to parse a huge XML document (too big to fit into RAM) using a
> stream parser. I can divide the stream into logical chunks which can be
> processed individually. If a particular chunk fails, I want to append it
> to an output XML file, which will contain all the failed chunks, and can
> be patched up and retried.
>

If you aren't completely against Perl - XML-Twig [1] has a tool called
xml_split [2] which works rather well at splitting xml files. You might wish
to split up your files into smaller files prior to even beginning the
processing and then if a file fails to process you just have the file in
hand. When finished you could smash the failed files back together using
xml_merge [3] from the same perl package.

If there is some ruby variant of this I couldn't locate it but that never
means much :)

John

[1] - http://search.cpan.org/~mirod/XML-Twig-3.34/
[2] - http://search.cpan.org/~mirod/XML-Twig-3.34/tools/xml_split/xml_split
[3] - http://search.cpan.org/~mirod/XML-Twig-3.34/tools/xml_merge/xml_merge

--005045015b64633bcc0485786f83--