From: Thomas Gagne Date: 2010-05-06T02:40:18+09:00 Subject: Re: Inverse of stream parser On 4/30/2010 12:26 PM, Brian Candler wrote: > I plan to parse a huge XML document (too big to fit into RAM) using a > stream parser. I can divide the stream into logical chunks which can be > processed individually. If a particular chunk fails, I want to append it > to an output XML file, which will contain all the failed chunks, and can > be patched up and retried. > > To do this, I want to be able to regenerate the XML of the failed chunk, > preferably identical to how it was seen. > > > Depending on how complicated the XML is, you may be able to use a combination of self-parsing and XML libraries. I've needed to handle arbitrarily large XML "streams" before in C, Smalltalk, and Python. The "outer" XML was really a wrapper around (or to connect) a bunch of XML fragments that were not large. We parsed the "outer" XML until we located the fragment we were interested in, then parsed it as though it were a complete XML document of its own. This way we were able to handle XML files of infinite size by biting off individual chunks. In our case, the XML was coming across a network and there was no knowing how big it would be. We also had the advantage that if the /whole/ XML document was not well-formed (maybe a network error interrupted it) we didn't lose the fragments. -- Visit for more great reading.