From: Chilkat Software <support@...>
Date: 2006-11-08T06:33:55+09:00
Subject: Re: REXML


I tested the Chilkat XML parser (an in-memory DOM) on a 21MB XML file
that looks like this:

<phonebook>
<address><company>yuy25uiFfaku</company><street>A7ZbA3jP48rp</street><city>fSgWAhn3i3lD</city><state>p3rfNqf6kzUq</state><postal_code>lqVZ0b4daYWQ</postal_code><country>VjfXvb0AdxSt</country><extra>TEST</extra></address>
<address><company>Ki78Ypx8FlbZ</company><street>340PK6u2DsZQ</street><city>EqbFawBo0mCi</city><state>fTZK5YT0Tur8</state><postal_code>EXP29c5Hi2Hj</postal_code><country>sfGB4EzWR3Ft</country><extra>TEST</extra></address>
...
</phonebook>

(the data is random garbage...)

The XML is parsed in 11.5 seconds on a 18.Ghz Pentium 4.  Peak memory 
usage is 180MB.
I don't think the parser would break a sweat on the 6MB file...  ;-)

I uploaded the XML test data to: 
http://www.example-code.com/downloads/bigXml.zip
The code for parsing the iTunes XML is easy: 
http://www.example-code.com/ruby/ruby-parse-itunes-xml.asp

-Matt




At 02:45 PM 11/7/2006, you wrote:

>Hi David (or others)
>
>I am still not sure I get it, could you explain a little more?
>
>Thanks,
>Paul.
>
>David Vallner wrote:
> > pdg wrote:
> > > Assuming I go with the Ruby pull parser, how do I use this in my code.
> > > I see from the link the code sample, but I have no idea how to throw
> > > that into my code and make it work. Any suggestions.
> > >
> >
> > Generally, you should have some layer between XML input, and processing
> > the records themselves. E.g. a trivial Song class, or at least a hash.
> > Personally, I'd make a XMLSongList class that's enumerable (implements
> > #each), and rework the REXML code that works for small files into one
> > that yields a Song object for each of the records in succession by
> > querying the tree accordingly.
> >
> > That shouldn't then be too hard to rework so that while #each is
> > running, it opens a pull parser, and for each yield, builds up a Song
> > object going through the record in the order how the elements appear in
> > the XML file, instead of a random one. Once you isolate the code that
> > manipulates the XML to the smallest significant unit (a song record in
> > this case, I presume), it shouldn't be conceptually that difficult to
> > rework from a tree parser to a pull parser. The code probably will get a
> > little messier and verbose, but the main shift of thinking is in not
> > asking the XML for what your object needs, but feeding an object what
> > the XML has.
> >
> > > PS: idiot (slaps head). Yes it was 5-6meg not gig!
> > >
> >
> > 6MB is still Huge (tm) for a XML file.
> >
> >
> > --------------enigF9D6B5236ACE2603700BC85A
> > Content-Type: application/pgp-signature
> > Content-Disposition: inline;
> >       filename="signature.asc"
> > Content-Description: OpenPGP digital signature
> > X-Google-AttachSize: 188
>
>
>
>
>
>--
>No virus found in this incoming message.
>Checked by AVG Free Edition.
>Version: 7.1.409 / Virus Database: 268.13.31/522 - Release Date: 11/7/2006


-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.13.31/522 - Release Date: 11/7/2006