From: slumos@...
Date: 2002-06-21T10:08:47+09:00
Subject: Re: REXML in C

Tobias Reif <tobiasreif@pinkjuice.com> writes:
> slumos@unlv.edu wrote:
> 
> 
> > It seems like the claim that I should have to give up the nice
> > interface just because the problem gets large is fundamentally flawed
> > somehow.  The acceptance of such rules of thumb is surely one of the
> > reasons why the XML world sucks so much.
> 
> 
> Err; if you want to be able to randomly manipulate a tree, it will have 
> to be stored in memory. If that's getting slow, then get a faster 
> machine or more memory, or go with streaming or pull. I can't see how 
> this requirement to balance tradeoffs is in any way specific to XML.

What seems specific to XML is that any time you mention you have a
large problem you get an immediate "oh, you should use a stream parser
and go away".  Doesn't that seem a little too HIN to anyone else? As
in I could have 10GB machines with XML parsers in silicon and somebody
would still say that? (Hey, for all anyone knows, I do!) 

I tend to think that it's what I have to *do* with the data that
decides which API I want to use.

> > The files are only 13MB or so, which isn't even large by new PC
> > standards.
> 
> 
> Then your hardware sure is capable of handling it. If not, find a good 
> compromise between hardware, flexibility, convenience, and speed. Again; 
> this is to be faced with every other notational system.
> 
> Tobi

Let me see if I can make my position clear:

 1. REXML is good.
 2. REXML is not "fast enough"[1] for really big files.
 3. Therefore I wasn't able to use REXML recently (as in I already
    used something else instead) and that sucks, because 
 4. I want to use REXML or something like it for XML, period (see 1).
 5. Therefore I need a faster REXML.
 6. I already know that libxml2 is "fast enough"[1] because I ended up
    using Perl/libxml2 in the end.  (This also sucks via corollary 1: 
    Ruby is good.)
 7. For my purposes, a fast parser isn't enough, I need fast XPath
    (that's why I couldn't just use REXMLBuilder).

Maybe whatever I said didn't come across the way I meant it to.  My
impression (or imagination) was:

Radu: I'm thinking about rewriting parts of REXML in C to make it
      faster, anyone like that idea?

Me:   Hey, that's great because I wasn't able to use REXML recently for
      this project which requires these really big files.  And
      actually I've been thinking about backing REXML with libxml2 to
      make it faster too.  And by the way, I'm really interested in
      helping out.

Bryan: (meaning well)  You shouldn't be trying to do that anyway,
       doctrine says you should use a stream parser.

Me (to self): oh yes, *another* stream parser, then instead of adding
    rules to my system by adding a method and using introspection to
    automatically find it, as I was planning, I can add a rule by
    spreading the code out through a half dozen cases in a switch
    instead. :-)

Steve
---
[1] Since I'm mostly a research programmer, the ability to reimplement
major parts of a system quickly is much more important than whether
the result takes 5 hours instead of 1 hour to run.  As long as it
doesn't take 1 day instead of 1 hour.