From: James Edward Gray II <james@...>
Date: 2007-04-05T23:40:55+09:00
Subject: Re: fast XML parser, other than libxml

On Apr 4, 2007, at 7:45 PM, James Edward Gray II wrote:

> On Apr 4, 2007, at 5:05 PM, Arto Bendiken wrote:
>
>> On Apr 4, 12:00 pm, Peter Szinek <p...@rubyrailways.com> wrote:
>>
>>> Should I interpret this as 'decide between REXML and libxml'?
>>> There are really no other alternatives?
>>
>> You may find Tim Bray's recent in-depth experiments in attempting to
>> write a fast pure-Ruby XML parser instructive and informative:
>>
>> http://www.tbray.org/ongoing/When/200x/2006/11/09/Optimizing-Ruby
>> http://www.tbray.org/ongoing/When/200x/2006/11/15/RS-Redux
>
> And:
>
> http://www.tbray.org/ongoing/When/200x/2006/11/23/RX-plus-YARV
>
> The series is an interesting read.  Tim's pretty focused on the  
> character based parsing and in my experience that's always death in  
> Ruby.  It's the primary reason the standard CSV library is so slow,  
> for example.
>
> He says it's because Ruby's regex engine isn't really up to the  
> task of handling non-UTF-8 input.  I'm pretty sure I understand why  
> that is, but he also basically admits that at least the lexing  
> stage of XML reading is just looking for < and &.  I guess the  
> problem becomes that a UTF-16 document actually encodes that as two  
> bytes?  Well, surely the regex could be adapted to handle that.

Or, another thought, introduce an Iconv filter that normalizes the  
input to UTF-8.  This probably degrades the performance against non- 
UTF-8 documents, but Tim's code had trouble in that area too.

Is it legal for a well behaved XML processor to expose character data  
to an application in an encoding other than the actual document  
encoding?  I didn't see anything in the specification to suggest it  
wasn't.

James Edward Gray II