From: James Edward Gray II <james@...>
Date: 2007-04-05T09:47:35+09:00
Subject: Re: fast XML parser, other than libxml

On Apr 4, 2007, at 5:05 PM, Arto Bendiken wrote:

> On Apr 4, 12:00 pm, Peter Szinek <p...@rubyrailways.com> wrote:
>>
>> Should I interpret this as 'decide between REXML and libxml'?
>> There are really no other alternatives?
>
> You may find Tim Bray's recent in-depth experiments in attempting to
> write a fast pure-Ruby XML parser instructive and informative:
>
> http://www.tbray.org/ongoing/When/200x/2006/11/09/Optimizing-Ruby
> http://www.tbray.org/ongoing/When/200x/2006/11/15/RS-Redux

And:

http://www.tbray.org/ongoing/When/200x/2006/11/23/RX-plus-YARV

The series is an interesting read.  Tim's pretty focused on the  
character based parsing and in my experience that's always death in  
Ruby.  It's the primary reason the standard CSV library is so slow,  
for example.

He says it's because Ruby's regex engine isn't really up to the task  
of handling non-UTF-8 input.  I'm pretty sure I understand why that  
is, but he also basically admits that at least the lexing stage of  
XML reading is just looking for < and &.  I guess the problem becomes  
that a UTF-16 document actually encodes that as two bytes?  Well,  
surely the regex could be adapted to handle that.  In fact, the key  
expressions could be swapped out for encoding-aware replacements.   
Then we can keep playing to Ruby's strengths, I hope.  Or is it true  
that there are some encodings we can't effectively build expressions  
for?

Sorry for thinking out loud here.  I'm just trying to better  
understand Tim's logic.  It's interesting stuff.

I'll go try to read his code now and see what else I can learn...

James Edward Gray II