From: James Edward Gray II <james@...>
Date: 2007-04-05T10:34:47+09:00
Subject: Re: fast XML parser, other than libxml

On Apr 4, 2007, at 7:53 PM, Keith Fahlgren wrote:

> On 4/4/07, James Edward Gray II <james@grayproductions.net> wrote:
>> The series is an interesting read.  Tim's pretty focused on the
>> character based parsing and in my experience that's always death in
>> Ruby.  It's the primary reason the standard CSV library is so slow,
>> for example.
>
> Is the inverse the reason that FasterCSV is so fast (because it uses
> regular expressions)?

That is one of the two key reasons, yes:

1.  This first one is summarized by this comment from Aristotle  
Pagaltzis in Tim's RX article series, "The fastest way to do  
something in Perl is frequently the one that implements the most  
costly step in the fewest ops.  You can substitute Ruby, Python or  
the like for Perl; the basic statement holds in any case. For string  
processing, it generally means doing as much work as possible with  
pattern matching. The more time you spend inside the VM�s  
implementation of its opcodes rather than inside the opcode loader/ 
dispatcher, the faster the code will go."  For a comparison, have a  
peak at CSV::parse_body().  It's CSV's primary parser and it has a  
lot of steps.

2.  Method calls are expensive in Ruby.  You can see that CSV is  
calling things all over the place.  For example, if you call  
CSV::parse() the primary call chain is something like:

CSV::parse()
CSV::Reader::create()
CSV::IOReader::new()  # or StringReader
CSV::Reader#each()
CSV::IOReader#get_row()
CSV::parse_row()
CSV::parse_body()

The same call chain for FasterCSV is:

FasterCSV::parse()
FasterCSV::new()
FasterCSV::each()
FasterCSV::shift()

The object construction doesn't much matter, because it's one-time  
cost stuff.  But look at each() down in both examples.  CSV is  
iterating over a three method call chain.  FasterCSV is just  
iterating over one.  That adds up.

There are many other little tricks to speed up FasterCSV.  But those  
two easily bring us 90% of the distance.

Just to be clear, I'm not trying to attack the standard CSV library.   
It's pretty proven and has more users than FasterCSV does.  ;)  All  
of those calls make its interface more flexible and some prefer its  
design.

I'm just trying to share what I learned in my process of speeding it up.

James Edward Gray II