Re: very bad character performance on ruby1.9

Hi,

> I'd like to bring up the issue of how characters are represented in  
> ruby 1.9 from a performance standpoint.  In a recent ruby-quiz  
> (parsing JSON), the fastest pure-ruby solution was simply an LL(1)  
> parser that looked at one character at a time (it beat various  
> Regexp solutions).  With ruby 1.9, the runtime increased by 4X  
> making it a slow solution.  A simple benchmark is at the end of this  
> message that counts words in an LL(1) fashion.  With ruby 1.8.6, it  
> can could the words in Homer's Iliad in 1.46s on my machine and in  
> ruby 1.9 (from ubuntu gutsy) it takes 52.87s (36X increase in  
> runtime).

I'm surprised that the fastest parsing done in Ruby was with a  
handwritten parser. When I tried to code a small XML parser in Ruby  
the fastest solution was using StringScanner. And for your small  
example, I rewrote a version using StringScanner that's faster in both  
1.8 and 1.9 (and it's faster in 1.9). And it's shorter and (I think)  
more readable.

require 'strscan'

strscan = StringScanner.new(text)
punctuation = spacing = words = 0
while not strscan.eos?
   if strscan.skip(/[a-zA-Z_]+/)
     words += 1
   elsif strscan.skip(/\s+/)
     spacing += 1
   else
     strscan.skip(/./)
     punctuation += 1
   end
end

Most popular parsers written in Ruby (ERB, json-pure, RedCloth) use  
Regexps (some with and some without StringScanner).

> Please consider this significant performance issue in ruby 1.9.

I am not sure this particular case is really a significant issue.