very bad character performance on ruby1.9

I'd like to bring up the issue of how characters are represented in
ruby 1.9from a performance standpoint.  In a recent ruby-quiz (parsing
JSON), the
fastest pure-ruby solution was simply an LL(1) parser that looked at one
character at a time (it beat various Regexp solutions).  With ruby 1.9, the
runtime increased by 4X making it a slow solution.  A simple benchmark is at
the end of this message that counts words in an LL(1) fashion.  With ruby
1.8.6, it can could the words in Homer's Iliad in 1.46s on my machine and in
ruby 1.9 (from ubuntu gutsy) it takes 52.87s (36X increase in runtime).

I'm writing a ruby DSL parser/lexer generator (could also replace Regexp
functionality).  This performance issue in ruby 1.9 is a serious problem.

The problem of course is that every character in ruby 1.9 becomes a normal
ruby object (String) in ruby 1.9, whereas in ruby 1.8 they where immediates
(Fixnums).

I'd like to propose that at least ASCII characters in ruby 1.9 be made into
immediates:

* at a minimum, characters should be read-only/frozen.  Allowing them to be
mutable will inhibit many future optimizations.
* give (small) characters a separate class with string-like (read-only)
functionality.
* possibly add a base class that String and this new character class would
be a descendent of.
* eventually make this small (i.e. ASCII or even unicode) character class
have immediate objects

If the above was done, one of these immediate characters would be to a
Fixnum as a frozen String would be to Bignum.  A possible base class of
these would be in line with the Integer class.

Please consider this significant performance issue in ruby 1.9.

Eric


#!/usr/bin/env ruby

require 'benchmark'
require 'stringio'

def io_getc(io)
    io.rewind
    io0 = io.getc
    words = 0
    strings = 0
    spacing = 0
    punctuation = 0
    while (true)
         case io0
         when ?a..?z, ?A..?Z, ?_
             words += 1
             io0 = io.getc
             io0 = io.getc while (case io0;when
?a..?z,?A..?Z,?_,?0..?9;1;end)
         when ?\s,?\t,?\n,?\r
             spacing += 1
             io0 = io.getc
             io0 = io.getc while (case io0;when ?\s,?\t,?\n,?\r;1;end)
         when nil
             break
         else
             punctuation += 1
             io0 = io.getc
         end
    end
    return words, strings, spacing, punctuation
end

file_name = "Homer - Iliad.txt"
system("wget  http://www.e-text.org/text/Homer%20-%20Iliad.txt") unless
File.exist?(file_name)
text = IO.read(file_name)

io = StringIO.new(text)
#io = File.open(file_name)

Benchmark.bmbm { |b|
    b.report("IO#getc") { p io_getc(io) }
}