From: Robert Klemme <shortcutter@...>
Date: 2009-12-15T01:55:51+09:00
Subject: Re: Ruby 1.9 string slicing and StringScanner pointers

2009/12/14 Brian Candler <b.candler@pobox.com>:
> Robert Klemme wrote:
>> I don't know StringScanner internals, but does this have to be so?  I
>> mean, with $' you get the remainder of the string so when not using
>> positions you could handle it that way at the expense of an additional
>> String instance for each match.
>
> Even if it did, I think the point remains that StringScanner#pos
> wouldn't be of much use if it gave a character offset, since str[n..m]
> is an expensive operation in ruby 1.9.
>
> If all you want is the rest of the string, then StringScanner#post_match
> gives you that already, doesn't it? But I think the OP wanted to get the
> buffer between two arbitrary match positions.

You're right.  Frankly, I did not read the stackoverflow question
initially but from that it is obvious.

Still this leaves an awkward feeling: you have a String which can
informally be defined as a sequence of _characters_.  Now, for most
application cases accessing the nth character seems to be a more
natural operation than accessing the nth byte.  I know the internal
reasons for the fact that accessing the nth character is expensive
(variable length encodings) but from an interface perspective this is
not good IMHO.

Java did solve this with a specialized character type so you can have
arrays of char, but from what I recall about Matz's comments the Java
model is flawed because it does not work well with non western
languages, namely Asian languages.

Btw, although UTF-16 is a fixed length encoding, char based accesses
are *really* slow:

#!ruby19

require 'benchmark'

REP = 100

s1 = "abcdeABCDE" * 1_000_000

encodings =["ASCII", "UTF-8", "UTF-16BE", "UTF-16LE"]
strings = {}

encodings.each do |enc|
  strings[enc] = s1.encode(enc)
end

idx = s1.length - 10

Benchmark.bmbm 30 do |b|
  encodings.each do |enc|
    str = strings[enc]
    rep = /16/ =~ enc ? REP : REP * 1000

    b.report "enc %-10s rep %11d" % [enc, rep] do
      rep.times do
        s = str[idx..-1]
      end
    end
  end
end


Cheers

robert


-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/