From: Urabe Shyouhei <shyouhei@...>
Date: 2008-09-09T13:06:11+09:00
Subject: [ruby-core:18515] Re: Ruby 1.9 strings & character encoding

# First off, I'm neutral to this issue

Tim Bray wrote:
>> When you use each_code?
>> If you want to use it to iterate CHARACTERS, they may go wrong.
>
> In Unicode, each_character and each_codepoint would mean the same
> thing. For serious low-level text processing in Unicode, you really
> must have the codepoint-by-codepoint access.  By  "serious" I mean
> things like full-text-indexers, markup parsers, and typesetting
> software.   It might be useful to look at
> http://www.w3.org/TR/charmod/ - individual "characters" do not
> correspond to units of display, nor to units of sound, nor to units of
> input, nor to units of collation, nor to units of storage.  
> Developers simply have to live with these facts, and once again, in
> Unicode, the only way to stay sane is to process a codepoint at a
> time.  In my work on efficient and (unlike REXML) correct XML parsing,
> I had to do everything via String#unpack, very memory-inefficient,
> when all I wanted was next-Unicode-codepoint.

Yui's point was that your unicode definition of "character" might differ
from what programmers want to have.  Unicode's concept of codepoints are
not universal among those other encodings such as Shift_JIS.  Ruby did
not take unicode-centric architecture so something specific to unicode
might not always be adopted.  What you should show was how iterations
over codepoints are useful (among other encodings); hence his question
"When you use each_code?". Manfred's use case is one of those.

And if you wanted a codepoint next to a string, just try String#succ. 
It sometimes overkills, but normally works.