From: Tim Bray <Tim.Bray@...>
Date: 2008-09-10T12:46:45+09:00
Subject: [ruby-core:18530] Re: Ruby 1.9 strings & character encoding

On Sep 8, 2008, at 9:06 PM, Urabe Shyouhei wrote:

> Yui's point was that your unicode definition of "character" might  
> differ
> from what programmers want to have.  Unicode's concept of codepoints  
> are
> not universal among those other encodings such as Shift_JIS.

Correct.

>  Ruby did
> not take unicode-centric architecture so something specific to unicode
> might not always be adopted.  What you should show was how iterations
> over codepoints are useful (among other encodings); hence his question
> "When you use each_code?". Manfred's use case is one of those.

Yes, there are lots of others.  For example, a full-text indexing  
system dealing with a word like Qu�bec, which needs to index it the  
same whether the � appears as one codepoint or two.

Actually, for many programmers working in Unicode, what they need  
isn't String#each_codepoint but IO#each_codepoint, because with  
variable-length encodings it would be very nice if the library took  
care of the necessary buffer juggling.

  -T