From: Curt Sampson <cjs@...>
Date: 2002-08-01T21:50:36+09:00
Subject: Re: Unicode in Ruby now?

On Thu, 1 Aug 2002, Tobias Peters wrote:

> > Not just political reasons, but practical reasons. Unicode is designed
> > to work if you restrict yourself to using only 16-bit chars, and I
> > expect most programs are going to limit themselves to that.
>
> I thought this was a design error, and Unicode has now roughly 32 bits for
> code points....

No, and no. As specified in section 2.2 of the standard (I quote
from 3.0, but 3.1 and 3.2 do not appear to have updated these
sections):

    Plain Unicode text consists of sequences of 16-bit Unicode character
    codes.... From the full range of 65,536 code values, 63,486 are
    available to represent characters with single 16-bit code values,
    and 2,048 code values are available to represent an additional
    1,048,544 chaaracters through paired 16-bit code values.

So note that the character codes are still 16-bit. though sometimes
it takes two character codes to represent a character.

Section 2.9, "Conforming to the Unicode Standard" states as its
first precept that, "An implementation that conforms to the Unicode
Standard... treats characters as 16-bit units." It goes on to say
that the standard, "does not require that an application be capable
of interpreting and rendering all Unicode characters so as to be
conformant."

Section 5.4 really gets into the details of handling surrogate pairs.
There are essentially three levels at which you can do this: none, where
you treat each 16-bit code as an individual, unknown character and do
not guarantee integrity of the pairs (i.e., you don't guarantee that
you won't split them); weak where you interpret some pairs, but treat
the others as "none" treats them; and "strong" where you interpet some
pairs, and guarantee the integrity of them.

It continues with:

    As with text-element boundaries, the lowest-level string-handling
    routines (such as wcschr) do not necessarily need to be modified
    to prevent surrogates from being damaged. In practice, it is
    sufficient that only certain higher-level processes...be aware of
    surrogate pairs; the lowest-level routines can continue to function
    on sequences of 16-bit Unicode code values without having to treat
    surrogates specially.

So from all of this, it's pretty obvious that the expectation is that
only those systems that really need to work with surrogates are going to
be doing more than the bare minimum to support them, and those that do
support it will be doing it at something above the basic language level.

I should also point out that Java has no special support for surrogates
in the String class; String.length() returns the number of code values
in the string, not the number of characters. It's not a problem in practice.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC