[ruby-core:18547] Re: Ruby 1.9 strings & character encoding

On Sep 10, 2008, at 12:55 AM, Tanaka Akira wrote:

> "=C3=A9" is a character, even if it is represented as two
> codepoints.
>
> So ruby should treat it as a character.

Unfortunately it's generally impossible to agree on a definition of =
=20
"character".  There is no notion of character that is compatible =
=20
across all the different systems for encoding text in all the world's=
 =20
languages; unfortunate but true.

Also, remember, there is no system of character encoding in which =
=20
"characters" correspond exactly to units of input or or storage or =
=20
display.  At least unicode is honest about this.

> I know current ruby doesn't do that.  But it is desirable.
>
> NFC (Normalization Form C) can be a solution for "=C3=A9".  But
> there are characters which don't have single codepoint (as
> some characters defined in JIS X 0213, for example).

Unfortunately NFC isn't a solution because it isn't widely respected,=
 =20
so a developer has to deal with nonstandard normalizations. :(

> I think codepoint is implementation details.  Although it
> may be useful for unicode experts, non-experts will be
> confused with the difference of characters and codepoints.
> I think it should not be provided by default.

I agree.  The programmer who wants to search a list for filenames lik=
e =20
%r{/([^/]*\.atom)$}, or the zoo catalog for "=E3=82=B5=E3=83=AB" shou=
ld never have =20
to think carefully about what a "character" is.  But someone who's =
=20
writing an XML/HTML processor or a typesetter or a full-text search =
=20
system has to think about these things all the time.  And when the =
=20
text is traveling over the internet, the probability is quite high =
=20
that it's in Unicode.

I don't want to weaken ruby's support for legacy encodings.  I just =
=20
want to make it straightforward and efficient to process Unicode.  An=
d =20
for that, you need straightforward and efficient access to =20
codepoints.  BTW, one reason why Unicode is popular on the Internet i=
s =20
that its notion of a character (a 21-bit number, with some associated=
 =20
metadata and a suggested graphic rendition) is not perfect, but it's =
=20
clean and clear and makes it possible for programmers to produce =
=20
decent results with multilingual texts.

  -Tim