From: Curt Sampson <cjs@...>
Date: 2002-08-06T11:18:56+09:00
Subject: Re: Unicode in Ruby now?

On Tue, 6 Aug 2002, Clifford Heath wrote:

> Curt Sampson wrote:
> > NOTHING IS BEING DROPPED! ALL CHARACTERS ARE REPRESENTABLE IN UTF-16! HELLO!
>
> Hey, no need to shout. Why is there a Mojikyo standard if no-one's being
> shut out?

Because certain very specialized applications, mainly to do with
academic research into Kanji, need to represent more kanji than Unicode
cares to represent.

If you need it, you need it. If you don't, you don't want to get
near it because it's extremely unwiedly and expensive to use. Also,
note that it's only useful for kanji, not for other writing systems.

> I can only assume that they want to represent characters that
> are excluded from Unicode but could have been easily included via UCS-4
> and UTF-8.

One doesn't "include" things in Unicode via UCS-4 and UTF-8".
Those are merely encoding schemes for Unicode; different ways of
representing it. All Unicode characters can be represented in UTF-8,
UTF-16 and UTF-32.

You can encode other character sets in these encoding as well, but then
(tautology here) you're not encoding Unicode. From the spec, section 3.8:

    The definition of UTF-8 in Amendment 2 to ISO/IEC 10646 also
    allows for the use of five- and six-byte sequences to encode
    characters that are outside the range of the Unicode character
    set; those five- and six-=byte sequences are illegal for the
    use of UTF-8 as a transformation of Unicode characters.

> > Aside from a few kanji specialists,...
>
> Those few also need to use computers - linguistic study is a valid use for
> computers - and there's no need to adopt a standard that forces them to
> use an encoding other than that used by the rest of the world.

Yes there is. They have completely different requirements that are
basically impossible to met with something also intended for general
purpose use by the rest of the world. (One of these is the ability
to change the standard very quickly.) And they're not interested
in supporting many of the things required by other languages
(right-to-left text, combining characters, etc.)

Would you make the entire world stop using regular filesystems and
start using Lotus Notes just because a few people need to have
full-text search and additional indexed fields in some applications?

Anyway, it appears to me that you do not have a very good understanding
of Unicode, so I am going to drop this argument until you go read
the specification carefully and can point out which parts of it
you want to argue with. Nothing I've been saying here is my own
original idea; it's all in the Unicode specification itself.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC