[ruby-talk:7515] Re: Unicode Issues

Schulman:
>|In converting to UTF-16, it's the Western languages that
>|would suffer a "hit" in terms of storage and processing
>|time. UTF-8, accordingly, will probably remain common in
>|Western end users shops for some time to come but not, I
>|hope, as the internal encoding of system software.

Matz:
>Why not?  Although its variable length nature, I think UTF-8 is good
>for internal encoding too.  E.g.
>
>  * ASCII superset
>  * no NULL (\0) in string
>  * no endian problems

Processing overhead and developer pain is considerable in
trying to do multilingual client server applications in
which some layers are UTF-8, others UCS-2 or UTF-16, and
still others are legacy encodings. There need to be
continual computer-cycle-devouring conversions from one
format to the other as the data passes from layer to layer.
One often sees multiple character-encoding conversions at
the client, into and out of the network, at the server, and
still again into and out of the database!

Multiple clients, networks, databases, and servers in a
distributed environment add further to unnecessary
development complexity and processing overhead.

Databases work most efficiently with fixed character lengths
and predictable table and index sizes. Variable length
encodings of characters work against this.

Although UTF-16 is a variable length format, most users will
be using the approximately 49K two-byte characters in the
main Unicode plane and won't have a requirement for the
extensions. Accordingly, the situation will approximate the
predictable storage and processing requirements of UCS-2.

Thanking Matz for his informative posts on this subject --
Richard Schulman