From: Martin Elzen <martinelzen@...>
Date: 2004-05-04T03:27:56+09:00
Subject: Re: Strange behaviour of Strings in Range

Hi everyone.

> > |So then what's Unicode for in the first place?  I thought the aim was 
>to
> > |have a universal encoding for all chars.  Did I miss something?
>
> > It's _their_ intention.  Whether it succeeds or not is another story.
> > I think they tried their best, but it is virtually impossible to
> > satisfy all requirement for internationalization.
>
>Why is that?  Is there not enough room for every character known to
>man, or is there some other problem?

I've been reading up on Unicode recently, and it strikes me as a standard 
that is:
a) awesome, if you only consider what it could help you do
b) a *MAJOR* headache for those who need to implement it

Each Unicode 'character' can be represented in a 21 bit integer, so 
representing them with a 32 bit wide integer would be no problem (that is, 
if you're looking for the *easiest* way to store them in memory, and don't 
care too much about how much memory it would cost).

The hard part is everything that comes after that.  Only a relatively few 
ranges of Unicode characters can be capitalized, for example.  The ASCII 
subset is easy to (un)capitalise, just add (or subtract) a constant and 
there you go.  But for other Unicode ranges the capitalised and 
non-capitalised forms of each character are right next to each other.  
Although in my understanding the Japanese Hiragana 'characters' don't 
actually have a non-capitalised form, they do have a  large and a small form 
in Unicode.  And those small and normal size forms are next to each other.  
Not to mention the issue of normal width and half width Asian 
'characters'...  Coming up with a generalised to_upper_case / to_lower_case 
function is a problem.

A bigger problem is how Unicode lets you compose characters.  I'm not saying 
such a character exists, but what if there was a character consisting of a 
lower case a with an umlaut and a '^' character.  Unicode would let you 
define that character by saying it's a lowercase a with_a umlaut with_a '^'. 
  But it would also let you compose it by saying it's a lowercase a with_a 
'^' with_a umlaut.  And here's where it gets fun: if you have 2 strings 
containing the same character, no matter which order you used to compose the 
character, it has to compare as equal if the result of all the composing 
elements is the same.  And, mind you, this is a relatively simple example!

Unicode also likes to assume that the programs that render different 
languages know how to 'do the right thing' for that language in different 
situations, while some of those 'right things' seem very tough to get right.

Given all of the complexity of Unicode, working with the ASCII character set 
seems delightfully simple!

Martin

_________________________________________________________________
Ontvang je Hotmail & Messenger berichten op je mobiele telefoon 
http://www.msn.nl/communicatie/smsdiensten/hotmailsmsv2/