From: MikkelFJ <mikkelfj-anti-spam@...>
Date: 2002-08-03T07:15:18+09:00
Subject: Re: Unicode in Ruby now?


"Curt Sampson" <cjs@cynic.net> wrote in message
news:Pine.NEB.4.44.0208021752530.442-100000@angelic.cynic.net...

> > But isn't this what matz suggest?
> > Each stream is tagged, that is the same as having different types. It's
> > basically just  a different way to store the type while having a lot of
> > common string operations.
>
> No, because then you have to deal with conversions. Most popular
> character sets are convertable to Unicode and back without loss. That is
> not true of any arbitrary pair of character sets, though, even if you go
> through Unicode.

Not really, you just produce a runtime error that the data cannot be - say -
concatenated. You just use the same vehicle to carry different cargo.
You can have special functions for to-Unicode and From-unicode. And similar
for popular Asian scripts.

> But there's an even better reason than this for converting to
> Unicode on input, rather than doing internal tagging. If you don't
> have conversion tables for a particular character encoding, it's
> much better to find out at the time you try to get the information
> in to the system than at some arbitrary later point when you try
> to do a conversion. That way you know where the problem information
> is coming from.

That is essentially static versus dynamic typing. I'd say in most situations
the application would be pretty well aware about what they are doing. The
tagging allows a generic rutine to handle multiple formats if it so chooses,
or could error if it got anything but Unicode (or whatever).

>     2. Add a UString or similar for dealing with UTF-16 data. There's

Obviously you know more about Unicode than most. What is the practical
difference between UCS-4, UCS-2 and UTF-16. Is it that "extended
characerts" - or surrogates - will take on more space than UCS-4 but
typically take up the same space as UCS-2?

> > and options for future extensions.
>
> Not that I know of. Can you explain what these are?

I guess you know more about this than I. I can't give you details I only
have it from memory. It is possible that it is covered by reserved ranges of
code pairs.
In that case UCS-4 should be sufficient.
>
> > Hence UCS-4 is a strategy with limited timespan.

> Not at all, unless they decide to change Unicode to the point where it
> no longer uses 16-bit code values, or add escape codes, or something
> like that. That would be backward-incompatable, severely complicate
> processing, and generally cause all hell to break lose. So I'd rate
> this as "Not Likely."

It wouldn't be the first time hell breaks loose in this area though.

>     2. There are many situations where, even if surrogate pairs
>     are present, you don't know or care, and need do nothing to
>     correctly deal with them.

Does this means that UCS-2 is the best format?

> So I propose just what the Unicode standard itself proposes in
> section 5.4: UString (or whatever we call it) should have the
> Surrogate Support Level "none"; i.e., it completely ignores the
> existence of surrogate pairs. Things that use UString that have
> the potential to encounter surrogate pair problems or wish to
> interpret them can add simple or complex code, as they need, to
> deal with the problem at hand. (Many users of UString will need to
> do nothing.)

Is that UCS-2 or UTF-16 then?

>
> Note that there's a big difference between this and your UTF-8
> proposal: ignoring multibyte stuff in UTF-8 is going to cause much,
> much more lossage because there's a much, much bigger chance of
> breaking things when using Asian languages. With UTF-16, you probably
> won't even encounter surrogates, whereas with Japanese in UTF-8,
> pretty much every character is multibyte.

I did not mean so that you should ignore the content. But you can process it
as if it were ASCII because in many languages everthing that is not text is
found in the ASCII range. Due to the way UTF-8 is encoded you never risc
getting a spurious ASCII character following this path. For example, you can
find delimited text simply scanning from one double quote to the next.
Everything in between is a sound text possibly in UTF-8 - you do not need to
care about this. Subsequently you may wish to convert the delimited string
into UCS-2 or whatever.

This approach avoids complex character type lookups when parsing text. It
will for example work for XML and Ruby.

In order to add international identifier support to a UTF-8 stream processed
as ASCII you can use the following pattern (I believe Ruby already does
something similar).

identifier = /[A-Za-z_\x80-\0xfd][A-Za-z_\x0080-\0xfd]*/

Mikkel