From: Henning von Rosen <henning@...>
Date: 2001-12-03T23:31:11+09:00
Subject: [ruby-talk:27348] Re: Unicode string for the standard library?


> > the utf-8 representation works fine for me, as regexes do support them.
>
> Is UTF-8 really ok ? Qt uses QString, which is a collection of QChar,
> which is UCS-2 (er, 2 bytes, I think that's the right name.)
> I would guess that the reason for this is because byte != character, and
> with an Unicode string, you want to deal with characters.
> Hmm, I'm no Unicode expert :|
>
> Would it be better for Ruby to have 'UChar' and 'UString' classes, or is
> it really a better solution to store string data as UTF-8 ?
>
> Rik

Implementing the full unicode width fixed-width characters _does_ cost cost
4 bytes per character. For the time being, no, if any, signs are outside of
UCS-2, but unicode means the _full_ unicode. There is a scheme for encoding
UCS-4 as 2-byte legas UCS-2, now named UTF-16. UTF-8 does the same thing,
encoding in one-byte ascii-compatible byte-stream.

So, setteling for fixed-width 2-byte, is not a "permanent solution". So it
shouldn't be the _only_ option.

The reason for using fixed-width representation is efficiency when doing
certain operations. Many operations though can be performed on
un-transformed utf-8. F ex searching on substrings. Regexes do not seem to
suffer noticeably. Input and output are often utf-8.

Would be interesting to hear the needs and solutions in relation unicode.

> Hmm, I'm no Unicode expert :|

neither am I, but, non-experts should at least be able to use Ruby for
text-processing.

Interesting theme.

/h