From: Mark Hahn Date: 2001-12-04T07:20:46+09:00 Subject: [ruby-talk:27380] Re: Unicode string for the standard library? Aren't native ruby strings 31 bits per char? The most efficient unicoding in Ruby would be something that took advantage of that fact. Then UTF-8 or other standards could be imput and output. Java takes this approach with a unique internal 16-bit format. -----Original Message----- From: Henning von Rosen [mailto:henning@ikso.net] Sent: Monday, December 03, 2001 6:31 AM To: ruby-talk ML Subject: [ruby-talk:27348] Re: Unicode string for the standard library? > > the utf-8 representation works fine for me, as regexes do support them. > > Is UTF-8 really ok ? Qt uses QString, which is a collection of QChar, > which is UCS-2 (er, 2 bytes, I think that's the right name.) > I would guess that the reason for this is because byte != character, and > with an Unicode string, you want to deal with characters. > Hmm, I'm no Unicode expert :| > > Would it be better for Ruby to have 'UChar' and 'UString' classes, or is > it really a better solution to store string data as UTF-8 ? > > Rik Implementing the full unicode width fixed-width characters _does_ cost cost 4 bytes per character. For the time being, no, if any, signs are outside of UCS-2, but unicode means the _full_ unicode. There is a scheme for encoding UCS-4 as 2-byte legas UCS-2, now named UTF-16. UTF-8 does the same thing, encoding in one-byte ascii-compatible byte-stream. So, setteling for fixed-width 2-byte, is not a "permanent solution". So it shouldn't be the _only_ option. The reason for using fixed-width representation is efficiency when doing certain operations. Many operations though can be performed on un-transformed utf-8. F ex searching on substrings. Regexes do not seem to suffer noticeably. Input and output are often utf-8. Would be interesting to hear the needs and solutions in relation unicode. > Hmm, I'm no Unicode expert :| neither am I, but, non-experts should at least be able to use Ruby for text-processing. Interesting theme. /h