From: Stefan Lang <perfectly.normal.hacker@...>
Date: 2008-04-10T21:31:21+09:00
Subject: Re: Ruby's not ready - an indepth essay

2008/4/8, Phillip Gawlowski <cmdjackryan@googlemail.com>:
> -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>
>  Austin Ziegler wrote:
>  | On Tue, Apr 8, 2008 at 12:05 AM, Phillip Gawlowski
>  | <cmdjackryan@googlemail.com> wrote:
>  |>  Austin Ziegler wrote:
>  |>  | (Yes, Virginia. Most people don't need full-on Unicode munging in
>  |>  | their code. It's necessary when you do need it, but most people don't
>  |>  | need it.)
>  |>  http://www.joelonsoftware.com/articles/Unicode.html
>  |>
>  |>  I need it. Most of Europe needs it. Not to mention Arabia, Japan, and
>  |>  everybody else not speaking English.
>  |
>  | You didn't read what I said. I said "most people don't need full-on
>  | Unicode munging." This is true. There are some cases where it's
>  | absolutely necessary, but most people just need to know that they're
>  | not going to screw up things when they work with Unicode.
>
>  Without Unicode support, a string operation in a non-English alphabet
>  will work as expected. Any language with more characters in the alphabet
>  that English, needs full Unicode support.

This is wrong. My native language is German, and although the
situation is not ideal, the provided UTF-8 support in
combination with iconv was sufficient for me.

There are a number of features that seem to be little known:

* Just because Ruby doesn't have full (whatever that means)
  Unicode support, it does not throw away bytes not in ASCII.
  A Ruby string can hold _any_ byte sequence.

* Ruby comes with the Iconv library for encoding conversion.

* UTF-8 is backwards compatible with ASCII. Any byte in an UTF-8
  string that looks like an ASCII character _is_ that character.
  Thus you can safely split any UTF-8 strings on ASCII
  characters. You can safely split UTF-8 strings on UTF-8
  strings, search for UTF-8 strings, safely concatenate
  UTF-8 with UTF-8 or ASCII strings, etc.

* Ruby's regex engine has some support for UTF-8.

* As Vidar Hokstad pointed out, you can turn UTF-8 strings into
  arrays of unicode code points via unpack.

  "€äöü".unpack("U*").length  # => 4
  "€äöü".unpack("U*").reverse.pack("U*") # => "üöä€"

* In some cases, like case-insensitive comparison, you can let
  your database work for you.

Not to mention that what I've seen so far of Ruby 1.9's encoding
and Unicode support looks good. (In contrast to say, Java.  I
wonder how many Java methods deal correctly with surrogate pairs
and how many Java programmers know that the char type has little
to do with characters.)

Stefan