Re: locale in ruby

At 14:17 08/01/01, Tanaka Akira wrote:
>In article <6.0.0.20.2.20071231200106.0b1bfb00@localhost>,
>  Martin Duerst <[e-mail]> writes:
>
>> I'm not sure this was a good idea. It will change Ruby behavior
>> in various areas, even when not intended.
>
>Do you have concreate examples now?

Ok, I just found one. What we will try is something very simple:

$ ruby -e "puts 'abc'.force_encoding('US-ASCII').encoding"

Usually, that just prints the string "US-ASCII".

But now try switching to e.g. a Turkish locale
(the example that follows is on a Fedora box with tcsh,
your mileage may vary):

$ setenv LANG tr_TR.iso88599
$ ruby -e "puts 'abc'.force_encoding('US-ASCII').encoding"
-e:1:in `force_encoding': unknown encoding name - US-ASCII (ArgumentError)
        from -e:1:in `<main>'

The reason for this is that in a Turkish locale, the lowercase
i is mapped to the uppercase I-with-dot, and the uppercase I
is mapped to the lowercase i-without-dot.
The same problem happens with US-ASCii. I guess this is because
encoding names are kept in upper case, while the linkable extensions
have lower-case filenames. The same happens with all other encoding
names that contain an i/I, except for ASCII-8BIT if written
with all-uppercase I. The later case is probably due to the
fact that ASCII-8BIT is somehow hard-coded.

The same thing also happens for a Turkish locale using utf-8,
and for an Azeri locale using utf-8. There may be others.

>Some application require this clearly, such as curses.
>Some aplication may have trouble with this.

Truble is more per locale than per application.
Ruby 1.9 doesn't have many Turkish users yet, I guess,
or we might already have heard some complaints.

>I'd like to know Ruby 1.9.0-0 causes trouble or not.
>
>However I guess it is not so harmful.  It is because
>libreadline invokes setlocale.  So readline-enabled irb
>invokes setlocale.  But I don't see locale related trouble
>on irb.

If you set your locale before invoking irb, you might.

As I said earlier, if we want to keep setlocale, we have
to check every libc call for locale effects. I looked around
for strcasecmp, which I knew changes behavior in some locales.
There are quite a few strcasecmp in Ruby. Some are harmless
(e.g. checking for ".exe" in ruby.c, because there are no
locales where '.', 'e', or 'x' are treated differently).
Others are dangerous. Sometimes it may be difficult to
figure out what's the right way to do, because some
Turkish systems may e.g. use Turkish rules for case
correspondence of environment variables, while other
systems may use basic C locale rules.

Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]