[ruby-core:54800] Re: Call setlocale(LC_ALL, "") instead of setlocale(LC_CTYPE, "")

On Sun, May 5, 2013 at 12:56 AM, Tanaka Akira <[e-mail]> wrote:
> 2013/5/5 Nikolai Weibull <[e-mail]>:

>> Why isn=E2=80=99t setlocale called with LC_ALL instead of LC_CTYPE in ma=
in.c?

> Changing setlocale(LC_CTYPE, "") to setlocale(LC_ALL, "") still cause pro=
blems.

It causes problems that are based precisely on the fact that they=E2=80=99r=
e
locale sensitive.

> make check with changed Ruby on GNU/Linux fails as follows with
> tr_TR.UTF-8 locale.
> (tr_TR locale changes case related bahavior.  [ruby-core:14662])

> % LANG=3Dtr_TR.UTF-8 make check

> [  247/13312] DL::TestDL#test_call_double =3D 0.09 s
>   1) Failure:
> DL::TestDL#test_call_double [/home/akr/ruby/tst1/ruby/test/dl/test_dl2.rb=
:90]:
> Expected |0.1 - 0.0| (0.1) to be <=3D 0.001.

Bugs because atof is called, which is locale sensitive.

> [  279/13312] DL::TestFunc#test_strtod =3D 0.09 s
>   2) Failure:
> DL::TestFunc#test_strtod [/home/akr/ruby/tst1/ruby/test/dl/test_func.rb:1=
52]:
> Expected |12.34 - 12.0| (0.33999999999999986) to be <=3D 0.001.

Same here, but with locale-sensitive function strtod.

> [  439/13312] Fiddle::TestFunc#test_strtod =3D 0.09 s
>   3) Failure:
> Fiddle::TestFunc#test_strtod
> [/home/akr/ruby/tst1/ruby/test/fiddle/test_func.rb:56]:
> Expected |12.34 - 12.0| (0.33999999999999986) to be <=3D 0.001.

And again.

> [ 2133/13312] TestArgf#test_unreadable =3D 0.00 s
>   4) Error:
> TestArgf#test_unreadable:
> ArgumentError: invalid byte sequence in US-ASCII
>     /home/akr/ruby/tst1/ruby/test/ruby/test_argf.rb:803:in `block in
> test_unreadable'

This seems to be a problem with paths being locale sensitive and
therefore encoding sensitive, but not created with the right encoding.
 This is a more complicated problem than the previous ones, but it
surely is a bug in Ruby, not a reason not to enable locale support in
Ruby.

> [ 2379/13312] TestBigDecimal#test_BigMath_exp =3D 0.00 s
>   5) Failure:
> TestBigDecimal#test_BigMath_exp
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:1280]:
> Expected |485165195.4097903 - 0.48516519540979027797E9|
> (485165195.4097903) to be <=3D 0.48516519540979027797E6.
>
> [ 2383/13312] TestBigDecimal#test_BigMath_log_with_2 =3D 0.00 s
>   6) Failure:
> TestBigDecimal#test_BigMath_log_with_2
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:1365]:
> Expected |0.6931471805599453 -
> 0.6931471805599453094172321214581765674099931010273832E0|
> (0.6931471805599453) to be <=3D 0.001.
>
> [ 2384/13312] TestBigDecimal#test_BigMath_log_with_42 =3D 0.01 s
>   7) Failure:
> TestBigDecimal#test_BigMath_log_with_42
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:1375]:
> Expected |3.7376696182833684 -
> 0.3737669618283368305917830101823881909945797027394132E1|
> (3.7376696182833684) to be <=3D 0.001.
>
> [ 2386/13312] TestBigDecimal#test_BigMath_log_with_exp_1 =3D 0.00 s
>   8) Failure:
> TestBigDecimal#test_BigMath_log_with_exp_1
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:1361]:
> Expected |1.0 -
> 0.10000000000000000000146094228462874163031819506060204E1| (1.0) to be
> <=3D 0.001.
>
> [ 2393/13312] TestBigDecimal#test_BigMath_log_with_reciprocal_of_42 =3D
> 0.00 s
>   9) Failure:
> TestBigDecimal#test_BigMath_log_with_reciprocal_of_42
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:1381]:
> Expected |-96.70857390574992 -
> -0.96708573905749918728755641096743294420914068228418976E2|
> (96.70857390574992) to be <=3D 0.001.
>
> [ 2425/13312] TestBigDecimal#test_exp_with_1 =3D 0.00 s
>  10) Failure:
> TestBigDecimal#test_exp_with_1
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:1275]:
> Expected |2.718281828459045 - 0.27182818284590452354E1|
> (2.718281828459045) to be <=3D 0.002718281828459045.
>
> [ 2469/13312] TestBigDecimal#test_power_of_three =3D 0.00 s
>  11) Failure:
> TestBigDecimal#test_power_of_three
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:1007]:
> Expected |0.012345679012345678 -
> 0.12345679012345679012345679012345679012345679012345679012345679E-1|
> (0.012345679012345678) to be <=3D 0.001.
>
> [ 2488/13312] TestBigDecimal#test_sqrt_bigdecimal =3D 0.00 s
>  12) Failure:
> TestBigDecimal#test_sqrt_bigdecimal
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:786]:
> Expected |0.3 - 0.3E0| (0.3) to be <=3D 0.001.
>
> [ 2495/13312] TestBigDecimal#test_to_f =3D 0.00 s
>  13) Failure:
> TestBigDecimal#test_to_f
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigdecimal.rb:557]:
> <1.0> expected but was
> <0.0>.
>
> [ 2510/13312] TestBigMath#test_atan =3D 0.00 s
>  14) Failure:
> TestBigMath#test_atan
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigmath.rb:57]:
> Expected |0.7853981633974483 -
> 0.78539816339744830961566084581987572097996482338035675E0|
> (0.7853981633974483) to be <=3D 0.001.
>
> [ 2511/13312] TestBigMath#test_const =3D 0.00 s
>  15) Failure:
> TestBigMath#test_const
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigmath.rb:13]:
> Expected |3.141592653589793 -
> 0.3141592653589793238462643383279502883919859293521427E1|
> (3.141592653589793) to be <=3D 0.001.
>
> [ 2512/13312] TestBigMath#test_cos =3D 0.00 s
>  16) Failure:
> TestBigMath#test_cos
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigmath.rb:42]:
> Expected |1.0 - 0.1E1| (1.0) to be <=3D 0.001.
>
> [ 2513/13312] TestBigMath#test_sin =3D 0.00 s
>  17) Failure:
> TestBigMath#test_sin
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigmath.rb:29]:
> Expected |0.7071067811865476 -
> 0.70710678118654752440084436210484903917782034136995079E0|
> (0.7071067811865476) to be <=3D 0.001.
>
> [ 2514/13312] TestBigMath#test_sqrt =3D 0.00 s
>  18) Failure:
> TestBigMath#test_sqrt
> [/home/akr/ruby/tst1/ruby/test/bigdecimal/test_bigmath.rb:18]:
> Expected |1.4142135623730951 -
> 0.141421356237309504880168872420969807825E1| (1.4142135623730951) to
> be <=3D 0.001.

These I can=E2=80=99t understand with only a cursory glance.  Do you know w=
hy
they=E2=80=99re failing?  I see that BigDecimal_to_f uses strtod (which is
locale dependent and should therefore most likely not be used) and
perhaps that=E2=80=99s the culprit?

> [ 4903/13312] TestGemExtCmakeBuilder#test_self_build =3D 0.02 s
>  19) Skipped:
> TestGemExtCmakeBuilder#test_self_build
> [/home/akr/ruby/tst1/ruby/test/rubygems/test_gem_ext_cmake_builder.rb:11]=
:
> cmake not present

This fails for me as I don=E2=80=99t have cmake.

> [ 4904/13312] TestGemExtCmakeBuilder#test_self_build_fail =3D 0.02 s
>  20) Skipped:
> TestGemExtCmakeBuilder#test_self_build_fail
> [/home/akr/ruby/tst1/ruby/test/rubygems/test_gem_ext_cmake_builder.rb:11]=
:
> cmake not present
>
> [ 4905/13312] TestGemExtCmakeBuilder#test_self_build_has_makefile =3D 0.0=
3 s
>  21) Skipped:
> TestGemExtCmakeBuilder#test_self_build_has_makefile
> [/home/akr/ruby/tst1/ruby/test/rubygems/test_gem_ext_cmake_builder.rb:11]=
:
> cmake not present

These two as well.

> [ 4994/13312] TestGemInstaller#test_build_extensions_extconf_bad =3D
> 0.10 s
>  22) Failure:
> TestGemInstaller#test_build_extensions_extconf_bad
> [/home/akr/ruby/tst1/ruby/test/rubygems/test_gem_installer.rb:86]:
> Expected /\/home\/akr\/ruby\/tst1\/ruby\/ruby: No such file/ to match
> "/home/akr/ruby/tst1/ruby/ruby
> extconf.rb\n/home/akr/ruby/tst1/ruby/ruby: B=C3=B6yle bir dosya ya da diz=
in
> yok -- extconf.rb (LoadError)\n".

OK, this fails because the error message generated by strerror is
being localized.  If this is really a problem beyond a test that
depends on the locale being en_US, then perhaps exclude LC_MESSAGES
from setlocale.

> [ 7181/13312] TestMiniTestUnitTestCase#test_capture_subprocess_io =3D
> 0.00 s
>  23) Skipped:
> TestMiniTestUnitTestCase#test_capture_subprocess_io
> [/home/akr/ruby/tst1/ruby/test/minitest/test_minitest_unit.rb:1401]:
> Dunno why but the parallel run of this fails

Hard-coded skip, so this isn=E2=80=99t related to locales.

> [ 8173/13312] TestProcess#test_execopts_env =3D 0.07 s
>  24) Error:
> TestProcess#test_execopts_env:
> ArgumentError: invalid byte sequence in US-ASCII
>     /home/akr/ruby/tst1/ruby/test/ruby/test_process.rb:260:in `sub'
>     /home/akr/ruby/tst1/ruby/test/ruby/test_process.rb:260:in `block
> in test_execopts_env'
>     /home/akr/ruby/tst1/ruby/test/ruby/test_process.rb:33:in `block (2
> levels) in with_tmpchdir'
>     /home/akr/ruby/tst1/ruby/test/ruby/test_process.rb:32:in `chdir'
>     /home/akr/ruby/tst1/ruby/test/ruby/test_process.rb:32:in `block in
> with_tmpchdir'
>     /home/akr/ruby/tst1/ruby/lib/tmpdir.rb:88:in `mktmpdir'
>     /home/akr/ruby/tst1/ruby/test/ruby/test_process.rb:30:in `with_tmpchd=
ir'
>     /home/akr/ruby/tst1/ruby/test/ruby/test_process.rb:251:in
> `test_execopts_env'

This is also related to strerror being used by Errno error messages.
Strerror returns a localized message if LC_MESSAGES is included in
setlocale() and in the codeset/encoding of the locale, if so.

> [10528/13312] TestRakeFunctional#test_signal_propagation_in_testsTerminat=
ed
>  =3D 0.08 s
>  25) Skipped:
> TestRakeFunctional#test_signal_propagation_in_tests
> [/home/akr/ruby/tst1/ruby/test/rake/test_rake_functional.rb:436]:
> Signal detect seems broken on this system

I don=E2=80=99t know why this fails, but perhaps it=E2=80=99s not locale de=
pendent?

> It seems four years are not enough to make ruby robust with locales.

> I think it is acceptable that make ruby more robust.
> (i.e. fixing above failures and errors are ok.)

> But changing setlocale invocation to LC_ALL is too dangerous.
> The change affects not only ruby itself but also other extension librarie=
s.

Precisely my point and precisely why setlocale should be called with
LC_ALL.  It=E2=80=99s impossible to write extensions tat use locale-depende=
nt
functionality unless Ruby actively supports it.  Your argument held
equally true before setlocale(LC_CTYPE) was added, but still it was
added.