[ruby-core:104569] [Ruby master Bug#15993] 'require' doesn't work if there are Cyrillic chars in the path to Ruby dir

Issue #15993 has been updated by jeremyevans0 (Jeremy Evans).

Status changed from Closed to Open

inversion (Yura Babak) wrote in #note-8:
> jeremyevans0 (Jeremy Evans) wrote in #note-7:
> > This appears to be fixed starting in Ruby 2.7 (also works in 3.0):
>  
> **Still, there is a problem.**
> `require 'bundler/setup'` fails if `LOAD_PATH` or `Gem.dir` contain Cyrillic chars, the error is similar to:
> ```
> incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
> ```
> 
> From the trace I have prepared the **minimum reproducible case** :
> 1. Put Ruby in a location where the path will contain Cyrillic chars, like `"D:\users\ミコミクム予Ruby"`
> 1. Prepare 2 files (saved in UTF-8 encoding) somewhere in a location where the path will contain Cyrillic chars (can be near that Ruby):
>    https://gist.github.com/Inversion-des/75949795cc5be707c19d31901e79d1cf
> 1. Open cmd and ensure to do `chcp 1251` in the current console session.
> 1. run `"[this Ruby path]" f1.rb`

I was able to reproduce the issue, but only when I installed Ruby into a path not supported by the `Windows-1251` encoding:

```
d:\ミ片イミウミオミスミクミケ>d:\zz-kテカnnen2\Ruby31-x64\bin\bundle install --local
d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler.rb:94:in `expand_path': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler.rb:94:in `expand_path'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler.rb:94:in `bundle_path'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler.rb:682:in `configure_gem_home'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler.rb:663:in `configure_gem_home_and_path'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler.rb:80:in `configure'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler.rb:193:in `definition'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/cli/install.rb:57:in `run'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/cli.rb:259:in `block in install'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/settings.rb:133:in `temporary'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/cli.rb:258:in `install'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/cli.rb:30:in `dispatch'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/cli.rb:24:in `start'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/gems/3.1.0/gems/bundler-2.3.0.dev/libexec/bundle:49:in `block in <top (required)>'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/3.1.0/bundler/friendly_errors.rb:130:in `with_friendly_errors'
        from d:/zz-kテカnnen2/Ruby31-x64/lib/ruby/gems/3.1.0/gems/bundler-2.3.0.dev/libexec/bundle:37:in `<top (required)>'
        from d:/zz-k?nnen2/Ruby31-x64/bin/bundle:31:in `load'
        from d:/zz-k?nnen2/Ruby31-x64/bin/bundle:31:in `<main>'
```

Part of the underlying issue seems to be that `__FILE__` and `__dir__` are not UTF-8 encoded for the main script, unlike required files.  I'm not sure if changing that alone will fix the issue, though.

When I run the following script (`f3.rb`):

```
p ['__FILE__', __FILE__, __FILE__.encoding]
p ['__dir__', __dir__, __dir__.encoding]
p ['Gem.dir', Gem.dir, Gem.dir.encoding]
puts 'Gem.path'
Gem.path.each do |s|
  p [s, s.encoding]
end
puts '$:'
$:.each do |s|
  p [s, s.encoding]
end
```

I get the following when using Ruby installed in a non-ASCII path:

```
d:\ミ片イミウミオミスミクミケ>d:\zz-kテカnnen2\Ruby31-x64\bin\ruby D:\ミ片イミウミオミスミクミケ\f3.rb
["__FILE__", "D:/\xC5\xE2\xE3\xE5\xED\xE8\xE9/f3.rb", #<Encoding:Windows-1251>]
["__dir__", "D:/\xC5\xE2\xE3\xE5\xED\xE8\xE9", #<Encoding:Windows-1251>]
["Gem.dir", "d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/gems/3.1.0", #<Encoding:ASCII-8BIT>]
Gem.path
["C:/Users/jeremye/.gem/ruby/3.1.0", #<Encoding:UTF-8>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/gems/3.1.0", #<Encoding:ASCII-8BIT>]
$:
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/site_ruby/3.1.0", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/site_ruby/3.1.0/x64-ucrt", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/site_ruby", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/vendor_ruby/3.1.0", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/vendor_ruby/3.1.0/x64-ucrt", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/vendor_ruby", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/3.1.0", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/3.1.0/x64-mingw-ucrt", #<Encoding:ASCII-8BIT>]
```

and when installed into an ASCII path:

```
d:\ミ片イミウミオミスミクミケ>C:\Ruby30-x64\bin\ruby d:\ミ片イミウミオミスミクミケ\f3.rb
["__FILE__", "d:/\xC5\xE2\xE3\xE5\xED\xE8\xE9/f3.rb", #<Encoding:Windows-1251>]
["__dir__", "d:/\xC5\xE2\xE3\xE5\xED\xE8\xE9", #<Encoding:Windows-1251>]
["Gem.dir", "C:/Ruby30-x64/lib/ruby/gems/3.0.0", #<Encoding:ASCII-8BIT>]
Gem.path
["C:/Users/jeremye/.gem/ruby/3.0.0", #<Encoding:UTF-8>]
["C:/Ruby30-x64/lib/ruby/gems/3.0.0", #<Encoding:ASCII-8BIT>]
$:
["C:/Ruby30-x64/lib/ruby/site_ruby/3.0.0", #<Encoding:Windows-1251>]
["C:/Ruby30-x64/lib/ruby/site_ruby/3.0.0/x64-msvcrt", #<Encoding:Windows-1251>]
["C:/Ruby30-x64/lib/ruby/site_ruby", #<Encoding:Windows-1251>]
["C:/Ruby30-x64/lib/ruby/vendor_ruby/3.0.0", #<Encoding:Windows-1251>]
["C:/Ruby30-x64/lib/ruby/vendor_ruby/3.0.0/x64-msvcrt", #<Encoding:Windows-1251>]
["C:/Ruby30-x64/lib/ruby/vendor_ruby", #<Encoding:Windows-1251>]
["C:/Ruby30-x64/lib/ruby/3.0.0", #<Encoding:Windows-1251>]
["C:/Ruby30-x64/lib/ruby/3.0.0/x64-mingw32", #<Encoding:Windows-1251>]
```

It looks like the difference in the non-ASCII path case is that `ASCII-8BIT` encoding is used even if the path itself is valid UTF-8.  This is true even if you force a UTF-8 code page (though that does fix `__FILE__` and `__dir__`):

```
d:\ミ片イミウミオミスミクミケ>chcp 65001
Active code page: 65001

d:\ミ片イミウミオミスミクミケ>d:\zz-kテカnnen2\Ruby31-x64\bin\ruby D:\ミ片イミウミオミスミクミケ\f3.rb
["__FILE__", "D:/ミ片イミウミオミスミクミケ/f3.rb", #<Encoding:UTF-8>]
["__dir__", "D:/ミ片イミウミオミスミクミケ", #<Encoding:UTF-8>]
["Gem.dir", "d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/gems/3.1.0", #<Encoding:ASCII-8BIT>]
Gem.path
["C:/Users/jeremye/.gem/ruby/3.1.0", #<Encoding:UTF-8>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/gems/3.1.0", #<Encoding:ASCII-8BIT>]
$:
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/site_ruby/3.1.0", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/site_ruby/3.1.0/x64-ucrt", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/site_ruby", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/vendor_ruby/3.1.0", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/vendor_ruby/3.1.0/x64-ucrt", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/vendor_ruby", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/3.1.0", #<Encoding:ASCII-8BIT>]
["d:/zz-k\xC3\xB6nnen2/Ruby31-x64/lib/ruby/3.1.0/x64-mingw-ucrt", #<Encoding:ASCII-8BIT>]
```

Since there does appear to be an issue, I'll reopen this.  Hopefully someone with more knowledge in this area can suggest a possible fix.

----------------------------------------
Bug #15993: 'require' doesn't work if there are Cyrillic chars in the path to Ruby dir
https://bugs.ruby-lang.org/issues/15993#change-92849

* Author: inversion (Yura Babak)
* Status: Open
* Priority: Normal
* ruby -v: ruby 2.6.3p62 (2019-04-16 revision 67580) [x64-mingw32]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
I窶冦 trying to build a cross-platform portable application with Ruby onboard and there is a problem on Windows. 
A user usually installs it to the Roaming folder which sits inside a user folder which can often have not a Latin name or contain spaces). 
When there is a Cyrillic character (maybe just not Latin) in the path 窶require of any gem doesn’t work:

```
D:\users\киї\Ruby\2.6\bin>ruby -v
ruby 2.6.3p62 (2019-04-16 revision 67580) [x64-mingw32]

D:\users\киї\Ruby\2.6\bin>ruby -e "require 'logger'"
Traceback (most recent call last):
        1: from <internal:gem_prelude>:2:in `<internal:gem_prelude>'
<internal:gem_prelude>:2:in `require': No such file or directory -- D:/users/РєРёС—/Ruby/2.6/lib/ruby/2.6.0/rubygems.rb (LoadError)

D:\users\киї\Ruby\2.6\bin>ruby --disable=rubyopt -e "require 'logger'"
Traceback (most recent call last):
        1: from <internal:gem_prelude>:2:in `<internal:gem_prelude>'
<internal:gem_prelude>:2:in `require': No such file or directory -- D:/users/РєРёС—/Ruby/2.6/lib/ruby/2.6.0/rubygems.rb (LoadError)

D:\users\киї\Ruby\2.6\bin>gem list
Traceback (most recent call last):
        1: from <internal:gem_prelude>:2:in `<internal:gem_prelude>'
<internal:gem_prelude>:2:in `require': No such file or directory -- D:/users/РєРёС—/Ruby/2.6/lib/ruby/2.6.0/rubygems.rb (LoadError)

```

We can see such encoding transformations in the output:

```
киї (utf-8) == РєРёС— (win1251)
```

I have an old Ruby installation that works fine:

```
D:\users\киї\Ruby\2.0\bin>ruby -e "require 'logger'"

D:\users\киї\Ruby\2.0\bin>ruby -v
ruby 2.0.0p451 (2014-02-24) [i386-mingw32]
```
The same is for `ruby 2.0.0p643 (2015-02-25) [i386-mingw32]` .

I also checked that require fails in the same case for
`ruby 2.1.9p490 (2016-03-30 revision 54437) [i386-mingw32]`



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>