[ruby-core:61907] [ruby-trunk - Bug #9715] [Open] ENV data yield ASCII-8BIT encoded strings under Windows with unicode username

Issue #9715 has been reported by Thomas Thomassen.

----------------------------------------
Bug #9715: ENV data yield ASCII-8BIT encoded strings under Windows with unicode username
https://bugs.ruby-lang.org/issues/9715

* Author: Thomas Thomassen
* Status: Open
* Priority: Normal
* Assignee: cruby-windows
* Category: platform/windows
* Target version: current: 2.2.0
* ruby -v: ruby 2.2.0dev (2014-04-07 trunk 45530) [i386-mswin32_100]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
My testing scenario:
English Windows, Unicode username: てすと

Home directory: C:\Users\てすと\

The values returned from ENV have different encoding depending on their content. It appear to be OEM encoding label to most value, except when they contain characters not included in the OEM codepage. When they are not, for instance `ENV['HOME']` when the username is "てすと" will have ASCII-8BIT.

(I find the "ASCII-8BIT" name for an encoding confusing, as ASCII is 7bit - byte range 0-127)
But it appear that "ASCII-8BIT" is also aliased as "binary"? So Ruby is here returning a binary string when ENV contain byte characters not included in the OEM code page?

Reading the docs for Encoding:

> Returns default internal encoding.  Strings will be transcoded to the default internal encoding in the following places if the default internal encoding is not nil:
> ...
> ::default_internal is initialized by the source file's internal_encoding or -E option.

This includes `ENV` - but, even when I run ruby with the `-E` flag the `ENV` encoding doesn't change. It's still using the OEM code page - or ASCII-8BIT.
However, regardless of having set `-E` or not, ENV do appear to return UTF-8 bytes in the strings that contain the Unicode username.

This is one of several areas where I have found -E to have no effect on Ruby's string handling. I understand that some of Ruby's file handling is for backwards compatibility reasons, but I'm finding it difficult to set up a system which can properly handle Unicode files under Windows. Is this deliberate due to backwards compatibility decisions? Or have I simply not found the correct configuration flags for it? To me it appear bugged - inconsistent with what the documentation says. But please enlighten me if I am incorrect. My ideal situation would be for all strings to default to UTF-8.


Examples:

~~~
C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['ProgramFiles'].encoding"
#<Encoding:CP850>

C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['ProgramFiles'].bytes"
[67, 58, 92, 80, 114, 111, 103, 114, 97, 109, 32, 70, 105, 108, 101, 115, 32, 40, 120, 56, 54, 41]
~~~

~~~
C:\ruby-220\usr\bin>ruby -e "p ENV['HOME']"
"C:/Users/\xE3\x81\xA6\xE3\x81\x99\xE3\x81\xA8"

C:\ruby-220\usr\bin>ruby -e "p ENV['HOME'].encoding"
#<Encoding:ASCII-8BIT>

C:\ruby-220\usr\bin>ruby -e "p ENV['HOME'].bytes"
[67, 58, 47, 85, 115, 101, 114, 115, 47, 227, 129, 166, 227, 129, 153, 227, 129, 168]

C:\ruby-220\usr\bin>ruby -e "p __ENCODING__"
#<Encoding:CP850>

C:\ruby-220\usr\bin>ruby -e "p Encoding.default_internal"
nil

C:\ruby-220\usr\bin>ruby -e "p Encoding.default_external"
#<Encoding:CP850>

C:\ruby-220\usr\bin>ruby -e "p Encoding.find('filesystem')"
#<Encoding:Windows-1252>

C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['HOME'].encoding"
#<Encoding:ASCII-8BIT>

C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['HOME'].bytes"
[67, 58, 47, 85, 115, 101, 114, 115, 47, 227, 129, 166, 227, 129, 153, 227, 129, 168]
~~~



-- 
https://bugs.ruby-lang.org/