From: Joseph Jones Date: 2015-12-17T21:06:40-07:00 Subject: [ruby-core:72279] [Ruby trunk - Bug #9715] ENV data yield ASCII-8BIT encoded strings under Windows with unicode username --56738650_2a487cb0_16c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Joseph Jones liked your message with Boxer. On December 4, 2015 at 07:09:= 28 MST, thomas=40thomthom.net wrote:Issue =239715 has been updated by Tho= mas Thomassen.Trying to bump this. Even just some feedback on what an acc= eptable change would be and I could try to produce a patch.--------------= --------------------------Bug =239715: ENV data yield ASCII-8BIT encoded = strings under Windows with unicode usernamehttps://bugs.ruby-lang.org/iss= ues/9715=23change-55236* Author: Thomas Thomassen* Status: Open* Priority= : Normal* Assignee: cruby-windows* ruby -v: ruby 2.2.0dev (2014-04-07 tru= nk 45530) =5Bi386-mswin32=5F100=5D* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOW= N----------------------------------------My testing scenario:English Wind= ows, Unicode username: =E3=81=A6=E3=81=99=E3=81=A8Home directory: C:=5CUs= ers=5C=E3=81=A6=E3=81=99=E3=81=A8=5CThe values returned from ENV have dif= ferent encoding depending on their content. It appear to be OEM encoding = label to most value, except when they contain characters not included in = the OEM codepage. When they are not, for instance =60ENV=5B'HOME'=5D=60 w= hen the username is =22=E3=81=A6=E3=81=99=E3=81=A8=22 will have ASCII-8BI= T.(I find the =22ASCII-8BIT=22 name for an encoding confusing, as ASCII i= s 7bit - byte range 0-127)But it appear that =22ASCII-8BIT=22 is also ali= ased as =22binary=22=3F So Ruby is here returning a binary string when EN= V contain byte characters not included in the OEM code page=3FReading the= docs for Encoding:> Returns default internal encoding. Strings will be t= ranscoded to the default internal encoding in the following places if the= default internal encoding is not nil:> ...> ::default=5Finternal is init= ialized by the source file's internal=5Fencoding or -E option.This includ= es =60ENV=60 - but, even when I run ruby with the =60-E=60 flag the =60EN= V=60 encoding doesn't change. It's still using the OEM code page - or ASC= II-8BIT.However, regardless of having set =60-E=60 or not, ENV do appear = to return UT=46-8 bytes in the strings that contain the Unicode username.= This is one of several areas where I have found -E to have no effect on R= uby's string handling. I understand that some of Ruby's file handling is = for backwards compatibility reasons, but I'm finding it difficult to set = up a system which can properly handle Unicode files under Windows. Is thi= s deliberate due to backwards compatibility decisions=3F Or have I simply= not found the correct configuration flags for it=3F To me it appear bugg= ed - inconsistent with what the documentation says. But please enlighten = me if I am incorrect. My ideal situation would be for all strings to defa= ult to UT=46-8.Examples:=7E=7E=7EC:=5Cruby-220=5Cusr=5Cbin>ruby -E UT=46-= 8:UT=46-8 -e =22p ENV=5B'Program=46iles'=5D.encoding=22=23C:=5Cruby-220=5C= usr=5Cbin>ruby -E UT=46-8:UT=46-8 -e =22p ENV=5B'Program=46iles'=5D.bytes= =22=5B67, 58, 92, 80, 114, 111, 103, 114, 97, 109, 32, 70, 105, 108, 101,= 115, 32, 40, 120, 56, 54, 41=5D=7E=7E=7E=7E=7E=7EC:=5Cruby-220=5Cusr=5Cb= in>ruby -e =22p ENV=5B'HOME'=5D=22=22C:/Users/=5CxE3=5Cx81=5CxA6=5CxE3=5C= x81=5Cx99=5CxE3=5Cx81=5CxA8=22C:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p ENV=5B= 'HOME'=5D.encoding=22=23C:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p ENV=5B'HOM= E'=5D.bytes=22=5B67, 58, 47, 85, 115, 101, 114, 115, 47, 227, 129, 166, 2= 27, 129, 153, 227, 129, 168=5DC:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p =5F=5F= ENCODING=5F=5F=22=23C:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p Encoding.defau= lt=5Finternal=22nilC:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p Encoding.defaul= t=5Fexternal=22=23C:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p Encoding.find('f= ilesystem')=22=23C:=5Cruby-220=5Cusr=5Cbin>ruby -E UT=46-8:UT=46-8 -e =22= p ENV=5B'HOME'=5D.encoding=22=23C:=5Cruby-220=5Cusr=5Cbin>ruby -E UT=46-8= :UT=46-8 -e =22p ENV=5B'HOME'=5D.bytes=22=5B67, 58, 47, 85, 115, 101, 114= , 115, 47, 227, 129, 166, 227, 129, 153, 227, 129, 168=5D=7E=7E=7E-- http= s://bugs.ruby-lang.org/ --56738650_2a487cb0_16c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Joseph Jones liked your message with Boxer.


= On December 4, 2015 at 07:09:28 MST, thomas=40thomthom.net wrote:
Issue =239715 has been update= d by Thomas Thomassen.


Trying to bump this. Even just so= me feedback on what an acceptable change would be and I could try to prod= uce a patch.

----------------------------------------
Bug= =239715: ENV data yield ASCII-8BIT encoded strings under Windows with un= icode username
https://bugs.ruby-lang.org/issues/9715=23change-55236=

* Author: Thomas Thomassen
* Status: Open
* Priorit= y: Normal
* Assignee: cruby-windows
* ruby -v: ruby 2.2.0dev (2= 014-04-07 trunk 45530) =5Bi386-mswin32=5F100=5D
* Backport: 2.0.0: U= NKNOWN, 2.1: UNKNOWN
----------------------------------------
M= y testing scenario:
English Windows, Unicode username: =E3=81=A6=E3=81= =99=E3=81=A8

Home directory: C:=5CUsers=5C=E3=81=A6=E3=81=99=E3= =81=A8=5C

The values returned from ENV have different encoding= depending on their content. It appear to be OEM encoding label to most v= alue, except when they contain characters not included in the OEM codepag= e. When they are not, for instance =60ENV=5B'HOME'=5D=60 when the usernam= e is =22=E3=81=A6=E3=81=99=E3=81=A8=22 will have ASCII-8BIT.

(= I find the =22ASCII-8BIT=22 name for an encoding confusing, as ASCII is 7= bit - byte range 0-127)
But it appear that =22ASCII-8BIT=22 is also = aliased as =22binary=22=3F So Ruby is here returning a binary string when= ENV contain byte characters not included in the OEM code page=3F
Reading the docs for Encoding:

> Returns default internal = encoding. Strings will be transcoded to the default internal encoding in= the following places if the default internal encoding is not nil:
>= ...
> ::default=5Finternal is initialized by the source file's inte= rnal=5Fencoding or -E option.

This includes =60ENV=60 - but, e= ven when I run ruby with the =60-E=60 flag the =60ENV=60 encoding doesn't= change. It's still using the OEM code page - or ASCII-8BIT.
However= , regardless of having set =60-E=60 or not, ENV do appear to return UT=46= -8 bytes in the strings that contain the Unicode username.

Thi= s is one of several areas where I have found -E to have no effect on Ruby= 's string handling. I understand that some of Ruby's file handling is for= backwards compatibility reasons, but I'm finding it difficult to set up = a system which can properly handle Unicode files under Windows. Is this d= eliberate due to backwards compatibility decisions=3F Or have I simply no= t found the correct configuration flags for it=3F To me it appear bugged = - inconsistent with what the documentation says. But please enlighten me = if I am incorrect. My ideal situation would be for all strings to default= to UT=46-8.


Examples:

=7E=7E=7E
C:=5Cru= by-220=5Cusr=5Cbin>ruby -E UT=46-8:UT=46-8 -e =22p ENV=5B'Program=46iles'= =5D.encoding=22
=23

C:=5Cruby-220=5Cusr=5C= bin>ruby -E UT=46-8:UT=46-8 -e =22p ENV=5B'Program=46iles'=5D.bytes=22=5B67, 58, 92, 80, 114, 111, 103, 114, 97, 109, 32, 70, 105, 108, 101,= 115, 32, 40, 120, 56, 54, 41=5D
=7E=7E=7E

=7E=7E=7E
C:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p ENV=5B'HOME'=5D=22
=22C:/Use= rs/=5CxE3=5Cx81=5CxA6=5CxE3=5Cx81=5Cx99=5CxE3=5Cx81=5CxA8=22

C= :=5Cruby-220=5Cusr=5Cbin>ruby -e =22p ENV=5B'HOME'=5D.encoding=22
=23=

C:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p E= NV=5B'HOME'=5D.bytes=22
=5B67, 58, 47, 85, 115, 101, 114, 115, 47, 2= 27, 129, 166, 227, 129, 153, 227, 129, 168=5D

C:=5Cruby-220=5C= usr=5Cbin>ruby -e =22p =5F=5FENCODING=5F=5F=22
=23
C:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p Encoding.default=5Finter= nal=22
nil

C:=5Cruby-220=5Cusr=5Cbin>ruby -e =22p Encodin= g.default=5Fexternal=22
=23

C:=5Cruby-220= =5Cusr=5Cbin>ruby -e =22p Encoding.find('filesystem')=22
=23

C:=5Cruby-220=5Cusr=5Cbin>ruby -E UT=46-8:UT=46= -8 -e =22p ENV=5B'HOME'=5D.encoding=22
=23

C:=5Cruby-220=5Cusr=5Cbin>ruby -E UT=46-8:UT=46-8 -e =22p ENV=5B'H= OME'=5D.bytes=22
=5B67, 58, 47, 85, 115, 101, 114, 115, 47, 227, 129= , 166, 227, 129, 153, 227, 129, 168=5D
=7E=7E=7E


--
https://bugs.ruby-lang.org/
--56738650_2a487cb0_16c--