From: Joseph Jones <joeyi5216@...>
Date: 2015-12-17T21:06:39-07:00
Subject: [ruby-core:72278] [Ruby trunk - Bug #9713] __FILE__ return unexpected encoding - breaks Dir.glob

--5673864f_70a64e2a_16c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Joseph Jones liked your message with Boxer. On December 4, 2015 at 06:35:=
43 MST, thomas=40thomthom.net wrote:Issue =239713 has been updated by Tho=
mas Thomassen.Revisiting this issue again. Is there a resolution to what =
can be done to improve this and still satisfy compatibility concerns=3F--=
--------------------------------------Bug =239713: =5F=5F=46ILE=5F=5F ret=
urn unexpected encoding - breaks Dir.globhttps://bugs.ruby-lang.org/issue=
s/9713=23change-55235* Author: Thomas Thomassen* Status: Assigned* Priori=
ty: Normal* Assignee: cruby-windows* ruby -v: ruby 2.2.0dev (2014-04-07 t=
runk 45528) =5Bi386-mswin32=5F100=5D * Backport: 2.0.0: UNKNOWN, 2.1: UNK=
NOWN----------------------------------------**C:/=E3=81=A6=E3=81=99=E3=81=
=A8/=46ILE.rb:**=7E=7E=7E=23 encoding: UT=46-8puts =22Encoding.find 'file=
system': =23=7BEncoding.find('filesystem').inspect=7D=22puts =22Encoding.=
find 'locale': =23=7BEncoding.find('locale').inspect=7D=22puts =22Encodin=
g.default internal: =23=7BEncoding.default=5Finternal.inspect=7D=22puts =22=
Encoding.default external: =23=7BEncoding.default=5Fexternal.inspect=7D=22=
puts =22Encoding.locale=5Fcharmap: =23=7BEncoding.locale=5Fcharmap.inspec=
t=7D=22puts =22=5F=5F=46ILE=5F=5F: =23=7B=5F=5F=46ILE=5F=5F.encoding.insp=
ect=7D=22puts =22'foobar': =23=7B'foobar'.encoding.inspect=7D=22=7E=7E=7E=
**C:/=46ILE.rb:**=7E=7E=7E=23 encoding: UT=46-8puts =22Encoding.find 'fil=
esystem': =23=7BEncoding.find('filesystem').inspect=7D=22puts =22Encoding=
.find 'locale': =23=7BEncoding.find('locale').inspect=7D=22puts =22Encodi=
ng.default internal: =23=7BEncoding.default=5Finternal.inspect=7D=22puts =
=22Encoding.default external: =23=7BEncoding.default=5Fexternal.inspect=7D=
=22puts =22Encoding.locale=5Fcharmap: =23=7BEncoding.locale=5Fcharmap.ins=
pect=7D=22puts =22=5F=5F=46ILE=5F=5F: =23=7B=5F=5F=46ILE=5F=5F.encoding.i=
nspect=7D=22puts =22'foobar': =23=7B'foobar'.encoding.inspect=7D=22puts =22=
=22puts =22Loading C:/=E3=81=A6=E3=81=99=E3=81=A8/=46ILE.rb ...=22require=
 =22C:/=E3=81=A6=E3=81=99=E3=81=A8/=46ILE.rb=22=7E=7E=7E**Results:**=21=5B=
=5D(media-20140407.png)=7E=7E=7Ec:=5Cruby-220=5Cusr=5Cbin>ruby =22C:=5C=46=
ILE.rb=22Encoding.find 'filesystem': =23Encoding.find 'locale': =23Encodi=
ng.default internal: nilEncoding.default external: =23Encoding.locale=5Fc=
harmap: =22CP437=22=5F=5F=46ILE=5F=5F: =23'foobar': =23Loading C:/=3F=3F=3F=
/=46ILE.rb ...Encoding.find 'filesystem': =23Encoding.find 'locale': =23E=
ncoding.default internal: nilEncoding.default external: =23Encoding.local=
e=5Fcharmap: =22CP437=22=5F=5F=46ILE=5F=5F: =23'foobar': =23c:=5Cruby-220=
=5Cusr=5Cbin>=7E=7E=7ENow, lets see how this affects Dir.glob:Test scenar=
io - a folder structure like this:=7E=7E=7EC:/test/C:/test/foo/C:/test/=E3=
=81=A6=E3=81=99=E3=81=A8/=7E=7E=7E**C:/=46ILE.rb**=7E=7E=7E=23 encoding: =
UT=46-8puts =22Encoding.find 'filesystem': =23=7BEncoding.find('filesyste=
m').inspect=7D=22puts =22Encoding.find 'locale': =23=7BEncoding.find('loc=
ale').inspect=7D=22puts =22Encoding.default internal: =23=7BEncoding.defa=
ult=5Finternal.inspect=7D=22puts =22Encoding.default external: =23=7BEnco=
ding.default=5Fexternal.inspect=7D=22puts =22Encoding.locale=5Fcharmap: =23=
=7BEncoding.locale=5Fcharmap.inspect=7D=22puts =22=5F=5F=46ILE=5F=5F: =23=
=7B=5F=5F=46ILE=5F=5F.encoding.inspect=7D=22puts =22'foobar': =23=7B'foob=
ar'.encoding.inspect=7D=22puts =22=22pattern =3D =46ile.join(=46ile.dirna=
me(=5F=5F=46ILE=5F=5F), =22test=22, =22*=22)puts =22pattern.encoding: =23=
=7Bpattern.encoding.inspect=7D=22result =3D Dir.glob(pattern)p resultp re=
sult.map =7B =7Cfile=7C file.encoding =7Dputs =22=22puts =22force encodin=
g:=22pattern.force=5Fencoding(=22UT=46-8=22)result =3D Dir.glob(pattern)p=
 resultp result.map =7B =7Cfile=7C file.encoding =7D=7E=7E=7E**Result:**=7E=
=7E=7Ec:=5Cruby-220=5Cusr=5Cbin>ruby =22C:=5C=46ILE.rb=22Encoding.find 'f=
ilesystem': =23Encoding.find 'locale': =23Encoding.default internal: nilE=
ncoding.default external: =23Encoding.locale=5Fcharmap: =22CP437=22=5F=5F=
=46ILE=5F=5F: =23'foobar': =23pattern.encoding: =23=5B=22C:/test/foo=22, =
=22C:/test/=3F=3F=3F=22=5D=5B=23, =23=5Dforce encoding:=5B=22C:/test/foo=22=
, =22C:/test/=5Cu3066=5Cu3059=5Cu3068=22=5D=5B=23, =23=5Dc:=5Cruby-220=5C=
usr=5Cbin>=7E=7E=7EObserve how when Dir.glob is fed a string based on =5F=
=5F=46ILE=5F=5F it will return strings in the same encoding, even though =
the string should include Unicode characters. The Unicode characters are =
replaced by question marks. (Actual ASCII bytes for question mark: 63)Jus=
t by forcing the input string to UT=46-8 will make Dir.glob return the ex=
pected strings with correct Unicode characters.I'm unsure of where the bu=
g lies, but in terms of what I expected I would not have expected =5F=5F=46=
ILE=5F=5F to return different encoding depending on the executing file co=
ntaining Unicode characters. All files have been marked as UT=46-8 in the=
 file header.---=46iles--------------------------------media-20140407.png=
 (83.1 KB)-- https://bugs.ruby-lang.org/     
--5673864f_70a64e2a_16c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<html><body><div>Joseph Jones liked your message with <a href=3D=22http:/=
/bxr.io/PBIqo=22>Boxer</a>.</div><br/><br/><div><div class=3D=22quote=22>=
On December 4, 2015 at 06:35:43 MST, thomas=40thomthom.net wrote:<br/><bl=
ockquote type=3D=22cite=22 style=3D=22border-left-style:solid;border-widt=
h:1px;margin-left:0px;padding-left:10px;=22>Issue =239713 has been update=
d by Thomas Thomassen.<br /><br /><br />Revisiting this issue again. Is t=
here a resolution to what can be done to improve this and still satisfy c=
ompatibility concerns=3F<br /><br />-------------------------------------=
---<br />Bug =239713: =5F=5F=46ILE=5F=5F return unexpected encoding - bre=
aks Dir.glob<br />https://bugs.ruby-lang.org/issues/9713=23change-55235<b=
r /><br />* Author: Thomas Thomassen<br />* Status: Assigned<br />* Prior=
ity: Normal<br />* Assignee: cruby-windows<br />* ruby -v: ruby 2.2.0dev =
(2014-04-07 trunk 45528) =5Bi386-mswin32=5F100=5D	<br />* Backport: 2.0.0=
: UNKNOWN, 2.1: UNKNOWN<br />----------------------------------------<br =
/>**C:/=E3=81=A6=E3=81=99=E3=81=A8/=46ILE.rb:**<br /><br />=7E=7E=7E<br /=
>=23 encoding: UT=46-8<br />puts =22Encoding.find 'filesystem': =23=7BEnc=
oding.find('filesystem').inspect=7D=22<br />puts =22Encoding.find 'locale=
': =23=7BEncoding.find('locale').inspect=7D=22<br />puts =22Encoding.defa=
ult internal: =23=7BEncoding.default=5Finternal.inspect=7D=22<br />puts =22=
Encoding.default external: =23=7BEncoding.default=5Fexternal.inspect=7D=22=
<br />puts =22Encoding.locale=5Fcharmap: =23=7BEncoding.locale=5Fcharmap.=
inspect=7D=22<br />puts =22=5F=5F=46ILE=5F=5F: =23=7B=5F=5F=46ILE=5F=5F.e=
ncoding.inspect=7D=22<br />puts =22'foobar': =23=7B'foobar'.encoding.insp=
ect=7D=22<br />=7E=7E=7E<br /><br />**C:/=46ILE.rb:**<br />=7E=7E=7E<br /=
>=23 encoding: UT=46-8<br />puts =22Encoding.find 'filesystem': =23=7BEnc=
oding.find('filesystem').inspect=7D=22<br />puts =22Encoding.find 'locale=
': =23=7BEncoding.find('locale').inspect=7D=22<br />puts =22Encoding.defa=
ult internal: =23=7BEncoding.default=5Finternal.inspect=7D=22<br />puts =22=
Encoding.default external: =23=7BEncoding.default=5Fexternal.inspect=7D=22=
<br />puts =22Encoding.locale=5Fcharmap: =23=7BEncoding.locale=5Fcharmap.=
inspect=7D=22<br />puts =22=5F=5F=46ILE=5F=5F: =23=7B=5F=5F=46ILE=5F=5F.e=
ncoding.inspect=7D=22<br />puts =22'foobar': =23=7B'foobar'.encoding.insp=
ect=7D=22<br /><br />puts =22=22<br />puts =22Loading C:/=E3=81=A6=E3=81=99=
=E3=81=A8/=46ILE.rb ...=22<br />require =22C:/=E3=81=A6=E3=81=99=E3=81=A8=
/=46ILE.rb=22<br />=7E=7E=7E<br /><br />**Results:**<br /><br />=21=5B=5D=
(media-20140407.png)<br /><br />=7E=7E=7E<br />c:=5Cruby-220=5Cusr=5Cbin>=
ruby =22C:=5C=46ILE.rb=22<br />Encoding.find 'filesystem': =23<Encoding:W=
indows-1252><br />Encoding.find 'locale': =23<Encoding:IBM437><br />Encod=
ing.default internal: nil<br />Encoding.default external: =23<Encoding:IB=
M437><br />Encoding.locale=5Fcharmap: =22CP437=22<br />=5F=5F=46ILE=5F=5F=
: =23<Encoding:IBM437><br />'foobar': =23<Encoding:UT=46-8><br /><br />Lo=
ading C:/=3F=3F=3F/=46ILE.rb ...<br />Encoding.find 'filesystem': =23<Enc=
oding:Windows-1252><br />Encoding.find 'locale': =23<Encoding:IBM437><br =
/>Encoding.default internal: nil<br />Encoding.default external: =23<Enco=
ding:IBM437><br />Encoding.locale=5Fcharmap: =22CP437=22<br />=5F=5F=46IL=
E=5F=5F: =23<Encoding:UT=46-8><br />'foobar': =23<Encoding:UT=46-8><br />=
<br />c:=5Cruby-220=5Cusr=5Cbin><br />=7E=7E=7E<br /><br />Now, lets see =
how this affects Dir.glob:<br /><br />Test scenario - a folder structure =
like this:<br />=7E=7E=7E<br />C:/test/<br />C:/test/foo/<br />C:/test/=E3=
=81=A6=E3=81=99=E3=81=A8/<br />=7E=7E=7E<br /><br />**C:/=46ILE.rb**<br /=
><br />=7E=7E=7E<br />=23 encoding: UT=46-8<br />puts =22Encoding.find 'f=
ilesystem': =23=7BEncoding.find('filesystem').inspect=7D=22<br />puts =22=
Encoding.find 'locale': =23=7BEncoding.find('locale').inspect=7D=22<br />=
puts =22Encoding.default internal: =23=7BEncoding.default=5Finternal.insp=
ect=7D=22<br />puts =22Encoding.default external: =23=7BEncoding.default=5F=
external.inspect=7D=22<br />puts =22Encoding.locale=5Fcharmap: =23=7BEnco=
ding.locale=5Fcharmap.inspect=7D=22<br />puts =22=5F=5F=46ILE=5F=5F: =23=7B=
=5F=5F=46ILE=5F=5F.encoding.inspect=7D=22<br />puts =22'foobar': =23=7B'f=
oobar'.encoding.inspect=7D=22<br /><br />puts =22=22<br />pattern =3D =46=
ile.join(=46ile.dirname(=5F=5F=46ILE=5F=5F), =22test=22, =22*=22)<br />pu=
ts =22pattern.encoding: =23=7Bpattern.encoding.inspect=7D=22<br />result =
=3D Dir.glob(pattern)<br />p result<br />p result.map =7B =7Cfile=7C file=
.encoding =7D<br /><br />puts =22=22<br />puts =22force encoding:=22<br /=
>pattern.force=5Fencoding(=22UT=46-8=22)<br />result =3D Dir.glob(pattern=
)<br />p result<br />p result.map =7B =7Cfile=7C file.encoding =7D<br />=7E=
=7E=7E<br /><br />**Result:**<br /><br />=7E=7E=7E<br />c:=5Cruby-220=5Cu=
sr=5Cbin>ruby =22C:=5C=46ILE.rb=22<br />Encoding.find 'filesystem': =23<E=
ncoding:Windows-1252><br />Encoding.find 'locale': =23<Encoding:IBM437><b=
r />Encoding.default internal: nil<br />Encoding.default external: =23<En=
coding:IBM437><br />Encoding.locale=5Fcharmap: =22CP437=22<br />=5F=5F=46=
ILE=5F=5F: =23<Encoding:IBM437><br />'foobar': =23<Encoding:UT=46-8><br /=
><br />pattern.encoding: =23<Encoding:IBM437><br />=5B=22C:/test/foo=22, =
=22C:/test/=3F=3F=3F=22=5D<br />=5B=23<Encoding:IBM437>, =23<Encoding:IBM=
437>=5D<br /><br />force encoding:<br />=5B=22C:/test/foo=22, =22C:/test/=
=5Cu3066=5Cu3059=5Cu3068=22=5D<br />=5B=23<Encoding:UT=46-8>, =23<Encodin=
g:UT=46-8>=5D<br /><br />c:=5Cruby-220=5Cusr=5Cbin><br />=7E=7E=7E<br /><=
br />Observe how when Dir.glob is fed a string based on =5F=5F=46ILE=5F=5F=
 it will return strings in the same encoding, even though the string shou=
ld include Unicode characters. The Unicode characters are replaced by que=
stion marks. (Actual ASCII bytes for question mark: 63)<br />Just by forc=
ing the input string to UT=46-8 will make Dir.glob return the expected st=
rings with correct Unicode characters.<br /><br />I'm unsure of where the=
 bug lies, but in terms of what I expected I would not have expected =5F=5F=
=46ILE=5F=5F to return different encoding depending on the executing file=
 containing Unicode characters. All files have been marked as UT=46-8 in =
the file header.<br /><br />---=46iles--------------------------------<br=
 />media-20140407.png (83.1 KB)<br /><br /><br />-- <br />https://bugs.ru=
by-lang.org/<br /></blockquote></div></div></body></html>
--5673864f_70a64e2a_16c--