From: Aaron Patterson <aaron@...>
Date: 2010-11-04T01:02:33+09:00
Subject: [ruby-core:33037] Re: [Ruby 1.9-Bug#4010] YAML fails to roundtrip non ASCII String


--24zk1gE8NUlDmwG9
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Nov 03, 2010 at 11:36:35AM +0900, Heesob Park wrote:
> 2010/11/3 Aaron Patterson <aaron@tenderlovemaking.com>:
> > On Tue, Nov 02, 2010 at 09:58:27PM +0900, Heesob Park wrote:
> >> Issue #4010 has been updated by Heesob Park.
> >>
> >>
> >> The same result with psych.
> >>
> >> $ ruby -v -ryaml -e 'YAML::ENGINE.yamler =3D "psych"; s=3D"=ED=95=9C=
=EA=B8=80";pYAML.load(YAML.dump(s))=3D=3Ds'
> >> ruby 1.9.3dev (2010-11-02 trunk 29667) [i686-linux]
> >> /usr/local/lib/ruby/1.9.1/psych/deprecated.rb:79: warning: method rede=
fined; discarding old to_yaml_properties
> >> /usr/local/lib/ruby/1.9.1/syck/rubytypes.rb:13: warning: previous defi=
nition of to_yaml_properties was here
> >> false
> >>
> >> FYI the current encoding is 'EUC-KR'.
> >> I know it works when the encoding is 'UTF-8'.
> >
> > I think the problem is that your default_internal encoding isn't being
> > set. =C2=A0YAML is usually stored as UTF-8, but psych will automatically
> > transcode the string to whatever your default_internal encoding is set
> > to.
> >
> > Maybe this script will help illustrate the problem:
> >
> >
> > =C2=A0 =C2=A0# coding: utf-8
> >
> > =C2=A0 =C2=A0require 'psych'
> >
> > =C2=A0 =C2=A0eucjp =3D "=E3=81=93=E3=82=93=E3=81=AB=E3=81=A1=E3=81=AF=
=EF=BC=81".encode('EUC-JP')
> > =C2=A0 =C2=A0string =3D Psych.load(Psych.dump(eucjp))
> >
> > =C2=A0 =C2=A0p string.encoding # =3D> #<Encoding:UTF-8>
> > =C2=A0 =C2=A0p eucjp =3D=3D string # =3D> false
> >
> > =C2=A0 =C2=A0Encoding.default_internal =3D 'EUC-JP'
> >
> > =C2=A0 =C2=A0string =3D Psych.load(Psych.dump(eucjp))
> > =C2=A0 =C2=A0p string.encoding # =3D> #<Encoding:EUC-JP>
> > =C2=A0 =C2=A0p eucjp =3D=3D string # =3D> true
> >
> > Try running your Ruby like this:
> >
> > =C2=A0$ ruby -EEUC-KR:EUC-KR -v -ryaml -e 'YAML::ENGINE.yamler =3D "psy=
ch"; s=3D"=ED=95=9C=EA=B8=80";pYAML.load(YAML.dump(s))=3D=3Ds'
> Here is the result
>=20
> $ ruby -EEUC-KR:EUC-KR -v -ryaml -e 'YAML::ENGINE.yamler =3D "psych";
> s=3D"=ED=95=9C=EA=B8=80";p YAML.load(YAML.dump(s))=3D=3Ds'
> ruby 1.9.3dev (2010-11-02 trunk 29667) [i686-linux]
> true
>=20
> Did you mean it is not a bug and I must specify the default external
> and internal character encodings?

Yes.  YAML is stored as UTF-8 or 16 (possibly 32 as well, depending on
the spec you choose).  Strings you pull out of Psych will have an
encoding that matches the source document.

> $ irb
> irb(main):001:0> Encoding.default_external
> =3D> #<Encoding:EUC-KR>
> irb(main):002:0> Encoding.default_internal
> =3D> nil
> Why ruby cannot detect Encoding.default_internal ?

"default_external" indicates the encoding that files on disk probably
have.  A good default is the OS setting.

magic comments indicate the encoding of string literals within that file.

"default_internal" indicates the encoding that you want strings internal
to your programs to have.  Making that decision is not so easy.  Magic
comments cannot be used because there can exist many magic comments in
your programs.

"default_internal" should be used by things like database adapters (or
in this case YAML parsers) where the encoding of the external entity may
differ from the encoding the user wants to use.  The external entity
should transcode to the user's "default_internal" setting.  Because of
this logic, there exists another problem with setting "default_internal"
to something for the user.

Here is an example of what *could* happen if default_internal was
automatically set:

"default_internal" is automatically set to something, say Shift-JIS, and
you load a YAML file.  You know that your YAML file is stored as UTF-8,
and yet when you output the encoding from your program, you're surprised
to see it report Shift-JIS!  You wanted the original encoding of the file,
but now you must call encode() to get it back to UTF-8.  Even worse,
because you went from UTF-8 =3D> Shift-JIS =3D> UTF-8, now there may be data
loss due to encoding round trip problems.

Hope that helps!

--=20
Aaron Patterson
http://tenderlovemaking.com/

--24zk1gE8NUlDmwG9
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)

iQEcBAEBAgAGBQJM0YeTAAoJEJUxcLy0/6/GbVkH/1fy1/VvkQUJQkj6JJ0U2OWB
n1qn1eTZ+ZlQSxIXPSA5JDMrCES1/GuqjIQ3g+jdHLhIZPqpS8BFX6sdtQsVZLHy
g6p12dFd90VSi9yxui/2681ndKpE/ezAnLGkRsenjEF3XpA4lMg+atL/eQ6JZXfM
FnT/+2YRMtc6M0IeeH24nxAA4Ln1UJTSbxQ3KMKONx2XIbD4U8itXevQOIJxCmR8
dkkQpZ23hsmOBPt1QVJKCL34LbtwV9w3mROniRhjrs4PYNzPJJV40Q1t7DPjX+RP
u8t7cVCwxdC1HziA/3S+LDlqcwvNSVh25VqIeRyXXbE1vo/YrIFn6VNI5nyotD0=
=fw1q
-----END PGP SIGNATURE-----

--24zk1gE8NUlDmwG9--