From: Aaron Patterson <tenderlove@...>
Date: 2012-05-04T00:44:28+09:00
Subject: Re: Psych dumping in binary


--f2QGlHpHGjS2mn6Y
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Thu, May 03, 2012 at 06:41:57AM +0900, Jim Hranicky wrote:
> On 05/02/2012 05:10 PM, Jeremy Kemper wrote:
>=20
> > If you can, try US-ASCII encoding for 7-bit clean ASCII. Psych will
> > dump that as you expect.
>=20
> Well, I'd rather avoid this when reading and parsing syslog
> lines:=20
>=20
>   ruby -Eus-ascii -e 'x =3D "foo"; x.force_encoding("US-ASCII"); puts x.e=
ncoding; x +=3D "\xf0\xff";   x.force_encoding("US-ASCII"); puts x.match(/f=
oo/); puts x' | m
>   -e:1:in `match': invalid byte sequence in US-ASCII (ArgumentError)
>           from -e:1:in `match'
>           from -e:1:in `<main>'
>   US-ASCII
>=20
> You never know what'll be in there, and I'd rather not have to run=20
> force_encoding on every processed line.=20
>=20
> If there's a better way to handle strings with naughty characters
> I'd be grateful for pointers.=20

For now, you can use String#ascii_only?

    def tag(string)
      string.force_encoding("US-ASCII") if string.ascii_only?
    end

    x =3D "foo"
    tag(x)
    puts x.encoding
    x +=3D "\xf0\xff"
    tag(x)
    puts x.match(/foo/)
    puts x

I will push this up to Psych so that ascii only ASCII-8BIT strings are
dumped as UTF-8.

--=20
Aaron Patterson
http://tenderlovemaking.com/

--f2QGlHpHGjS2mn6Y
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)

iQEcBAEBAgAGBQJPoqfXAAoJEJUxcLy0/6/G+rgH/1dapJ5r9LFkLbU/NBeOg2+8
xFSDmfy041nO2rwSrR4oFPKLpkM2L5J3TIr9pyopZ9zkYwx9z+dIUXDxeghjx+aa
fedSZhZpQVLxpy+DpBchfk3GbEZb2DOyQfo3cTRI3HCSXUHGrNXPe2GCSpJBzllV
RHIBhjRMDKYCgBS0dV9MQvv4QsgwoXRwwolVKhYU4s5OpBt4KDU2LPyWFoZ/xBI6
klb9bq2IZGPivLrAIa3DCf/+Kwn6ey/ruBXhnNXHIQgwvRbxfWXFGtBqxdI61V2O
v2qQaosEgDS+eZktO7I4dbeg2wAStPyJw4UG2XdBjrwvLb+bTgk9e63rSDCN9d4=
=dcXM
-----END PGP SIGNATURE-----

--f2QGlHpHGjS2mn6Y--