From: Benoit Daloze <eregontp@...>
Date: 2009-12-27T20:48:51+09:00
Subject: Re: ruby 1.9 hates you and me and the encodings we rode in on so just 	get used to it.

--0016e65aef4838dd43047bb45fbb
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

2009/12/27 Brian Candler <b.candler@pobox.com>

> DJ Jazzy Linefeed wrote:
> > compare.rb:64:in `gsub': broken UTF-8 string (ArgumentError)
>
> Yep. Ruby 1.9 will raise exceptions in all sorts of odd places,
> dependent on both the tagged encoding of the string *and* its content at
> that point in time.
>
> I got as far as recording 200 behaviours of String in ruby 1.9 before I
> gave up:
> http://github.com/candlerb/string19/blob/master/string19.rb
>
> The solution I use is simple: stick to ruby 1.8.x. When that branch
> dies, perhaps reia will be ready. If not I'll move to something else.
>
> IMO, both python 3 and erlang have got the right idea when it comes to
> handling UTF8.
> --
> Posted via http://www.ruby-forum.com/.
>
>
Hi,

I got this kind of problem yesterday too.

While taking some file names with Dir#[], I got some special results.

I was searching for "bad" file names, I mean file names with =C3=A9,=C3=AA =
or
whatever. When I print the String given in the block directly, no problem.

But then I come with things like:
/Users/benoitdaloze/Library/GlestGame/data/lang/espan>=CC=83<ol.lng

(The ~ is separated from the n and then is not =C3=B1). The Regexp is actin=
g like
it is 2 different characters. How to handle that easily? I tried to change
the script encoding in MacRoman, but it produced an error of bad encoding
not matching UTF-8.

as output of this script (which is then not able to rename any wrong file,
because tr! seem to not work either on name) :

path =3D ARGV[0] || "/"

ALLOWED_CHARS =3D "A-Za-z0-9 %#:$@?!=3D+~&|'()\\[\\]{}.,\r_-"

Dir["#{File.expand_path(path)}/**/*"].each { |f|
    name =3D File.basename(f)
    unless name =3D~ /^[#{ALLOWED_CHARS}]+$/
        puts File.dirname(f) + '/' + name.gsub(/([^#{ALLOWED_CHARS}]+)/,
">\\1<")

        if name.tr!('=C3=A9=C3=A8=C3=AA', 'e') =3D~ /^[#{ALLOWED_CHARS}]+$/=
 # Here it is not
complete, it is just a test, but it doesn't work even for 'fil=C3=A9name'
            File.rename(f, File.dirname(f) + '/' + name)
            puts "\trenamed in #{name}"
            break
        end
    end
}

--0016e65aef4838dd43047bb45fbb--