From: DJ Jazzy Linefeed <john.d.perkins@...>
Date: 2008-05-20T19:50:08+09:00
Subject: Re: ruby 1.9 hates you and me and the encodings we rode in on so just 	get  used to it.

On May 19, 1:53 pm, Yukihiro Matsumoto <m...@ruby-lang.org> wrote:
> Hi,
>
> In message "Re: ruby 1.9 hates you and me and the encodings we rode in on so just get      used to it."
>     on Sat, 17 May 2008 06:10:05 +0900, DJ Jazzy Linefeed <john.d.perk...@gmail.com> writes:
> |
> |def prep_file(path)
> |
> |  ret = ''
> |
> |  x = File.open(path)
> |
> |  x.lines.each do |l|
> |    l.gsub!('\n', ' ')
> |    ret << l
> |  end
> |
> |  puts ret
> |
> |end
> |...
> |compare.rb:64:in `gsub': broken UTF-8 string (ArgumentError)
> |       from compare.rb:64:in `block in prep_file'
> |       from compare.rb:63:in `each_line'
> |       from compare.rb:63:in `call'
> |       from compare.rb:63:in `each'
> |       from compare.rb:63:in `prep_file'
> |       from compare.rb:144:in `<main>'
>
> Regular expression operation does not work fine on broken strings.  It
> seems that you specify utf-8 for your locale, yet the content of
> reading file is not.  If you know the encoding of the content, say
> iso-8859-1, you can open it with the explicit encoding:
>
>   x = File.open(path, "r:iso-8859-1")
>
> if not, you can say it
>
>   x = File.open(path, "r:ascii-8bit")
>
> unless the file content is non ASCII like UTF-16.
>
>                                                         matz.

It makes no sense, Matz.

I don't get to know what the encoding is before hand, that's just it -
there may be every encoding. I just deal with a pile of files, I
think...

case encoding
when ascii-8bit
# ... 12 lines of boilerplate encoder extraction
when iso-8859
# ... 12 more?
end

is a case to avoid

should move forward or die, yes, but not
and die.

Ruby is accessible to the masses because you don't have to understand
encodings, not in spite of the fact.
Food for thought.
-DJ J\N