From: Yukihiro Matsumoto <matz@...>
Date: 2008-05-20T22:51:38+09:00
Subject: Re: ruby 1.9 hates you and me and the encodings we rode in on so just 	get  used to it.

Hi,

In message "Re: ruby 1.9 hates you and me and the encodings we rode in on so just 	get  used to it."
    on Tue, 20 May 2008 19:50:08 +0900, DJ Jazzy Linefeed <john.d.perkins@gmail.com> writes:

|> Regular expression operation does not work fine on broken strings.  It
|> seems that you specify utf-8 for your locale, yet the content of
|> reading file is not.  If you know the encoding of the content, say
|> iso-8859-1, you can open it with the explicit encoding:
|>
|>   x = File.open(path, "r:iso-8859-1")
|>
|> if not, you can say it
|>
|>   x = File.open(path, "r:ascii-8bit")
|>
|> unless the file content is non ASCII like UTF-16.

|It makes no sense, Matz.
|
|I don't get to know what the encoding is before hand, that's just it -
|there may be every encoding. I just deal with a pile of files, I
|think...

Since today's OSes do not provide encoding information for files, you
HAVE TO know the encoding of the files if you want to handle them
correctly, unfortunately.  That's life, no matter how you expect.

If you don't need exact encoding handling, and know the file is mostly
ASCII, use ASCII-8BIT for encoding.  It works most of the cases.

							matz.