From: Tom Link <micathom@...>
Date: 2009-02-17T00:33:12+09:00
Subject: Re: invalid byte sequence in US-ASCII (ArgumentError)

> Result in Ruby 1.8:
> * My stuff works fine on Linux/Unix. Somebody else runs
>   the script on Windows, the script corrupts data because
>   Windows does line ending conversion.
>
> Result in Ruby 1.9:
> * On the first run on my Linux machine, I get an EncodingError.
>    I fix the problem by specifying the "b" flag on open. Done.

Actually, this is a point I have never quite understood. Why does only
the windows version convert line endings? Is it out of question that
somebody could want to process a text file created under windows on a
linux box or virtual machine? Regular expressions that check only for
\n but not \r won't work then. Now you could of course take the stance
that you simply have to check for \r too, but then why automatically
convert line separators under windows? Or did I miss something
obvious?

This is also the reason why I think opening text files as binary isn't
really a solution. It leads to either convoluted regexps or non-
portable code. (Unless I missed something obvious, which is quite
possible.)

I personally find it somewhat confusing having to juggle with
different encodings. IMHO it would have been preferable to define a
fixed internal encoding (uft16 or whatever) and to transcode every
string that is known to be text and identifiers to that canonical/
uniform encoding and to deal with everything else as a sequence of
bytes.

BTW I recently skimmed through the python3000 user guide. From what I
understand, they seem to distinguish between strings as (binary) data
and strings as text (encoded as utf).