From: Stefan Lang <perfectly.normal.hacker@...>
Date: 2009-02-17T00:57:50+09:00
Subject: Re: invalid byte sequence in US-ASCII (ArgumentError)

2009/2/16 Tom Link <micathom@gmail.com>:
>> Result in Ruby 1.8:
>> * My stuff works fine on Linux/Unix. Somebody else runs
>>   the script on Windows, the script corrupts data because
>>   Windows does line ending conversion.
>>
>> Result in Ruby 1.9:
>> * On the first run on my Linux machine, I get an EncodingError.
>>    I fix the problem by specifying the "b" flag on open. Done.
>
> Actually, this is a point I have never quite understood. Why does only
> the windows version convert line endings? Is it out of question that
> somebody could want to process a text file created under windows on a
> linux box or virtual machine? Regular expressions that check only for
> \n but not \r won't work then. Now you could of course take the stance
> that you simply have to check for \r too, but then why automatically
> convert line separators under windows? Or did I miss something
> obvious?

It's the underlying C API that does the line ending conversion.
Ruby inherited that behavior.

> This is also the reason why I think opening text files as binary isn't
> really a solution. It leads to either convoluted regexps or non-
> portable code. (Unless I missed something obvious, which is quite
> possible.)
>
> I personally find it somewhat confusing having to juggle with
> different encodings. IMHO it would have been preferable to define a
> fixed internal encoding (uft16 or whatever) and to transcode every
> string that is known to be text and identifiers to that canonical/
> uniform encoding and to deal with everything else as a sequence of
> bytes.

Ruby does that when you set the internal encoding with
Encoding.default_internal=

> BTW I recently skimmed through the python3000 user guide. From what I
> understand, they seem to distinguish between strings as (binary) data
> and strings as text (encoded as utf).

There were many and long discussions about the encoding API,
mostly on Ruby core. If you search the archives you can find
why the current API is how it is.

IIRC, these were important issues:

* We don't have a single internal string encoding (like Java and Python)
  because there are many Ruby users, especially Asian, that still
  have to work with legacy encodings for which a lossless Unicode
  round-trip is not possible. They'd be forced to use the
  binary API.

* Because Ruby already has a rich String API, and because it simplifies
  porting of 1.8 code, there is no separate data type for binary strings.

Stefan