From: Run Paint Run Run <runrun@...>
Date: 2010-08-31T09:31:55+09:00
Subject: [ruby-core:31963] Re: not use system for default encoding

>> Well, firstly, that���s a pretty odd encoding to use. In general, you���ll have
>> a far easier time if you use UTF-8 for everything, and legacy encodings only
>> when necessary or, possibly, writing in a CJK script. In any case, even if
>> you continue using that encoding, the external encoding only needs to be
>> specified explicitly if the files contain non-ASCII characters. And if they
>> do, and interoperability is your goal, then why are you using IBM437 in the
>> first place?

> So are you saying that regardless of what
> Encoding.default_external is set to, if I read a file that only has ascii
> characters, it will always show up as the same string, regardless?
> Thanks!

Almost always. :-) Specifically, the encodings both need to be
ASCII-compatible, meaning their repertoire should consist of a superset of
ASCII. You can confirm with the Encoding#ascii_compatible? predicate, e.g.:

  >> Encoding::IBM437.ascii_compatible?  #=> true
  >> Encoding::UTF_8.ascii_compatible?   #=> true

The implication is that a file containing only ASCII is valid in any of these
encodings, i.e. a given ASCII character is represented by the same byte
sequence when transcoded to any ASCII-compatible encoding.

In practise, most of the encodings supported by Ruby are ASCII-compatible, with
the main exceptions being UTF-16 and UTF-32, in both their big- and
little-endian variants.

  Encoding.list.reject(&:ascii_compatible?)
  #=> [#<Encoding:UTF-16BE>, <Encoding:UTF-16LE>, #<Encoding:UTF-32BE>,
  #<Encoding:UTF-32LE>, <Encoding:ISO-2022-JP (dummy)>,
  #<Encoding:ISO-2022-JP-2 (dummy)>, <Encoding:CP50220 (dummy)>,
  #<Encoding:CP50221 (dummy)>, #<Encoding:UTF-7 (dummy)>,
  #<Encoding:ISO-2022-JP-KDDI (dummy)>]

For example, the first three encodings below are ASCII-compatible, so an
ASCII-only string is represented by the same set of bytes; the last three are
not ASCII-compatible, so the byte sequences differ:

  >> s="y%$;"
  >> pp %w{ibm437 utf-8 ascii utf-16le utf-16be utf-32le}.
  >>    map{|e| [e, *s.encode(e).bytes]}
  [
    ["ibm437", 121, 37, 36, 59],
    ["utf-8", 121, 37, 36,59],
    ["ascii", 121, 37, 36, 59],
    ["utf-16le", 121, 0, 37, 0, 36, 0, 59, 0],
    ["utf-16be", 0, 121, 0, 37, 0, 36, 0, 59],
    ["utf-32le", 121, 0, 0, 0, 37, 0, 0, 0, 36, 0, 0, 0, 59, 0, 0, 0]
  ]