From: Joshua Ballanco <jballanc@...>
Date: 2010-07-22T18:02:01+09:00
Subject: [ruby-core:31441] Re: [Feature #3595] Theres no encoding to differentiate a stream of Binary data from an 8-Bit ASCII string

On Jul 22, 2010, at 1:46 AM, Joshua Ballanco wrote:

> On Jul 21, 2010, at 11:41 AM, Dreamcat Four wrote:
> 
>> Issue #3595 has been updated by Dreamcat Four.
>> 
>> 
>> And lets face it, if Encoding::BINARY were its own seperate encoding that shouldnt really hurt anybody. Given the definition of what binary data is. If a ruby programmer wants to continue using 8-Bit Ascii strings that shouldn't get broken.
> 
> Oh, one more thing...there is no such thing as an ASCII-8BIT string that isn't data. The high bit values don't map to any characters. There are derivative encodings that use these values for other characters, but they have their own names like "ISO8859_2". If someone wants an ASCII string, they'll use US-ASCII. I'd think it's safe to assume that ASCII-8BIT always implies binary data.

To illustrate the point:

> irb1.9
irb(main):001:0> "\xAA".force_encoding('ASCII-8BIT').encode('UTF-8')
Encoding::UndefinedConversionError: "\xAA" from ASCII-8BIT to UTF-8
	from (irb):1:in `encode'
	from (irb):1
	from /usr/local/bin/irb1.9:12:in `<main>'
irb(main):002:0> "\xAA".force_encoding('US-ASCII').encode('UTF-8')
Encoding::InvalidByteSequenceError: "\xAA" on US-ASCII
	from (irb):2:in `encode'
	from (irb):2
	from /usr/local/bin/irb1.9:12:in `<main>'
irb(main):003:0> "\xAA".force_encoding('ISO8859-2').encode('UTF-8')
=> "��"

Notice how, in the first case you get an "UndefinedConversionError"? That's because there's not a character that corresponds to this value in ASCII-8BIT encoding. In the second case, you get an "InvalidByteSequenceError" because you've put an 8 bit value into an ASCII string, which is only valid for 7 bits. Finally, using one of the "Extended ASCII" encodings, you can successfully convert to Unicode because now this value, 0xAA, actually represents a character. So, in other words, ASCII-8BIT is already what you're looking for.