[ruby-core:55984] [ruby-trunk - Bug #8630][Open] Transcoding high-bit bytes from ASCII-8BIT to a text encoding should be :invalid, not :undef

Issue #8630 has been reported by headius (Charles Nutter).

----------------------------------------
Bug #8630: Transcoding high-bit bytes from ASCII-8BIT to a text encoding should be :invalid, not :undef
https://bugs.ruby-lang.org/issues/8630

Author: headius (Charles Nutter)
Status: Open
Priority: Normal
Assignee: 
Category: 
Target version: 
ruby -v: 2.0.0
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN


When transcoding from ASCII-8BIT (BINARY) to a text encoding (e.g. UTF-8), MRI will raise an error for high-bit bytes:

"\xC3".encode("utf-8", "binary") # => Encoding::UndefinedConversionError

This can be disabled by passing :undef => :replace as an option to the encode call.

I believe that "undef" is the wrong treatment for this error. Undef means that the input character has no representation in the target encoding. In this case, the error is raised because only US-ASCII range of bytes are *valid* for transcoding, so the transcoding of high-bit bytes is by definition *invalid*, not undefined. In other words, high-bit bytes in ASCII-8BIT/BINARY are *invalid* as characters.

The error raised should be InvalidByteSequenceError and it should be prevented by using :invalid => :replace option.


-- 
http://bugs.ruby-lang.org/