From: Eric Hodel <drbrain@...7.net>
Date: 2009-07-24T07:44:36+09:00
Subject: Re: Byte–stream parsing in Ruby

On Jul 23, 2009, at 10:02, Brian Candler wrote:
> If I cannot predict what will happen when string A (encoding X)  
> encounters
> string B (encoding Y), and I have to keep forcing the encodings to X,
> then there's no benefit in having the capability for strings to carry
> about their own encodings.

I think you have a misconception about what #force_encoding does.  It  
does not do any conversion.  Use Encoding::Converter for that.

While #force_encoding does approximately what you want in the examples  
you've shown (ASCII, binary data and UTF-8 encodings) it won't work  
when you're reading one multibyte encoding (say, Shift-JIS from an IO)  
and adding it to another multibyte encoding (say, a UTF-8 String).   
You'll only end up with garbage if you don't use a converter.

For 1.9, I don't think io.read(1) is correct.  #getc is better since  
it'll read what you want:

$ cat file
π
$ irb19
irb(main):001:0> open 'file' do |io| p io.getc end
"π"
=> "π"
irb(main):002:0> open 'file' do |io| io.set_encoding 'binary'; p  
io.getc end
"\xCF"
=> "\xCF"

Even for control characters:

$ ruby19 -e 'p $stdin.getc'
^I
"\t"
$