From: James Gray <james@...>
Date: 2009-07-23T20:49:03+09:00
Subject: Re: Byte–stream parsing in Ruby

On Jul 23, 2009, at 2:47 AM, Brian Candler wrote:

> Eric Hodel wrote:
>> On Jul 22, 2009, at 00:46, Brian Candler wrote:
>>> => "hello\xFF\xFA"
>>>>> str.valid_encoding?
>>> => false
>>>
>>> This is why I hate ruby 1.9.
>>
>> I don't think that's a valid UTF-8 byte sequence...
>
> That's the whole point. The OP wanted to append bytes to a string, and
> detect whether the resulting string was a valid set of complete UTF-8
> codepoints, or whether it was necessary to wait for more byte(s) for  
> it
> to become complete.
>
> Ruby 1.9's valid_encoding? method seems to do that for you - except  
> that
> all the automagical and undocumented mutation of Strings gets in the
> way.

I'm pretty sure I document all the behavior we've seen in this thread  
(and much more), in this single article on my blog:

http://blog.grayproductions.net/articles/ruby_19s_string

I'm really not sure why you seem totally unwilling to count my  
articles as a valid source of information after all this time.  They  
continually explain what you say is unexplained.  I've asked you in  
the past to list what they don't cover, but aside from the C API side  
of things (which I admit I don't cover) you're just all out of  
excuses.  I assume you simply have no desire to read them.  Fair  
enough, but hopefully others do.  I feel that means we should list  
them as an available resource.

I'm not sure what "automagical" means in this context either, but I  
don't feel it's a good description.  I assume "auto" is for  
"automatic."  Is Ruby automatically changing the Encoding?  I don't  
think so.  The programmer is asking Ruby to add two Strings with  
different Encodings.  Ruby could just say no, but in this case there  
is a way it can be done, so it makes the choice, assuming that's what  
you wanted.

I guess "magical" may just mean you don't understand what's happening  
here.  I do though, so there's certainly a process we can break down  
and understand.

>> Now let's use 1.9's built-in encoding stuff with our valid byte
>> sequence:
>>
>> $ cat conv.rb
>> # encoding: utf-8
>> str = "hello "
>> p :encoding => str.encoding
>> str << 0xE2.chr
>> str << 0x98.chr
>> str << 0x83.chr
>>
>> puts str
>> $ ruby19 conv.rb
>> {:encoding=>#<Encoding:UTF-8>}
>> hello ☃
>>
>> huh, it worked fine.
>
> Yes, but you forgot to add another
>
>  p :encoding => str.encoding
>
> to the end. This shows that the string's encoding has magically  
> mutated
> without a by-your-leave.

That's not true.  You asked Ruby to combine those Strings of differing  
content.  You gave your permission.

> So now to test whether the encoding is valid or not, you have to  
> mutate
> the string back again:
>
>  str.force_encoding("UTF-8")
>  puts "is valid" if str.valid_encoding?
>
> OK, then what happens if you concatenate another byte?
>
>  str << 0xFF.chr    # boom
>
> Argh, you need to mutate it back to ASCII-8BIT first.

As always, you are just not explaining what these examples show.  The  
str variable contains some UTF-8 content.  There is another String  
involved here though and we should examine its Encoding:

 >> 0xFF.chr.encoding
=> #<Encoding:ASCII-8BIT>

So what you are really asking Ruby to do is to combine data in two  
different Encodings.  There is a way to do that here, thanks to Ruby's  
concept of compatible Encodings.  Given that, the conversion is made.   
If you had wanted to keep that data in UTF-8, you should have added  
more UTF-8 bytes to it:

 >> ("abc".force_encoding("UTF-8") <<  
0xFF.chr.force_encoding("UTF-8")).encoding
=> #<Encoding:UTF-8>

There's no magic here.  It's a process.  We can explain it.  I have.

James Edward Gray II