Re: Byte窶都tream parsing in Ruby

On Jul 23, 2009, at 2:47 AM, Brian Candler wrote:

> Eric Hodel wrote:
>> On Jul 22, 2009, at 00:46, Brian Candler wrote:
>>> =3D> "hello\xFF\xFA"
>>>>> str.valid_encoding?
>>> =3D> false
>>>
>>> This is why I hate ruby 1.9.
>>
>> I don't think that's a valid UTF-8 byte sequence...
>
> That's the whole point. The OP wanted to append bytes to a string, and
> detect whether the resulting string was a valid set of complete UTF-8
> codepoints, or whether it was necessary to wait for more byte(s) for =20=

> it
> to become complete.
>
> Ruby 1.9's valid_encoding? method seems to do that for you - except =20=

> that
> all the automagical and undocumented mutation of Strings gets in the
> way.

I'm pretty sure I document all the behavior we've seen in this thread =20=

(and much more), in this single article on my blog:

http://blog.grayproductions.net/articles/ruby_19s_string

I'm really not sure why you seem totally unwilling to count my =20
articles as a valid source of information after all this time.  They =20
continually explain what you say is unexplained.  I've asked you in =20
the past to list what they don't cover, but aside from the C API side =20=

of things (which I admit I don't cover) you're just all out of =20
excuses.  I assume you simply have no desire to read them.  Fair =20
enough, but hopefully others do.  I feel that means we should list =20
them as an available resource.

I'm not sure what "automagical" means in this context either, but I =20
don't feel it's a good description.  I assume "auto" is for =20
"automatic."  Is Ruby automatically changing the Encoding?  I don't =20
think so.  The programmer is asking Ruby to add two Strings with =20
different Encodings.  Ruby could just say no, but in this case there =20
is a way it can be done, so it makes the choice, assuming that's what =20=

you wanted.

I guess "magical" may just mean you don't understand what's happening =20=

here.  I do though, so there's certainly a process we can break down =20
and understand.

>> Now let's use 1.9's built-in encoding stuff with our valid byte
>> sequence:
>>
>> $ cat conv.rb
>> # encoding: utf-8
>> str =3D "hello "
>> p :encoding =3D> str.encoding
>> str << 0xE2.chr
>> str << 0x98.chr
>> str << 0x83.chr
>>
>> puts str
>> $ ruby19 conv.rb
>> {:encoding=3D>#<Encoding:UTF-8>}
>> hello =E2=98=83
>>
>> huh, it worked fine.
>
> Yes, but you forgot to add another
>
>  p :encoding =3D> str.encoding
>
> to the end. This shows that the string's encoding has magically =20
> mutated
> without a by-your-leave.

That's not true.  You asked Ruby to combine those Strings of differing =20=

content.  You gave your permission.

> So now to test whether the encoding is valid or not, you have to =20
> mutate
> the string back again:
>
>  str.force_encoding("UTF-8")
>  puts "is valid" if str.valid_encoding?
>
> OK, then what happens if you concatenate another byte?
>
>  str << 0xFF.chr    # boom
>
> Argh, you need to mutate it back to ASCII-8BIT first.

As always, you are just not explaining what these examples show.  The =20=

str variable contains some UTF-8 content.  There is another String =20
involved here though and we should examine its Encoding:

 >> 0xFF.chr.encoding
=3D> #<Encoding:ASCII-8BIT>

So what you are really asking Ruby to do is to combine data in two =20
different Encodings.  There is a way to do that here, thanks to Ruby's =20=

concept of compatible Encodings.  Given that, the conversion is made.  =20=

If you had wanted to keep that data in UTF-8, you should have added =20
more UTF-8 bytes to it:

 >> ("abc".force_encoding("UTF-8") << =20
0xFF.chr.force_encoding("UTF-8")).encoding
=3D> #<Encoding:UTF-8>

There's no magic here.  It's a process.  We can explain it.  I have.

James Edward Gray II