From: Brian Candler <b.candler@...>
Date: 2009-07-23T16:47:12+09:00
Subject: Re: Byte–stream parsing in Ruby

Eric Hodel wrote:
> On Jul 22, 2009, at 00:46, Brian Candler wrote:
>> => "hello\xFF\xFA"
>>>> str.valid_encoding?
>> => false
>>
>> This is why I hate ruby 1.9.
> 
> I don't think that's a valid UTF-8 byte sequence...

That's the whole point. The OP wanted to append bytes to a string, and 
detect whether the resulting string was a valid set of complete UTF-8 
codepoints, or whether it was necessary to wait for more byte(s) for it 
to become complete.

Ruby 1.9's valid_encoding? method seems to do that for you - except that 
all the automagical and undocumented mutation of Strings gets in the 
way. Sometimes, ruby lets you concatenate an arbitrary byte to a UTF-8 
string without an exception; sometimes it does not. It appears this is 
something to do with the concept of "compatible encodings".

> Now let's use 1.9's built-in encoding stuff with our valid byte
> sequence:
> 
> $ cat conv.rb
> # encoding: utf-8
> str = "hello "
> p :encoding => str.encoding
> str << 0xE2.chr
> str << 0x98.chr
> str << 0x83.chr
> 
> puts str
> $ ruby19 conv.rb
> {:encoding=>#<Encoding:UTF-8>}
> hello ☃
> 
> huh, it worked fine.

Yes, but you forgot to add another

  p :encoding => str.encoding

to the end. This shows that the string's encoding has magically mutated 
without a by-your-leave.

So now to test whether the encoding is valid or not, you have to mutate 
the string back again:

  str.force_encoding("UTF-8")
  puts "is valid" if str.valid_encoding?

OK, then what happens if you concatenate another byte?

  str << 0xFF.chr    # boom

Argh, you need to mutate it back to ASCII-8BIT first.

> So you're mad that Ruby doesn't let you shoot yourself in the foot?

I'm mad that Ruby has behaviour which is (a) undocumented, and (b) IMO 
just plain stupid, and you have to expend ridiculous effort both to 
understand it and to work around it.

I'm actually attempting to document it in my spare time, in the form of 
a Test::Unit script. It looks like I'm going to have over 200 
assertions. This is time I should probably have spent migrating code to 
Erlang - which incidentally has a very sensible proposal for Unicode 
handling.

Thank goodness for those people maintaining 1.8.6 and related forks like 
Ruby Enterprise Edition.

Regards,

Brian.
-- 
Posted via http://www.ruby-forum.com/.