From: Markus Fischer <markus@...>
Date: 2011-03-28T21:18:26+09:00
Subject: Re: Even more 1.9 encoding issues

On 28.03.2011 02:10, brabuhr@gmail.com wrote:
> The credentials in Sinatra are unpacked in Rack:
>
> ruby 1.9.2p0:
>> Encoding.default_external
> =>  #<Encoding:UTF-8>
>> Encoding.default_internal
> =>  nil
>> s = 'hello'
> =>  "hello"
>> s.encoding
> =>  #<Encoding:UTF-8>
>> [s].pack('m*')
> =>  "aGVsbG8=\n"
>> [s].pack('m*').unpack('m*')
> =>  ["hello"]
>> [s].pack('m*').unpack('m*').first.encoding
> =>  #<Encoding:ASCII-8BIT>
>
> rack/blob/master/lib/rack/auth/basic.rb:
>          def credentials
>            @credentials ||= params.unpack("m*").first.split(/:/, 2)
>          end
>
> http://redmine.ruby-lang.org/issues/show/4279
> pack.c (pack_unpack): the resulted string of unpack('M') must have
> ASCII-8BIT encoding (and ENC_CODERANGE_VALID). [ruby-core:34482]
>
> I don't know if there is any reasonable way for Rack to reliably
> return a more correct encoding for the credentials, but that seems to
> be where your incompatible encoding is coming from.

On 28.03.2011 02:12, brabuhr@gmail.com wrote:
 >> http://redmine.ruby-lang.org/issues/show/4279
 >> pack.c (pack_unpack): the resulted string of unpack('M') must have
 >> ASCII-8BIT encoding (and ENC_CODERANGE_VALID). [ruby-core:34482]
 >
 > Oops, that's unpack('M'), Rack is using unpack('m').
 >

Thanks, that very informative! Guess proper encoding handling just comes 
with cost and confusion. Although it's very confusing, it actually looks 
right what ruby is doing. When it decodes base64, how should it know 
it's encoding? My example, involving PHP:

$ php -r 'echo base64_encode("this is a t�st"), "\n";'
dGhpcyBpcyBhIHTDtnN0

$ irb
ruby-1.9.2-p180 :001 > "dGhpcyBpcyBhIHTDtnN0".unpack('m')[0].encoding
  => #<Encoding:ASCII-8BIT>
ruby-1.9.2-p180 :002 > "dGhpcyBpcyBhIHTDtnN0".unpack('m')[0]
  => "this is a t\xC3\xB6st"
ruby-1.9.2-p180 :003 > 
"dGhpcyBpcyBhIHTDtnN0".unpack('m')[0].force_encoding('UTF-8')
  => "this is a t�st"
ruby-1.9.2-p180 :004 >

So it simply can't know it, slaps ASCII-8BIT at it and the application 
needs to deal with the proper encoding. Me not knowingly that what I get 
is a direct result from unpack() [and if I would have knew it, I 
wouldn't have knew'd that it's ASCII-8BIT either ;)] treated it like a 
source string when it's more a network string.

Good to know, case solved, still long road ahead :)

thank you,
- Markus