From: brabuhr@...
Date: 2011-03-28T09:10:51+09:00
Subject: Re: Even more 1.9 encoding issues (was: Re: SQLite3/Sinatra not returning results)

On Sun, Mar 27, 2011 at 5:14 PM, Markus Fischer <markus@fischer.name> wrote:
> Within Sinatra it is as you said:
>
>  if @auth.provided? && @auth.basic? && @auth.credentials
>     sql = "SELECT * FROM user WHERE username = ? AND password = ?"
>     username = @auth.credentials[0]
>     password = @auth.credentials[1]
>     pp username.encoding, password.encoding, $dbi.select_one(sql,
> username, password)
>     username.force_encoding('US-ASCII')
>     password.force_encoding('US-ASCII')
>     pp username.encoding, password.encoding, $dbi.select_one(sql,
> username, password)[0].encoding
>  end
>
> This is the output
>
> #<Encoding:ASCII-8BIT>
> #<Encoding:ASCII-8BIT>
> nil
> #<Encoding:US-ASCII>
> #<Encoding:US-ASCII>
> #<Encoding:UTF-8>

> Getting back to this specific case, where's the
> problem exactly here? Is it Sinatra? My source
> file? The sqlite database?

The credentials in Sinatra are unpacked in Rack:

ruby 1.9.2p0:
> Encoding.default_external
=> #<Encoding:UTF-8>
> Encoding.default_internal
=> nil
> s = 'hello'
=> "hello"
> s.encoding
=> #<Encoding:UTF-8>
> [s].pack('m*')
=> "aGVsbG8=\n"
> [s].pack('m*').unpack('m*')
=> ["hello"]
> [s].pack('m*').unpack('m*').first.encoding
=> #<Encoding:ASCII-8BIT>

rack/blob/master/lib/rack/auth/basic.rb:
        def credentials
          @credentials ||= params.unpack("m*").first.split(/:/, 2)
        end

http://redmine.ruby-lang.org/issues/show/4279
pack.c (pack_unpack): the resulted string of unpack('M') must have
ASCII-8BIT encoding (and ENC_CODERANGE_VALID). [ruby-core:34482]

I don't know if there is any reasonable way for Rack to reliably
return a more correct encoding for the credentials, but that seems to
be where your incompatible encoding is coming from.