From: Iain Barnett <iainspeed@...>
Date: 2010-12-17T03:43:20+09:00
Subject: Re: do your bit for my mental health - how to find the difference between two strings?


On 16 Dec 2010, at 18:17, Brian Candler wrote:

> Iain Barnett wrote in post #968894:
>>>   service = params[:service]
>>>   service.replace('servicename')    # <<<<
>>>   STDERR.puts service.inspect, service.encoding
>>>   mymethod( service )
>>> 
>>> Try running this with the marked line present, and with that line
>>> commented out.
>> 
>> this bore some info. I'm running Ruby 1.92 (sorry, I should have
>> mentioned that) with Sinatra 1.1.0. With the replace function the
>> encoding is US-ASCII, or UTF-8 if I set the `# encoding: utf-8` at the
>> top of the file.
>> 
>> Without the replace function the encoding is ASCII-8BIT. I've been
>> searching around for how to make all this work now I know this, and
>> poking around in the Sinatra and Rack source, but haven't found a
>> (working) answer yet. I'll post it when I do.
> 
> OK, well you should now be able replicate this without Sinatra:
> 
> foo = "servicename"
> foo.force_encoding("ASCII-8BIT")
> mymethod(foo)
> 
> Note that Sinatra gives you a parameter tagged as ASCII-8BIT, more or 
> less by accident (i.e. it's not documented as far as I know); it's 
> simply that Ruby defaults to ASCII-8BIT when reading data from a socket, 
> as opposed to from a file.
> 
> If you really want the gorey details, I attempted to document about 200 
> ruby 1.9 string behaviours at 
> https://github.com/candlerb/string19/blob/master/string19.rb
> 
> Anyway, you can now investigate what's going on in mymethod() which 
> might be affected by the encoding. If you are using sqlite3 inside that 
> method, you may have hit upon the bug already discovered at
> http://www.ruby-forum.com/topic/476119
> 
> If you don't want to understand what's going on, you can do
> service.force_encoding("UTF-8")
> and cross your fingers. It might fix the problem, or it might be hiding 
> some other more serious problem which will explode on you at some 
> indeterminate time in the future.
> 
> But as far as I'm concerned, this is all total crap. I stick with ruby 
> 1.8 for exactly this reason.
> 
> -- 
> Posted via http://www.ruby-forum.com/.

Yep, I am using Sqlite3 so I tried the force_encoding just after I replied and bingo! You're right. I'm going to look into Rack Request and Response and perhaps patch them to enforce it there, but for now I'm a lot happier. It is a stupid bug, but I like the lambda syntax in 1.9 so I'm not giving it up :)

I'll take a look at the string behaviours too, thanks for the link.

Regards,
Iain