From: Perry Smith <pedzsan@...>
Date: 2011-10-22T09:52:14+09:00
Subject: [ruby-core:40267] Re: Counseling


On Oct 21, 2011, at 6:30 PM, Bill Kelly wrote:

> Gon�alo Silva wrote:
>> 
>> While I believe that this is a necessary evil for the real encoding
>> support that ruby currently provides, it would be great to have a
>> compile time flag to revert back to the old behavior (or something
>> similar). I don't think this is implemented, but I could be wrong.
> 
> It appears that defining NO_LOCALE_CHARMAP when building ruby, may
> have this effect:
> 
> $ git branch
> * ruby_1_9_3
> 
> $ cflags='-DNO_LOCALE_CHARMAP' ./configure --prefix=/opt/ruby193 --program-suffix=19 --enable-shared --disable-install-doc
> $ make
> $ sudo make install
> 
> $ /opt/ruby193/bin/ruby19 -v -e 'puts "".encoding'
> ruby 1.9.3dev (2011-10-11 revision 33457) [x86_64-linux]
> ASCII-8BIT
> 
> $ /opt/ruby193/bin/ruby19 -e 'puts `ls`.encoding'
> ASCII-8BIT
> 
> 
> Hope this helps,

Thanks Bill.  I will try that.  That looks very promising.  Just as good of an alternative would be to change my default to UTF-8 instead of US-ASCII.

To the others of interest: I am running a Mac Lion (10.7) (development) as well as AIX 5.3 (production).  Rails 2.3.x.  Ruby 1.9.2 p290.  I've had encoding problems running Mac 10.6, Rails 2.3.5 up, and Ruby 1.9.1 and up.  Essentially, when I switched to Ruby 1.9, I started having encoding problems.  I'm using PostgreSQL as my database backend with "pg" as the Ruby interface to ActiveRecord.  All my gems are fairly up to date.  (Oh... I also have ActiveLdap for interfacing with an LDAP server too.)

The Rails web site is used world wide and a lot of my encoding problems come with users in Korea but I can hit problems on my own Mac with my own requests.  I try (very hard) to keep all i/o as UTF-8.

My first attempt to solve this was to put a UTF-8 coding into all my ruby files.  This appeared to help but upon reflection, I don't think it really did.  Adding the -KU in scripts like thin's startup script helps more.  In fact, I think it solves 99.99% of my problems.  But when I update thin (for example) I forget to add the -KU to the script and hit errors until I add the -KU back.

One recent saga involved memcache-client (which I've mentioned).  memcache-client tries to concatenate a command, key, and Marshall'ed data.  If the Marshalled data really is ASCII-8BIT, then the concatenation dies.  I cobbled around that by forcing the marshalled data back to UTF-8 but... I wonder if that is really the right solution.  Someone mentioned Dalli.  I plan to move to that as soon as possible. (this weekend I hope)

Rails tends to have erb files and often the response built up so far is not compatible with a new piece about to be added and there is no easy way to fix this that I've found yet.  Today's encoding problem was because the -KU had been left out of one of my startup files.  I need to add a step in my update process to put the -KU flags in.  Someone mentioned that Bundler has a way to do that.

I guess, deep down, my belief is that all the strings really really really are UTF-8 but they are being marked otherwise for some reason (as in the case of Marshalled data).

I hope that helps others understand my woes a bit better.

Thanks for your help,
pedz