From: James Gray <james@...>
Date: 2009-04-17T03:30:37+09:00
Subject: Re: querying using HTTP

On Apr 16, 2009, at 11:30 AM, Brian Candler wrote:

> James Gray wrote:
>> Ruby 1.8 had a single global variable that, when set, changed the
>> behavior of all code in the interpreter, including the stuff I didn't
>> write.  I hope that isn't your idea of a "sane" system.
>
> Ruby 1.8 treated strings as sequences of 8-bit bytes unless explicitly
> told otherwise. That is sane.

I'm pretty sure you are in the minority with this opinion.  You really  
like this?

$ ruby -e 'p "Résumé"[0..1]'
"R\303"

How often is that going to be the desired result?

There were a lot of complaints about Ruby's encoding support over the  
years.  A lot.  I'm pretty sure if we had all been saying, "Matz, we  
love the it's-all-bytes approach," there would be no m17n.  That just  
wasn't the case though.

> Here is an example of the sort of problems still being caused by  
> String
> in 1.9:
> http://groups.google.com/group/rack-devel/browse_thread/thread/99628ed37ac5f5b

And if we combed the Web for documented problems caused by the Ruby  
1.8 system, do you think we would find a few of those?  I would be  
willing to bet I've seen one encoding related problem post every  
couple of weeks I've been on Ruby Talk.

I just did a quick search in one place users are prone to report  
issues with my FasterCSV library and about 47% of all the issues ever  
reported were character encoding issues.

>> Ruby 1.9 probably does require us to learn the bare minimum about how
>> character encodings are handled.
>
> Which is rather difficult if it's not documented. Yes I know there  
> have
> been some third-party efforts, including your own, but I have yet to  
> see
> anything which is anywhere near complete.

Can you list what's not yet covered in my blog series?  I'm aware of  
two very small things that I've never once seen used in the wild.   
I'll add those, but let's say I feel my current coverage is about 98%  
complete.  How am I still failing to meet your needs?

>> It's about time.  How many years
>> have we tried to get by with crossed fingers and a prayer that it
>> would just work out?  Character encodings should have been required
>> reading long before now.
>
> Sure, people who process text need to understand character encodings.
> But text is a small subset of data. When you're processing JPEGs or  
> PDFs
> or ASN1 certificates or HTTP POSTs, you just want something that's 8- 
> bit
> clean.

Ruby 1.9 has an encoding for that too and it's very well documented.

>> For years the Ruby community has begged for more speed and robust
>> character encoding support.  The core team delivered that and much
>> more this January with a production release that's substantially
>> faster and has a very powerful new encoding engine.
>
> Nobody's complaining about improved performance.

That's a relief.  Now we just need to get everyone over to Ruby 1.9  
and those issues will be a thing of the past.  Thus, it would make me  
happy if you stop telling people not to do that.

> I'm saying there is still much pain to be had by using 1.9, and  
> advising that people may wish to avoid the pain until (hopefully)  
> most of it has gone away.
> Anyone who has had no pain with 1.9 or libraries which don't work  
> under
> 1.9 is free to speak up.

I am speaking up.  That's the point.  :)

I had quite a bit of pain when I adapted FasterCSV to be the standard  
CSV library.  There were two reasons for that.  First, the m17n  
implementation was still pretty raw and I ran into bugs and  
complications.  Those are almost completely resolved now.  The second  
reason was the lack of documentation, so I wrote some from what I had  
learned in converting the code.

Now there's a lot less pain.

>> To repay their monumental efforts, we complain and urge people to  
>> stick with Ruby 1.8.  We must truly be the most ungrateful lot of  
>> bums ever.
>
> Who's "we"? Speakly only for myself, I didn't ask for String to be
> changed in this way. And does the amount of effort which went in mean
> that I am forbidden from saying that I don't like the result?

It means that I think your comments are doing harm to the 1.9  
migration and I can't find the good you are doing to balance that.

James Edward Gray II