From: "NARUSE, Yui" <naruse@...>
Date: 2008-02-29T17:43:00+09:00
Subject: Re: Gems running aground on multibyte char

Hi,

Eric Hodel wrote:
> On Feb 27, 2008, at 18:27 PM, Nobuyoshi Nakada wrote:
>> At Thu, 28 Feb 2008 01:10:48 +0900,
>> Austin Ziegler wrote in [ruby-core:15674]:
>>> I'm releasing an updated version soon in any case (to just fix this
>>> problem). Something should be worked out to make it easier for Ruby
>>> 1.9 gems to have their spec in a non-US-ASCII format.
>>
>> Use magic comments.
> 
> There is a problem with backwards compatibility, I think.
> 
> The ruby file for a gem specification comes from a YAML file that ships 
> with the gem.

The encoding of YAML file is UTF-8 or UTF-16 by the yaml spec.
http://www.yaml.org/spec/1.1/

> I can add an encoding field to the gem specification for future gems, 
> and write that encoding into the ruby file, but old gems may still fail.

So gems can write magic comment "UTF-8" into the ruby file if "metadata"
is written in right encoding.

> Would it be sufficient to add the magic comment specifying ASCII-8BIT to 
> all ruby gem specifications unless otherwise specified through a gemspec 
> attribute?

Why you specify ASCII-8BIT?  You know the encoding of YAML file is UTF-8 or UTF-16,
and rubygems will write gemspec as UTF-8.  You should specify UTF-8.
# specifying may the business of rubygems

> I think so, because I don't know what character set any particular gem's 
> data is actually in.  Will there still be a problem with YAML, though?

Practical problem is that some person write YAML file in other than UTF-8 or
UTF-16.  How do we treat those invalid YAML file can be a problem.

Ruby 1.9 which default script encoding is US-ASCII insist such files will be error.

-- 
NARUSE, Yui  <naruse@airemix.com>
DBDB A476 FDBD 9450 02CD 0EFC BCE3 C388 472E C1EA