From: Jordi Date: 2008-03-01T02:50:43+09:00 Subject: Re: Gems running aground on multibyte char ------=_Part_33123_6548441.1204307450466 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Defaulting script encoding to UTF-8 would be a bad idea? I think most of the computing world is moving to UTF-8 as default. On Fri, Feb 29, 2008 at 5:43 PM, NARUSE, Yui wrote: > Hi, > > Eric Hodel wrote: > > On Feb 27, 2008, at 18:27 PM, Nobuyoshi Nakada wrote: > >> At Thu, 28 Feb 2008 01:10:48 +0900, > >> Austin Ziegler wrote in [ruby-core:15674]: > >>> I'm releasing an updated version soon in any case (to just fix this > >>> problem). Something should be worked out to make it easier for Ruby > >>> 1.9 gems to have their spec in a non-US-ASCII format. > >> > >> Use magic comments. > > > > There is a problem with backwards compatibility, I think. > > > > The ruby file for a gem specification comes from a YAML file that ships > > with the gem. > > The encoding of YAML file is UTF-8 or UTF-16 by the yaml spec. > http://www.yaml.org/spec/1.1/ > > > I can add an encoding field to the gem specification for future gems, > > and write that encoding into the ruby file, but old gems may still fail. > > So gems can write magic comment "UTF-8" into the ruby file if "metadata" > is written in right encoding. > > > Would it be sufficient to add the magic comment specifying ASCII-8BIT to > > all ruby gem specifications unless otherwise specified through a gemspec > > attribute? > > Why you specify ASCII-8BIT? You know the encoding of YAML file is UTF-8 > or UTF-16, > and rubygems will write gemspec as UTF-8. You should specify UTF-8. > # specifying may the business of rubygems > > > I think so, because I don't know what character set any particular gem's > > data is actually in. Will there still be a problem with YAML, though? > > Practical problem is that some person write YAML file in other than UTF-8 > or > UTF-16. How do we treat those invalid YAML file can be a problem. > > Ruby 1.9 which default script encoding is US-ASCII insist such files will > be error. > > -- > NARUSE, Yui > DBDB A476 FDBD 9450 02CD 0EFC BCE3 C388 472E C1EA > > -- Jordi Polo Carres NLP laboratory - NAIST http://www.bahasara.org ------=_Part_33123_6548441.1204307450466 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline
Defaulting script encoding to UTF-8 would be a bad idea?
I think most of the computing world is moving to UTF-8 as default.

On Fri, Feb 29, 2008 at 5:43 PM, NARUSE, Yui <naruse@airemix.com> wrote:

Hi,

Eric Hodel wrote:
> On Feb 27, 2008, at 18:27 PM, Nobuyoshi Nakada wrote:
>> At Thu, 28 Feb 2008 01:10:48 +0900,
>> Austin Ziegler wrote in [ruby-core:15674]:
>>> I'm releasing an updated version soon in any case (to just fix this
>>> problem). Something should be worked out to make it easier for Ruby
>>> 1.9 gems to have their spec in a non-US-ASCII format.
>>
>> Use magic comments.
>
> There is a problem with backwards compatibility, I think.
>
> The ruby file for a gem specification comes from a YAML file that ships
> with the gem.

The encoding of YAML file is UTF-8 or UTF-16 by the yaml spec.
http://www.yaml.org/spec/1.1/

> I can add an encoding field to the gem specification for future gems,
> and write that encoding into the ruby file, but old gems may still fail.

So gems can write magic comment "UTF-8" into the ruby file if "metadata"
is written in right encoding.

> Would it be sufficient to add the magic comment specifying ASCII-8BIT to
> all ruby gem specifications unless otherwise specified through a gemspec
> attribute?

Why you specify ASCII-8BIT? You know the encoding of YAML file is UTF-8 or UTF-16,
and rubygems will write gemspec as UTF-8. You should specify UTF-8.
# specifying may the business of rubygems

> I think so, because I don't know what character set any particular gem's
> data is actually in. Will there still be a problem with YAML, though?

Practical problem is that some person write YAML file in other than UTF-8 or
UTF-16. How do we treat those invalid YAML file can be a problem.

Ruby 1.9 which default script encoding is US-ASCII insist such files will be error.

--
NARUSE, Yui <naruse@airemix.com>
DBDB A476 FDBD 9450 02CD 0EFC BCE3 C388 472E C1EA

--
Jordi Polo Carres
NLP laboratory - NAIST
http://www.bahasara.org
------=_Part_33123_6548441.1204307450466--