[ruby-core:18388] Re: Gems running aground on multibyte char

Hi,

2008/2/29 NARUSE, Yui <[e-mail]>:
> Hi,
>
> Eric Hodel wrote:
>>
>> On Feb 27, 2008, at 18:27 PM, Nobuyoshi Nakada wrote:
>>>
>>> At Thu, 28 Feb 2008 01:10:48 +0900,
>>> Austin Ziegler wrote in [ruby-core:15674]:
>>>>
>>>> I'm releasing an updated version soon in any case (to just fix this
>>>> problem). Something should be worked out to make it easier for Ruby
>>>> 1.9 gems to have their spec in a non-US-ASCII format.
>>>
>>> Use magic comments.
>>
>> There is a problem with backwards compatibility, I think.
>>
>> The ruby file for a gem specification comes from a YAML file that ships
>> with the gem.
>
> The encoding of YAML file is UTF-8 or UTF-16 by the yaml spec.
> http://www.yaml.org/spec/1.1/
>
>> I can add an encoding field to the gem specification for future gems, and
>> write that encoding into the ruby file, but old gems may still fail.
>
> So gems can write magic comment "UTF-8" into the ruby file if "metadata"
> is written in right encoding.
>
>> Would it be sufficient to add the magic comment specifying ASCII-8BIT to
>> all ruby gem specifications unless otherwise specified through a gemspec
>> attribute?
>
> Why you specify ASCII-8BIT?  You know the encoding of YAML file is UTF-8 or
> UTF-16,
> and rubygems will write gemspec as UTF-8.  You should specify UTF-8.
> # specifying may the business of rubygems
>
>> I think so, because I don't know what character set any particular gem's
>> data is actually in.  Will there still be a problem with YAML, though?
>
> Practical problem is that some person write YAML file in other than UTF-8 or
> UTF-16.  How do we treat those invalid YAML file can be a problem.
>
> Ruby 1.9 which default script encoding is US-ASCII insist such files will be
> error.
>
> --
> NARUSE, Yui  <[e-mail]>
> DBDB A476 FDBD 9450 02CD 0EFC BCE3 C388 472E C1EA


How's this topic going?  I seem to have similar trouble with cairo:


> $ echo $LANG
> C
>
> $ gem19 install cairo
> Building native extensions.  This could take a while...
> Successfully installed cairo-1.7.0
> 1 gem installed
>
> $ gem19 uninstall cairo
> WARNING:  /home/mame/work/ruby19/local/lib/ruby/gems/1.9.0/specifications/cairo-1.7.0.gemspec:6: invalid multibyte char
> /home/mame/work/ruby19/local/lib/ruby/gems/1.9.0/specifications/cairo-1.7.0.gemspec:6: invalid multibyte char
> /home/mame/work/ruby19/local/lib/ruby/gems/1.9.0/specifications/cairo-1.7.0.gemspec:6: syntax error, unexpected $end, expecting ']'
> ... s.authors = ["Evan Marin", "##yvind Kol#s", "MenTaLguY", "...
> ...                               ^
> WARNING:  Gem::Specification.new do |s|
>   s.name = %q{cairo}
>   s.version = "1.7.0"
(snip)
> end
> ERROR:  While executing gem ... (Gem::InstallError)
>     Unknown gem cairo >= 0
>
> $ LANG=ja_JP.UTF-8 gem19 uninstall cairo
> Successfully uninstalled cairo-1.7.0


Index: lib/rubygems/source_index.rb
===================================================================
--- lib/rubygems/source_index.rb	(revision 18876)
+++ lib/rubygems/source_index.rb	(working copy)
@@ -80,7 +80,7 @@

     def load_specification(file_name)
       begin
-        spec_code = File.read(file_name).untaint
+        spec_code = File.read(file_name, :encoding => "UTF-8").untaint
         gemspec = eval spec_code, binding, file_name
         if gemspec.is_a?(Gem::Specification)
           gemspec.loaded_from = file_name

-- 
Yusuke ENDOH <[e-mail]>