ruby-dev

At 21:50 07/12/10, Nobuyoshi Nakada wrote:
>なかだです。
>
>At Mon, 10 Dec 2007 20:35:58 +0900,
>Martin Duerst wrote in [ruby-dev:32520]:
>> 中田さんの下記のパッチの一部は「全ての encode で使える
>> 文字コードは Encoding として存在する」という前提で
>> 行われていたが、それは必ずしもそうとも限りません。
>> その部分を元に戻していただきますでしょうか。
>> それともこちらから更なるパッチを送った方がいいだすか。
>
>直しました。

ありがとうございました。大変助かりました。

>一緒に、登録していないEncodingに変換した場合は、変換
>後にreplicateしてセットするようにしました。

以前まつもとさんがメールで「ASCII-8BIT でいい」と書きましたが、
このところは Encoding の専門家の中田さんにまかせます。

これで手元のテストは一つ以外全部通過します。

残るのは、encode! の場合変換後の Encoding が変換前のまま
ということです。これを rb_str_shared_replace か
rb_str_transcode_bang で修正した方がいいと思いますが、
どちらの方がいいでしょうか。

test/ruby/test_transcode.rb へのパッチを添付します。
テストの数が少し増えましたし、中田さんの Encoding の
設定の変更にあわせました。

宜しくお願いします。   Martin.



#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp

Attachments (1)

test_transcode_patch_20071211.txt (6.16 KB, text/x-diff)

Index: test_transcode.rb
===================================================================
--- test_transcode.rb	(revision 14189)
+++ test_transcode.rb	(working copy)
@@ -1,15 +1,8 @@
-# -*- encoding: US-ASCII -*-   # make sure this runs in binary mode
+# -*- encoding: ASCII-8BIT -*-   # make sure this runs in binary mode
 
-class String
-  # different name, because we should be able to remove this later
-  def fix_encoding (encoding)
-    force_encoding(encoding)
-  end
-end
-
 require 'test/unit'
 class TestConvert < Test::Unit::TestCase
-  def test_can_call
+  def test_basic
     # we don't have semantics for conversion without attribute yet
     # maybe 'convert to UTF-8' would be nice :-)
     assert_raise(ArgumentError) { 'abc'.encode }
@@ -20,25 +13,63 @@
     assert_raise(ArgumentError) { 'abc'.encode!('foo', 'bar') }
     assert_raise(ArgumentError) { 'abc'.force_encoding('utf-8').encode('foo') }
     assert_raise(ArgumentError) { 'abc'.force_encoding('utf-8').encode!('foo') }
-    assert_equal('abc'.force_encoding('utf-8').encode('iso-8859-1'), 'abc') # temporary, fix encoding
-    assert_equal("D\xFCrst".force_encoding('iso-8859-1').encode('utf-8').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-1').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-2').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-3').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-4').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-9').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-10').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-13').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-14').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-15').fix_encoding('utf-8'), "D\u00FCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-1'), "D\xFCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-2'), "D\xFCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-3'), "D\xFCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-4'), "D\xFCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-9'), "D\xFCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-10'), "D\xFCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-13'), "D\xFCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-14'), "D\xFCrst")
-    assert_equal("D\u00FCrst".encode('iso-8859-15'), "D\xFCrst")
+    assert_equal('abc'.force_encoding('utf-8').encode('iso-8859-1'), 'abc')
+    # check that encoding is kept when no conversion is done
+    assert_equal('abc'.force_encoding('Shift_JIS').encode('Shift_JIS'), 'abc'.force_encoding('Shift_JIS'))
+    assert_equal('abc'.force_encoding('Shift_JIS').encode!('Shift_JIS'), 'abc'.force_encoding('Shift_JIS'))
+    # assert that encoding is correctly set
+    assert_equal("D\xFCrst".force_encoding('iso-8859-1').encode('utf-8').encoding, "D\u00FCrst".encoding)
+    # check that Encoding can be used as parameter
+    assert_equal("D\xFCrst".encode('utf-8', Encoding.find('ISO-8859-1')), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode(Encoding.find('utf-8'), 'ISO-8859-1'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode(Encoding.find('utf-8'), Encoding.find('ISO-8859-1')), "D\u00FCrst")
+
+    # temporary, fix encoding
+    assert_equal("D\xFCrst".force_encoding('iso-8859-1').encode('utf-8'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-1'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-2'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-3'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-4'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-9'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-10'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-13'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-14'), "D\u00FCrst")
+    assert_equal("D\xFCrst".encode('utf-8', 'iso-8859-15'), "D\u00FCrst")
+    assert_equal("D\u00FCrst".encode('iso-8859-1'), "D\xFCrst".force_encoding('iso-8859-1'))
+    assert_equal("D\u00FCrst".encode('iso-8859-2'), "D\xFCrst".force_encoding('iso-8859-2'))
+    assert_equal("D\u00FCrst".encode('iso-8859-3'), "D\xFCrst".force_encoding('iso-8859-3'))
+    assert_equal("D\u00FCrst".encode('iso-8859-4'), "D\xFCrst".force_encoding('iso-8859-4'))
+    assert_equal("D\u00FCrst".encode('iso-8859-9'), "D\xFCrst".force_encoding('iso-8859-9'))
+    assert_equal("D\u00FCrst".encode('iso-8859-10'), "D\xFCrst".force_encoding('iso-8859-10'))
+    assert_equal("D\u00FCrst".encode('iso-8859-13'), "D\xFCrst".force_encoding('iso-8859-13'))
+    assert_equal("D\u00FCrst".encode('iso-8859-14'), "D\xFCrst".force_encoding('iso-8859-14'))
+    assert_equal("D\u00FCrst".encode('iso-8859-15'), "D\xFCrst".force_encoding('iso-8859-15'))
+    # test length extension
+    assert_equal(("\xA4"*20).encode('utf-8', 'iso-8859-15'), "\u20AC"*20)
+    assert_equal(("\xA4"*20).encode!('utf-8', 'iso-8859-15'), "\u20AC"*20)
+    
   end
+  
+  def test_all_bytes
+    encodings_8859 = [
+      'ISO-8859-1', 'ISO-8859-2',
+      #'ISO-8859-3', # not all bytes used
+      'ISO-8859-4', 'ISO-8859-5',
+      #'ISO-8859-6', # not all bytes used
+      #'ISO-8859-7', # not all bytes used
+      #'ISO-8859-8', # not all bytes used
+      'ISO-8859-9', 'ISO-8859-10',
+      #'ISO-8859-11', # not all bytes used
+      #'ISO-8859-12', # not available
+      'ISO-8859-13','ISO-8859-14','ISO-8859-15',
+      #'ISO-8859-16', # not available
+    ]
+    all_bytes = (0..255).collect {|x| x}.pack 'C*'
+    test_start = all_bytes
+    test_start.encode('UTF-8','ISO-8859-1').encode('ISO-8859-1')
+    encodings_8859.each do |enc|
+      test_start = all_bytes
+      assert_equal(test_start.encode('UTF-8',enc).encode(enc).force_encoding('ASCII-8BIT'), test_start) 
+    end
+  end
 end

Thread

Prev Next

In This Thread

Prev Next

[#32429] eval("\"\xfd".force_encoding("utf-8")) generages garbage in error message. — Tanaka Akira <akr@...>

[#32430] "wrong number of arguments (6 for 2)" by def m(a, b=:b, c, d, e) — Tanaka Akira <akr@...>

[#32432] eval("a"): illegal multibyte char — Tanaka Akira <akr@...>

[#32434] signature of exit() on C++ — "KISHIMOTO, Makoto" <ksmakoto@...4u.or.jp>

[#32447] ruby 1.9 trunk NKF and KCONV Encoding:ASCII-8BIT — WATANABE Tetsuya <Tetsuya.WATANABE@...>

[#32448] SEGV on "abcd\xf0".force_encoding("utf-8").reverse — Tanaka Akira <akr@...>

[#32451] -e:1:in `conv': no implicit conversion from nil to integer (TypeError) — Tanaka Akira <akr@...>

[#32452] `split': negative string size (or size too big) (ArgumentError) — Tanaka Akira <akr@...>

[#32459] Enumerator raises `dead fiber called' — "Yusuke ENDOH" <mame@...>

[#32462] SEGV by test/ruby/test_fiber.rb — Tanaka Akira <akr@...>

[#32468] Iconv.list patch for NetBSD/Citrus — "NARUSE, Yui" <naruse@...>

[#32473] about to_path and to_open — "Yusuke ENDOH" <mame@...>

[#32475] ext/tk/sample/tkextlib/vu/canvSticker2.rb — Tanaka Akira <akr@...>

[#32485] self-calling method dumps core — Tadashi Saito <shiba@...2.accsnet.ne.jp>

[#32493] Regexp#names, MatchData#names, MatchData#regexp — Tanaka Akira <akr@...>

[#32498] Re: [ruby-cvs:21399] Ruby:r14162 (trunk): * parse.y (expr): redefinable not (!) operator. — SASADA Koichi <ko1@...>

[#32502] :!@.inspect — Kazuhiro NISHIYAMA <zn@...>

[#32509] ruby-core:13961での問題(Error while bulding Ruby 1.9 from snapshot 2007/12/09) — KIMURA Koichi <kimura.koichi@...>

[#32512] Re: [ruby-cvs:21409] Ruby:r14172 (trunk): * transcode.c: new file to provide encoding conversion features. — Nobuyoshi Nakada <nobu@...>

[#32514] typo in lib/uri/common.rb — Ueda Satoshi <s-ueda@...>

[#32518] bug in Array#slice! — Satoshi Nakagawa <snakagawa@...>

[#32531] eigenmethod definition for BasicObjects dumps core — Tadashi Saito <shiba@...2.accsnet.ne.jp>

[#32536] timezone 関数が cygwin で見つかりません — Martin Duerst <duerst@...>

[#32550] Binary String — Hidetoshi NAGAI <nagai@...>

[#32556] default completion for irb1.9 — Tadashi Saito <shiba@...2.accsnet.ne.jp>

[#32563] transcoder loading — Nobuyoshi Nakada <nobu@...>

[#32565] Re: [ruby-cvs:21436] Ruby:r14199 (trunk): * io.c (rb_io_check_readable): set default external encoding to — Tanaka Akira <akr@...>

[#32566] hang with write — Tanaka Akira <akr@...>

[#32567] [nil, [...]] — Tanaka Akira <akr@...>

[#32569] Re: [ruby-cvs:21441] Ruby:r14204 (trunk): * io.c (rb_io_getc): use default external encoding if fptr->enc is — Tanaka Akira <akr@...>

[#32581] expandarray may mark dirty VALUE beyond stack — "Yusuke ENDOH" <mame@...>

[#32585] SEGV by IO.pipe — Tanaka Akira <akr@...>

[#32588] /(?<foo>...)/ =~ str assigns foo — Tanaka Akira <akr@...>

[#32590] Circular dependency on AIX — "Yutaka Kanemoto" <kinpoco@...>

[#32604] stack overflow by GC in a thread — Tanaka Akira <akr@...>

[#32606] encoding loading — Nobuyoshi Nakada <nobu@...>

[#32608] $! == 8 — Tanaka Akira <akr@...>

[#32609] Re: [ruby-cvs:21475] Ruby:r14238 (trunk): * configure.in (enc/Makefile): add external encoding objects list. — Yukihiro Matsumoto <matz@...>

[#32610] 1.9.1 issues left (as of 12/15) — Yukihiro Matsumoto <matz@...>

[#32622] binding of caller — Nobuyoshi Nakada <nobu@...>

[#32624] rss and base64 — Tanaka Akira <akr@...>

[#32625] Re: [ruby-cvs:21528] Ruby:r14292 (trunk): more tests. — Yukihiro Matsumoto <matz@...>

[#32628] Bignum#to_s doesn't check base — "Yusuke ENDOH" <mame@...>

[#32629] faster Bignum#* — "Yusuke ENDOH" <mame@...>

[#32631] Cygwin 上で Ruby最新版 をインストールする場合 — "Koji Tominaga" <crimsonshower@...>

[#32636] Re: [ruby-cvs:21552] Ruby:r14315 (trunk): * ext/nkf/nkf.c (NKF::_ENCODING): removed. — Tanaka Akira <akr@...>

[#32640] inspect closed dir. — Tanaka Akira <akr@...>

[#32641] interrupt in * — Tanaka Akira <akr@...>

[#32644] unparenthesized method call with block — "Yusuke ENDOH" <mame@...>

[#32645] unparenthesized method call with block — "Yusuke ENDOH" <mame@...>

[#32662] encode! は変換しないときに <nil> になってしまう。 — Martin Duerst <duerst@...>

[#32665] Net::POP3#enable_sslの仕様変更 — Shugo Maeda <shugo@...>

[#32668] syntax errors on ext/tk/sample — "U.Nakamura" <usa@...>

[#32675] /(?<break>a)/ =~ "a" 等によるローカル変数 — "Keita Yamaguchi" <keita.yamaguchi@...>

[#32677] nil.freeze in date.rb — Shugo Maeda <shugo@...>

[#32680] fail to compile version.c in trunk — Masaki Suketa <masaki.suketa@...>

[#32685] test/webrick/test_server.rb hang — Tanaka Akira <akr@...>

[#32686] enum_inject の説明文 — "Keita Yamaguchi" <keita.yamaguchi@...>

[#32687] Re: [ruby-cvs:21753] Ruby:r14516 (trunk): * lib/time.rb (Time.httpdate): use Time.utc for — Tadayoshi Funaba <tadf@...>

[#32691] Why only inject() accepts :symbols as if it's a Proc in Enumerable? — Tadashi Saito <shiba@...2.accsnet.ne.jp>

[#32693] String#== is not transitive — Tanaka Akira <akr@...>

[#32695] ISO-2022-JP output for transcode — "NARUSE, Yui" <naruse@...>

[#32699] cygwin 上で r14576 現在の test の失敗 — Martin Duerst <duerst@...>

[#32706] Fwd: RDoc: [FATAL] failed to allocate memory — Martin Duerst <duerst@...>

[#32708] Enumerable can't take multiple parameters — GOTOU Yuuzou <gotoyuzo@...>

[#32709] Re: [ruby-cvs:21826] Ruby:r14589 (trunk): Mon Dec 24 17:20:34 2007 NAKAMURA, Hiroshi <nahi@ruby-lang.org> — Tadayoshi Funaba <tadf@...>

[#32710] \e — "U.Nakamura" <usa@...>

[#32713] pre-release note for the christmas release. — Yukihiro Matsumoto <matz@...>

[#32715] issues left as of 12/25 2:00am JST — Yukihiro Matsumoto <matz@...>

[#32718] next dumps core when poped is 0 — "Yusuke ENDOH" <mame@...>

[#32725] more ANSI'ize — Tadashi Saito <shiba@...2.accsnet.ne.jp>

[#32726] Can't build on MacOSX 10.4(Tiger) (was Re: Re: 1.9.1 issues left (as of 12/15)) — "MOROHASHI Kyosuke" <moronatural@...>

[#32740] distinguish "w" and "w:euc-jp" — Tanaka Akira <akr@...>

[#32746] appendline Invalid read of size 1 — Tanaka Akira <akr@...>

[#32747] test-all on Debian/GNU etch — SASADA Koichi <ko1@...>

[#32748] resut of make test on cygwin — SASADA Koichi <ko1@...>

[#32756] make rdoc cause segv on OpenBSD — SASADA Koichi <ko1@...>

[#32763] Re: [ruby-cvs:21913] Ruby:r14676 (trunk): * trunk/common.mk, goruby.c, golf_prelude.rb: for golfers. — Yukihiro Matsumoto <matz@...>

[#32765] test/ruby/test_transcode.rb の問題 — Martin Duerst <duerst@...>

[#32767] result of test-all on FreeBSD 6.2 — SASADA Koichi <ko1@...>

[#32631] Cygwin 上で Ruby最新版をインストールする場合 — "Koji Tominaga" <crimsonshower@...>