From: Martin Duerst Date: 2007-12-19T14:17:24+09:00 Subject: Re: Another last-minute Ruby 1.9 question Hello David, At 03:35 07/12/19, David Flanagan wrote: >Martin Duerst wrote: >> Hello David, >> At 15:34 07/12/18, David Flanagan wrote: >>> Here's another last-minute question, probably for Matz or Nobu... >>> >>> Are the encoding supported by transcode.c going to be merged with the encodings in the enc/ directory? Or will only iso-8859-1 be fully supported in 1.9.1? > >Thanks for the clarifications, Martin! > >> We have had some discussions on the Japanese list. In terms of >> actual functionality, the encodings truely supported by Ruby >> internally (mostly equivalent to what you see in enc/) will >> always be a subset of the encodings supported by transcode.c. >> As a clear example, there is no plan to make Ruby work with EBCDIC >> internally, but adding a few EBCDIC variants to transcode.c isn't >> a big deal. > >If encode can't *always* return a string whose encoding method specifies the encoding, then I suggest that it should *always* return a binary string. With "always", do you mean even if the encoding is fully supported? >That's what the name "encode" means, after all: converting characters to bytes. Very good point, in my opinion. I had some discussions with Matz, proposing to use transcode instead of encode as the method name. I used the same argument as above. Matz argued that encode would be familiar to many more people than transcode. The problem with your proposal above is that it means that in a usual program, you would have to write string.encode('UTF-8').force_encoding('UTF-8'), which really seems unnecessarily long and not Rubylike. >> What Nobu proposed and implemented is to *label* each string with >> a correct encoding. So assume there is a transcoding to an EBCDIC >> variant, labeled EBCDIC-FOO. You should then be able to do >> 'abc'.encode('EBCDIC-FOO').encoding >> and get "EBCDIC-FOO", but this will be to some extent misleading >> because EBCDIC-FOO will just be a replica of ASCII-8BIT, and won't >> behave like an EBCDIC encoding at all. > >But what does "behave like" mean for encodings? As far as I can tell from encoding.{c,h}, the only non-trivial feature of single-byte encodings is case folding. It looks like Oniguruma encodings also support ctype testing, but I'm not sure that Ruby uses this feature. I don't think Ruby uses it directly, but you can use it in any Regexp you write, which means it's exposed in Ruby. For some encodings, there are quite a few more tests than the traditional ctype ones, for example, both EUC-JP and Shift_JIS can test for Hiragana or Katakana, and UTF-8 can test for a whole lot more, scripts and properties. But you are right that "behave like" is a rather shallow concept. >I would rather have case-folding and ctype testing on on an EBCIDIC encoding abort with an error than apply incorrect ASCII case-folding to the string. My point here is this: can't we trivially implement any single-byte encoding if we allow case folding to report an error? And isn't reporting an error better than pretending to be ASCII and doing case folding wrong? This would amount to making EBCDIC encodings replicas of a (as of yet not existing) BINARY encoding rather than of ASCII-8BIT. To me, that makes sense. It would mean that for all encodings that are supported by transcode.c, we would need to know whether it's ASCII-8BIT-like or BINARY-like. But then we get some new problems: iso-8859-x are clearly ASCII-8BIT-like, but if Oniguruma's iso-8859-1 is exposed, then should windows-1252 be ASCII-8BIT-like, or should it be derived from iso-8859-1? You get some more correct results with the later e.g. for case testing and folding, but they still won't be correct. And it will become more difficult to figure out what's supported to what level. And then there is also the issue of 2-byte encodings such as UTF-16. >Also, Oniguruma already defines the ISO-8859 encodings. When I compile them, the striped object files are just 1780 bytes each, so it wouldn't be too heavy handed to actually import these encodings into Ruby as pre-defined ones, would it? No, it wouldn't. I think that will eventually happen. Nobu already worked on dynamic linking for encodings recently, so I guess it should be easy. But it is probably somewhat low priority, and it won't solve the basic problem that transcode.c will support more encodings than the Encoding machinery. >> But we are not yet 100% sure whether this is the right behavior, >> or whether these should just be labeled with an encoding of ASCII-8BIT >> or maybe even something more like BINARY. >> >>> (Currently the encoding method of the string returned by encode("iso-8859-15") returns ASCII") >> The very latest version I have behaves differently, please see >> test/ruby/test_transcode.rb, around line 75: >> assert_equal("D\xFCrst".force_encoding('iso-8859-15'), "D\u00FCrst".encode('iso-8859-15')) >> (It may be that you hit the bug that I described in the bottom half >> of [ruby-dev:32591]. If yes, please send a test case, thanks.) > >I can't duplicate anymore. That one should actually be fixed by a very recent patch from Nobu, so that may be the reason. Regards, Martin. #-#-# Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University #-#-# http://www.sw.it.aoyama.ac.jp mailto:duerst@it.aoyama.ac.jp