From: Martin Duerst <duerst@...>
Date: 2007-12-19T14:17:24+09:00
Subject: Re: Last-minute encoding proposal

Hello David,

Many thanks for some more great thoughts.

At 05:25 07/12/19, you wrote:
>In my response to Martin's message about encodings supported by transcode.c vs. encodings supported by encoding.c, I suggested that if the String.encode() method cannot always return a string whose encoding method returns a valid Encoding object, then it should always return a binary string instead.  I bolstered my argument by saying that this is what "encode" means: converting characters into bytes.
>
>It occurs to me now that force_encoding does the opposite: it converts bytes into characters, and that therefore, a more appropriate name would be "decode".  While this might be a little more confusing at first, it is more orthogonal and more logical, and I think it would be a better name choice in the long run.

This proposal is appealing in its internal logic, but I'm affraid
that it will be heavily confusing to all newcommers.
'encode' and 'decode' fit very well with programming models that
use a single internal character representation. Typical examples
would be Java and Perl, the later actually using these terms.
The Ruby model, however, is different. The main purpose of
both 'transcode' (using that name for the moment for clarity,
the actual method name is currently 'encode') and 'force_encoding'
are to change from one encoding to another. In the case of
'force_encoding', the 'origin' encoding indeed often is just
binary, but 'transcode' can be used just as well to convert
from some (for Ruby) binary stuff such as EBCDIC to e.g. ASCII
or UTF-8 as to convert from ASCII or UTF-8 to some (for Ruby)
binary stuff such as EBCDIC or UTF-16.

Expressed in other terms, 'transcode' is "change the encoding,
keep the characters (change the bytes)", whereas 'force_encoding'
is "change the encoding, keep the bytes (change the characters)".
Using 'encode' and 'decode' to name this functionality makes it
very difficult to understand what's actually going on.


>I also think that the fact that there is one set of encodings supported by the Encoding class and another set supported for transcoding has the potential to be a big source of confusion.

Yes indeed. The confusion will be greatest where:
- An encoding is in daily use and people are thus assuming support
- The encoding is close to e.g. ASCII and thus many things "just work"
  (but of course sometimes, strange things will happen)

>One way to address this is to use strings for transcode.c encoding names and Encoding objects for encoding.c encodings.

Yes, but that wouldn't be very duck-typing-like.

>So I propose the following:
>
>1) Rename force_encoding to decode! and define a non-mutating version named decode.  The argument must be
>an Encoding object or nil.  Encoding names as strings are not allowed.
>
>2) Change the String.encoding method to return nil for byte strings
>
>3) Change the String.encode() method as follows:
>
>   If the encoding of the receiver is non-nil then it expects one
>   argument--the destination encoding.
>
>   If the encoding of the receiver is nil, then it must take two
>   arguments, the destination encoding and the source encoding.
>
>   If the destination encoding is specified as a string, then the return
>   string is a byte string with an encoding of nil.
>
>   If the destination encoding is specified as an Encoding object, then
>   the return value is a character string with the specified Encoding
>   object associated with it.
>
>   It would look something like this:
>
>   def encode(to, from=nil)
>
>     if !from
>       from = self.encoding
>       raise ArgumentError if !from
>     end
>
>     # Now perform the transcode operation somehow
>     result = transcode(self, to, from)
>
>     if (to.is_a? Encoding) result.decode!(to)
>
>     result
>   end
>
>The benefits of this proposal are:
>
>a) It gives us nice, logical method names encode and decode
>b) It clarifies the distinction between byte strings and character strings: byte strings have a nil encoding
>c) It helps (a little) to distinguish between Encoding objects that can be used to interpret bytes as characters and encoding names that can be used as transcoding targets but which are not otherwise supported.
>
>I assume that encoding issues have been discussed heavily in Japanese on ruby-dev already, and I'm well aware that it is late in the process, but I'm throwing these ideas out there anyway. My apologies if these ideas have already been considered and rejected.

It is indeed late in the game. My take is that this is an area where
we need more feedback from practical users. We have had quite a bit of
discussion on the Japanese list, but it hasn't really been conclusive
on all points. How encoding that Ruby doesn't support will be handled
may change in a future version.


Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp