From: Michal Suchanek <hramrach@...>
Date: 2008-01-12T01:18:38+09:00
Subject: Re: nil encoding as synonym for binary encoding

On 11/01/2008, Dave Thomas <dave@pragprog.com> wrote:
>
> On Jan 11, 2008, at 12:06 AM, David Flanagan wrote:
>
> >> I'm certainly not an expert on encoding, so take this with a grain
> >> of salt, but it seems to be that there _is_ a different between a
> >> string tagged ASCII-8BIT and a string with no encoding. A string
> >> with no encoding should remain unchanged under operations such as
> >> #upcase, whereas upcase on an ASCII-8BIT string could return a
> >> string with different content.
> >
> > Yes, but keep in mind that Encoding::BINARY is a synonym for
> > Encoding::ASCII_8BIT. I just think that nil should work like "no
> > encoding". The closest we've got right now is Encoding::BINARY.
>
> It is now. I'm suggesting that is a mistake: Encoding::Binary should
> not be an alias, but a separate encoding, so that
>
>     "cat".force_encoding("ascii-8bit").upcase  # => "CAT"
>     "cat".force_encoding("binary).upcase       # => "cat"
>
> The reason for this is that if a string is truly binary, it contains
> bytes, not characters. There's no way of breaking it into characters,
> so there's no way you can perform character-related operations on it.
> (For example, it might be UTF-32, for all you know, so upcasing it
> byte by byte would be silly). Or it might be a TCP packet, so upcasing
> it is again meaningless. Only when you know the encoding of a sequence
> of bits can you then treat them as characters.
>
> >> Or, looking at it another way, a string with no encoding contains
> >> bytes but no characters.
> >
> > Exactly.  And when you want to deal with bytes the best you can do
> > is to use Encoding::BINARY
>
> Right, not ASCII-8BIT.

Well, ascii-8bit is the way the strings are mostly used. If you
claimed "nothing known about this, no character operations allowed"
some operations would get very inconvenient.

For one, somebody suggested that
"aaa".force_encoding(Encoding::BINARY)[0] makes no sense as you do not
know how to break into characters. Perhaps byte indexing should be
allowed then?

Then if you want to find out whether there is "GIF"  at the start you
cannot because "GIF" is an ASCII string but your string is
unknown-binary. Or should the binary string allow byte comparison
(searching, ..) with strings of any encoding (even ecbdic or whatever
it is called)? If there is no searching these byte buffers are pretty
useless, right?

Since upcasing (and chomp, and splitting) can't work on binary (and in
this case it's probably even desirable) we got rid of quite a few
methods.

What about regexps then? These would be useful if we collected all the
variants of the GIF or JPEG signature for example. They could possibly
be used but regexps of multibyte encodings are troublesome. Should
they match anywhere - that is even at places other than character
boundaries?

Thanks

Michal