From: Martin Duerst <duerst@...>
Date: 2007-12-31T19:01:26+09:00
Subject: Re: multibyte strings & bucket-of-bytes efficiency under 1.9.0

Hello Akira,

Sorry for responding here on the English list.

At 12:17 07/12/31, Tanaka Akira wrote:
>In article <14544702.post@talk.nabble.com>,
>  Brent Roman <brent@mbari.org> writes:
>
>> Perhaps sysread and syswrite should be
>> defined in terms of ByteStrings while
>> the higher level I/O methods would deal
>> exclusively with encoded strings.
>
>sysread returns ASCII-8BIT string always regardless of IO's
>internal/external encoding.  Others which takes length also
>returns ASCII-8BIT string.  (readpartial, read with length,
>etc.)
>
>ASCII-8BIT is similar to ByteString except 0x00 to 0x7f
>represent ASCII characters.  0x80 to 0xff represent special
>characters dedicated for eight bit bytes.
>(Note that GNU Emacs has similar character sets:
>eight-bit-control and eight-bit-graphic.)
>
>I think ASCII-8BIT is more natural than ByteString.  We
>don't distinguish ASCII and BINARY strictly.  For example,
>I say, GIF file has a "GIF" at first 3byte.  This depends on
>ASCII.

I think it's a bit overkill to claim that we better use
ASCII-8BIT than BINARY if out of a file that can easily
be 100KB or more, and where virtually all byte values
that look like ASCII characters are not characters at all,
just because of three bytes at the start of the file.

>Apart from that, String methods depends on ASCII characters.
>For example, String#count uses HYPHEN-MINUS for ranges,
>CIRCUMFLEX ACCENT for complements, REVERSE SOLIDUS for
>escapes.  String#delete and String#squeeze also use them.
>
>ByteString has no such characters.  How do we specify
>ranges, complements and escapes?

Count uses just a very tiny part of the Regexp syntax,
so see below.

>Of course, Regexp highly depends on ASCII characters.  How
>do we specify repetition without ASTERISK and other ASCII
>characters?

We of course can't do this. But it's not necessary. There
are, simply put, two strings participating in a regexp
operation: The regular expression and the 'target string'.
The regular expression needs a 'real' encoding, in most
cases ASCII-8BIT will be sufficient. The target string
can be just bytes. We just need a few conventions to
do the right things, e.g. we can agree that '.' matches
one byte (rather than one character). That's very easy
to implement. We can also limit non-meta characters to
e.g. just \xHH notation, to make clear that we are just
matching byte values and not actual characters. But
imprementing that is probably quite a bit of a hassle
for little benefit.

Even currently, we can use an ASCII-8BIT regexp with
many, many other encodings, so using it with BINARY
isn't anything much new.


>So it is difficult to introduce non-ASCII encoding.  When
>binary encoding is introduced, all ASCII dependency should
>be examined and re-designed.

Well, it's actually not so difficult, and it will be needed.
We can't label a String as e.g. UTF-16, and claim that
UTF-16 is some kind of ASCII-compatible encoding.

Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp