From: Austin Ziegler <halostatue@...>
Date: 2006-06-26T23:51:51+09:00
Subject: Re: Unicode roadmap?

On 6/26/06, Julian 'Julik' Tarkhanov <listbox@julik.nl> wrote:
> On 26-jun-2006, at 3:11, Austin Ziegler wrote:
> >  st = File.open("file.txt", "rb") { |f| f.read(4056) }
> and you recieve a PART of a unicode string (because you cannot know
> where to stop reading before yoy look into the structure).
> The only way to make what you read valid now is to slide along the
> byte length and try to catch the bytes that you skipped.
> Should I continue?

Sure. It won't make you any more correct. Let's play with your example:

  st = File.open("file.txt", "rb", :encoding => :utf8) { |f| f.read(4096) }

Okay. Am I reading 4096 bytes or 4096 characters? The *correct* and
*least surprising* behaviour is to read the specified number of bytes.
Instead it would be better to expose the minimum amount required to
work with this:

  bv = File.open("file.txt", "rb") { |f| f.read(4096) }
  bv.encoding = :utf8
  bv.encoding_valid? # will return false if the whole string isn't a
valid UTF-8 sequence.

You're really looking for something that is, in the end, completely
unworkable and unnecessarily complex in doing so. The m17n String --
with byte vector characteristics retained -- maintains a clear, simple
API with few exceptions that would have to be memorised or understood.
Adding another class *doubles* the size of the class hierarchy that
has to be understood, and if there are *any* variances between them
the number of exceptions effectively doubles. If there *aren't* any
variances between the class APIs, then what's the point of separating
them in the first place?

A string is an ordered sequence of characters. A byte vector is an
ordered sequence of bytes. If your string is suitably flexible, then
it can say that a byte vector is a string where each character is one
byte long and that collation (etc.) are determined by the byte value.
We're not talking rocket science here. Stop trying to make it such.

-austin
-- 
Austin Ziegler * halostatue@gmail.com * http://www.halostatue.ca/
               * austin@halostatue.ca * http://www.halostatue.ca/feed/
               * austin@zieglers.ca