From: Patrick Hurley <phurley@...>
Date: 2006-06-15T22:13:41+09:00
Subject: Re: A plan for another unicode string hack

On 6/15/06, Austin Ziegler <halostatue@gmail.com> wrote:
> On 6/15/06, Paul Battley <pbattley@gmail.com> wrote:
> > On 15/06/06, Yukihiro Matsumoto <matz@ruby-lang.org> wrote:
> > > |I agree with Austin on this - the distinction is too vague. I'd leave
> > > |length and size the same and make a size_in_bytes method.
> > >
> > > On my latest prototype (not checked in anywhere), String#length and
> > > String#size behave same, and there is String#buffer_size to return
> > > size in bytes.  The method name might change in the future.
> > Actually, this makes a lot of sense. Why would you ever want to know
> > the actual byte length of a UTF-8 string? It's pretty meaningless for
> > most string-processing tasks: the main times you would need it would
> > be in allocation and interfacing with external systems and libraries.
> > Thus, something like buffer_size maps to real-world usage extremely
> > well, in my opinion.
>
> I agree. #buffer_size or #byte_size is probably sufficient. #size and
> #length should return the number of glyphs in the string.
>
> -austin
> --
> Austin Ziegler * halostatue@gmail.com * http://www.halostatue.ca/
>                * austin@halostatue.ca * http://www.halostatue.ca/feed/
>                * austin@zieglers.ca
>
>

I have to concur. If we are taking votes, I like the byte_ prefix:
byte_size, byte_length -- and these will remain important even within
pure ruby (e.g. when doing block based IO).

Which actually leads to an important question how does the proposed
(2.0 or this library) string deal with incomplete trailing data? In
particular, if I am gathering data off a socket and end up with half
of an utf-8 extended key code pair at the end of my buffer -- what
will happen? Also this will happen at beginning of strings as well
(when I continue reading on socket and the rest of the data arrives).

One solution is to collect data in a non-encoded string, but we still
need a reasonable method to encode "as much as possible" and leave any
trailing bytes available for future byte based operations.

pth