[ruby-core:40291] Re: Counseling

On Oct 22, 2011, at 9:16 PM, Joshua Ballanco wrote:

> On Saturday, October 22, 2011 at 12:43 PM, Jon wrote:
>> =20
>>> What Ruby needs (IMHO), is the equivalent of Obj-C's NSData class. =
That is, something which can hold a contiguous span of raw bytes without =
encoding, but with the ability to access ranges and iterate over the =
data like a String. I regret that I did not recall this desire of mine =
for the original Ruby 2.0 feature list (I originally encountered the =
need for this when writing the ControlTower server for MacRuby; which, =
consequently, does make use of NSData). I would, however, like to =
propose such a class for Ruby 2.0.
>> =20
>> =20
>> What's your view regarding both the `bytes` (immutable) and =
`bytearray` (mutable) abstractions from
>> =20
>>   http://docs.python.org/py3k/library/functions.html#bytearray
>=20
>=20
> Yes, this sounds like a very similar idea (NSData is immutable and has =
an NSMutableData counterpart). I think the intro for the NSData =
documentation captures the motivation perfectly:
>=20
>> NSData and its mutable subclass NSMutableData provide data objects, =
object-oriented wrappers for byte buffers. Data objects let simple =
allocated buffers (that is, data with no embedded pointers) take on the =
behavior of Foundation objects.
>=20
> Basically, since the Array class in Ruby is designed to hold objects, =
there is an annoying amount of overhead required to use Ruby arrays to =
hold simple bytes (e.g. you have to manually decompose bytes on each =
append operation). On the other hand, since Ruby does its best to always =
do the right thing with encodings for String objects, it can get =
annoying to try and use Ruby strings to hold bytes (you never know when =
your BINARY string might be coerced into UTF-8).=20

I like and agree with this concept but I wonder if we are talking about =
two (subtly) different things.

There really is a problem if I take a string encoded with UTF-8 and try =
to concatenate it with a string encoded with 8859-1 (or one of the more =
exotic character sets).  What I have never understood (and the Ruby =
people have tried to educate me) is why, when I say "utf-8-string" + =
"8859-1-string", Ruby can't just convert the latter to the encoding of =
the first, do the concatenation and be done with it.  So, there is a =
second problem.

And there is a third problem (which is probably a set of problems).  In =
my application, all the data actually starts off as various EBCDIC code =
pages. (http://bit.ly/rtTO8F).  Using ICU =
(http://site.icu-project.org/), I convert these to UTF-8 strings.  I =
store these in a PostgreSQL database (9.0.4) that is set up with UTF-8 =
encoding.  But STILL, frequently, something creates strings that are not =
UTF-8 strings.  As previously stated, I've set all my files to UTF-8 =
coding as well as set -KU but there are still ways for things to get =
botched.  And my whole point here is that what Ruby has ended up doing =
is making simple libraries damn near impossible to write if you really =
really really really want to do things properly.  Any library that =
concatenate any strings is open to mistakes.

pedz