[ruby-core:40269] Re: Counseling

To try and cut to the core of the issue: in Ruby 1.8 it was common practi=
ce to use the String class to represent both =22proper strings=22 as well=
 as a =22bag-o-bytes=22. In Ruby 1.9, you can only properly use the Strin=
g class to represent =22proper strings=22. =46or a =22bag-o-bytes=22 we'r=
e left with Array, but there are times when Array is not the right abstra=
ction (e.g. reading data from a socket, identifying a start and stop toke=
n, and writing the bytes between to a file on disk). Also, the =22BINARY=22=
 encoding is not the right abstraction, because you still have an object =
which will worry about encodings and, due to Ruby always trying to do =22=
the right thing=22, bugs can be very difficult to track down. Consider:

    > a =3D =22test=22.force=5Fencoding('BINARY')
    > b =3D =22=5Cx=46=46=22.force=5Fencoding('BINARY')
    > a << =22test=22
    =22testtest=22
    > b << =22test=22
    =22=5Cx=46=46test=22
    > a << =22t=C3=A9st=22
    =22testt=C3=A9st=22
    > b << =22t=C3=A9st=22
    Encoding::CompatibilityError: incompatible character encodings: ASCII=
-8BIT and UT=46-8

What Ruby needs (IMHO), is the equivalent of Obj-C's NSData class. That i=
s, something which can hold a contiguous span of raw bytes without encodi=
ng, but with the ability to access ranges and iterate over the data like =
a String. I regret that I did not recall this desire of mine for the orig=
inal Ruby 2.0 feature list (I originally encountered the need for this wh=
en writing the ControlTower server for MacRuby; which, consequently, does=
 make use of NSData). I would, however, like to propose such a class for =
Ruby 2.0. =20


On =46riday, October 21, 2011 at 9:45 PM, Eric Hodel wrote:

> On Oct 21, 2011, at 9:43 AM, Perry Smith wrote:
> > Rails, thin, rvm, almost nothing is really and truly ruby 1.9 complia=
nt -- not really. Not when you include all the encoding problems that are=
 still very common in my life and I assume in the life of anyone trying t=
o use Ruby in any serious fashion.
> =20
> In order to generate correct documentation RDoc may need to transcode y=
our source files into the output encoding you desire.
> =20
> > Is there a compile time option (or can one be added) that says =22I d=
on't care=21=21=21 -- just cram the two strings together and eat your spi=
nach=21=22
> =20
> Mashing strings of different encodings together destroys data, but Ruby=
 will automatically handle compatible encodings. You can concatenate a UT=
=46-8 string with a US-ASCII string, for example.
> =20
> > Because, ultimately, I've yet to find anything except ruby that actua=
lly cares.
> =20
> I care, because I hate to see text like 'This =C3=A2=E2=82=AC=C5=B8enco=
ding=C3=A2=E2=82=AC=C2=9D stuff is =E2=80=A6' on a website. (Yeah, I know=
 fancy-quotes are an equal abomination, but it's not that hard to be awar=
e of encodings, is it=3F)
> =20
> On Oct 21, 2011, at 5:52 PM, Perry Smith wrote:
> > Just as good of an alternative would be to change my default to UT=46=
-8 instead of US-ASCII.
> =20
> This will not fix your problem, nor will -KU fix your problem. They'll =
only mask your problem.
> =20
> The correct solution is to add the encoding magic comment to files that=
 matches the expected encoding of the strings they create. Blindly forcin=
g all strings to UT=46-8 will break libraries that depend on their string=
s being in US-ASCII encoding.
> =20
> See:
> =20
> https://github.com/rdoc/rdoc/commit/ca7651a8b9e6ef32dfa56f4ca618d9cff6b=
a8b74
> =20
> https://github.com/rdoc/rdoc/issues/63
> =20
> You will need to send patches to the library maintainers to mark their =
required encodings correctly, or file tickets.
> =20
> > My first attempt to solve this was to put a UT=46-8 coding into all m=
y ruby files. This appeared to help but upon reflection, I don't think it=
 really did. Adding the -KU in scripts like thin's startup script helps m=
ore. In fact, I think it solves 99.99% of my problems. But when I update =
thin (for example) I forget to add the -KU to the script and hit errors u=
ntil I add the -KU back.
> =20
> Let's get concrete.
> =20
> Show us an error you get when running thin without any modification and=
 I can help you and the maintainer of thin (or whatever other library) fi=
nd the appropriate changes to make for it to work correctly.
> =20
> Through our combined efforts at a concrete task we may even be able to =
make it easier for authors to avoid such a pitfall.
> =20
> > One recent saga involved memcache-client (which I've mentioned). memc=
ache-client tries to concatenate a command, key, and Marshall'ed data. If=
 the Marshalled data really is ASCII-8BIT, then the concatenation dies.
> =20
> Marshal data is always ASCII-8BIT. If memcache-client doesn't set the e=
ncoding to US-ASCII (compatible with ASCII-8BIT) then -KU will break it:
> =20
> =24 ruby19 -e 'a =3D =22text=22; b =3D =22=5Cx=46=46=22; a.force=5Fenco=
ding Encoding::US=5FASCII; b.force=5Fencoding Encoding::BINARY; p (a + b)=
.encoding'
> =23<Encoding:ASCII-8BIT>