[ruby-core:31439] Re: [Feature #3595] Theres no encoding to differentiate a stream of Binary data from an 8-Bit ASCII string

On Jul 21, 2010, at 11:26 PM, Dreamcat Four wrote:

> Issue #3595 has been updated by Dreamcat Four.
> 
> 
> Hi,
> Well I was unaware of this. In that case the argument Bill has can be seen as an issue. Reading a file with the IO object would read the ASCII tags, and you wouldn't know what to do. The tags map to both Ascii 7-bit and ascii 8-bit anyway.

Keep in mind, there isn't really any such thing as 8-bit ASCII. The ASCII standard only defines 128 characters. This is why ASCII-8BIT == BINARY. The "ASCII" part implies that there are no multibyte characters, the "8BIT" implies that there are no invalid character bytes (even though there are only, strictly speaking, 7 bits worth of meaningful characters).

> What prompted me to report this:
> 
> Translating data from a Ruby hash object and simple Ruby types into a Plist representation. To give users a standard and appropriate way to differentiate between their Ruby strings which are either textual (ascii or unicode), and their persistent binary data.

I'm assuming that you're referring to OS X's plists? And I'm also assuming that you want to differentiate between dumping <string> nodes versus <data> nodes, yes? If so, the I'm afraid that encoding is not your issue. The issue is that Ruby (ab)uses String for "Strings" in the traditional sense and "Just someplace to put an arbitrary array of bytes".

This is something that I've been dealing with a lot, lately, and I'd been meaning to put forward a more thought out proposal. In Cocoa, there is NSString and friends for traditional strings, and NSData for arbitrary length data. I think a divide like this could be useful for Ruby in the future. In particular, as I said before the issue is not specific encodings, it's that encodings don't mean anything for binary data.

Think of it this way: What are the contents of a "String" object? Characters, right? The fact that these characters are stored as bytes is an implementation detail, and one that, if we could all just move toward a common encoding like Unicode, most programmers could forget about. Even today, the only reason you would need to know anything about encodings is if you were consuming characters from a source outside your control. More importantly, though, is that if you change the bytes as a consequence of a change to the encoding, the content of the String (i.e. the array of characters it represents) does not change. On the other hand, if you're overloading a String to store bytes, then a change in encoding is catastrophic. That's because what you were really storing wasn't a "String".

To get to the root of your problem: If you are using RubyCocoa or MacRuby, NSData is there for you (and you'll probably get improved performance to boot). If you want to do something in pure ruby to generate plists with <data> nodes, I would suggest creating a custom "Data" class for this purpose, and leave alone the topic of encodings.

Cheers,

Josh