From: Austin Ziegler <halostatue@...>
Date: 2006-06-29T07:01:03+09:00
Subject: Re: Unicode roadmap?

On 6/28/06, gwtmp01@mac.com <gwtmp01@mac.com> wrote:
> On Jun 28, 2006, at 4:25 PM, Jim Weirich wrote:
>>  raise "Not PNG." unless (SINGLE_BYTE_ENCODINGS.include?
>>  (@top.encoding) && @top[0, 8] == "\x89PNG\x0d\x0a\x1a\x0a")
>>
>> Of course, if you can guarantee that @top is indeed a single byte
>> encoded BEFORE hitting this line, then the encoding test is not
>> needed (and I think you assume that).
> Does it even make sense to talk about 'encodings' in the context of
> binary data?
>
> I suppose you could extend the concept of encoding to capture some
> sort of mime type characterization of the data but isn't that a bit
> beyond what this thread has been talking about?
>
> I like Austin's idea of an encoding as a 'lens' with respect to the
> raw data.
>
> It is getting pretty hard to follow this entire discussion in the
> absence of some concrete examples of the imagined APIs as well as some
> sort of taxonomy of use cases with which to evaluate the APIs.  For
> example:
>   - create a copy of a text file when the text encoding is unknown
>   - transmit a copy of a text file across a TCP/IP connection when the
>     encoding is unknown

If you don't know the encoding, you must use binary (unencoded) data.
That will be unchanged ... unless we have a ByteArray.

  without ByteArray, without pragma:
    File.open(a, "r") { |b| File.open(c, "w") { |d| d.write b.read } }

  without ByteArray, with pragma:
    File.open(a, "r", encoding: "binary") { |b|
      File.open(c, "w", encoding: "binary") { |d|
        d.write b.read
      }
    }

  with ByteArray:
    File.open(a, "r") { |b|
      File.open(c, "w") { |d|
        d.write_bytes b.read_bytes
      }
    }

>   - analyze binary data and guess at its text encoding

This is a "hard" problem and not demonstratable easily; there are
expensive programs out there that do this. The problem is that encodings
like ISO-8859-1 and ISO-8859-5 *mean* different things and you'd have
serious textual analysis to determine which you're looking at. On the
other hand, simple analysis (e.g., determining ISO-8859-* but not which
one, as opposed to UTF-8) may be possible.

>   - convert PNG image to a GIF image, in memory, to/from disk

Use RMagick. ;)

>   - input n characters from the keyboard/stdin/tty

I'm not sure, to be honest. This is unlikely to be cross-platform.

-austin
-- 
Austin Ziegler * halostatue@gmail.com * http://www.halostatue.ca/
               * austin@halostatue.ca * http://www.halostatue.ca/feed/
               * austin@zieglers.ca