From: danielcavanagh@...
Date: 2008-12-19T08:41:29+09:00
Subject: [ruby-core:20667] Re: 1.9 character encoding (was: encoding of      symbols)

>> well we know how to solve that don't we? make read() read characters
>> not bytes ;)
>>
>> honestly, that seems to be only proper solution. it makes no sense
>> to work with characters everywhere but then read only bytes. reading
>> only bytes should set the string's encoding to binary, and only when
>> the programmer is sure the string is valid utf-8 should he change
>> the encoding.
>
> You can certainly do that.  CSV does:
>
>    #
>    # Builds a String in <tt>@encoding</tt>.  All +chunks+ will be
> transcoded to
>    # that encoding.
>    #
>    def encode_str(*chunks)
>      chunks.map { |chunk| chunk.encode(@encoding.name) }.join
>    end
>
>    #
>    # Reads at least +bytes+ from <tt>@io</tt>, but will read up 10
> bytes ahead if
>    # needed to ensure the data read is valid in the ecoding of that
> data.  This
>    # should ensure that it is safe to use regular expressions on the
> read data,
>    # unless it is actually a broken encoding.  The read data will be
> returned in
>    # <tt>@encoding</tt>.
>    #
>    def read_to_char(bytes)
>      return "" if @io.eof?
>      data = @io.read(bytes)
>      begin
>        encoded = encode_str(data)
>        raise unless encoded.valid_encoding?
>        return encoded
>      rescue  # encoding error or my invalid data raise
>        if @io.eof? or data.size >= bytes + 10
>          return data
>        else
>          data += @io.read(1) until data.valid_encoding? or
>                                    @io.eof?             or
>                                    data.size >= bytes + 10
>          retry
>        end
>      end
>    end

that doesn't read based on characters though. i meant read(1) would read
one whole character, no matter the number of bytes. this reads based on
bytes and just makes sure that there are no broken characters. and does
everyone who wants to work with exotic characters (ie, everyone but
english speakers nowadays, i would imagine) have to put this code into
their applications every time? shouldn't ruby just do it for them? but
this is a separate issue from the one i raised so perhaps we shouldn't go
into that just yet :)