From: Nikolai Weibull <now@...>
Date: 2007-10-29T17:50:36+09:00
Subject: Re: Implementation of "guessUTF" method - final questions

On 10/29/07, Wolfgang N�dasi-Donner <ed.odanow@wonado.de> wrote:

> 1) First of all the method will receive the name "guessUTF", and I build
>     it as an instance method in an extension of class "String". The
>     method will be in the file "guessutf.rb". Is this acceptable for you?
>
>     For me it seem the most natural way to ask a String object, if it's
>     first bytes give some hints for a possible Unicode encoding.

Wouldn't it be a lot neater to have something like Encoding::scan that
would take a String and go through all registered encodings and see if
any of them accept the input as encoded with their encoding schemes.
The return value would be an array of matching Encodings.

Any Encoding that can then define a class method that Encoding::scan
would call to see if the Encoding can decode the input.

class Encoding
  def self.scan(input)
    registered_encodings.inject([]) do |matches, encoding|
      matches << encoding if encoding.decodable?(input)
      matches
    end
  end
end

class Encoding::UTF8
  def self.decodable?(input)
    got_one = false
    i = 0
    while i < input.length
      byte = input[i]
      if (byte & 0x80) == 0
        case byte
        when 0..7, 11, 14..26, 28..31, 127
          return false
        end
      elsif (byte & 0x40) == 0
        return false
      else
        following =
          case
          when (byte & 0x20) == 0: 1
          when (byte & 0x10) == 0: 2
          when (byte & 0x08) == 0: 3
          when (byte & 0x04) == 0: 4
          when (byte & 0x02) == 0: 5
          else
            return false
          end
        following.times do
          i += 1
          return got_one if i >= data.length
          return false (data[i] & 0x80) == 0 or (data[i] & 0x40) == 1
        end
        got_one = true
      end
    end
    got_one
  end
end

class Encoding::UTF8BOM
  def self.decodable?(input)
    input[0..2] == "\xef\xbb\xbf" and Encoding::UTF8.decodable?(input[3,-1])
  end
end

> 2) At present the method returns a number between 0 and 5

No magic numbers, please.