Re: Implementation of "guessUTF" method - final questions

Hello Wolfgang,

Some comments below.

At 12:11 07/10/29, Wolfgang N〓dasi-Donner wrote:
>Dear Ruby designers, developers, and testers!
>
>As recently announced in topic "Hopefully Complete List of Possible
>Encoding Specifications - Existing Ones", I provide the method "guessBOM"
>for discussion now. If it may be helpful for user that the method may be
>in Ruby 1.9.1 standard distribution, than I would like to discuss details
>before finishing the method (it isn't as complicated and large).
>
>The original intention for a module "UTFchecker" containing methods "utf8?",
>"utf16?", "utf32?" , and "utfBOM?" was to identify incomplete and corrupted
>textual data, which I usually expect to receive for analysis. The method
>"utfBOM?" is different from the other insofar, as it should only give a hint
>what Unicode encoding might be used in the file, if it is Unicode encoded
>at all. The others are very slow and much more complicated, because they
>support several possibilities in reporting and other reactions.
>
>For a possible usage inside standard distribution I suggest to ship the
>BOM checking method without the others and make it as handy and small as
>possible to avoid overhead. The other methods will be a different project
>in future.
>
>My proposal:
>
>1) First of all the method will receive the name "guessUTF", and I build
>    it as an instance method in an extension of class "String". The
>    method will be in the file "guessutf.rb". Is this acceptable for you?
>
>    For me it seem the most natural way to ask a String object, if it's
>    first bytes give some hints for a possible Unicode encoding.

First, I think guessUTF is too general, if the method only returns
the values below. Also, as far as I understand, true Ruby style
naming would be guess_utf.

String is definitely a good place to have this. It may also be
nice to have this on an IO stream, in which case it would ideally
drop the BOM and make the remaining output to be in the encoding
detected.

>2) At present the method returns a number between 0 and 5 with the
>    following meaning:
>
>    0 - no BOM recognized (either there is no BOM or the String object
>        is too short)
>    1 - UTF-8 BOM recognized
>    2 - UTF-16LE BOM recognized
>    3 - UTF-16BE BOM recognized
>    4 - UTF-32LE BOM recognized or UTF-16LE with 0x0000 after BOM
>    5 - UTF-32BE BOM recognized
>
>    I don't know if you prefer different return codes, maybe Symbol
>    objects or predefined constants. It will be a simple change if
>    there are other preferences.

One way of dealing with this that I have once used was to
return the 'guessed' encoding (e.g. as a string) and the length
of the BOM in bytes. Returning the length of the BOM in bytes
is helpful to skip the BOM, which is one of the most important
operations in case there is a BOM.

>3) The method is commented and readable structered about 150 Lines
>    of simple Ruby code (it's a finite state machine).

I don't understand that. Why do you need a finite state machine
to check the first few bytes for the presence of a BOM? Or are
you checking the encoding also? If you do the later, why are you
throwing away the information (i.e. you return 0 for UTF-8, but
without BOM as well as for some arbitrary data that isn't UTF-8).

Regards,     Martin.



#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]