Re: Implementation of "guessUTF" method - final questions

Martin Duerst schrieb:
> First, I think guessUTF is too general, if the method only returns
> the values below.
Hmm - but "guessing the UTF" is what this small method does.
It goes through the beginning of a String object an looks, if
the first bytes can be interpreted as one of the possible BOM
patterns, which describe an specific UTF encoding. "guess_bom"
is also possible. I leave the decision up to you here in
Ruby-Core (...but I've already asked for a Project "guessUTF"
in Ruby-Forge).


Martin Duerst schrieb:
> Also, as far as I understand, true Ruby style
> the values below. Also, as far as I understand, true Ruby style
> naming would be guess_utf.
Accepted, I will rename it to "guess_utf".


Martin Duerst schrieb:
> String is definitely a good place to have this. It may also be
> nice to have this on an IO stream, in which case it would ideally
> drop the BOM and make the remaining output to be in the encoding
> detected.
I think it can be used in IO if wanted. The problem is, that
it cannot be guaranteed, that a (e.g.) utf-8 BOM preceeds uft-8
encoded data, because it can be binary or some 8 Bit encoding
which allows the byte patterns as characters. I see this
method more as a hint, which encoding the data may have. If I
expect corrupted data, I can take further analysis actions. On
the other hand, when expecting well formatted data, I can use
the resulting value for a decision to delete the BOM.

In addition I see only support for utf-8 encoding in Ruby 1.9.n.
Insofar it would only make sense to use the method in IO to
remove a found utf-8 BOM. In all other cases the data should not
be changed.

I prefer to have it as a method, that gives me something like a
"structured hint" about a possible utf-encoding of a String object.


Martin Duerst schrieb:
> One way of dealing with this that I have once used was to
> return the 'guessed' encoding (e.g. as a string) and the length
> of the BOM in bytes. Returning the length of the BOM in bytes
> is helpful to skip the BOM, which is one of the most important
> operations in case there is a BOM.
I agree. This additional information is really necessary, and
the guessed utf-encoding is a better return value than a number.
This means, that the return value will be a Hash or an Array object.
Again its up to you in Ruby-Core to decide what you want.


Martin Duerst schrieb:
> I don't understand that. Why do you need a finite state machine
> to check the first few bytes for the presence of a BOM?
Oh - it is a misunderstanding. I wanted to say, that I don't use
backtracking at all. I go straight trought the maximum of four bytes
to find a decision. I wrote this only to explain, that it is fast.


I want to leave this small method as simple and fast as possible.
Extended "guessing methods" are much more complicated and expensive,
because the results are usually not definit (binary data is always
possible, ISO-8859-nn also), and this extended functionality is
seldom used. The BOM is, one can hate it or love it, a standard at
least on Windows systems, so everybody should be able to find out,
if this data is there. Ruby is large enough (as often stated in
posts), so I don't want to enlarge it artificially.

Wolfgang N叩dasi-Donner