From: "Wolfgang Nádasi-Donner" <ed.odanow@...>
Date: 2007-11-03T06:33:48+09:00
Subject: Re: Implementation of "guessUTF" method - final questions

Hi!

I am now finishing the methods and the description. I hope
the result will be usable in a simple and fast way.

I added the following instance methods for class String:

1) "check_bom" as described before

2) "remove_utf8_bom" as described before (removes bom only,
    if self.encoding is utf-8)

3) "remove_utf8_bom!" as described before (removes bom only,
    if self.encoding is utf-8)

4) "add_utf8_bom", which adds an utf-8-BOM in front
    of a copy of self, if self.encoding is utf-8. Otherwise
    a copy of self is returned

5) "add_utf8_bom!", which adds an utf-8-BOM in front
    of self, if self.encoding is utf-8, and returns self.
    Otherwise "nil" is returned.

6) "check_utf8_encoding" checks self completely on basis of
    "Table 3-7. Well-Formed UTF-8 Byte Sequences" (Unicode 5.0),
    or "Table 3-6. Well-Formed UTF-8 Byte Sequences"
    (Unicode 4.0). Both have the same contents.
    The method returns "nil" if the encoding of self isn't
    utf-8 or if the contents of self doesn't follow the rules
    described in the Unicode tables. If the check succeeds, the
    number of recognized utf-8 characters (Unicode codepoints)
    will be returned (this may include a BOM if present).

I've added the last method to complete the stuff around utf-8.
The method doesn't make detailed tests for unused ranges or
else, this will be left to more complex Unicode analysis methods.

The software will delivered as a rc-version on sunday, 4th
(November 2007 ;-) ) to Ruby-Forge (project "guessUTF").

Required changes will be done afterwards, with one exception...

..."check_utf8_encoding" expects an encoding of "utf-8",
otherwise it will not test the string. I'm not sure about my
decision there. The rational behind is, that I expect an
utf-8 encoded string, which I want to check, and not an
unknown encoding, for which I hope, it will conform to utf-8
rules.

What do you think about it?

Wolfgang N��dasi-Donner