Re: Implementation of "guessUTF" method - final questions

Michal Suchanek schrieb:
> On 29/10/2007, Nikolai Weibull <[e-mail]> wrote:
>> On 10/29/07, Martin Duerst <[e-mail]> wrote:
>>> At 17:50 07/10/29, Nikolai Weibull wrote:
>>>> On 10/29/07, Wolfgang N疆asi-Donner <[e-mail]> wrote:
>>>>
>>>>> 1) First of all the method will receive the name "guessUTF", and I build
>>>>>     it as an instance method in an extension of class "String". The
>>>>>     method will be in the file "guessutf.rb". Is this acceptable for you?
>>>>>
>>>>>     For me it seem the most natural way to ask a String object, if it's
>>>>>     first bytes give some hints for a possible Unicode encoding.
>>>> Wouldn't it be a lot neater to have something like Encoding::scan that
>>>> would take a String and go through all registered encodings and see if
>>>> any of them accept the input as encoded with their encoding schemes.
>>>> The return value would be an array of matching Encodings.
>>> In theory, yes. However, for many encoding (e.g. the ISO-8859-X
>>> series), the answer would always be 'yes'. That would make these
>>> methods easy to implement, but the result would be rather useless.
>> Not necessarily.  There are still ranges in 0..255 that the ISO-8859-X
>> don't use, for example,
>> ISO-8859-1 = UNION(7..10, 12..13, 27, 32..126, 133, 164..255).  Also,
>> the question of what is useful here is rather much up to the user.
>> The returned list of Encodings that believe themselves capable of
>> decoding the input would have to be discriminated somehow.  In an
>> interface, for example, the application could present the user with
>> the list of Encodings and let them choose one.
>>
> It is also possible to determine the encoding more reliably if you
> know the language. There is a tool called enca that can determine the
> encoding for a few languages.

I think we should not underestimate the effort guessing the encoding, if the 
list of possible encodings will increase. I had to do with problems like these 
during the last decades very often, and I'm still confronted with data encoded 
in some EBCDIC variants, old mac or Atari encodings, up to very exotic 
Telefunken-ZC1 for TR440.

The official unused ranges may not help very much, if the data isn't 100% 
correct, but hints about the natural language used in texts or - sometimes much 
more helpful - typical personal names, that should be inside the text, can give 
good ideas for the encoding of several characters. Even algorithms for wrong 
written names, like "Soundex" for English names or "Kölner Phonetik" for German 
ones, can help to guess an encoding.

These methods are usually of interest for (very) special usage, and not for 
everybody who processes texts, because (s)he usually knows the encoding of the 
presented text.

Additional things, that are not part of "standard Ruby", are the complete sets 
of Unicode character properties - I speak about this several megabytes C type 
files, downloadable from the Unicode homepage. May be, that I will take these 
data during the nexts months and prepare it for Ruby usage, because I may need 
it sometimes.

But - this are all features, that are very seldom used, and they are expensive 
in respect to space usage. I think, that these not very often used features 
should not shipped together with "standard Ruby", because it will increase the 
package with stuff, that can easy be taken from Ruby-Forge if necessary.

This is the reason I tried to put only a very small method String#check_bom in a 
package possibly shipped together with "standard Ruby", and will put the rest 
into Ruby-Forge projects, where it is available for everybody who needs this 
special functionality.

Wolfgang Nádasi-Donner