[ruby-core:19105] Re: Encoding.default_internal

----- Original Message ----- 
From: "Martin Duerst" <[e-mail]>
To: <[e-mail]>
Sent: Thursday, October 02, 2008 2:36 PM
Subject: [ruby-core:19103] Re: Encoding.default_internal


>>But I agree with you: if a library can be confident that its inputs are at
>>least ASCII-comptible, quite a bit of your efforts could be saved.
>>If on top of that, if it can be reasonably sure that all its inputs are
>>encoding compatible, then it's even better.
>
> I think this is not about confidence. In the software world,
> there is no confidence about input. It's much more about what
> expectation a library sets and documents.

I think you may have missed my point.

My previous posts talk about "encoding naive" libraries (see 
[ruby-core:19013]), and these are the ones I am referring to, and they are 
probably the bulk of Ruby's libraries. My aim is to make writing of such 
libraries, and conversion from 1.8 versions as painless as possible.

As an example, imagine a very cut-down version of the CSV library with no 
encoding awareness at all, which just takes 2 inputs: a "separator" string, 
and a data string to be split into fields. Let's say the separator string 
being used is a non-ascii UTF-8 character, and the input data string came 
over the network from somewhere. If the encoding of the data string is not 
UTF-8, then the CSV library is going to raise an Encoding Compatibility 
Error at some stage. It may also be quite hard to trace the source of the 
problem.

With "default_internal" set, hopefully this situation will be avoided.

So the question is:
Is it worth making the library encoding aware (like James did with the real 
CSV library) by adding code to check & convert encodings?

I would argue that the answer to that is "no" for most libraries, but only 
if "default_internal" is set. And that is because the library will then have 
a reaonable expectation that the inputs are in both the "default_internal" 
encoding.

Naturally programmers can change encodings, but they have to do it 
consciously if "default_internal" is set. So if they get an "Encoding 
Compatibility Error" from a non-encoding aware library then it possibly 
won't come as a much of a surprise to the programmer. So, as a liibrary 
developer, I may not care enough about this to code around it.

Cheers
Mike