[ruby-core:30146] Re: Suggestion regarding m17n

2010/5/10 Czarek <[e-mail]>:
> MY QUESTIONS
>
> 1. Does it make sense to expect libraries and frameworks for Ruby to
> work with an ASCII incompatible internal encoding, e.g "-E :UTF16_BE"?
> Could we consider failing to do so a bug?

No, ASCII imcompatible encoding is not a primal encoding.

This is because:
* Cannot run as shell script
* Cannot work with ASCII only strings without conversion

ASCII only strings are generated in many situations, for example
Object#to_s, foo.name and so on.
Auotomatic conversion of ASCII only strings for ASCII incompatible
but includes ASCII related characters was discussed before Ruby 1.9.0.
But it is gave up because of its tight schedule.

> 2. If so, would it be reasonable to expect the same of Ruby's standard
>  library?

Some library may fail. It depends the library's policy.

> 3. If so, which Ruby version should people test against?
>
> * a specific 1.9.2 revision?

This is about testing MRI and bug reporting?
We recommend trunk, 1.9.2-head and 1.9.1-head.

We MRIer develops each branches, don't develop already released version.
So if people check the branch's head, we needn't say "it is already
fixed in branch head".
It is happy for us.
(if it is fixed in trunk but not in branch, it is "backport request")

> 4. Would trying to add support to applications running in ASCII
> incompatible environments be useful to developers in the long run
> (taking into account future versions of Ruby) or just be an
> unnecessary activity (e.g. because of UTF-8 which is ASCII
> compatible)?

Supporting ASCII incompatible encoding is hard work.
Because we cannot use literals:
  str == ?a
  /foo/ =~ str

Because of such difficulty we gave up source encoding and so on.

> EXPLANATION
> REASONS

They looks reasonable.

> The downside would be:
>
> - many more problems will surface than would normally occur in
>   reality

Yes. I think most of trouble is from literals.

> - people interested only in ASCII compatible encodings would have to
>   do more work or give up on encoding support

If we recommend implement ASCII incompatible encodings support,
we should implement something to ease supporting them.

> - developers may have to become aware of regexp, 'filesystem',
>   'locale' and other encodings - even if they want things to "just
>   work"
>
> - problems would have to solved near the cause, which requires time
>   and knowledge to do properly - adding ".encoding" calls in random
>   places won't be enough
>
> - handling all the issues in the most correct and
>   backward-compatible way would make the code uglier, especially
>   with regexps
>
> - effort needed to persuade developers to pro-actively test their
>   software this way, instead of expecting test cases and patches.

Testing seems hard problem.

> CASE STUDY:
>
>As an experiment, I tried to get irb and then tests to run without
>warnings, starting with the following:
> ...
> Although getting all the tests running will still require touching
> quite a few files.
>
> NOTE: I tried my best to when choosing each workaround, but I believe
> there are much cleaner solutions out there. I also admit that I gave
> up on trying to get irb to actually do something other than just
> start.

It shouldn't be acceptable for users to have to write encode('filesystem')
for each require filenames. So at least automatic conversion is needed.
But it should need more, so we made the category: ASCII incompatible
and separete UTF-16 and UTF-32.

> The problems I encountered were usually related to:
> - regexp handling
> - ENV variables (reading and writing) in locale which were not
>   compatible with default_internal
> - using concat operations like File.join, split, interpolation
> - comparing strings with different encodings silently returns false

They should be fixed in Ruby itself if we make ASCII incompatible encoding
first class encoding.

> Some other issues that may become more apparent as a result of trying
> to get things to work:
> - filesystem encoding / locale mismatches

Locale encoding itself should be rarely used in real script.
It is Encoding.default_external, isn't it?

> - program argument encodings

Ah, this is locale encoding.

> - file system encoding differences for different mount points

Explicitly set encoding argument to API like Dir.entries(".",
encoding: "UTF-8").

> - environment variables containing non-ascii or non-utf data

This is considered hard to decide expected encoding.

>  - lack of encoding info from stdlib

Intended.

> - stdin/stdout issues

what?

> - gracefully handling corrupt data

Use case required.

> - combinations of the above and other

?

> As a side note, I was wondering if regexp support couldn't be extended
> to better support what was done in the CSV library - encoding the
> regexp to match the default/input. Perhaps with a cleaner syntax or
> making the existing notation more flexible, since regexp are used like
> a Swiss Army Knife for many different things. Then again, I may be
> missing something important.

Auto-recompiling regexp for ASCII incompatible encoding?

> I understand what I propose may be crazy and require a insane amount
> of work to support. Although I do *not* believe that the flexibility
> Ruby provides and the support of vague cases (e.g. ASCII-7BIT safe
> concat, default encodings) is meant to be treated as a standard for
> new applications now and in the future.

Ruby M17N can potentially support those case.
Most of current limitation for ASCII incompatible encoding is
because of human resource.

> Instead, IMHO this flexibility should be used appropriately: for easy
> transitions, backward compatibility, performance and other instances
> that are genuinely useful. In every other case, I believe putting
> additional effort into avoiding transcoding where possible and
> generally honoring the user's selection of encoding, whatever he or
> she may choose, will provide more benefit for everyone in the long
> run.

If you find some new API can help people, please tell us.
We welcome such suggestions.

> Please correct me if I am wrong.

You are correct in most part.
Thank you for your effort.

-- 
NARUSE, Yui
[e-mail]