From: Martin Duerst <duerst@...>
Date: 2008-09-21T16:59:54+09:00
Subject: [ruby-core:18769] Re: Reading non-ascii compatible files

At 22:40 08/09/18, Tanaka Akira wrote:

>ASCII incompatible encoding may be stateful as ISO-2022-JP.
>An IO opened by "w:enc" needs an encoding converter to hold
>the state.  A newline converter can be attached to that.
>
>An IO opened by "r:enc" don't need an encoding converter.

Some more questions/comments:

- There is no need for a converter on output for non-stateful
  encodings (the majority of encodings are non-stateful).
  What's the overhead of using a converter on non-stateful
  encodings?
- There still could be a converter on input if it's needed
  for newline conversion.
- A converter would usually convert from one encoding to another.
  But when somebody writes "w:iso-2022-jp", the only thing we
  know is the 'to' encoding. How is this handled?
- Keeping the state may work for data from other encodings being
  output with iso-2022-jp. But data labeled as iso-2022-jp
  internally is dummy, i.e. just bytes. That seems it might lead
  to a somewhat paradoxical situation: output will be clear
  and correct when actually converting (e.g. from EUC-JP or so)
  but not when writing data out directly (from iso-2022-jp).

Sorry for that many questions.   Regards,   Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp