Re: Encoding::primary_encoding

On 22/10/2007, David Flanagan <[e-mail]> wrote:
>
> Here's a section from chapter 2 of my forthcoming book. Search for
> "comment" to find the paragraph about magic encoding comments.  This
> section is my current understanding of -K, -E, --encoding, BOM, and
> magic comments. If you find that the current Ruby 1.9 behavior differs
> from this, I'd like to hear about it.
>
>      David
>
>        <sect3 id="specifyingencoding">
>         <title>Specifying Program Encoding</title>
>         <para>
>           By default, the Ruby interpreter assumes that programs are
>           encoded in ASCII. In Ruby 1.8 you can specify a different
>           encoding with the <literal>-K</literal> command-line option.
>           To run a Ruby program that includes Unicode characters encoded
>           in UTF-8, invoke the interpreter with the
>           <literal>-Ku</literal> option. Programs that include Japanese
>           characters in EUC or SJIS encodings can be run with
>           <literal>-Ke</literal> and <literal>-Ks</literal> options.
>         </para>
Yes, this is true. The -K option also specifies the way standard input
and output is handled. This also means that to run a program in a
different locale you have to *recode* the *source*. Quite awkward, and
I had to do it for a program :-S
However, in 1.8 this is inevitable because the encoding is global and
you cannot handle different strings differently so you need special
data for each encoding anyway.
>
>         <para>
>           Ruby 1.9 also supports the ASCII, Unicode, EUC and SJIS
>           encodings. New encodings can be plugged in, so your build or
>           distribution of Ruby may suppport additional encodings as
>           well.  The <literal>-K</literal> option works in Ruby 1.9 as
>           it doe sin Ruby 1.8.  Two new option, <literal>-E</literal>
>           and <literal>--encoding</literal> are also supported in Ruby
>           1.9 and allow you to specify an encoding by its full name or
>           alias.  For example:
>         </para>
>
>
> <programlisting><![CDATA[
> ruby -E utf-8            # Encoding name follows -E
> ruby -Eutf-8             # The space is optional
> ruby --encoding utf-8    # Encoding following --encoding with a space
> ruby --encoding=utf-8    # Or use an equals sign with --encoding
> ]]></programlisting>
>
>         <para>
>           Encoding names are not case sensitive and may be written in
>           uppercase or lowercase, or a mix. The following encodings (at
>           a minimum) are defined.  Note that some of these are aliases
>           for the same encoding:
>         </para>
>
> <programlisting><![CDATA[
> ASCII-8BIT      UTF-8           SHIFT_JIS       EUC-JP
> ASCII                           SJIS
> BINARY
> US-ASCII
> ]]></programlisting>
>
>         <para>
>           In Ruby 1.9 you can write your files so that they specify
>           their own encoding, and if you do this, then you don't need to
>           use <literal>-K</literal>, <literal>-E</literal> or
>           <literal>--encoding</literal> when invoking the interpreter.
>           (Though you can still use those options to override the
>           specification in a file)
>         </para>
>
>         <para>
>           A program encoded in UTF-8 identifies its encoding if it
>           starts with the three byte sequence <literal>0xEF 0xBB
>           0xBF</literal>. These bytes are known as the BOM or "Byte
>           Order Mark" and are optional in UTF-8 encoded files.
>         </para>
>
>         <para>
>           More generally, however, you can identify the encoding of a
>           Ruby file by placing a special comment on the first (or
>           second: see below) line of the Ruby file.  The comment must be
>           written entirely in ASCII, and must include the string
>           "coding" followed by colon or equals sign and the name of the
>           desired encoding (which cannot include spaces or punctuation
>           other than hyphen and underscore).  Whitespace is allowed on
>           either side of the colon or equals sign, and the string
>           "coding" may have any prefix, such as "en" to spell
>           "encoding". The entire comment, including "coding" and the
>           encoding name is case-insensitive and can be written with
>           uppercase or lowercase letters.
>         </para>

However, this also changes the encoding for standard input and output
so a program that specifies encoding for its source file will likely
garble any input and output (because it is likely that the system
locale is different from the source encoding).
I am not sure if Ruby has any logic at all to match the system locale
and the locale expected on the input and output. In 1.8 the default
encoding is ascii regardless of locale.

>
>         <para>
>           Encoding comments are usually written so that they also inform
>           a text editor of the file encoding.
>           <emphasis>Emacs</emphasis> users, for example can write:
>         </para>
>
> <programlisting><![CDATA[
> # -*- coding: utf-8 -*-
> ]]></programlisting>
>
>         <para>
>           And <emphasis>vi</emphasis> users might write:
>         </para>
>
> <programlisting><![CDATA[
> # vim: set fileencoding=utf-8
> ]]></programlisting>
>
>         <para>
>           An encoding comment like that shown above is usually only
>           valid on the first line of the file.  It may appear on the
>           second line, however, if the first line is a "shebang" comment
>           like this:
>         </para>
>
> <programlisting><![CDATA[
> #!/usr/bin/ruby -w
> ]]></programlisting>
>
>         <para>
>           A comment of this form makes a Ruby program directly
>           executable on Unix-like operating systems.
>         </para>
>        </sect3>
>
>
Thanks

Michal