From: David Flanagan <david@...>
Date: 2007-11-09T16:08:02+09:00
Subject: Re: primary encoding and source encoding

Nobuyoshi Nakada wrote:
> Hi,
> 
> At Fri, 9 Nov 2007 05:42:47 +0900,
> David Flanagan wrote in [ruby-core:13315]:
>> Q1) In step 1 above, should the default primary encoding come from the 
>> locale environment variables (LC_ALL, LC_CTYPE, and LANG) instead of 
>> defaulting to ASCII?
> 
> It's planned, but we have no mappings from locale name to
> encoding name.  Attached is a quick hack I tried the week
> before last.

Did you consider nl_langinfo(CODESET)?  Your code looks good to me, 
except that you don't check LC_CTYPE.  Is it your intent to be 
conservative and assume ASCII unless the locale explicitly specifies an 
encoding name following a .?  You're not going to choose either EUC-JP 
or SJIS as the default for Japanese locales?

The public-domain code at http://www.cl.cam.ac.uk/~mgk25/ucs/langinfo.c 
might be useful if you want to be more liberal about guessing encodings 
based on language codes.

>> Q2) If yes, to the above, then shouldn't we drop step 5 above?  If my 
>> locale specifies UTF-8 as my primary encoding then I don't think that 
>> should be changed just because I run a script developed by a Japanese 
>> programmer and encoded in EUC-JP.
> 
> Yes, I think.

Thanks.

	David

> 
> Index: ruby.c
> ===================================================================
> --- ruby.c	(revision 13847)
> +++ ruby.c	(working copy)
> @@ -133,4 +133,34 @@ usage(const char *name)
>  }
>  
> +static rb_encoding *
> +locale_encoding(void)
> +{
> +    const char *lang = getenv("LC_ALL");
> +    int idx = 0;
> +    rb_encoding *enc;
> +
> +    if (!lang) lang = getenv("LANG");
> +    if (lang && (lang = strchr(lang, '.')) != 0) {
> +	const char *at = strchr(++lang, '@');
> +	int len = (at ? at - lang : strlen(lang));
> +	char buf[32];
> +	if (len < sizeof(buf)) {
> +	    MEMCPY(buf, lang, char, len);
> +	    buf[len] = 0;
> +	    idx = rb_enc_find_index(buf);
> +	    if (idx < 0 && len > 3 &&
> +		(strncasecmp(buf, "euc", 3) == 0 ||
> +		 strncasecmp(buf, "utf", 3) == 0) &&
> +		buf[3]) {
> +		MEMMOVE(buf + 4, buf + 3, char, len - 2);
> +		buf[3] = '-';
> +		idx = rb_enc_find_index(buf);
> +	    }
> +	}
> +    }
> +    enc = rb_enc_from_index(idx);
> +    return enc ? enc : rb_enc_default();
> +}
> +
>  extern VALUE rb_load_path;
>  
> @@ -851,5 +881,5 @@ process_options(VALUE arg)
>      NODE *tree = 0;
>      VALUE parser;
> -    VALUE encoding;
> +    rb_encoding *enc;
>      const char *s;
>      int i = proc_options(argc, argv, opt);
> @@ -980,10 +1010,10 @@ process_options(VALUE arg)
>  
>      if (opt->enc_index >= 0) {
> -	encoding = rb_enc_from_encoding(rb_enc_from_index(opt->enc_index));
> +	enc = rb_enc_from_index(opt->enc_index);
>      }
>      else {
> -	encoding = rb_parser_encoding(parser);
> +	enc = locale_encoding();
>      }
> -    rb_set_primary_encoding(encoding);
> +    rb_set_primary_encoding(rb_enc_from_encoding(enc));
>  
>      return (VALUE)tree;
> 
>