From: murphy <murphy@...>
Date: 2007-09-11T11:12:19+09:00
Subject: Re: Encodings and String Literals

> If a string literal ends with \u, \U, \E, or \S (with no hex digits
> following) then the escape specifies the encoding of the string, even
> when the string does not contain any characters outside of the ASCII
> subset.
That's a very original idea, although a bit strange. It actually looks
like a syntax error:

  foo.bar("\uFFFFxyz\uEEEEabc\u")

(As if the programmer just forgot to complete the unicode char.) It also
has no apparent HEREDOC equivalent...

  puts <<MYUNICODESTRING
A lot of strange chars and a
\u
  MYUNICODESTRING

This shouldn't work.

A small thing: If the core team opts for a prefix, then Ruby should not
warn me as the often-seen literal prefix functions do:

  def u(str)
    str  # here be unicode magic
  end

  puts u'test'  # => nil
  >> parenthesize argument(s) for future version

Enhancing the popular %-notation for fancy strings seems like a nice
option, and it wouldn't even break old code. But it may not be intuitive:

  puts %u'Jupiter'    # nice
  puts %x'Saturn'     # how to get Unicode strings executed? %xu? %ux?
  puts %x:u'Mercury'  # ugly!
  puts %r'Venus'u     # regular expression = special case?
  puts %w(Mars Zeus Baldur Aton)u  # suffixes work for other literals
  puts :'Uranus'u     # even Symbols!
  puts <<NEPTUNEu     # and heredocs?
  puts <<"PLUTO"u     # quotes to the rescue :)

I like the suffix variant. Regexp literals already use it (although //u
means something different...I think Unicode encoding should imply
Unicode matching.) For more encodings, we can have e:weird42 as matz
suggested earlier.

We could use ''u'...' to ensure that the encoding modifier comes at the
beginning (Ruby's literal concatenation.) but multiline strings should
be HEREDOC'd anyway.

How would such strings handle interpretation? Maybe I have a US-ASCII
encoded instance variable @foo inside a UTF-8 literal:

  log "#{@foo} [Swahili debugging message suffix]"u

...or a Japanese @foo in an English log message:

  log "I don't want to be a #{@foo}!"

Will @foo be converted to UTF-8 in the first example? Will the final log
message be SJIS in the second? And what about mixing many encodings
together?

I suggest that Ruby 1.9 provides a way to define a default encoding for
a file (with the -*- pragma, perhaps?) *All* the string literals in such
a file would have this encoding by default. If you have a lot of string
literals in an application (think of an Arabic localized web
application), you wouldn't want to use "..."u (or something even longer)
for every one of them. It seems like noise to my Ruby-spoiled eyes.

[murphy]