From: Eric Hodel <drbrain@...7.net>
Date: 2011-02-11T09:53:45+09:00
Subject: Re: ruby 1.9.2 unicode with hex codepoint problem

On Feb 10, 2011, at 12:34 PM, Gilles Gilles wrote:
> I have the following test case:
> 
> $ cat test.in
> Der gro\xdfe BilderSauger
> 
> $ cat test.rb
> File.open('test.in', 'r').each_line do |line|
>  puts line
>  test = "Der gro\xdfe BilderSauger"
>  puts test
> 
> The result is
> Der gro\xdfe BilderSauger
> Der gro?e BilderSauger
> 
> I have tried to put an encoding in the File.open() or line.encode()
> without success. the '\' is recognized as a real '\', not as the
> beginning of an hex escape sequence.
> 
> How can I get \xdf to be recognized as ß when reading from a file?

Unicode codepoint 00df needs to be written in a particular encoding.  I'll choose UTF-8.

$ echo 'Der große BilderSauger' > test.in
$ hexdump -C test.in 
00000000  44 65 72 20 67 72 6f c3  9f 65 20 42 69 6c 64 65  |Der gro..e Bilde|
00000010  72 53 61 75 67 65 72 0a                           |rSauger.|
00000018

In test.rb you also need to set UTF-8 for this to work:

$ cat test.rb
# coding: UTF-8

File.open('test.in', 'r').each_line do |line|
  puts line
  test = "Der große BilderSauger"
  puts test

end
$ ruby19 test.rb
Der große BilderSauger
Der große BilderSauger