From: "Wolfgang Nádasi-Donner" <ed.odanow@...>
Date: 2009-02-25T01:36:20+09:00
Subject: [ruby-core:22418] [Question]utf-8 data contains BOM - by intention or by accident?

Hi!

One short question. When I'm reading an utf-8 encoded file (Windows, 
Ruby 1.9.1-p0, built using MinGW/MSYS) the Data contains the BOM if present.

File.open('text.txt','r:utf-8') do |f|
   s = f.read
   p s.encoding                       # => #<Encoding:UTF-8>
   p s.length                         # => 7
   p s.bytesize                       # => 9
   puts s.codepoints.to_a[0].to_s(16) # => feff
   p s                                # => "���Helau!"
end

The BOM is not part of data as namend in Unicode standard, so I mus 
remove it before processing the data.

Is this the planned and stable behaviour for Ruby 1.9.1 an future Ruby 
1.9 versions?

I don't have any problems with this, but it should not change later on 
because it the "remove BOM is there" code parts must be present in 
nearly every Ruby program.

Wolfgang N�dasi-Donner