From: Sam Roberts <sroberts@...>
Date: 2005-03-21T01:13:03+09:00
Subject: Re: Ruby UTF-8

Quoting mailing-lists.ruby-talk@rawuncut.elitemail.org, on Sun, Mar 20, 2005 at 11:25:27PM +0900:
> * Wolfgang N???dasi-Donner (Mar 15, 2005 19:10):
> > > I'm working with Japanese character sets in Windows. I can save my
> > > *.rb files with notepad using UTF-8 but I can't run them with Ruby.
> 
> > The Windows-Editor writes always a "Byte Order Mark" (BOM) at the
> > beginning of UTF-8/16LE/16BE coded files. In this case a UTF-8 coded
> > file begins with "EF BB BF" (hex). These non-characters should usually
> > be ignored (for more information see http://www.unicode.org/).
> 
> Why does it write a BOM for UTF-8 encoded files?  It's utterly
> meaningless to discuss byte order for UTF-8 encoded text,

It's a tag to indicate the data is UTF-8. If it wasn't there it could be
anything, iso-8859-*, koi8, ...

Apple does this in NSString when I convert UCS-2 data to UTF-8, too. It
confused me. I still don't like it, but I have a suspicion that its
partly because these OSes have a legacy "assumed encoding" for 8-bit
text, and that it isn't UTF-8.

In a way it means that there are two "dialects" of utf-8. This is
unfortunate.

Sam