From: James Gray <james@...>
Date: 2009-08-08T00:30:07+09:00
Subject: Re: R1.9 mixed encoding in file

On Aug 7, 2009, at 10:20 AM, Vít Ondruch wrote:

> James Gray wrote:
>> On Aug 7, 2009, at 9:47 AM, Vít Ondruch wrote:
>>
>> You don't really have to:
>>
>
> It is disturbing that this approach will fail as soon as the file is
> UTF-16 encoded or it has BOM for UTF-8, etc.

You are not allowed to set the source encoding to a non-ASCII  
compatible encoding, if memory serves.  That eliminates any issues  
with encodings like UTF-16.  This makes perfect sense as there's no  
way to reliably support the magic encoding comment unless we can count  
on being able to read at least that far.

A BOM could be handled similarly to what I showed.  You need to open  
the file in ASCII-8BIT and check the beginning bytes, then you could  
switch to US-ASCII and finish reading the first line (or to the second  
if a shebang line is includes), then switch encodings again if needed  
and finish processing.

James Edward Gray II