From: "Y. NOBUOKA" <nobuoka@...>
Date: 2011-04-26T18:47:37+09:00
Subject: Re: splitting binary data

On ruby 1.9, a String object knows the encoding of itself.
And, If a String object includes byte sequences unsuitable for the encoding,
the String#split method raises error.

Not using the magic comment, it's not the matter that a string literal includes
non-ASCII characters.

## example: OK!!
#-------------------------------------------------
#! ruby-1.9.2

str = "\xFF\xFF\x61\xFF\xFF\x62\xFF\xFF\x63\xFF\xFF\x64"
p str.encoding         #=> #<Encoding:ASCII-8BIT>
p str.valid_encoding?  #=> true

pattern = "\xFF\xFF"
p str.split( pattern ) #=> ["", "a", "b", "c", "d"]
#-------------------------------------------------

However, using the magic comment to tell the file encoding is UTF-8,
it's the matter that a string literal includes non-ASCII characters.

## example: NG
#-------------------------------------------------
#! ruby-1.9.2
# coding: UTF-8

str = "\xFF\xFF\x61\xFF\xFF\x62\xFF\xFF\x63\xFF\xFF\x64"
p str.encoding         #=> #<Encoding:UTF-8>
p str.valid_encoding?  #=> false

pattern = "\xFF\xFF"
p pattern.valid_encoding? #=> false
p str.split( pattern ) # ERROR OCCURS!!!
#-------------------------------------------------

Avoiding this problem, you must change the encoding of the string which include
non-ASCII characters into ASCII-8BIT.

## example: avoiding the problem
#-------------------------------------------------
#! ruby-1.9.2
# coding: UTF-8

str = "\xFF\xFF\x61\xFF\xFF\x62\xFF\xFF\x63\xFF\xFF\x64"
# change the encoding of the string
str.force_encoding Encoding::ASCII_8BIT
p str.encoding         #=> #<Encoding:ASCII-8BIT>
p str.valid_encoding?  #=> true

pattern = "\xFF\xFF".force_encoding Encoding::ASCII_8BIT
p pattern.valid_encoding? #=> true
p str.split( pattern ) #=> ["", "a", "b", "c", "d"]
#-------------------------------------------------

Kind regards,
-- 
NOBUOKA Yu