From: Stanley Xu <wenhao.xu@...>
Date: 2011-03-23T00:37:51+09:00
Subject: Re: How could I make the Ruby 1.9 string ignore the invalid utf-8 byte sequence in split?

--bcaec52e6017df6dd6049f14066f
Content-Type: text/plain; charset=ISO-8859-1

Hi Joey,

I don't think that's the problem. It is probably a file with utf-8
characters. Like 1 millions lines could be split well, but 1000 of them will
get the "invalid bytes sequence error".

Now I have a temporary solution like the following:

if !line.valid_encoding?
      line = line.unpack('C*').pack('U*')
end
fields = line.chomp.split("\t")

But I really doubt it is a good solution, for the invalid character might
means a valid sequence in gbk or something like that.

Isn't there a way I could split the string in ruby 1.9 in the old 1.8 "dirty
way"?

Best wishes,
Stanley Xu



On Tue, Mar 22, 2011 at 11:26 PM, Joey Zhou <yimutang@gmail.com> wrote:

> cannot transform to UTF-8.
>

--bcaec52e6017df6dd6049f14066f--