From: Adam Akhtar <adamtemporary@...>
Date: 2009-04-22T22:27:20+09:00
Subject: Re: reformatting a text file that has some binary in it

Im thoroughly confused and have spent a good 10 hours getting nowhere 
fast. Im gong to throw my monitor against the wall!

I have a file with text like the stuff in posts above. I dont create the 
file, its given to me as a standard text file. I dont know how it is 
encoded. Im assuming utf-8. There is your standard readable english 
lower 128 ascii and then there are bits of garbled crap that are 
supposed to be binary.

I do the following

$KCODE = "UTF8"

then i do

data_a = File.read('mn-scrape.txt')
data_b = File.open("mn-scrape.txt", "rb").readlines.join("")
data_a.scan(/./m).length ( ==> 170799 )
data_b.scan(/./m).length ( ==> 767702 )

why are they different?
When I look in notepad++ viewing the file under the utf-8 encoding it 
says the num of characters is 767702 which is nearly 4 times bigger that 
the .read version

Why is this happening?

What is the correct way to open this type of file? Any help whatsoever 
will be a great great great help!

-- 
Posted via http://www.ruby-forum.com/.