From: Junhui Liao <junhui.liao@...>
Date: 2010-08-02T06:10:48+09:00
Subject: Re: my script just read one line?

Hi, Jesus.

Thanks a lot for your help!
I modified a little to the script and  make it running as expected.

Here is the code:

def write_line_to_file every_line, base_time = Hash.new(0)
  every_line.each_slice(2).with_index do |(time,signal), index|
    File.open("header_split_#{index}"+".tsv" , "a") do |f|
      f << "#{time.to_f - base_time[index].to_f}\t#{signal}\n"
    end
  end
end

#count = 0
  first_line = file.readline.chomp.split("\t")
 # counter +=1
 # if counter >= 2
 # puts "here!"
  first_line_times = first_line.each_slice(2).map{|time,signal| time}
  file.each_line do |record|
    line_data = record.chomp.split("\t")
      write_line_to_file line_data, first_line_times
  end
  end


However, there existed two items need to be improved at least.
Item 1, this code took ~2 hours to save into 4096 files.
BTW, the original tsv file is around 250M. I wonder if there exist
some tricks to make it speed up?

Item 2, the original data has 21 lines header. Although it could be
deleted then read by the script. But I do want to update the script
to make it exclude the fist 21 lines header.

I tried two ways to do this job, but it failed.

The first way was referencing from cvs file's header reading.

require 'csv'
reader =  File.open("../data/test_7_10lines.tsv") do |file|
header = reader.shift
reader.each {|row| process(header,2)}#Suppose the first two lines are 
header.

The error code seemed no
undefined method `shift' for nil:NilClass (NoMethodError)

Another try is, insert the two lines like above code commented.

  counter +=1
  if counter >= 2

But it seemed the counter +=1 did not work at all, since
counter always <=1 !

What's wrong and any suggestions?

Best regards,
Junhui
-- 
Posted via http://www.ruby-forum.com/.