From: Pen Ttt <myocean135@...>
Date: 2010-08-14T10:03:00+09:00
Subject: synchronize  between download and save  in  multi-threads?

i want to download all usa listed company historial data from yahoo
finance,
at first i download  all compamy name list and save it in
/home/pt/usastock/totalname(amex nyse nasdaq,6000 companies),here is my
program to dowload their historial data from yahoo finance with
multi-threads:
require 'rubygems'
require  'open-uri'
threads = []
puts time0=Time.now
i=0
j=0
str=[]
open('/home/pt/usastock/totalname','r'){|item|
while line=item.gets
line=line.chomp
i=i+1
str<<"http://ichart.finance.yahoo.com/table.csv?s="+line
  if i==10   or  item.eof == true  then
    begin
      for page_to_fetch in str
             threads<<Thread.new(page_to_fetch) do |url|
              open(url){|urlfile|
              datafile="/home/pt/test/"+url.split("=")[1].to_s
              open(datafile,'w'){|file|
              file.puts  urlfile.read
              j=j+1
              puts  "#{url}  #{j} is over"
              }}
              end
              end
             threads.each {|thr| thr.join}
             i=0
             str=[]
             threads = []
    rescue Timeout::Error => e
    rescue => e
    end
  else
  end
end
}
puts Time.now-time0
it can run seccessfully,when i change
 if i==10   or  item.eof == true  then   (which means i open 10 threads)
into
 if i==100   or  item.eof == true  then  (which means i open 100
threads)
there is a problem,some data will not saved in my designated  position,
(datafile="/home/pt/axel/"+url.split("=")[1].to_s)
some of them are saved such as "/home/pt/table.csv?s=YHOO"
according to my program,it should be saved as  "/home/pt/test/YHOO"
i think that maybe it is a problem of synchronizimg  between download
and save
in my program.
would you mind to fix it?
thinks.
-- 
Posted via http://www.ruby-forum.com/.