From: Keegan Dunn <theweeg@...>
Date: 2004-12-14T03:53:04+09:00
Subject: Thread and HTTP troubles

I'm trying to write a threaded program that will run through a list of
web sites and download/process a set number of them at a
time(maintaining a pool of threads that can process page
downloads/processing).  I have something simple working, but I am
unsure how to approach the "pool" of threads idea.  Is that even the
way to go about processing multiple pages simultaneously?  Is there a
better way?

Also, how can I deal with a "socket read timeout" error?  I have the
http get call wrapped in a begin...rescue...end block, but it doesn't
seem to be catching it.  Here is the code in question:

def getHTTP(site)
	siteHost = site.gsub(/http:\/\//,'').gsub(/\/.*/,'')
	begin
		masterSite = Net::HTTP.new(siteHost,80)
		siteURL = "/" + site.gsub(/http:\/\//,'').gsub(siteHost,'')
		resp, data = masterSite.get2(siteURL, nil)
		return data
	rescue
		return "-999"
	end
end


Sorry about the two for one question :-P

Thanks!