From: Tony Arcieri Date: 2009-06-10T03:34:30+09:00 Subject: Re: Ruby 1.9.x Concurrency --0016368e1f86d99e0e046bee9bef Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On Tue, Jun 9, 2009 at 12:11 PM, s.ross wrote: > More to the point, I created an app using MRI that relied on downloading a > boatload of information from a Web service. Single threaded, this took about > 20 minutes, where using multiple threads, it was accomplished in 3-5 > minutes. However, this one involves a good deal of trickery so as not to > step on buffers in the net/http libraries (or something underlying). > > So I come back to the question: As we find ourselves with resources that > scale across processing units, how best does Ruby solve the problem and what > role do Fibers play in that solution, if any? Having written what's probably the fastest concurrent HTTP fetcher available in Ruby, here's a bit on how it worked in practice: We set the system up to allow N HTTP fetching "agents", each of which would attach to a message queue and indicate their availability for accepting jobs. Want it to go faster? Just make N bigger. A command and control process would then pick and idle fetcher agent and send it a batch of URLs to fetch. It used a lightweight concurrency library I wrote called Revactor which is based around Fibers. Each fetcher process used 64 Fibers which would pull from the URL buffer in a round robin fashion. If you're curious how this works, the core logic for this process is distributed as part of Revactor's standard library: http://github.com/tarcieri/revactor/blob/master/lib/revactor/http_fetcher.rb We ran one of these fetcher processes per CPU core of the systems we were running them on. They were rather CPU intensive as they did a lot of regex processing on the fetched documents. That said, it didn't take much: we were able to suck in 30 megabits of data at once using just four processes running on a single quad core system. -- Tony Arcieri medioh.com --0016368e1f86d99e0e046bee9bef--