From: "ara.t.howard" <ara.t.howard@...>
Date: 2008-07-12T10:31:10+09:00
Subject: Re: [ANN] threadify-0.0.1


On Jul 11, 2008, at 2:38 PM, Charles Oliver Nutter wrote:

> Ok, there's good news and bad news. First the good news.
>
> I've found several egregious threading bugs in JRuby's Enumerable  
> implementation that probably caused the bulk of errors you saw.  
> Basically, the runtime information for the main Ruby thread in JRuby  
> was getting reused by the blocks passed into threadify, causing all  
> sorts of wacky errors (multiple threads all sharing runtime thread  
> data...fun!). Fixing that seems to have resolved most of the errors.
>
> Now the bad news...
>
> What you're doing is a bit suspect. In this case, it works out  
> reasonable well, since you're just doing a map and gathering  
> results. There's some remaining bugs in JRuby wrt the temporary data  
> structure used to gather map results (it needs to be made thread- 
> safe) but it can work. However in general I don't think this use of  
> threadify is going to apply well to Enumera(ble|tor) since so many  
> of the operations depend on the result of the previous iteration.
>
> I'll have the remaining issues wrapped up shortly, but I'd love to  
> see someone come up with a safe set of Enumerable-like operations  
> that can run in parallel. For example, a detect that uses a cross- 
> thread trigger to stop all iterations (rather than the naive  
> threadification of detect which would not propagate a successful  
> detection out of the thread). Things like that could be very useful.
>
> I'd also love to see someone come up with a nice installable gem of  
> truly thread-safe wrappers around the core collections, since in  
> general I don't believe the core array and friends should suffer the  
> perf penalty that comes from always synchronizing.

check out 0.0.3, it allows this, but the sync overhead is prohibitive  
for in memory stuff - for network scraping it'd be great though.   
anyhow, 0.0.3 allows one the 'break' from parallel processing and the  
value broken with will be the same as if the jobs were run serially.   
damn tricky that.

cheers.

a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being  
better. simply reflect on that.
h.h. the 14th dalai lama