From: David Masover <ninja@...>
Date: 2008-09-11T12:51:58+09:00
Subject: Re: Drake: Distributed Rake

On Tuesday 09 September 2008 17:33:37 . wrote:

> > Still going to be a fair number of cases of (3), I imagine: use
> > locks to synchronize non-thread-safe libraries, for which there's
> > still a benefit to running those tasks in parallel.
> 
> If by "non-thread-safe libraries" you mean a library whose Rakefile is
> not j-safe, then you would just run it without -j.

Which is, by the way, one of the most irritating things about Makefiles.

While in the simple case, a Makefile author might not know about -j, and write 
a safe Makefile anyway (because that's really simpler, after all), it's 
really troublesome that there's no standard way to tell whether something's 
j-safe or not.

Seems like the best I can do is run something with -j, and if it seems to 
work, well, hope for the best.

That's one reason I like multitask -- it forces the programmer to be 
explicitly thinking about threading.

> If it is a library 
> inside a larger project, you have at least two options:
> 
> (a) Run single-threaded rake in a subprocess for that library.
> 
> (b) Use the Rake module directly, as the unit tests do.  The
> no-invoke-inside-invoke rule applies per TaskManager, so you could
> create a new TaskManager and do whatever you wish with it.

I was talking about an even simpler problem:

Let's pretend, for a moment, that we're talking to an HTTP library that's not 
thread-safe. Our Rakefile, for whatever reason, needs to download stuff and 
then work with it. So we can't use this HTTP library directly -- we need to 
wrap synchronization around it.

But, there's still an advantage to running the actual meat of the tasks in 
parallel -- maybe we're doing some complex hpricot parsing, and we're 
connecting to a potentially-slow server. Ideally, we want to download as fast 
as we can, but once it's downloaded, we want to start crunching in worker 
threads.

So there's still a benefit to Drake/Multitask, but there's the added 
complexity of having to wrap that non-thread-safe library.

Contrived example, I know.

That's an advantage to single-threaded Rake, by the way -- by default, you 
don't need to think about any of this. (-j1 isn't an excuse, unless the 
Rakefile can force it, because then it's up to the user to figure out what 
j-level to use. That should be transparent.)