Re: Drake: Distributed Rake

> On Tuesday 09 September 2008 05:03:43 [e-mail] wrote:
>
> > On Sep 9, 4:52 am, David Masover <[e-mail]> wrote:
>
> > > So what do you do with things which aren't thread-save? Or
> > > "j-safe"?
>
> > The same thing you do with a Makefile that isn't j-safe: (1) write
> > the dependencies correctly, which makes it j-safe, or (2) don't
> > run it with -j.
>
> Still going to be a fair number of cases of (3), I imagine: use
> locks to synchronize non-thread-safe libraries, for which there's
> still a benefit to running those tasks in parallel.

If by "non-thread-safe libraries" you mean a library whose Rakefile is
not j-safe, then you would just run it without -j.  If it is a library
inside a larger project, you have at least two options:

(a) Run single-threaded rake in a subprocess for that library.

(b) Use the Rake module directly, as the unit tests do.  The
no-invoke-inside-invoke rule applies per TaskManager, so you could
create a new TaskManager and do whatever you wish with it.

Keep in mind this is supposed to be stop-gap as you wait for the
author of the Rakefile in question to write the dependencies
correctly.

>
> > If 'task' became 'multitask', Rake would run all your tasks at
> > once -- all at the same time.  That's probably not what you want
> > :)
>
> Actually, no, I assumed that 'multitask' only ran that specific task
> in parallel.
>
> Actually, I hadn't thought about it thoroughly enough to realize
> that this wasn't what was happening:
>
> > The dependency graph tells us what can be run in parallel and what
> > can't.
>
> I understand make -j, and I think I understand the difference with
> multitask -- if I understand it:
>
> multitask :foo ...
> multitask :bar ...
>
> In the above example, will everything really run concurrently? I'd
> assumed that foo would run concurrently, and then bar would run
> concurrently.

When I mentioned that multitask is an alias of task, you asked about
the converse: what if task is an alias of multitask?  multitask
blindly executes all its prereqs at once, in parallel, one task per
thread.  So if everything is a multitask, applying the rule
recursively, all tasks run at once.

Getting back to your question, your code

> multitask :foo ...
> multitask :bar ...

does not say "do foo, then do bar" (whether it's a task or a multitask
is not relevant).  The code is only defining the dependency relations
and the tasks associated with each node, to be executed later.  If foo
lists bar as a prereq, then bar will run before foo.

>
> In either case, I see what Drake is doing (real make -j
> behavior). Thanks for explaining this -- it looks cool!
>
> One more thing: I'm not sure what the best way to do this is, but I
> think it would still be useful to have the task/multitask dichotomy,
> for legacy programs. Multitasks would operate as properly parallized
> Drake tasks. Plain old tasks would run in complete isolation, with
> the exception that if they invoke a multitask, that multitask (and
> all its remaining dependencies) run in j-parallized mode.

I had anticipated the suggestion that 'multitask' enable parallelism.
It cannot solve the problem.  See my other post on superstructures
around the graph concept.

As I mentioned, drake is 100% compatible with rake for -j1, the
default behavior.  The only thing we lose is the speedup for a -j1
multitask, which is recovered (and then some) once the dependencies
are written correctly and -jN may be run.

I don't think it is too much to ask: in exchange for fixing your
dependency logic, your build times will divided by N (best case) for
an N-core machine.  If you don't fix it, then expect a little slowdown
if you were using multitask for something nontrival.  That situation
seems rare enough, and harmless enough, to not worry much about.

Also, do we really want to create elaborate special cases, introducing
bugs and complexity, just to save someone a few moments (perhaps
nothing) during a build?  Especially considering this is being done in
order to allow the person to retain the logic errors in his code?

>
> That would certainly break the purity of it, and it would be a bit
> more work, but I think it could be made to work. The benefit is, you
> could translate an existing project iteratively, without having to
> verify that the whole thing is correct, first.
>

You can already translate it piecemeal, if that is what you meant by
iteratively.  It simply requires testing the bottom-level tasks
individually and working your way up.  When I explained this
previously, I mistakenly said bottom-level-last-executed.  I meant the
opposite, of course -- bottom level (most-depended-upon) is executed
first.