From: "." Date: 2008-09-10T07:33:37+09:00 Subject: Re: Drake: Distributed Rake > On Tuesday 09 September 2008 05:03:43 quixoticsycoph...@gmail.com wrote: > > > On Sep 9, 4:52 am, David Masover wrote: > > > > So what do you do with things which aren't thread-save? Or > > > "j-safe"? > > > The same thing you do with a Makefile that isn't j-safe: (1) write > > the dependencies correctly, which makes it j-safe, or (2) don't > > run it with -j. > > Still going to be a fair number of cases of (3), I imagine: use > locks to synchronize non-thread-safe libraries, for which there's > still a benefit to running those tasks in parallel. If by "non-thread-safe libraries" you mean a library whose Rakefile is not j-safe, then you would just run it without -j. If it is a library inside a larger project, you have at least two options: (a) Run single-threaded rake in a subprocess for that library. (b) Use the Rake module directly, as the unit tests do. The no-invoke-inside-invoke rule applies per TaskManager, so you could create a new TaskManager and do whatever you wish with it. Keep in mind this is supposed to be stop-gap as you wait for the author of the Rakefile in question to write the dependencies correctly. > > > If 'task' became 'multitask', Rake would run all your tasks at > > once -- all at the same time. That's probably not what you want > > :) > > Actually, no, I assumed that 'multitask' only ran that specific task > in parallel. > > Actually, I hadn't thought about it thoroughly enough to realize > that this wasn't what was happening: > > > The dependency graph tells us what can be run in parallel and what > > can't. > > I understand make -j, and I think I understand the difference with > multitask -- if I understand it: > > multitask :foo ... > multitask :bar ... > > In the above example, will everything really run concurrently? I'd > assumed that foo would run concurrently, and then bar would run > concurrently. When I mentioned that multitask is an alias of task, you asked about the converse: what if task is an alias of multitask? multitask blindly executes all its prereqs at once, in parallel, one task per thread. So if everything is a multitask, applying the rule recursively, all tasks run at once. Getting back to your question, your code > multitask :foo ... > multitask :bar ... does not say "do foo, then do bar" (whether it's a task or a multitask is not relevant). The code is only defining the dependency relations and the tasks associated with each node, to be executed later. If foo lists bar as a prereq, then bar will run before foo. > > In either case, I see what Drake is doing (real make -j > behavior). Thanks for explaining this -- it looks cool! > > One more thing: I'm not sure what the best way to do this is, but I > think it would still be useful to have the task/multitask dichotomy, > for legacy programs. Multitasks would operate as properly parallized > Drake tasks. Plain old tasks would run in complete isolation, with > the exception that if they invoke a multitask, that multitask (and > all its remaining dependencies) run in j-parallized mode. I had anticipated the suggestion that 'multitask' enable parallelism. It cannot solve the problem. See my other post on superstructures around the graph concept. As I mentioned, drake is 100% compatible with rake for -j1, the default behavior. The only thing we lose is the speedup for a -j1 multitask, which is recovered (and then some) once the dependencies are written correctly and -jN may be run. I don't think it is too much to ask: in exchange for fixing your dependency logic, your build times will divided by N (best case) for an N-core machine. If you don't fix it, then expect a little slowdown if you were using multitask for something nontrival. That situation seems rare enough, and harmless enough, to not worry much about. Also, do we really want to create elaborate special cases, introducing bugs and complexity, just to save someone a few moments (perhaps nothing) during a build? Especially considering this is being done in order to allow the person to retain the logic errors in his code? > > That would certainly break the purity of it, and it would be a bit > more work, but I think it could be made to work. The benefit is, you > could translate an existing project iteratively, without having to > verify that the whole thing is correct, first. > You can already translate it piecemeal, if that is what you meant by iteratively. It simply requires testing the bottom-level tasks individually and working your way up. When I explained this previously, I mistakenly said bottom-level-last-executed. I meant the opposite, of course -- bottom level (most-depended-upon) is executed first.