From: David Masover Date: 2008-09-11T13:07:53+09:00 Subject: Re: Drake: Distributed Rake On Tuesday 09 September 2008 17:38:40 . wrote: > > There is a mathematical reality we cannot avoid, from which > special-case syntax and backwards-compatibility acrobatics cannot save > us. The problem is in our thinking. We didn't specify what depends > on what. We thought we did, but it turns out we were fooling > ourselves all along. That's not always the problem. Given that Rake itself doesn't guarantee any kind of ordering, we have to assume that dependencies are specified correctly, or close to it. But we're not writing Erlang, which means spec-ing dependencies correctly isn't enough. > Trans suggested that this > > task :a => [:x, :y, :z] > > should be translated into this > > task :a => :z > task :z => :y > task :y => :x > > while this > > task :a => [[:x, :y, :z]] > > is translated into this > > task :a => :x > task :a => :y > task :a => :z > > OK, but there are a million ways in which a programmer can > insufficiently define dependencies. This will not come close to > saving us. No, but it does take us back to the behavior of Rake, or of Drake -j1. If you really want to provide bug-for-bug compatibility, dig into the Rake code and figure out what the ordering should be. > There is already a historical precedent with Makefiles. A new syntax > could have been added to Makefiles, but none was. The Makefiles had > bugs, but instead of timidly skirting around the problems while > praising the gods of backwards compatibility, people faced them > head-on, solving them one at at time. Some did, yes. And some let their Makefiles remain, with the existing syntax and bugs, and left it to their users to figure out whether they could be parellized or not. I'm sorry, but if you're already asking me to manually run a rake task, you don't get to also ask me to read the source code of your Rakefile and figure out whether or not it will work with -j2. Nor should I have to use trial and error, potentially with very subtle bugs, to figure out what's happened. And it's worth mentioning again: We're not writing Erlang, we're writing Ruby. That means shared memory. It means locking issues. And it means thread-unsafe libraries. It means that a Rakefile could very well crash if run with -j2. Understand, I don't mean it will be run in the wrong order, or that the dependencies are wrong. The dependencies may well be perfect, and it will run exactly as designed to. Except that at some point, two separate tasks will simultaneously do something a library won't like, and that library will deadlock. Or segfault. Or worse, give corrupt data. Which means that the Rakefile author is responsible, then, for fixing the deficiencies in the library. Or they have to contact the library author, and attempt to get the library fixed. Making every single Ruby library thread-safe is a laudable goal, but also not going to happen. You could solve a lot of that, I suppose, by forking instead -- but that introduces its own problems.