From: "Joe Wölfel" <joe@...>
Date: 2008-09-30T01:19:59+09:00
Subject: Re: Rake dependencies unknown prior to running tasks

I've had a chance to play with your solution a bit.   It is a much  
better solution than my other earlier crude solution that relaunched  
Rake.   With your solution it seems all you need to do is call  
restart at the end of a task that creates new tasks.  It's very  
simple.  Everything just seems to work and previously executed tasks  
aren't executed twice.  In my solution they are executed twice, which  
is bad (or at least expensive).  Also, in my solution I have to hard  
code Rake parameters and it gets especially hairy when I have more  
than one task that creates other tasks.

I  noticed that your patch doesn't seem to work with multitask for  
some reason.  I'm not sure why.  Also, as simple as it is to throw  
restart I'm wondering if it's possible that this could be done  
automatically - maybe with a warning for users who do it  
inadvertently.   If a task is defined at any point while a task is  
executing then restart could be thrown automatically when the  
executing task completes.   Then Rake could automatically support  
dynamic task creation.  Would that make sense?


On 25 sept. 08, at 19:23, James M. Lawrence wrote:

>
> The following is a better implementation which could be made to work
> with Drake.  A patch for regular Rake follows.
>
>   task :setup_a do
>     puts "setup_a"
>   end
>
>   task :setup_b do
>     puts "setup_b"
>   end
>
>   task :setup => [:setup_a, :setup_b] do
>     puts "setup phase complete. defining new tasks..."
>
>     task :main_a do
>       puts "main_a"
>     end
>
>     task :main_b do
>       puts "main_b"
>     end
>
>     task :main => [:main_a, :main_b] do
>       puts "main phase complete."
>     end
>
>     puts "restarting..."
>     throw :restart
>   end
>
>   task :main
>
>   task :default => [:setup, :main] do
>     puts "all done."
>   end
>
> diff --git a/lib/rake.rb b/lib/rake.rb
> index 36c2734..1e360a6 100755
> --- a/lib/rake.rb
> +++ b/lib/rake.rb
> @@ -573,8 +573,8 @@ module Rake
>            puts "** Invoke #{name} #{format_trace_flags}"
>          end
>          return if @already_invoked
> -        @already_invoked = true
>          invoke_prerequisites(task_args, new_chain)
> +        @already_invoked = true
>          execute(task_args) if needed?
>        end
>      end
> @@ -1994,7 +1994,14 @@ module Rake
>          elsif options.show_prereqs
>            display_prerequisites
>          else
> -          top_level_tasks.each { |task_name| invoke_task(task_name) }
> +          catch(:done) {
> +            loop {
> +              catch(:restart) {
> +                top_level_tasks.each { |task_name|
> invoke_task(task_name) }
> +                throw :done
> +              }
> +            }
> +          }
>          end
>        end
>      end
>
> Joe Wölfel wrote:
>> Thanks for the patch.   Here's a clunkier variation on your
>> suggestion that seems to work with Drake.  Stage 1 serializes an
>> unpredictable set of tasks.   Stage 2 creates instances of them and
>> runs them if necessary.  There might be a better way that involves
>> making the dependency tree modifiable dynamically.  I think allowing
>> all possible dependency changes would get complicated.  Maybe that
>> would require reevaluating the entire tree constantly and there's no
>> way to un-execute a task anyway.  But most of the real world problems
>> I can think of seem to involve adding tasks that wouldn't have been
>> exercised yet anyway.  Could this be solved with an improved
>> dependency tree walking algorithm?
>
> I think the best strategy is to cache the dynamic changes and update
> only when the :restart flag is thrown.  Fortunately, Drake is already
> structured to work this way.
>
> Drake does a dry run to collect all tasks to be executed, then passes
> the dependency graph to my CompTree package which executes it in
> parallel (CompTree is a kind of modest Erlang-in-Ruby).
>
> It would be safe to add tasks during execution, as CompTree is  
> running a
> shallow copy of the dependency graph and will be unaware of any new
> tasks or dependencies.  I don't foresee any serious issues with simply
> restarting the computation with a new shallow copy of the appended
> graph.
>
> Though Drake copies the dependency tree for unrelated reasons, it  
> turns
> out to be coincidentally useful here because it acts as a cache while
> the user can append the original.
>
> Though this is mostly brainstorming, I do see a need for a restart
> feature, whether or not these ideas pan out.  The :setup phase may
> execute non-trivial tasks which will be obliviously re-executed  
> by :main
> in the separate process.  We could assume the two stages comprise
> disjoint sets, however it would be difficult to enforce.  It's an
> artificial restriction which will eventually fail.
>
> In the example above, the :main gets executed in Rake and Drake for
> entirely different reasons.  In single-threaded Rake, the restart
> happens before it even gets to :main, so :main did not get marked as
> @already_invoked.  In multi-threaded Drake, :main *does* get marked,
> however after the restart its newly-created child nodes will still be
> executed because CompTree will not even *consider* a node until all  
> its
> children have been computed.
>
> One difference: in single-threaded Rake you must be careful to add  
> tasks
> "in the future", some place ahead in the sequential order of  
> execution.
> In my example :setup modifies :main, which is fine since the order  
> given
> is [:setup, :main].  In multi-threaded Drake you don't have to worry
> about it, for reasons mentioned in the previous paragraph.
>
> James M. Lawrence
> -- 
> Posted via http://www.ruby-forum.com/.
>