[ANN] forkoff-1.1.0

NAME

  forkoff

SYNOPSIS

  brain-dead simple parallel processing for ruby

URI

  http://rubyforge.org/projects/codeforpeople
  http://github.com/ahoward/forkoff

INSTALL

  gem install forkoff

DESCRIPTION

  forkoff works for any enumerable object, iterating a code block to
run in a
  child process and collecting the results.  forkoff can limit the
number of
  child processes which is, by default, 2.

SAMPLES


  <========< samples/a.rb >========>

  ~ > cat samples/a.rb

    # forkoff makes it trivial to do parallel processing with ruby,
the following
    # prints out each word in a separate process
    #

      require 'forkoff'

      %w( hey you ).forkoff!{|word| puts "#{ word } from #
{ Process.pid }"}

  ~ > ruby samples/a.rb

    hey from 7907
    you from 7908


  <========< samples/b.rb >========>

  ~ > cat samples/b.rb

    # for example, this takes only 4 seconds or so to complete (8
iterations
    # running in two processes = twice as fast)
    #

      require 'forkoff'

      a = Time.now.to_f

      results =
        (0..7).forkoff do |i|
          sleep 1
          i ** 2
        end

      b = Time.now.to_f

      elapsed = b - a

      puts "elapsed: #{ elapsed }"
      puts "results: #{ results.inspect }"

  ~ > ruby samples/b.rb

    elapsed: 4.19184589385986
    results: [0, 1, 4, 9, 16, 25, 36, 49]


  <========< samples/c.rb >========>

  ~ > cat samples/c.rb

    # forkoff does *NOT* spawn processes in batches, waiting for each
batch to
    # complete.  rather, it keeps a certain number of processes busy
until all
    # results have been gathered.  in otherwords the following will
ensure that 3
    # processes are running at all times, until the list is complete.
note that
    # the following will take about 3 seconds to run (3 sets of 3 @ 1
second).
    #

    require 'forkoff'

    pid = Process.pid

    a = Time.now.to_f

    pstrees =
      %w( a b c d e f g h i ).forkoff! :processes => 3 do |letter|
        sleep 1
        { letter => ` pstree -l 2 #{ pid } ` }
      end


    b = Time.now.to_f

    puts
    puts "pid: #{ pid }"
    puts "elapsed: #{ b - a }"
    puts

    require 'yaml'

    pstrees.each do |pstree|
      y pstree
    end

  ~ > ruby samples/c.rb


    pid: 7922
    elapsed: 3.37899208068848

    ---
    a: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward (ruby)
       \-+- 07925 ahoward ruby -Ilib samples/c.rb

    ---
    b: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward ruby -Ilib samples/c.rb
       \-+- 07925 ahoward ruby -Ilib samples/c.rb

    ---
    c: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward (ruby)
       \-+- 07925 ahoward ruby -Ilib samples/c.rb

    ---
    d: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07932 ahoward ruby -Ilib samples/c.rb
       |--- 07933 ahoward ruby -Ilib samples/c.rb
       \--- 07934 ahoward ruby -Ilib samples/c.rb

    ---
    e: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07932 ahoward (ruby)
       |-+- 07933 ahoward ruby -Ilib samples/c.rb
       \-+- 07934 ahoward (ruby)

    ---
    f: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07932 ahoward (ruby)
       |-+- 07933 ahoward ruby -Ilib samples/c.rb
       \-+- 07934 ahoward ruby -Ilib samples/c.rb

    ---
    g: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07941 ahoward ruby -Ilib samples/c.rb
       |--- 07942 ahoward ruby -Ilib samples/c.rb
       \--- 07943 ahoward ruby -Ilib samples/c.rb

    ---
    h: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07941 ahoward (ruby)
       |-+- 07942 ahoward ruby -Ilib samples/c.rb
       \--- 07943 ahoward ruby -Ilib samples/c.rb

    ---
    i: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07942 ahoward (ruby)
       \-+- 07943 ahoward ruby -Ilib samples/c.rb



  <========< samples/d.rb >========>

  ~ > cat samples/d.rb

    # forkoff supports two strategies of reading the result from the
child: via
    # pipe (the default) or via file.  you can select which to use
using the
    # :strategy option.
    #

      require 'forkoff'

      %w( hey you guys ).forkoff :strategy => :file do |word|
        puts "#{ word } from #{ Process.pid }"
      end

  ~ > ruby samples/d.rb

    hey from 7953
    you from 7954
    guys from 7955



HISTORY
  1.1.0
    - move to a model with one work queue and signals sent from
consumers to
    producer to noitify ready state.  this let's smaller jobs race
through a
    single process even while a larger job may have one sub-process
bound up.
    incorporates a fix from http://github.com/fredrikj/forkoff which
meant
    some processes would lag behind when jobs didn't have similar
execution
    times.

  1.0.0
    - move to github

  0.0.4
    - code re-org
    - add :strategy option
    - default number of processes is 2, not 8

  0.0.1

    - updated to use producer threds pushing onto a SizedQueue for
each consumer
      channel.  in this way the producers do not build up a massize
parllel data
      structure but provide data to the consumers only as fast as they
can fork
      and proccess it.  basically for a 4 process run you'll end up
with 4
      channels of size 1 between 4 produces and 4 consumers, each
consumer is a
      thread popping of jobs, forking, and yielding results.

    - removed use of Queue for capturing the output.  now it's simply
an array
      of arrays which removed some sync overhead.

    - you can configure the number of processes globally with

        Forkoff.default['proccess'] = 4

    - you can now pass either an options hash

        forkoff( :processes => 2 ) ...

      or plain vanilla number

        forkoff( 2 ) ...

      to the forkoff call

    - default number of processes is 8, not 2


  0.0.0

    initial version


enjoy.