From: Vance A Heron <heron@...>
Date: 2004-11-25T04:09:24+09:00
Subject: ThreadWait problem

Rubyists,

I have a bug - probably in my code or understanding, but
possibly in the ThreadWait library. I'd appreciate any 
help in understanding and/or fixing it.

My application retrieves and parses data from a tree 
structured, XML/HTTP accessible datastore.

I retrieve objects which contain 0 or more leaf objects 
and  0 or more container objects. Container objects are 
queued for subsequent retrieval and processing.  To improve 
performance, I decided to use threads to process multiple 
objects simultaneously.

Originally (non threaded), container objects were appended 
to a list, and the main loop shifted targets off as long 
as there were any in the list.  With the threaded version, 
I changed that to use a queue.

The program works fine for small to medium size trees,
but fails for large ones, with several processes
in the wait group having a status of nil.

Here's my original attempt at the thread dispatcher.
---
  tgroup = ThreadsWait.new
  $tgts = Queue.new

  $tgts.enq(top_tgt)

  # have more nodes to process
  while $tgts.length > 0
    cur_tgt = $tgts.deq
    t = Thread.new do 
      proc_tgt(cur_tgt, param, file_mode) 
    end
    tgroup.join_nowait(t)

    # running threads can add targets to Q
    # only allow t_max threads to run 
    while ((tgroup.threads.length > 0) && ($tgts.length == 0)  ||
           (tgroup.threads.length >= t_max))
      tgroup.next_wait
    end
  end
---

For large trees, the code never exited, and looking I found
tgroup to contain several threads with a status of nil.

I first added the line
  abort_on_execption = true
thinking that some of the threads were dying unexpectedly,
and being ignored.  Stil no joy.

In desperation i've added the following.  
 near the begining of the file after including thwait
----
  class ThreadsWait
    attr_accessor :threads
  end
----

Immediately prior to the 2nd while
---
    tgroup.threads.delete_if {|t| t.status == nil }
---

This seems to work, but is *UGLY*.  I wish I could give
a short example, but the original code only fails on 
data sets > ~10K items - which take > 30 minutes to run.

If it makes any difference, I've had t_max set to 10 and
20 for these

----
$ ruby -v
ruby 1.8.2 (2004-11-06) [i686-linux]
----

Thanks in advance for any help/thoughts/ideas
Vance