From: Matthew Kerwin <lists@...>
Date: 2013-03-17T21:37:50+09:00
Subject: Re: control the concurrent thread numbers

Ken Paul wrote in post #1101934:
> The real problem is that there are plenty of files on disks and a long
> list of words stored in an array. I'd count the total appearing times
> for every words among these files.
>
> I need to run this task with a multiple threads program. Each thread
> would take one word from the array by shifting a word to ensure other
> thread will not do the same task at the same time, then do the counting.
> However, the total numbers of thread should be controlled at a number
> less than 10.

As Joel said, partition the array outside the threads.  I made a tiny
proof-of-concept: https://gist.github.com/phluid61/5180668  However it's
probably simpler to just do:

  threads = []
  matches = []
  mtx = Mutex.new
  big_word_list.each_slice(10) do |smaller_word_list|
    threads << Thread.new(smaller_word_list) do |ary|
      data = []
      ary.each do |word|
        data << scan_files_for(word)
      end
      mtx.synchronize{ matches += data }
    end
  end
  threads.each{|t| t.join }

Or whatever.  It might even make sense to decimate the files, and make
each thread handle the full word list.

  threads = []
  matches = []
  mtx = Mutex.new
  all_files.each_slice(10) do |some_files|
    threads << Thread.new(some_files) do |ary|
      data = []
      ary.each do |file|
        data << scan_file(file, big_word_list)
      end
      mtx.synchronize{ matches += data }
    end
  end
  threads.each{|t| t.join }

Depends how the scan_* function works, and how easy it is to get the
list of all_files.

-- 
Posted via http://www.ruby-forum.com/.