[ruby-core:70601] [Ruby trunk - Feature #10600] [PATCH] Queue#close

Issue #10600 has been updated by Tomoyuki Chikanaga.


Hello,
I'm interrested in this topic.
I have some opinions about API design.

I'd like to specify the object to be returned by closed Queue#pop. Application could push nil to Queue as a significant value and.

And I think whether Queue#pop return nil(or something indicate the `EOQ') or raise exception should be determined by parameter of Queue#pop/deq.

If the behavior of Queue#pop is specified by Queue#close, you should know how the queue could be closed to write the code call Queue#pop, but it could be written by different programmers. And the worse the both could be happen.

How about adding keyword argument to Queue#pop,deq?

    queue.pop(exception: false, eoq: nil) # raise ClosedQueue if exception is true, otherwise return eoq.

At last, if you can set counter to Queue#close `really close` the Queue, it easy to write multiple producer pattern. This is an advanced functionality and could be discussed on another ticket. How do you think?

ex)

    def produce(q)
      while obj = get_something
        q.push(obj)
      end
      q.close
    end
    writers = 4
    q = Queue.new(writers_count: writers)
    writers.times { Thread.start { produce(q) } }
    while obj = q.pop  # q.pop return nil after q.close was called 4 times
      # do something
    end


----------------------------------------
Feature #10600: [PATCH] Queue#close
https://bugs.ruby-lang.org/issues/10600#change-54009

* Author: John Anderson
* Status: Assigned
* Priority: Normal
* Assignee: Koichi Sasada
----------------------------------------
In a multiple-producer / multiple-consumer situation using blocking enq and deq, closing a queue cleanly is difficult. It's possible using a queue poison token, but unpleasant because either producers have to know how to match up number of poison tokens with number of consumers, or consumers have to keep putting the poison back into the queue which complicates testing for empty and not blocking on deq.

This patch (from trunk at b2a128f) implements Queue#close which will close the queue to producers, leaving consumers to deq the remaining items. Once the queue is both closed and empty, consumers will not block. When an empty queue is closed, all consumers blocking on deq will be woken up and given nil.

With Queue#close, clean queue shutdown is simple:

~~~ ruby
queue = SizedQueue.new 1000

consumer_threads = lots_of.times.map do
  Thread.new do
    while item = queue.pop
      do_work item
    end
  end
end

source = somewhat_async_enumerator

producer_threads = a_few.times.map do
  Thread.new do
    loop{queue << source.next}
  end
end

producer_threads.each &:join
queue.close
consumer_threads.each &:join
~~~


---Files--------------------------------
queue-close.diff (5.18 KB)
queue-close-2.diff (10.2 KB)
patch-25f99aef.diff (25.2 KB)
queue_benchmark.rb (2.95 KB)


-- 
https://bugs.ruby-lang.org/