[ruby-core:19763] [Bug #738] Repeated calls to popen cause thread problems

Bug #738: Repeated calls to popen cause thread problems
http://redmine.ruby-lang.org/issues/show/738

Author: Michal Suchanek
Status: Open, Priority: Normal

This is a repost of an older issue form the other issue tracker.

I am trying to interface a legacy program in my code. This program reads text from stdin and writes them to stdout with some additional information added. Unfortunately it also caches a bit of the output so I cannot just write a piece of text and read it back, I have to 

a) write the results into a temporary file and get them from there

b) use threads in Ruby

Also I have to execute a new instance of the program each time because the cache is only flushed completely when the program exits.

However, (b) is completely out of question because there are various bugs in various Ruby implementation regarding threading and popen.

There are two ways of handling the threading - either have the main thread read the results or have the main thread write the text into the pipe.

Although these should be theoretically equivalent the number of invocations before the interpreter locks up may vary wildly depending on the way roles are assigned to the two threads.

Also there is a problem with the pipe one gets from popen. Using the same IO in both threads is the only safe way in MRI (although caution would advise to duplicate the IO as shown in testt.rb). Duplicating the IO results in deadlock error within about 200 calls in any recent MRI Ruby I tried but is required in JRuby.

The swapping of thread roles had no noticeable affect in MRI Ruries up to some 1.8.5 version but lately only the way shown in testt2.rb works. Although I cannot reproduce any lockup with the testt.rb if I remove the IO duplication running my application this way results in occasional intermittent thread deadlock messages that can be worked around by repeatedly processing the same data until the whole task finishes.

Note that in JRuby the required approach is exactly the opposite to what is required in MRI. The IO has to be duplicated for JRuby and the threads have to be swapped. This lack of reliability and portability is quite disappointing.

$ ruby -w testt.rb
testt.rb:24: warning: `*' interpreted as argument prefix
    99.deadlock 0x7fadea3a6610: sleep:F(3)  - testt.rb:6
deadlock 0x7fadea3fe360: sleep:S (main) - testt.rb:15
testt.rb:15:in `write': Thread(0x7fadea3fe360): deadlock (fatal)
	from testt.rb:15:in `puts'
	from testt.rb:15:in `try_analyze'
	from testt.rb:13:in `each'
	from testt.rb:13:in `try_analyze'
	from testt.rb:24
	from testt.rb:22:in `upto'
	from testt.rb:22


----------------------------------------
http://redmine.ruby-lang.org