[ruby-core:20031] Re: Fibers in 1.8

On Fri, Nov 21, 2008 at 7:49 AM, Brian Mitchell <[e-mail]> wrote:
> On Fri, Nov 21, 2008 at 09:27, Paul Brannan <[e-mail]> wrote:
>> On Fri, Nov 21, 2008 at 05:45:23PM +0900, Urabe Shyouhei wrote:
>>> Ko1 explained me about this.  According to him, 1.8 threads are slower
>>> because context switching on 1.8 needs a lot of memory copies.  1.8
>>
>> 1.8 threads do a lot of memory copies on context switch, but they are
>> not all necessary.  The stack can be switched on some platforms using
>> swapcontext.
>>
>> (but this isn't a trivial change either; the implementation of threads
>> on 1.8 is complex and is tightly coupled to eval.c)
>
> From what I understand, the NeverBlock [0] has used 1.9 Fibers to
> achieve its goal. I believe they have ported it to 1.8 in some effect
> [1] (most likely not at the Thread level but at the C ext. level with
> stack allocation and swapping). Anyone know what the caveats are with
> their implementation?

NeverBlock uses my pure ruby implementation
(http://gist.github.com/4631) which relies on Thread and Queue. This
is not ideal because the thread timer and scheduler continue to run,
and the use of Queue makes deadlocks possible.

# Poor Man's Fiber (API compatible Thread based Fiber implementation
for Ruby 1.8)
# (c) 2008 Aman Gupta (tmm1)

unless defined? Fiber
 require 'thread'

 class FiberError < StandardError; end

 class Fiber
   def initialize
     raise ArgumentError, 'new Fiber requires a block' unless block_given?

     @yield = Queue.new
     @resume = Queue.new

     @thread = Thread.new{ @yield.push [ *yield(*@resume.pop) ] }
     @thread.abort_on_exception = true
     @thread[:fiber] = self
   end
   attr_reader :thread

   def resume *args
     raise FiberError, 'dead fiber called' unless @thread.alive?
     @resume.push(args)
     result = @yield.pop
     result.size > 1 ? result : result.first
   end

   def yield *args
     @yield.push(args)
     result = @resume.pop
     result.size > 1 ? result : result.first
   end

   def self.yield *args
     raise FiberError, "can't yield from root fiber" unless fiber =
Thread.current[:fiber]
     fiber.yield(*args)
   end

   def self.current
     Thread.current[:fiber] or raise FiberError, 'not inside a fiber'
   end

   def inspect
     "#<#{self.class}:0x#{self.object_id.to_s(16)}>"
   end
 end
end

if __FILE__ == $0
 f = Fiber.new{ puts 'hi'; p Fiber.yield(1); puts 'bye'; :done }
 p f.resume
 p f.resume(2)
end

__END__

$ ruby fbr.rb
hi
1
2
bye
:done

$ ruby1.9 fbr.rb
hi
1
2
bye
:done

> I would imagine it would be possible to have a C extension provide
> working fibers as long as the GC was made aware of the stack so it
> could scan the stacks appropriately. Threads would also probably have
> to be modified a bit to work in concert (jumping back to the proper
> fiber before making the context switch I would guess).

I am currently attempting to backport the C implementation from 1.9
(http://github.com/tmm1/ruby187/commits/fiber_backport). So far the
simple API is functional:

$ ./miniruby -e "f = Fiber.new{ |sym|
 p(sym)
 puts 'hi'
 p(Fiber.yield 1)
 puts 'bye'
 :end
}
p(f.resume :begin)
p(f.resume 2)
"
 :begin
 hi
 1
 2
 bye
 :end

The first three tests from 1.9 test suite also pass:

 $ ./miniruby -I lib test/ruby/test_fiber.rb Loaded suite
 test/ruby/test_fiber
 Started
 ...
 Finished in 8.902874 seconds.

 3 tests, 7 assertions, 0 failures, 0 errors

The API is implemented using rb_thread_save_context and
rb_thread_restore_context, which memcpy the stack and are much slower
when compared to 1.9 implementation:

 $ /opt/ruby19/bin/ruby19 test/ruby/test_fiber.rb
 Loaded suite test/ruby/test_fiber
 Started
 ...
 Finished in 0.597483 seconds.

 3 tests, 7 assertions, 0 failures, 0 errors, 0 skips

Aman.