[ruby-core:51521] [ruby-trunk - Bug #7696] Lazy enumerators with state can't be rewound

Issue #7696 has been updated by marcandre (Marc-Andre Lafortune).


The same idea will work for zip too; the arguments must be converted to enumerators only when yielder.memo is not set, i.e. every new yielder.

The arguments are never really rewound, but the first yielder holding them will not be reused after enum.rewind; a new yielder is given.
I haven't checked if that's the current behavior in other implementations, but if yielder is to be the state holder, that's the way it should work.

    enum = (1..2).lazy.zip(1..2)
    enum.next # => yielder.memo was nil, so (1..2).each is called and stored in the memo
    enum.rewind # => the original yielder is discarded
    enum.next # => the second yielder.memo is nil, so (1..2).each is called again and stored in the memo

I'm using the same idea in my `backports` gem to implement this in pure Ruby:

    https://github.com/marcandre/backports/blob/master/lib/backports/2.0.0/enumerator/lazy.rb

The only other possibility that would work is to pass yield extra 'state' argument when required, like:

  def drop(n)
      Lazy.new(self, state: ->{{remain: n}}) do |yielder, state, *values|
        if state[:remain] > 0
          state[:remain] -= 1
        else
          yielder.yield(*values)
        end
      end
    end  

Maybe the :state option could be an object instead of a lmabda, which would be dupped before each enumerations:

  def drop(n)
      Lazy.new(self, state: {remain: n}) do |yielder, state, *values|
        if state[:remain] > 0
          state[:remain] -= 1
        else
          yielder.yield(*values)
        end
      end
    end  

Both solutions seem complex to me, but would definitely put the correct emphasis on how to deal with state.


So, does the patch look acceptable as far as MRI is concerned?
For the public api, should there be a public Yielder#memo and a guarantee that there is exactly one yielder object per iteration? or instead an extra state yielded when required?

----------------------------------------
Bug #7696: Lazy enumerators with state can't be rewound
https://bugs.ruby-lang.org/issues/7696#change-35488

Author: marcandre (Marc-Andre Lafortune)
Status: Open
Priority: High
Assignee: 
Category: core
Target version: 2.0.0
ruby -v: r38800


The 4 lazy enumerators requiring internal state, i.e. {take|drop}{_while}, don't work as expected after a couple `next` and a call to `rewind`.

For example:

    e=(1..42).lazy.take(2)
    e.next # => 1
    e.next # => 2
    e.rewind
    e.next # => 1
    e.next # => StopIteration: iteration reached an end, expected 2

This is related to #7691; the current API does not give an easy way to handle state.

Either there's a dedicated callback to rewind, or data must be attached to the yielder.


-- 
http://bugs.ruby-lang.org/