[ruby-core:112297] [Ruby master Feature#19061] Proposal: make a concept of "consuming enumerator" explicit

Issue #19061 has been updated by zverok (Victor Shepelev).


@matz Thanks for your answer. I'll gather more evidence/real-life examples and will adjust the proposal.

My main concern though was not as much some particular usage but general awareness of the difference between the two types of enumerators. 

The latest evidence of the fact that it is a problem is bug #19294 in the new feature of Ruby 3.2, where even the core team member implementing new functionality hasn't considered that some enumerators would be "consumed" by the first iteration.

I believe it to be a pretty important distinction frequently leading to idiosyncrasies and not just a random feature request. But I need to think about how to communicate my intentions and proposals better.

----------------------------------------
Feature #19061: Proposal: make a concept of "consuming enumerator" explicit
https://bugs.ruby-lang.org/issues/19061#change-101732

* Author: zverok (Victor Shepelev)
* Status: Open
* Priority: Normal
----------------------------------------
**The problem**

Let's imagine this synthetic data:
```ruby
lines = [
  "--EMAIL--",
  "From: [e-mail]",
  "To; [e-mail]",
  "Subject: Consuming Enumerators",
  "",
  "Here, I am presenting the following proposal.",
  "Let's talk about consuming enumerators..."
]
```
The logic of parsing it is more or less clear:
* skip the first line
* take lines until meet empty, to read the header
* take the rest of the lines to read the body

It can be easily translated into Ruby code, almost literally:
```ruby
def parse(enumerator)
  puts "Testing: #{enumerator.inspect}"
  enumerator.next
  p enumerator.take_while { !_1.empty? }
  p enumerator.to_a
end
```

Now, let's try this code with two different enumerators on those lines:
```ruby
require 'stringio'

enumerator1 = lines.each
enumerator2 = StringIO.new(lines.join("\n")).each_line(chomp: true)

puts "Array#each"
parse(enumerator1)

puts
puts "StringIO#each_line"
parse(enumerator2)
```
Output (as you probably already guessed):
```
Array#each
Testing: #<Enumerator: [...]:each>
["--EMAIL--", "From: [e-mail]", "To; [e-mail]", "Subject: Consuming Enumerators"]
["--EMAIL--", "From: [e-mail]", "To; [e-mail]", "Subject: Consuming Enumerators", "", "Here, I am presenting the following proposal.", "Let's talk about consuming enumerators..."]

StringIO#each_line
Testing: #<Enumerator: #<StringIO:0x00005581018c50a0>:each_line(chomp: true)>
["From: [e-mail]", "To; [e-mail]", "Subject: Consuming Enumerators"]
["Here, I am presenting the following proposal.", "Let's talk about consuming enumerators..."]
```

Only the second enumerator behaves the way we wanted it to.
Things to notice here:
1. Both enumerators are of the same class, "just enumerator," but they behave differently: one of them is **consuming** data on each iteration method, the other does not; but there is no programmatic way to tell whether some enumerator instance is consuming
2. There is no easy way to **make a non-consuming enumerator behave in a consuming way**, to open a possibility of a sequence of processing "skip this, take that, take the rest"

**Concrete proposal**

1. Introduce an `Enumerator#consuming?` method that will allow telling one of the other (and make core enumerators like `#each_line` properly report they are consuming).
2. Introduce `consuming: true` parameter for `Enumerator.new` so it would be easy for user's code to specify the flag
3. Introduce `Enumerator#consuming` method to produce a consuming enumerator from a non-consuming one:
```ruby
# reference implementation is trivial:
class Enumerator
  def consuming
    source = self
    Enumerator.new { |y| loop { y << source.next } }
  end
end

enumerator3 = lines.each.consuming
parse(enumerator3)
```
Output:
```
["From: [e-mail]", "To; [e-mail]", "Subject: Consuming Enumerators"]
["Here, I am presenting the following proposal.", "Let's talk about consuming enumerators..."]
```



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- [e-mail]
 To unsubscribe send an email to [e-mail]
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/