Proc / def / yield semantics (long)

ABSTRACT

     The semantics of Proc object calling was changed between 1.8.0 and
1.8.1 to resolve a disparity with the semantics of yield(); I feel this
was a mistake & would like to see the change reversed.  Specifically:

      * For 1.8.2 I would like to see the semantics revert to the way
        they were in 1.8.0; this leaves the disparity between yield and
        Proc#call, but does not break anything.

      * For 2.0 I would like the semantics to be homogenized, either by:

              * Changing yield (and assignment) to match the semantics
                of the rest of the language, 
        or
              * Making the behavior controllable by the user (on a case
                by case basis) so that people who want the new behavior
                can get it.


BACKGROUND

     Ruby supports parallel assignment and array expansion with
relatively clean semantics.  Arrays used as the last rvalue or the last
lvalue can be expanded by prefixing them with an "*"; the only
inconsistency come from the fact that: 

        This is not necessary if the rvalue is the only thing on the
        right hand side--the array will be expanded automatically.
        -- (Pickaxe I, page 77)

     This syntactic "shortcut" can lead to some decidedly unobvious
behavior.  For example, adding a rvalue to the right hand side of an
assignment statement can cause the class of any of the other lvalues to
change, but only under some conditions:

        x = [1]
        
        a,b = x 
        p a.class    #Fixnum
        
        a,b = x,4 
        p a.class    #Array
        
        x = "1"
        
        a,b = x 
        p a.class    #String
        
        a,b = x,4 
        p a.class    #String
        
     In 1.8.0, only assignment and yield, of the half dozen or so ways
to assign values to a collection of variables, special case single
rvalues which happen to be arrays this way.  (The code in listing one
can be used to test the behavior of the different version.)  

     In the case of assignment, the "feature" is of very slight utility,
since the same effect can be had (if desired) by prefixing the single
rvalue with an "*".

     Where it is must useful is in the case of yield, where it
facilitates constructs such as:

    my_hash.each { |key,value| ... }

which are widely used.

THE CHANGE

     A consequence of this is that using yield on a block has different
semantics than converting the block to a Proc and calling it (as noted
in http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-dev/21726):

        def foo
            yield([:key, :value])
            end
        foo {|k, v| p [k, v]}     #[:key, :value]
        
        def bar(&blk)
            blk.call([:key, :value])
            end
        bar {|k, v| p [k, v]}     #[[:key, :value], nil]
        

     I am somewhat hampered by my illiteracy (at best I can "read"
Japanese at a rate of 1 page a month or so, given sufficient reference
materials) and therefore I am guessing in what follows.

     There are two ways that the discrepancy could be resolved: either
the behavior of yield could be changed, or that of Proc#call.  If the
former path had been taken, we would have:


        def foo
            yield([:key, :value])
            end
        foo {|k, v| p [k, v]}     #[[:key, :value], nil]
        
        def bar(&blk)
            blk.call([:key, :value])
            end
        bar {|k, v| p [k, v]}     #[[:key, :value], nil]
        
and the prior behavior could have been reproduced (if desired) by writing:

        def foo
            yield(*[:key, :value])
            end
        foo {|k, v| p [k, v]}     #[:key, :value]
        
or more simply (in the example case at least):

        def foo
            yield(:key, :value)
            end
        foo {|k, v| p [k, v]}     #[:key, :value]
        
     (This is the choice I would have advocated had I been following the
debate.  But I wasn't and so I am making this 11th hour plea.)

     Instead, it was decided to change Proc.  This, I surmise, led to
some problems and a compromise appears to have been hammered out along
these lines (using the tags from the program in listing 1):

        inline:          special case single arrays
        inline (array):  no special casing
        def:             no special casing
        Proc.new:        special case single arrays
        proc:            no special casing
        My_proc.new:     special case single arrays
        my_proc:         special case single arrays
        yield:           special case single arrays
        
     Contrast this with the situation in 1.8.0:

        inline:          special case single arrays
        inline (array):  no special casing
        def:             no special casing
        Proc.new:        no special casing
        proc:            no special casing
        My_proc.new:     no special casing
        my_proc:         no special casing
        yield:           special case single arrays
        
and you will note that:

      * Kernel#proc is now almost but not quite a synonym for Proc.new

      * It does not appear to be possible to subclass Proc without
        getting the special case behavior

      * It does not appear to be possible to write "wrapper" Procs (a'la
        CLOS) that act like methods

      * It does not appear to be possible to define something that works
        like Kernel#proc

      * The potential for odd behavior noted in the background section
        has not gone away; instead, it has spread to many more contexts.

CONSEQUENCES

     There is a venerable pattern (dating back decades before we even
called them patterns) for dealing with sequences recursively by at each
stage treating the list as a pair: the first item and the rest of the
list.  (CAR/CDR)

     Up until 1.8.0 (and, I hope, in 1.8.2 on) it is easy to implement
this pattern in ruby.  For example:

        def tickle(head,*rest)
            head.giggle if head.respond_to? :giggle
            tickle(rest) if rest
            end

     This works, but breaks subtly under the new semantics if you try to
make it an object:

        tickeler = Proc.new { |head,*rest|
            head.giggle if head.respond_to? :giggle
            self.call(rest) if rest   
            }


     Another useful construct that is possible with the old semantics
but not with the new is before methods (borrowed from CLOS):

        $observers = {}
        class Module
            def before(method,&block)
                new_name = Symbol.unique(method)
                alias_method new_name, method
                block_name = Symbol.unique(method)
                $observers[block_name] = block
                module_eval %{
                    def #{method}(*args, &block)
                        $observers[:#{block_name}].
                            call(self,*args,&block)
                        #{new_name}(*args,&block)
                        end
                    }
                return [self, method, new_name, block_name]
                end
            end
        
with which we can write:

        class Animal
             def dinner(...)
                  :
                  end
             :
             end
        
        Animal.before :dinner do |animal,*args|
            :
            end
        
        
     Of course, this only works so long as the semantics of calling a
block are the same as the semantics of calling a method.  In other
words, it works fine under 1.8.0 but fails mysteriously under 1.8.1

     There are many such examples, but most of them are much harder to
abstract (or at least, harder to abstract concisely).  For example, the
Action pattern and its kin are much easier to implement if Procs and
methods don't have different special cases.


PROPOSED RESOLUTION

     I gather the deadline for 1.8.2 is looming.  Therefore, for 1.8.2,
I would propose going back to the 1.8.0 semantics.  This could be done
(I believe) by simply reversing the change made by Matz on 30 September
2003.  Anyone who wants the new behavior should be able to get it by the
judicious use of "*".

     For 2.0 (which, as I understand it, is not committed to 100%
backward compatibility), I would recommend one of the following:

      * If no one can devise a case in which it would not be possible to
        reproduce the special case behavior by prefixing single rvalues
        with an "*", I would recommend eliminating the special case
        altogether.  This would mean changing the semantics of yield and
        assignment slightly, through I would be surprised if there was
        much code at all (out side of iterators over structures such as
        hashes) that _wants_ the new behavior.

      * If there is some reason why yield and assignment need to keep
        the special case, I would like to see a flag added to the Proc
        object that determines its behavior.  Rather than having
        Kernal#proc return one type of Proc and Proc#new return a subtly
        different type of Proc, both should return the same thing and
        its behavior should be adjustable either at creation time (via a
        parameter to the creation method) or by sending a message to the
        object (Proc#expand_array_parameters=, or some such). 



     Thank you for taking time to read this; I'm sorry I wasn't alert
enough to offer it a year ago.  As always, I am more than willing to
help in whatever way is needed.

           -- Markus



LISTING 1

def show(h,r)
    print " head: #{h.inspect}, rest: #{r.inspect}    "
    end


#-----------------------------------------------
print "inline:          "
#head,*rest = 1,2,3   ; show(head,rest)
#head,*rest = [1],2,3 ; show(head,rest)
head,*rest = [1,2],3 ; show(head,rest)
head,*rest = [1,2,3] ; show(head,rest)
print "\n"

#-----------------------------------------------
print "inline (array):  "
#a = [1,2,3]   ; h,*r = *a; show(head,rest)
#a = [[1],2,3] ; h,*r = *a; show(head,rest)
a = [[1,2],3] ; h,*r = *a; show(head,rest)
a = [[1,2,3]]  ; h,*r = *a; show(head,rest)
print "\n"

#-----------------------------------------------
print "def:             "
def test1(head,*rest)
    show(head,rest)
    end
#test1(1,2,3)
#test1([1],2,3)
test1([1,2],3)
test1([1,2,3])
print "\n"

#-----------------------------------------------
print "Proc.new:        "
test2 = Proc.new { |head,*rest|
    show(head,rest)
    }
#test2.call(1,2,3)
#test2.call([1],2,3)
test2.call([1,2],3)
test2.call([1,2,3])
print "\n"

#-----------------------------------------------
print "proc:            "
test3 = proc { |head,*rest|
    show(head,rest)
    }
#test3.call(1,2,3)
#test3.call([1],2,3)
test3.call([1,2],3)
test3.call([1,2,3])
print "\n"

#-----------------------------------------------
print "My_proc.new:     "
class My_proc < Proc
    end
test4 = My_proc.new { |head,*rest|
    show(head,rest)
    }
#test4.call(1,2,3)
#test4.call([1],2,3)
test4.call([1,2],3)
test4.call([1,2,3])
print "\n"

#-----------------------------------------------
print "my_proc:         "
def my_proc(&b)
    b
    end
test5 = my_proc { |head,*rest|
    show(head,rest)
    }
#test5.call(1,2,3)
#test5.call([1],2,3)
test5.call([1,2],3)
test5.call([1,2,3])
print "\n"

#-----------------------------------------------
print "yield:           "
def test6
    #yield(1,2,3)
    #yield([1],2,3)
    yield([1,2],3)
    yield([1,2,3])
    end
test6 { |head,*rest|
    show(head,rest)
    }
print "\n"