From: Markus Date: 2004-09-25T13:58:51+09:00 Subject: Proc / def / yield semantics (long) ABSTRACT The semantics of Proc object calling was changed between 1.8.0 and 1.8.1 to resolve a disparity with the semantics of yield(); I feel this was a mistake & would like to see the change reversed. Specifically: * For 1.8.2 I would like to see the semantics revert to the way they were in 1.8.0; this leaves the disparity between yield and Proc#call, but does not break anything. * For 2.0 I would like the semantics to be homogenized, either by: * Changing yield (and assignment) to match the semantics of the rest of the language, or * Making the behavior controllable by the user (on a case by case basis) so that people who want the new behavior can get it. BACKGROUND Ruby supports parallel assignment and array expansion with relatively clean semantics. Arrays used as the last rvalue or the last lvalue can be expanded by prefixing them with an "*"; the only inconsistency come from the fact that: This is not necessary if the rvalue is the only thing on the right hand side--the array will be expanded automatically. -- (Pickaxe I, page 77) This syntactic "shortcut" can lead to some decidedly unobvious behavior. For example, adding a rvalue to the right hand side of an assignment statement can cause the class of any of the other lvalues to change, but only under some conditions: x = [1] a,b = x p a.class #Fixnum a,b = x,4 p a.class #Array x = "1" a,b = x p a.class #String a,b = x,4 p a.class #String In 1.8.0, only assignment and yield, of the half dozen or so ways to assign values to a collection of variables, special case single rvalues which happen to be arrays this way. (The code in listing one can be used to test the behavior of the different version.) In the case of assignment, the "feature" is of very slight utility, since the same effect can be had (if desired) by prefixing the single rvalue with an "*". Where it is must useful is in the case of yield, where it facilitates constructs such as: my_hash.each { |key,value| ... } which are widely used. THE CHANGE A consequence of this is that using yield on a block has different semantics than converting the block to a Proc and calling it (as noted in http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-dev/21726): def foo yield([:key, :value]) end foo {|k, v| p [k, v]} #[:key, :value] def bar(&blk) blk.call([:key, :value]) end bar {|k, v| p [k, v]} #[[:key, :value], nil] I am somewhat hampered by my illiteracy (at best I can "read" Japanese at a rate of 1 page a month or so, given sufficient reference materials) and therefore I am guessing in what follows. There are two ways that the discrepancy could be resolved: either the behavior of yield could be changed, or that of Proc#call. If the former path had been taken, we would have: def foo yield([:key, :value]) end foo {|k, v| p [k, v]} #[[:key, :value], nil] def bar(&blk) blk.call([:key, :value]) end bar {|k, v| p [k, v]} #[[:key, :value], nil] and the prior behavior could have been reproduced (if desired) by writing: def foo yield(*[:key, :value]) end foo {|k, v| p [k, v]} #[:key, :value] or more simply (in the example case at least): def foo yield(:key, :value) end foo {|k, v| p [k, v]} #[:key, :value] (This is the choice I would have advocated had I been following the debate. But I wasn't and so I am making this 11th hour plea.) Instead, it was decided to change Proc. This, I surmise, led to some problems and a compromise appears to have been hammered out along these lines (using the tags from the program in listing 1): inline: special case single arrays inline (array): no special casing def: no special casing Proc.new: special case single arrays proc: no special casing My_proc.new: special case single arrays my_proc: special case single arrays yield: special case single arrays Contrast this with the situation in 1.8.0: inline: special case single arrays inline (array): no special casing def: no special casing Proc.new: no special casing proc: no special casing My_proc.new: no special casing my_proc: no special casing yield: special case single arrays and you will note that: * Kernel#proc is now almost but not quite a synonym for Proc.new * It does not appear to be possible to subclass Proc without getting the special case behavior * It does not appear to be possible to write "wrapper" Procs (a'la CLOS) that act like methods * It does not appear to be possible to define something that works like Kernel#proc * The potential for odd behavior noted in the background section has not gone away; instead, it has spread to many more contexts. CONSEQUENCES There is a venerable pattern (dating back decades before we even called them patterns) for dealing with sequences recursively by at each stage treating the list as a pair: the first item and the rest of the list. (CAR/CDR) Up until 1.8.0 (and, I hope, in 1.8.2 on) it is easy to implement this pattern in ruby. For example: def tickle(head,*rest) head.giggle if head.respond_to? :giggle tickle(rest) if rest end This works, but breaks subtly under the new semantics if you try to make it an object: tickeler = Proc.new { |head,*rest| head.giggle if head.respond_to? :giggle self.call(rest) if rest } Another useful construct that is possible with the old semantics but not with the new is before methods (borrowed from CLOS): $observers = {} class Module def before(method,&block) new_name = Symbol.unique(method) alias_method new_name, method block_name = Symbol.unique(method) $observers[block_name] = block module_eval %{ def #{method}(*args, &block) $observers[:#{block_name}]. call(self,*args,&block) #{new_name}(*args,&block) end } return [self, method, new_name, block_name] end end with which we can write: class Animal def dinner(...) : end : end Animal.before :dinner do |animal,*args| : end Of course, this only works so long as the semantics of calling a block are the same as the semantics of calling a method. In other words, it works fine under 1.8.0 but fails mysteriously under 1.8.1 There are many such examples, but most of them are much harder to abstract (or at least, harder to abstract concisely). For example, the Action pattern and its kin are much easier to implement if Procs and methods don't have different special cases. PROPOSED RESOLUTION I gather the deadline for 1.8.2 is looming. Therefore, for 1.8.2, I would propose going back to the 1.8.0 semantics. This could be done (I believe) by simply reversing the change made by Matz on 30 September 2003. Anyone who wants the new behavior should be able to get it by the judicious use of "*". For 2.0 (which, as I understand it, is not committed to 100% backward compatibility), I would recommend one of the following: * If no one can devise a case in which it would not be possible to reproduce the special case behavior by prefixing single rvalues with an "*", I would recommend eliminating the special case altogether. This would mean changing the semantics of yield and assignment slightly, through I would be surprised if there was much code at all (out side of iterators over structures such as hashes) that _wants_ the new behavior. * If there is some reason why yield and assignment need to keep the special case, I would like to see a flag added to the Proc object that determines its behavior. Rather than having Kernal#proc return one type of Proc and Proc#new return a subtly different type of Proc, both should return the same thing and its behavior should be adjustable either at creation time (via a parameter to the creation method) or by sending a message to the object (Proc#expand_array_parameters=, or some such). Thank you for taking time to read this; I'm sorry I wasn't alert enough to offer it a year ago. As always, I am more than willing to help in whatever way is needed. -- Markus LISTING 1 def show(h,r) print " head: #{h.inspect}, rest: #{r.inspect} " end #----------------------------------------------- print "inline: " #head,*rest = 1,2,3 ; show(head,rest) #head,*rest = [1],2,3 ; show(head,rest) head,*rest = [1,2],3 ; show(head,rest) head,*rest = [1,2,3] ; show(head,rest) print "\n" #----------------------------------------------- print "inline (array): " #a = [1,2,3] ; h,*r = *a; show(head,rest) #a = [[1],2,3] ; h,*r = *a; show(head,rest) a = [[1,2],3] ; h,*r = *a; show(head,rest) a = [[1,2,3]] ; h,*r = *a; show(head,rest) print "\n" #----------------------------------------------- print "def: " def test1(head,*rest) show(head,rest) end #test1(1,2,3) #test1([1],2,3) test1([1,2],3) test1([1,2,3]) print "\n" #----------------------------------------------- print "Proc.new: " test2 = Proc.new { |head,*rest| show(head,rest) } #test2.call(1,2,3) #test2.call([1],2,3) test2.call([1,2],3) test2.call([1,2,3]) print "\n" #----------------------------------------------- print "proc: " test3 = proc { |head,*rest| show(head,rest) } #test3.call(1,2,3) #test3.call([1],2,3) test3.call([1,2],3) test3.call([1,2,3]) print "\n" #----------------------------------------------- print "My_proc.new: " class My_proc < Proc end test4 = My_proc.new { |head,*rest| show(head,rest) } #test4.call(1,2,3) #test4.call([1],2,3) test4.call([1,2],3) test4.call([1,2,3]) print "\n" #----------------------------------------------- print "my_proc: " def my_proc(&b) b end test5 = my_proc { |head,*rest| show(head,rest) } #test5.call(1,2,3) #test5.call([1],2,3) test5.call([1,2],3) test5.call([1,2,3]) print "\n" #----------------------------------------------- print "yield: " def test6 #yield(1,2,3) #yield([1],2,3) yield([1,2],3) yield([1,2,3]) end test6 { |head,*rest| show(head,rest) } print "\n"