[ruby-core:50465] [ruby-trunk - Feature #4085] Refinements and nested methods

Issue #4085 has been updated by headius (Charles Nutter).


Trying to think aloud how things should be structured. Hopefully this will be helpful to others. I base this on my understanding of refinements up to this point.

# VIRTUAL HIERARCHY AS A REPRESENTATION OF REFINEMENTS

I may be starting to form a picture of how refinements are structured. I'll try to summarize a bit...let me know what I have wrong.

Given a simple class hierarchy:

class Foo
  def hello; puts "hello in Foo"; end
end

class Bar < Foo
  def hello; puts "hello in Bar"; super; end
end

We have a hierarchy like this: Bar < Foo < Object

Method lookup for normal unrefined Ruby proceeds against this hierarchy. First Bar's implementation of "hello" is found and called, then its super call finds and calls Foo's implementation of "hello".

Foo.new.hello # sees hierarchy Foo < Object
Bar.new.hello # sees hierarchy Bar < Foo < Object

Now we add a refinement:

module M1
  refine Foo do
    def hello
      puts "before hello in M1"
      super
      puts "after hello in M1"
    end
  end
end

The refine call creates an anonymous module associated with the Foo class and containing a single method definition "hello" which acts around the primary "hello" implementation on Foo-related class hierarchies.

For purposes of lookup, a refined call site now sees a different hierarchy. Let's use the refinement and call a method:

using M1
Foo.new.hello # sees hierarchy [M1 refinement] < Foo < Object
Bar.new.hello # sees hierarchy [M1 refinement] < Bar < Foo < Object

Refinements are activated by the "using" method. Conceptually, when a refinement is brought into a scope, it is seen as being between the target object and its class, intercepting lookup at the call site.

So if normal call-site lookup works like this:

1. Get target object's class
2. Search class for method implementation

Refined call-site lookup works like this:

1. Get target object's class
2. Get the refinements that are active for this call site
3. Search the refinements for method implementation
4. Failing that, search the original class for method implementation

# METHOD LOOKUP AGAINST REFINED HIERARCHIES

Where things start to get fuzzy for me is in step (3). I will summarize how I believe lookup is supposed to proceed

For each class in the target class's hierarchy, look for an active refinement for that class. Refinements on descendants will mask refinements on ancestors, but may super to them if both are active. Super from the innermost refinement returns to searching the normal class hierarchy.

In this case, the call-site-specific hierarchy above makes sense. With method tables in place we have:

For the call:

Bar.new.hello

The call site sees a structure like:

Foo
  {:hello => Foo's hello}
Bar < Foo
  {:hello => Bar's hello}
M1 refinement < Bar
  {:hello => M1's hello}

In this case, it does look similar to prepend, but it's a virtual prepend that only lives at certain call sites. The behavior of "super" here starts to be a bit more clear, since M1's hello naturally supers into Bar's hello, even though M1 refines Foo.

Am I correct so far?

If we add a refinement to Bar:

module M2
  refine Bar do
    def hello; puts "before hello in M2"; super; puts "after hello in M2"; end
  end
end

Now if we're using both refinements:

using M1
using M2 # order is important...it defines the virtual hierarchy
Bar.new.hello

The hierarchy seen at the "hello" call site looks like this: [M2 refinement] < [M1 refinement] < Bar < Foo

And the call proceeds as follows:
M2's hello supers to...
  M1's hello supers to...
    Bar's hello supers to...
      Foo's hello

The virtual hierarchy looks this way for two reasons:

* M2 was used after M1, so it appears first in the search
* The classes that M2 and M1 refine both appear in the normal hierarchy, so both get searched

If we use the modules in a different order, we should see the M1 and M2 results reversed, since they are searched in most-recently-used order *ahead* of the hierarchy and independent of its structure:

using M2
using M1
Bar.new.hello

The virtual hierarchy here is: [M1] < [M2] < Bar < Foo

This would mean that the refinements on Foo fire before the refinements on Bar and M1's hello supers into M2's hello (which supers into Bar's hello). This means that the structure of the original hierarchy has no bearing on the ordering of refinements. The virtual hierarchy from refinements is always based on the order in which they are used in a given scope, regardless of what classes they refine.

Still correct?

# REFINEMENTS ARE BOTTOM-HIERARCHY, SCOPE-LOCAL PREPENDS

If so, then it seems like refinements are essentially bottom-of-the-hierarchy prepends, searched for a given call site only if activated in the current scope and if the target object contains a refined class/module in its hierarchy. Conceptually, that's a bit more clear than other definitions so far.

This also explains the terminology "overlay modules" in Shugo's patch. The modules defined by a refinement are "overlaid" at the call site and searched before searching the original class hierarchy.

The process of looking up a normal direct method call at a refined call site, in more detail:

1. Get the target object's class
2. Get the list of refinements active for this call site
3. For each refinement (in order of "using"), look for the refined class or module in the target object's class hierarchy
4. If it appears in the hierarchy, search it for the method name in question
5. If the method exists, use it for the call (and ideally cache it in some way)
6. If the method in question does not appear on any of the refinements, continue searching the normal class hierarchy

Super lookup proceeds in much the same way, starting from the point in the virtual hierarchy where the method appears and looking up the virtual hierarchy from that point.

# CALLS WITHIN REFINED METHODS ARE REFINED

What about other calls within refined methods?

class Chicken
  def cluck
    puts bok * 3
  end
  def bok; "bok"; end
end

Refining a method does not force down-stream calls to see it. Only call sites with that refinement active see the method.

module RefineChicken
  refine Chicken do
    def bok
      "buck" + super
    end
  end
end

using RefineChicken
Chicken.new.cluck # => returns "bokbokbok" because Chicken's primary "cluck" method is not refined.

But if we refine both methods, the "bok" call site is also refined and sees the refined "bok" method:

module RefineChicken2
  refine Chicken do
    def cluck
      puts bok * 2
    end
    def bok
      "buck" + super
    end
  end
end

using RefineChicken2
Chicken.new.cluck # => "buckbokbuckbok"

The search logic for the "cluck" call sees the following hierarchy: [RefineChicken2] < Chicken < Object

The search logic for the "bok" call sees the same hierarchy, because it is made from within a scope where RefineChicken2 is active.

---

Hopefully this is all correct and describes the basic structure of refinements in a way people can understand. I'll give some thought to the troublesome cases from this thread and see how they should behave.
----------------------------------------
Feature #4085: Refinements and nested methods
https://bugs.ruby-lang.org/issues/4085#change-34302

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal
Assignee: matz (Yukihiro Matsumoto)
Category: core
Target version: 2.0.0


=begin
 As I said at RubyConf 2010, I'd like to propose a new features called
 "Refinements."
 
 Refinements are similar to Classboxes.  However, Refinements doesn't
 support local rebinding as mentioned later.  In this sense,
 Refinements might be more similar to selector namespaces, but I'm not
 sure because I have never seen any implementation of selector
 namespaces.
 
 In Refinements, a Ruby module is used as a namespace (or classbox) for
 class extensions.  Such class extensions are called refinements.  For
 example, the following module refines Fixnum.
 
   module MathN
     refine Fixnum do
       def /(other) quo(other) end
     end
   end
 
 Module#refine(klass) takes one argument, which is a class to be
 extended.  Module#refine also takes a block, where additional or
 overriding methods of klass can be defined.  In this example, MathN
 refines Fixnum so that 1 / 2 returns a rational number (1/2) instead
 of an integer 0.
 
 This refinement can be enabled by the method using.
 
   class Foo
     using MathN
 
     def foo
       p 1 / 2
     end
   end
 
   f = Foo.new
   f.foo #=> (1/2)
   p 1 / 2
 
 In this example, the refinement in MathN is enabled in the definition
 of Foo.  The effective scope of the refinement is the innermost class,
 module, or method where using is called; however the refinement is not
 enabled before the call of using.  If there is no such class, module,
 or method, then the effective scope is the file where using is called.
 Note that refinements are pseudo-lexically scoped.  For example,
 foo.baz prints not "FooExt#bar" but "Foo#bar" in the following code:
 
   class Foo
     def bar
       puts "Foo#bar"
     end
 
     def baz
       bar
     end
   end
 
   module FooExt
     refine Foo do
       def bar
         puts "FooExt#bar"
       end
     end
   end
 
   module Quux
     using FooExt
 
     foo = Foo.new
     foo.bar  # => FooExt#bar
     foo.baz  # => Foo#bar
   end
 
 Refinements are also enabled in reopened definitions of classes using
 refinements and definitions of their subclasses, so they are
 *pseudo*-lexically scoped.
 
   class Foo
     using MathN
   end
 
   class Foo
     # MathN is enabled in a reopened definition.
     p 1 / 2  #=> (1/2)
   end
 
   class Bar < Foo
     # MathN is enabled in a subclass definition.
     p 1 / 2  #=> (1/2)
   end
 
 If a module or class is using refinements, they are enabled in
 module_eval, class_eval, and instance_eval if the receiver is the
 class or module, or an instance of the class.
 
   module A
     using MathN
   end
   class B
     using MathN
   end
   MathN.module_eval do
     p 1 / 2  #=> (1/2)
   end
   A.module_eval do
     p 1 / 2  #=> (1/2)
   end
   B.class_eval do
     p 1 / 2  #=> (1/2)
   end
   B.new.instance_eval do
     p 1 / 2  #=> (1/2)
   end
 
 Besides refinements, I'd like to propose new behavior of nested methods.
 Currently, the scope of a nested method is not closed in the outer method.
 
   def foo
     def bar
       puts "bar"
     end
     bar
   end
   foo  #=> bar
   bar  #=> bar
 
 In Ruby, there are no functions, but only methods.  So there are no
 right places where nested methods are defined.  However, if
 refinements are introduced, a refinement enabled only in the outer
 method would be the right place.  For example, the above code is
 almost equivalent to the following code:
 
   def foo
     klass = self.class
     m = Module.new {
       refine klass do
         def bar
           puts "bar"
         end
       end
     }
     using m
     bar
   end
   foo  #=> bar
   bar  #=> NoMethodError
 
 The attached patch is based on SVN trunk r29837.
=end



-- 
http://bugs.ruby-lang.org/