From: The 8472 <the8472@...>
Date: 2012-12-01T21:21:20+09:00
Subject: [ruby-core:50440] Re: [ruby-trunk - Feature #4085] Refinements and nested methods

On 30.11.2012 20:41, headius (Charles Nutter) wrote:
> respond_to? brings up a really good point: there's lots of methods that might need special refinement care...but I think it makes things more confusing rather than less.
>
> Let's take the respond_to? example. You want respond_to? to reflect refinements. That seems reasonable on the surface, even though respond_to? is not refined itself. The first peculiarity there is that you could no longer wrap respond_to?, respond_to_missing?, method, instance_method, and so on, because wrapping them would break their visibility to refinements (due to the intervening unrefined Ruby frame). We'd be reducing the metaprogrammability of many, many core methods.
>
> And then there's methods that *call* respond_to? (or coercion methods like to_s). Example:
>
> class X; end
>
> module M1
>    refine X
>      def to_path; '/tmp'; end
>    end
> end
>
> using M1
>
> File.open(X.new) # ????
>
> File.open checks respond_to?(:to_path) and if that succeeds it calls to_path. But the above code will never work unless File.open is also made refinement-aware.
>
> I think this is a pretty significant problem, and it shows how much more limited refinements will actually be. The bottom line is that making any core methods reflect refinements may be *more* confusing, because only direct invocations will work...not wrapped invocations, called-method invocations, or double-dispatched invocations.

The common problem of all these methods is that they happen down-stack. 
The obvious solution that problem would be the option to make 
refinements optionally apply on a thread-scope too, but the performance 
implications obviously are horrible as they would could impact any 
callsite anywhere in the whole application.

So I think the proper solution is to provide a much much more low-level 
metaprogramming alternative: The modification of individual, selected 
callsites and methods. I.e. Allow metaprogramming on the AST itself.

Once we can do that we can also inspect the stack - or as potentially 
more performant alternative as it wouldn't need to reify the whole stack 
- examine thread-local variables.

Modifying the AST obviates the problem of inheritance as the programmer 
has precise control of which method or callsite gets modified where.

To express some commonly thrown-around refinement examples as 
AST-transforms:

(Concepts shamelessly stolen from Java's MethodHandle/invokedynamic and 
AspectJ)


case a) Traditional monkeypatching

    class A
      def foo
        puts "in A"
        raise "woops"
      end
    end

    class B < A
      def foo
        super
        puts "after exception!"
      end
    end

    Ruby::AST.methods(A, :foo).enter do |method_name, *args|
      puts "inserted in #{self.name}" # we're inside the method here
    end

    A.new.foo
    # inserted in A
    # in A
    # Exception: woops

    # match super call
    sites = Ruby::Ast.methods(B, :foo).callsites(A, :foo)
    # match existing AST callsites and those defined in the future
    transform = sites.transforms(:existing, :future)

    # add a new transform
    sites.wrap do |source_self,target_self,target_as_proc, *args, &block|
      nil # don't execute target method
    end

    B.new.foo
    # after exception!


case b) Simple, scoped Intercept

    class C
      def m1
        "Foo".downcase
      end
    end

    # match methods in C, then select callsites inside those methods
    sites = Ruby::AST.methods(C, [:m1,:m2]).callsites(String, :downcase)

    # don't modify existing code
    sites.transforms(:future).after do |target, result, *args|
      result + "x"
    end

    class C
      def m2
        "Foo".downcase
      end

      def m3
        "Foo".downcase
      end
    end


    C.new.m1 # => foo
    C.new.m2 # => foox
    C.new.m3 # => foo



case c) advanced example, down-stack "refinement"

    f = future_transforms = []

    # not scoped to specific methods here!
    sites = Ruby::AST.callsites(String, :dasherize)
    transform = sites.transforms(:future, :existing)
    transform = transform.guard{Thread.token?(:refine_dasherize)}


    # modifies callsites in the whole application
    # megamorphic ones will suffer a performance loss from a typecheck.
    # optimized, monomorphic callsites will only
    #  suffer from the thread-local variable check
    # since it's not volatile it can be hoisted in loops
    f << transform.wrap do |target,target_method_as_proc, *args, &block|
       if target_method_as_proc != nil
          # ok, target already exists, let's call that
          target_method_as_proc.call(*args, &block)
       else
          # roll our own implementation
          target.gsub(%r_/, '-')
       end
    end

    sites = Ruby::AST.callsites(String, :send)
    transform = sites.transforms(:future, :existing)
    transform = transform.thread_guard(:refine_dasherize)

    # same performance impact as above
    f << transform.wrap do |target, target_method_as_proc, *args, &block|
      if args[0] == :dasherize
        # invoke directly -> this is a normal callsite
        # -> gets wrapped by previous transform
        target.dasherize
      else
        target_method_as_proc.call(*args,&block)
      end
    end

    # above deals with .send()
    # similar could be achieved with Module.prepend
    # but callsite modification is invisible to the stack
    # and to the module hierarchy
    # it's a matter of picking the right tools for the right job




    transforms = Ruby::AST.methods(D).transforms(:future)

    # enable refinement for all methods in ClassB and downstack
    transforms.enter do |method_name, *args, &block|
      # thread tokens should be a MultiSet to allow reentrancy
      Thread.add_token(:refine_dasherize)
    end
    transforms.exit do |method_name, return_value, *args, &block|
      Thread.remove_token(:refine_dasherize)
      return_value
    end

    class D
      def m1
        "a_b".dasherize
      end

      def m2
        "a_b".send(:dasherize)
      end
    end

    # transforms(:future) -> code gets patched
    require "some_gem"


    future_transforms.each{|t| t.suspend}

    # code doesn't get patched anymore
    require "other_gem"


Is this complex? Yes
Is this verbose? Yes
Can you shoot yourself in the foot with it? Yes

This is intentional. It's a low-level API meant to provide as much 
freedom to the programmer as possible. Higher-level abstractions (such 
as refinements) can be built ontop of it. These abstractions could cover 
many of the conflicting use-cases we have discussed in this thread.

Inheritance? A matter of module/method selectors
File scope? A matter of existing/future transform selectiveness
Super calls? You have full control over them, they're just callsites
OOP integration? Only when you want to go the extra mile

Actually, if we combine AST modification and Module.prepend we can 
actually get down-stack refinements with complete OOP-integration if 
needed. Assuming an .unprepend is also possible.

In fact, let me write this down:

case c) now with metaclass coercion:

   module StringRefinement
     def downcase
       super + "x"
     end
   end

   Class E
     def m1
       "Foo".send(:downcase)
     end
   end

   sites = Ruby::Ast.methods(E).callsites(String)
   transforms = sites.transform(:existing)

   # an alternative to .wrap
   # similar to method .enter/.exist. Except it's for callsites
   transforms.before do |target,method_name, *args, &block|
     target.metaclass.send(:prepend, StringRefinement)
   end
   transforms.after do |target,method_name, return_value, *args, &block|
     target.metaclass.send(:unprepend, StringRefinement)
     return_value
   end

   E.new.m1 # => foox



I think such a fine-grained API is what we will need. No single 
high-level API can cover all the use-cases provided for Refinements and 
retain performance at the same time. Simply because it is too coarse.