Re: new method dispatch rule (matz' proposal)

On 6/13/07, David Flanagan <[e-mail]> wrote:
> This is a late response to the very long thread that started back in
> January with this message:
>
> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/9996
>
> As I understand it, the problem Matz is seeking to address is that
> private methods are inherited by and may be overridden by subclasses.
> This means that a subclass writer can inadvertently alter the behavior
> of its superclass simply by defining a method that has the same name as
> an internal helper method of the superclass.  The only way to defend
> against this is for subclass writers to be familiar with the internal
> implementation of the superclass.

I would argue that there's really no technical defense against the
issues raised by inheriting from classes outside of your control, or
having classes outside of your control subclass your class.
Inheritance is implementation sharing and it inherently breaks
encapsulation.  That's why it's almost always better to delegate to
classes you 'buy' rather than subclassing them.

Back when I was evangelizing OOP for IBM, I used to talk about this
and say that inheritance was like sex, it's much more socially
acceptable when it's done between committed consensual adults.

This isn't to say that such subclassing is always bad, just that it
requires care.

The case isn't that subclass writers can inadvertently alter the
behavior of a superclass by overriding a private helper method.
Without actually opening up the superclass and changing it, the
superclass and it's instances will still work.  What gets broken is
the subclass and its instances, and I'd argue that that's no different
than any other bug in the subclass, and it's not limited to
inadvertantly changing a PRIVATE method. Of course private methods are
more likely to be undocumented and missed by a drive-by subclasser.

And if the concern about breaking classes is real, then the solution
would need to include doing away with the ability to open existing
classes. That would really mean that Ruby wouldn't be Ruby anymore
IMHO.

My sense is that if the changes were to be implemented they would just
provide a false sense of security, better to avoid introducing further
complexity, I'd argue for better "inheritance education" which
advocated delegation over promiscuous inheritance, and
"safe-inheritance" practices using the existing mechanisms.

> Matz's proposal was to resolve this problem by altering the method
> resolution algorithm.  If I understand correctly, the problems were:
>
> 1) It was kind of complicated
> 2) The method lookup algorithm depended on whether a method was called
> functionally without an object prefix or in OO form with an object
> prefix.  (That is called as foo() or o.foo())
> 3) There were backward compatibility issues
>
> Part of the discussion was about the question of whether to alter the
> semantics of private methods, or add a new "local" or "personal"
> visibility level for methods that were truely local to the defining
> class and could not be inherited or overridden.
>
> With all that as review, here are my thoughts on the matter.
>
> 1) It is reasonable to expect private methods (or local methods if we
> call them that) to be looked up and dispatched differently than
> non-private methods.

Personally, my instincts say no.

> 2) The problem, though, is that since Ruby is a dynamic language, the
> interpreter can't tell whether a method is private or not until it has
> looked it up.

And here I think that we run into the tensions between the concepts of
static and dynamic languages. Introducing static features can have
some unpleasant effects.  There are already areas where Ruby falls
down slightly on properly handling dynamic changes, for example the
double module inclusion problem where a change to module A to include
module B isn't seen by modules/classes which have already included
module A:
http://eigenclass.org/hiki/The+double+inclusion+problem

It seems to me that this is just an implementation issue, fixing it
would involve a little more internal bookkeeping so that the method
lookup chain of mdules including A when A was changed.

Or the issues involved in handling "re-including" a module which is
already included by an ancestor:
http://talklikeaduck.denhaven2.com/articles/2006/10/09/a-subtle-change-to-mixin-semantics-in-ruby-1-9

Ruby 1.8 ignores the 're-inclusion' at one point 1.9 altered this
behavior, then it went back, I haven't re-built 1.9 in a while so I'm
not sure where the matter lies now.

The 1.8 semantics on module 're-inclusion' always seemed wrong to me,
I'm not sure I fully understand the rationale, I haven't thought it
through but I thnk that it might be wrapped up in a relationship with
the double inclusion problem.

These edge-cases are really small flaws which make Ruby just a little
brittle and less dynamic than it could be.  I think of them like the
stress hardening which happens when you bend a piece of metal back and
forth.  Do it enough and the metal breaks at those hardened points.
Fixing them would be like annealing the metal to get back its original
malleability.

Getting back  to my main point my fear is that implementing the kind
of tinkering with method lookup being proposed will  increase the
number of these edge cases.

> 3) Therefore, solutions like Matz's proposal depend on the method
> invocation syntax (functional style versus object-oriented style) to
> determine which method lookup algorithm is to be used.  This leads to
> the confusing and hard-to-justify situation in which the invocation
> "foo" might do something different than the invocation "self.foo".
>
> I propose, therefore, that some new syntax be introduced (avoiding
> backward incompatibility) either for making local method names obviously
> different from other method names.  Or that a  new method invocation
> syntax be introduced and that this new syntax be used when we want to
> invoke a local method defined in (or inherited by) the caller without
> considering any methods defined further down the class hierarchy.

IMHO, Ruby already has enough syntax, of course that comes from my
Smalltalk background.


> I don't grok parse.y well enough to know whether any syntax I would
> propose would actually result in an unambiguous grammar, but my
> suggestion is that local methods have an @ prefix just like instance
> variables do.  Then the interpreter knows from the name of the method
> that it is to be dispatched differently.
>
> In fact, I might actually go further than this and propose that local
> methods are not actually methods, but are syntactic sugar for lambdas.

Which means that the same thing should be possible with DSL like
metaprogramming and without syntax changes to the language.

> In Ruby 1.8, we can write code like this:
>
> class Test
>    def initialize(greeting)
>      @greeting = greeting
>      @greeter = lambda { |x| puts "#@greeting #{x}" }
>    end
>
>    def greet(x)
>      @greeter[x]
>    end
> end
>
> t = Test.new("hello")
> t.greet("world")
>
> In this code, the instance variable @greeter refers to a local function
> that is completely hidden from subclasses and cannot be altered.

Actually, it can since it's an instance variable.

class TestSub < Test
     def initialize
          super
          @greeter = { rand }
      end
end

or even

class TestSub2 < Test
    def set_greeter(&b)
        @greeter = lambda(&b)
    end
end


> Perhaps Ruby 1.9 could take this idiom and add syntax sugar to make the
> definition and invocation of local functions more like the definition
> and invocation of regular methods.

My opinion is that the cure for this 'problem' is worse than the disease.


-- 
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/