[ruby-core:18008] Improving the metaprogramming facilities of Ruby.

G'day.  I hope this is the right place to introduce this discussion, 
as the ruby-lang.org website is not entirely clear on where language
design and specification discussion takes place.

If this is the wrong place then, please, be forgiving and direct me to
the correct venue to address this.

Also, if this has been previously discussed then please direct me to the
appropriate threads.  I searched the mailing list but turned up nothing
obvious.[1]

Finally, I am sorry for the length of this message.  I wanted to be as
clear and concise as possible in presenting this, to try and avoid
confusion.  Clarity lead to more text than I expected, though.


I have recently been working on a Ruby / ERB template issue where,
essentially, we wanted to have the ERB templates run in an environment
that was almost, but not exactly, the same as Ruby.

Specifically, in the problem domain undefined variables are forbidden,
and we want to expose that rule to our ERB templates.  

This was solved, previously, using 'method_missing', and pretending that
variables were actually local methods; that was a losing proposition as
soon as a variable named 'raise' came along. ;)

It could also be solved with an 'instance_variable_missing' method,
analogous to the 'method_missing' method, but thinking about that
brought me to thinking about metaprogramming in Ruby overall.


At the moment Ruby has two distinct and parallel methods for managing
instance variables[2], being:

    @foo
    instance_variable_get("@foo")

I had, initially, expected that I could simply define a method, 
'instance_variable_get', on my object which would allow me to override
the standard ivar lookup process for our own.

What I found, to my surprise, was that @foo acts directly on the ivar
table, accessing or setting the instance variable directly.  The
instance_variable_get method does the same, and there is no actual
relationship between the two.


So, after thinking about this and how I would have expected Ruby to
handle the situation, I have two questions:

First, is these a reason why these two operations are distinct?

In consulting the Ruby 1.8 and 1.9 code I see that rb_obj_ivar_get in
object.c and the handling of NODE_IVAR in eval.c both call directly into
the rb_ivar_set function.

Other than differences in error checking[3] both seem to be more or less
identical wrappers over the rb_ivar_set method.

To validate this I experimentally patched eval.c to call the
instance_variable_get (and set) methods rather than directly applying
the call; the results of that (and my tiny test code) are attached.

This seemed to support my theory: both methods seem identical, in
effect, and Ruby continues to pass the packages test suite with the
changes in place.


Second, would it be acceptable to change Ruby (1.9, and ideally 1.8) so
that these two paths /were/ uniform, at least in principal?

I am, obviously, happy to invest the time to write the needed code and
testing, and I am willing to work to see this happen.  (eg, to seek
agreement on the specification of the changes, etc, and implement to
something that the various Ruby implementers all considered valid.)

To this effect I have included a rough draft of the specification I
think would be appropriate for this small area of Ruby.  This is
intended to strike an appropriate balance between the users and the
implementers in terms of performance and power.


One of the specific goals I have kept in mind in the proposed
modification is that the implementation should be free to optimize the
behaviour of direct (@foo) and indirect (instance_variable_get) access
to ivar storage.

I believe that a conforming implementation could ensure that these
methods directly accessed the ivar table rather than making a method
call without too much trouble, while still supporting the user in
overriding the methods; a combination of caching and a sufficiently
smart compiler should be able to ensure safety.


In the text I have use the following terms:

"the results ... are undefined":

    Ruby code may do this, but the behaviour is not guaranteed by Ruby
    in any fashion.  The code may behave as they expect, or it may cause
    an exception, cause Ruby to malfunction, or any other result.

    Implementers are encouraged to document the behaviour and limits of
    their implementation, as well as to provide sensible results when
    undefined actions are taken, such as raising an appropriate
    exception.

"conforming code":

    Ruby code that is written within the defined behaviour of the core
    system, rather than code relying on the behaviour of any particular
    implementation of Ruby.


The documentation below covers the range of Object instance_* methods
associated with ivars; the quoted text is the Ruby 1.9 documentation,
and my proposed additions are included.

Editorial comments -- the "why" I added the text are in square brackets,
such that:

> This is Ruby 1.9 documentation.

This is my modified text.  [This is my editorial comment on my text.]


The modified method specifications:

> obj.instance_variables    => array
>
> Returns an array of instance variable names for the receiver. Note
> that simply defining an accessor does not create the corresponding
> instance variable.

The array returned from this method may not be modified by the caller;
the results of doing so are undefined.

[This text was added so that anyone overriding this method knows that
 they can safely return an internal array of names without risk of it
 being modified -- for performance reasons, mostly.]


> obj.remove_instance_variable(symbol)    => obj
>
> Removes the named instance variable from <i>obj</i>, returning that
> variable's value.

This method may or may not be called during object destruction;
conforming code should not rely on this to perform variable
finalization.

[Added so that implementers are guaranteed to be free of the burden of
 calling this during object destruction, and to ensure users know where
 they stand.]


> obj.instance_variable_defined?(symbol)    => true or false
>
> Returns <code>true</code> if the given instance variable is defined in
> <i>obj</i>.

If instance_variable_get or instance_variable_set are defined then it is
the responsibility of the user to ensure that this method must also be
return accurate results.  The results are undefined if this rule is not
respected.

[Added so that Ruby libraries can assume that this method and the
 get/set methods are symmetric, and to make it clear to the user that
 they are expected to preserve this behaviour.]


> obj.instance_variable_get(symbol)    => obj
>
> Returns the value of the given instance variable, or nil if the
> instance variable is not set. 

This method is called by Ruby when you write <code>@variable</code>
within the scope of your object methods.  You can override this method
to change the rules for instance variable lookup, as long as you abide
by the specified rules.

[Added to make it clear that the user /is/ permitted to override this
 method, and that this will result in *all* ivar references calling
 their method.]

> The <code>@</code> part of the variable name should be included for
> regular instance variables.  Throws a <code>NameError</code> exception
> if the supplied symbol is not valid as an instance variable name.

The default implementation of this method returns <code>nil</code> if
the instance variable has not been set.  Conforming code is free to
override this in user defined classes, but the results of overriding
this method in system classes is not defined.

[This is added to make it clear to the user that, for example, returning
 false rather than nil from Object.instance_variable_get would be a bad
 idea, and to allow the implementer and third party library authors to
 assume the standard behaviour in core classes.]

Care should be taken to avoid recursion in user supplied implementations
of this method, as the results of directly referencing an instance
variable inside the method are undefined.

[This is added so that the user, and the implementer, know that they can
 just infinitely recurse in the face of the pathological code:

 class BadlyBroken; def instance_variable_get name; @recurse; end; end ]


> obj.instance_variable_set(symbol, obj)    => obj
>
> Sets the instance variable names by <i>symbol</i> to <i>object</i>,
> thereby frustrating the efforts of the class's author to attempt to
> provide proper encapsulation. 

[This is the original text, and is probably not terribly appropriate
 after the changes here, but I preserve it for the moment.]

> The variable did not have to exist prior to this call.

[The proceeding should be replaced with:]

This method is called to set the initial, and all subsequent, values of
the instance variable.  The variable will not exist prior to setting the
initial value.

Ruby will call this method whenever an instance variable is set,
including for <code>@variable = value</code> expressions within the
scope of your object.

[Added to make it clear this is always called by Ruby to set a
 variable.]

Conforming code may override this method, provided the specified
behaviour is implemented.  However, the results of overriding this
method on core Ruby classes is undefined.

[Added to make it clear what the user can, and can't, do.]

Care should be taken to avoid recursion in this method; the results of
directly setting an instance variable within this method are undefined,
and you should call your superclass instance_variable_set method
instead.

[Added to avoid forcing implementers to support pathological code:

 def BadlyBroken; 
   def instance_variable_set name, value; @recurse = true; end
 end 

 Implementers are free to support that, of course, if they wish, but
 conforming code cannot assume that it will do anything remotely
 sensible.]


Regards,
        Daniel


Footnotes: 

[1]  Since I have not previously contributed to the Ruby core I am
     somewhat nervous about making such a large proposal for the
     language, and I would really like to avoid offending anyone.

[2]  This is also true of pretty much every other type, such as
     constants, methods, and so forth.  Assuming that the general
     principals of this message are accepted I intend to also address
     those areas for consistency.

[3]  Which, in eval.c, is handled at a higher level, so both perform
     identical checks and have an identical environment, as far as I can
     tell.