[ruby-talk:14537] Rajah: Ruby-Java integration

Hi.

Our names are Dan Munk and Matthew Spencer. We have noticed the recent threads
of discussion regarding Ruby/Java integration, and decided that it was time for
us to announce a project we have been quietly researching and developing for a
while now.

We would like to introduce Rajah, a bridge between the Ruby and Java languages.

Below you will find an outline of the development we have done to date. Please
excuse us if the doc below is a little rough. We want to introduce Rajah to
developers in a detailed way, but bear in mind that this document is also
a submission to the Ruby community outlining a work in progress.

Before we get into the discussion of Rajah itself, we should mention that we
would be very interested in any ideas or feedback regarding this document and
Rajah itself. We are very open minded about the future of this project, and we
would like all of the input we can get. We are both relatively new to Ruby, and
we are most concerned about making Rajah a *complete* Ruby/Java solution, not
simply an extension that is functionally sufficient. We look forward to hearing
from you!

      Dan Munk      
      [e-mail]

      Matthew Spencer
      [e-mail]


================================================================================

1. Introduction
2. Roadmap
3. Design
4. Known Issues


1. Introduction
--------------------------------------------------------------------------------

Rajah is an open source project that provides a bridge between the Ruby and Java
languages.  Within the Rajah project there are three planned Milestones:

    + From Ruby to Java.  The majority of this work is complete, but as we are
       fairly new to Ruby we are worried about correctness issues.  Thus this
       post. :)  A later iteration of this milestone would be Ruby extensions
       written in Java.  This piece has not started, but we would like to mimic
       the existing C extension architecture as much as possible.

    + The second milestone is from the JVM to Ruby.  The general approach and
       architecture has been considered, but no real implementation work has
       been done.

    + The third milestone would be a pure Java implementation of the Ruby
       interpreter.

The rest of this document is focused on design and implementation details of
the first milestone, from Ruby to Java.   Any ideas/feedback related to any
portion of this doc would be greatly appreciated.


2. Roadmap
--------------------------------------------------------------------------------

Once we implement the changes from the feedback that we are hoping to receive
from this post we are planning to release an alpha build on May 14 to the Ruby
application archive.  It is possible that this date will slip a week or two 
depending on the number and scope of changes.  At this time CVS on Sourceforge
will be used for versioning and snapshots will be continuously available.

We are members of the Extreme Programming camp (thank you for the RubyUnit 
effort) and internally we have created several iterations for each milestone.
The iterations for the first milestone will be updated and published on
Sourceforge once they are adjusted to include tasks from your input.

Following milestones will not be started until the previous milestone(s) have
been met and are of production quality.

We can test on Solaris(Sparc 7, 8), some Linux flavors, and Windows 98/NT/2000.
Help testing with additional platforms would be greatly appreciated.

The Rajah code is designed to run under any Java environment 1.1+. However, we
will only be testing with 1.2 and later, and for the first iterations (at least)
will not claim to support any Java environments prior to 1.2.


3. Design
--------------------------------------------------------------------------------

One of the major factors influencing all design decisions is that we wanted to
keep the Java side of the interface as transparent as possible.  Code to open a
JFrame would be:

        require 'Rajah'
        frmCls = Java.forName('javax.swing.JFrame')
        frame = frmCls.new("Example")
        frm.pack
        frm.show

The remainder of this section outlines the design of Rajah's Ruby to Java 
interface.

The Rajah library introduces a module named "Java". This module encapsulates the
Java functionality, including methods for loading Java classes. Once a Java
class is loaded into Ruby using Rajah, the class can be treated as a Ruby
class, including features such as method invocation, reflection, sub-classing,
and just about everything else you can do with a Ruby class.


Loading Java classes

The most fundamental capability of Rajah is loading Java classes into memory,
and allowing them to be accessed as Ruby classes from within the Ruby language.
This process involves the following primary steps:

+ Find the Java class.
+ Load the super class of the Java object.
+ Register the class with Ruby.
+ Introspect the Java class for all of it's public constructors.
+ Introspect the Java class for all of it's public methods (both static
   and instance methods).
+ Introspect the Java class for all of it's public fields (both static
   and instance fields).

Let us look at these steps in detail.

+ Find the Java class

   In order to load a class into Ruby, the embedded virtual machine must be
   able to find the Java class. In order to correctly find the class, the
   fully qualified class name must be provided (e.g., "java.lang.String").
   The classpath is then searched for the class. If the class is not found,
   an exception will be thrown.


+ Load the super class of the Java object

   In order to properly load a Java class into Ruby, it's super class must first
   be loaded. An attempt will be made to load the super class of the Java object
   (this is a recursive process, stopping when the top of the class hierarchy is
   reached).


+ Register the class with Ruby

   Registering a Java class with Ruby involves creating a new Ruby class that
   directly references the Java class. If the Java object has no super class
   (and thus is the "java.lang.Object" object), the new Ruby class is created
   with a super class of object Java::JavaObject. Otherwise, the new Ruby class
   is created with a super class of the Ruby representation of the Java super
   class (which must have already been registered with Ruby since super classes
   must be loaded before registration of a Java class can take place).

   Note that when the class is initially registered with Ruby, none of the
   Java constructors, methods, or fields are added to Ruby immediately.
   Introspection of the Java class does not occur until after the class is
   registered with Ruby (but before the class is finished being loaded by
   Rajah).


+ Introspect the Java class for all of it's public constructors

   The Java class is then introspected for all of the publicly accessible
   constructors. These constructors are added to the in-memory representation
   of the Java class. When the first constructor is added to the class, an
   "initialize" method with a variable number of parameters is added to the
   class. Every constructor for every class loaded via Rajah shares the same
   C function callback for handling the "initialize" method. See the section
   on object creation for more information.


+ Introspect the Java class for all of it's public methods

   The Java class is then introspected for all of the publicly accessible
   methods. These methods are added to the in-memory representation of the Java
   class. Every method of every class loaded via Rajah shares the same C
   function callback for handling the method. See the section on methods
   for more information.


+ Introspect the Java class for all of it's public fields

   The Java class is then introspected for all of the publicly accessible
   fields.


Object creation

Objects are created in Ruby using a class' "new" singleton method.
When Java objects are registered with Ruby, a handler for the singleton method
"new" is registered. When the new method is invoked, a new Java object of the
Java class is instantiated. However, a constructor is not immediately invoked.
Using JNI allows Rajah to instantiate a Java object without any constructor
being invoked, thus we postpone construction of the object till later. After the
new Java object is instantiated, it is wrapped in a Ruby object. Then, the
initialize method is invoked with the same parameters the new method was invoked
with. See the next section for more information on method invocation. The
initialize method will invoke the appropriate constructor on the Java object
created in the new method. If no matching constructor is found using the given
Ruby arguments, an exception will be generated.


Methods

Before delving into the details of how Java methods are handled in Ruby, let us
examine a few of the requirements for Java methods:

+ Support for static and instance methods of Java classes.
+ Support for overloaded Java methods.
+ Support for a lightweight invocation interface (i.e., Java method 
  invocation should impose as little overhead as possible).

All Java methods registered with Ruby use a variable length argument list to
process the method. This serves a few purposes:

1. Allows every instance method of every class to use the same C function to
    handle the method. This minimizes code complexity, and keeps the C API
    simple. Likewise, every static method of every class can use another C
    function to handle the method.
2. Allows support for overloaded Java methods.
3. Since parameters need to wrapped into an array internally eventually anyway,
    the performance of this is negligible.

Method invocation process

When a method is invoked, the following steps occur:

+ The name of the method that was invoked is retrieved.
+ The Java class that the method was invoked on is searched for a method that
   matches the name, the argument count, static or non-static criteria, and
   the argument types. If no method is found that matches all criteria, an
   exception is raised.
+ The Ruby arguments are marshaled (see below) into corresponding Java
   parameters.
+ The Java method is invoked via JNI. If any Java exceptions occur, a
   corresponding exception will be generated and raised in Ruby.
+ The return value of the method invocation is then unmarshaled into a Ruby
   object and returned from the method invocation.

Rules for mapping Ruby types to Java types

When a Java method or constructor is invoked using Ruby arguments, the arguments
must be converted into appropriate Java primitives or objects. This process we
refer to as marshaling. Likewise, the return value of a method (if the method is
not declared to return void) must be converted into the appropriate Ruby object.
This process we refer to as unmarshaling.

Java provides two fundamentally different entities that need to be examined for
the marshaling and unmarshaling processes: primitives and objects. There are
eight primitive types in the Java language:

+ boolean (values values are the Java keywords "true" and "false")
+ byte (signed 8-bit integer)
+ char (16 bit Unicode character)
+ short (signed 16-bit integer)
+ int (signed 32-bit integer)
+ long (signed 64-bit integer)
+ float (signed 32-bit floating point number)
+ double (signed 64-bit floating point number)

These primitives are handled by Rajah to correctly map Ruby values to Java, as
well as map Java values back to Ruby.

In Java, every object (either directly or indirectly) extends the Java class
"java.lang.Object". The single exception to this rule is the "java.lang.Object"
class itself, which has no super class.

Handling of primitives and objects should be fairly transparent from within
Ruby. However, when overloaded methods and constructors surface, some
familiarity with Rajah's mapping marshaling and unmarshaling rules is
necessary.

Marshaling

Marshaling of each individual parameter occurs sequentially, starting with the
first parameter. (Note that since the method is matched before any real
marshaling occurs, errors during the marshaling process should be infrequent.)
The Java type of the parameter is examined, and the Ruby value is marshaled into
an appropriate Java type. Here is a list of the marshaling rules when the given
Java type is encountered:

+ boolean - Ruby "true" will map to a Java "true". Ruby "false" will map to a
   Java "false".
+ byte - Fixnums within the range -128 to 127.
+ char - ???
+ short - Fixnums within the range -32768 to 32767.
+ int - Fixnums or Bignums in the range -2^31 to (2^31 - 1).
+ long - Fixnums of Bignums in the range -2^63 to (2^63 - 1).
+ float - Floating poit values in the appropriate range.
+ double - Floating poit values in the appropriate range.
+ java.lang.String object - Ruby string objects will be marshaled to
   java.lang.String objets. Of course, java.lang.String objects are also
   acceptable.
+ arrays - Ruby arrays will be marshaled into Java arrays. However, this places
   restrictions on what the values in the Ruby array can be so that marshaling
   can properly occur for every element in the array.

Unmarshaling

Unmarshaling values returned from Java (either via return values from invoked
methods or from field access) will map a Java object to a Ruby object. 
If the return type is a Java class that has not been loaded into Ruby yet,
Rajah will go through the class loading process for the return value as 
outlined above. This section will be explained in more detail in the Rajah
documentation.



4. Known Issues (in no particular order)
--------------------------------------------------------------------------------

- Support for arrays is not complete.

- There is currently a conflict between Java methods with the name "initialize"
   and the predefined "initialize" defined for each Java class when it is
   loaded into Ruby. Currently, no special checks are done when a Java method
   with the name "initialize" is loaded. The results are undefined if an attempt
   is made to load a Java class with a Java defined initialize method.

- Support for exceptions from Java is not complete.

- In Java, throwable types (java.lang.Throwable objects) extend
   java.lang.Object. In Ruby, the convention is to create exceptions that
   extend the class Exception. It would probably be nice if we made all of the
   Java Throwable objects somehow extend the Ruby Exception class. However,
   technically the Throwable object should extend the java.lang.Object class.
   Is it "bad" Ruby style to throw objects that do not extend Exception?

- Support for Ruby objects implementing Java interfaces is being examined, but
   not currently implemented (no pun intended). JDK 1.3 might alleviate parts of
   this problem, as JDK 1.3 introduces runtime interfaces (Proxies) for objects.
   However, even this approach would probably require helper classes written in
   Java.

- What about abstract classes? Abstract methods? Behavior needs to be defined
   when abstract classes are loaded.
   
- What about Java interfaces?

- When no Java constructor is supplied in the class, does introspecting the
   class still give us a no-parameter public constructor? If not, should we
   supply one ourselves?
  
- How are methods that begin with capital letters or _'s handled in Ruby?

- How are classes that begin with lowercase letters handled from Ruby?

- If a method is invoked that is not found in the Java class, should we attempt
   to invoke the method directly on the super class, or is that even necessary?
   
- Currently only public methods/fields are loaded. Do we want to load
   protected methods/fields too, in case a ruby class is sub-classing one of the
   java classes and wants access to protected fields? This might not even be
   possible with the reflection API, but it is something we should look in to.
   
- Subclassing Java classes loaded into Ruby is not supported right now.

- We should make the Java method "String toString()" map to "to_s" in Ruby.