From: Benjohn Barnes <benjohn@...>
Date: 2006-02-23T06:51:33+09:00
Subject: Dynamic code generation and linking

I'm very interested in being able to dynamically generate, compile,  
and link in c code (and potentially other languages, including  
assembly - but c is my primary interest), from within Ruby. When I  
say "link in", I mean have the compiled code immediately become  
available for use by the Ruby script that asked for the compilation.

So, I'm imagining something like...

# Creating a parameterless c function.
compiler.compile( 'printf( "Rock on, dudes!" );'  ).call

# A c function with an integer argument.
proc = compiler.compile( 'x' => Int, 'printf( "%d * %d = %d", x, x,  
x*x);' )
(1..10).each {|x| proc.call(x)}

A module that encapsulated this (particularly as part of the Ruby  
standard distribution) would allow a "pure ruby" module to have  
native speed [1].

It would also be awesome for dynamic code generation for building an  
algorithm optimised for a very specific situation. Coming from an  
image processing background, I find the idea quite exciting :) As an  
example, I could be using artificial evolution to grow functions that  
evaluate to a pretty picture. To evaluate these functions at maximum  
speed, I could compile them down to c, and execute them.

Another example would be creating a parser from a BNF like notation.  
It should be possible (making use of this module) to specify a  
grammar with a Rubyish domain specific language, and then compile  
that to native code and immediately make use of it from within the  
same Ruby program.

A further possibility is that it could pull all that pesky linking  
with external libraries (well I don't get on with it anyway, but it's  
not really my bag) in to the domain of Ruby, and make it a fun thing  
to explore from within IRB, for example. An offshoot from that would  
be an explorative SWIG like system: a simple c parser that you can  
point at a library's header file, allowing you to start hacking about  
with the c library as if you were back programming BASIC on your BBC  
[2] :)

Has anyone come across work in this direction, or is anyone working  
on it themselves? Could anyone comment on difficulties involved, or  
serious practical problems? I think the major issues are likely to be:

	Interfacing with the native linker.
	Passing stuff in and out of the c easily.
	Giving the c compiler header files (or bundles, or whatever it needs  
on OS X).

It's likely I'm completely missing many problems though - I'm coming  
to this having never written a c extension for Ruby!

My dream would be to encapsulate all of the platform specific nitty  
gritty in one place, and so make it reasonably simple for anyone with  
some c and Ruby knowledge to just dive in and start writing super  
fast code, or linking with a library they want to play with.

Sorry for a long post. I hope it will give you ideas, or stimulate  
interest.

Thank you for your time,
	Benjohn Barnes


[1] Although one would want a mechanism to be able to cache compiled  
code between runs eventually, as the compile time could become annoying.

[2] I think Ruby is such an enjoyable experience for me because it  
recreates the fun, and frankly joy, I used to have with BASIC, when I  
first learnt to program, but it allows me to concisely specify my  
ideas in ways that I've come to need after 20 years of programming.  
What I want to do here is get close to having BBC BASIC's build in  
assembly :) Total expressiveness, and raw touching the metal speed  
and control.