Bytecode handling (compilation) extensions to Ruby 1.9

Hello,

Since 1.9 is YARV VM powered it is technically possible to dump &  
restore VM code (instructions) to/from file. Which means that we could  
load already compiled Ruby code directly into 1.9 VM without related  
sourcecode.

The most important application would be commercial Ruby programs  
deployment. Often when we deploy our programs at our client's servers,  
or some web hosting providers, we would love to keep source-code and  
the ideas behind in secret. Since ruby up to 1.8 was dynamic  
interpreter it wasn't possible before, but it is now possible with  
YARV & 1.9.

I'm aware that it would be also possible to use JRuby to get Java's  
JAR with compiled Java classes, but I just feel that 1.9 may have more  
potential and is more lightweight than Java, moreover it sets kind of  
standard for other distributions.
I know that 1.9 VM bytecode could be reverse-engineered, however due  
optimizations and removal of comments etc, it would not present such  
as value as original sourcecode.

So I wish to propose extension for Ruby 1.9 for generating (emission)  
of VM bytecode to the file and reading/loading VM bytecode back to VM.
My main aim is to NOT change the current Ruby behavior, NEITHER add  
new file type for "compiled" Ruby, but instead of that do few light  
additions:

1. "ruby" command line additions:
  -o outputfile      emit VM bytecode into outputfile instead of  
executing (-o - for stdout)
  --omit-bootstrap   omit VM loading bootstrap in outputfile (output  
file won't be valid Ruby program)
  -f[level]          follow statically required files and emit their  
bytecode too; 0=don't follow, 1=only current and subdirs, 2=all except  
system, 3=all

2. System API additions

VM#exec(input)   Loads VM bytecode from input (IO) and executes it
VM#emit(output)  Emits all current VM instructions as bytecode into  
output (IO)

So:

$ ruby -o myprogram_dist.rb myprogram.rb

Will emit VM bytecode with bootstrap "VM.exec DATA" as below:

--------------- myprogram_dist.rb -----------------
#!/usr/bin/env ruby
VM.exec DATA
__END__
RubyVM-1.9.0!@#!@!@%&()*@$*!)@$!@$....
---------------------------------------------------

So produced myprogram_dist.rb will be valid Ruby program, that raises  
an class-not-found or method-not-found exception on all distributions  
or versions that don't implement VM#exec, also I believe it should  
produce another exception in version that has VM#exec but passed  
bytecode structure format isn't valid (or incompatible), however in  
our VM#exec matching version it will successfully execute the code  
into the VM. (Maybe checking if magic is RubyVM-1.9.0)

Also we could have an option to omit bootstrap i.e. --omit-bootstrap,  
so we get instead:

--------------- myprogram_dist.rb -----------------
RubyVM-1.9.0!@#!@!@%&()*@$*!)@$!@$....
---------------------------------------------------

Of course then myprogram_dist.rb won't be valid Ruby program anymore,  
but it may be useful to store some compiled code chunks in database,  
produce some more advanced bundles or bootstraps, or maybe on demand  
downloadable & executable code with VM#exec.

Note that VM#exec should load and execute the loaded code, and once it  
is done return back and execute code that follows VM#exec, so it is  
possible to stack them:

VM.exec my_library
VM.exec my_magic_routine
# rest of the code

Finally it would be nice to add option to follow Ruby 'requires' that  
are static (aka. flatten the code)

Lets take such file structure for out sample project we want to  
"compile" prior deploying:

myprogram/
   main.rb
   db.rb
   scripts.rb
   lib/
     extra.rb

We want to make single file bytecode bundle myprogram.rb instead of  
several "compiled" files (per each source file).

Our entry point is here:
-------------- myprogram/main.rb ------------------
require 'db' # this is static require, we can follow it
require 'scripts' # this is static require, we can also follow it
require 'lib/extra' # this is static require, we can also follow it

require 'sequel' # this is static require, but not in the current folder

# (...)
---------------------------------------------------

We do:

$ ruby -f -o myprogram.rb myprogram/main.rb

And get myprogram.rb which has VM code for all main.rb and it's  
dependencies from myprogram/ folder.

Our -f new option will work as follows:
  -f0 would mean 'don't follow',
  -f1 'follow only current dir and subdirs',
  -f2 'follow all except system',
  -f3 'follow all'.

When -f not specified default behavior 'don't follow', when -f without  
level specified default 'follow current dir and subdirs'.

$ ruby -f2 -o myprogram.rb myprogram/main.rb

Will also include 'sequel' VM bytecode.

-f will not work obviously for any:
require SomeModule.getdependencies
require Config.getpath . 'tests'

As they can be only evaluated only at runtime. But that's not a  
problem. But it would be nice that ruby tells what it is compiling.

$ ruby -f -o myprogram.rb myprogram/main.rb
Compiling:
   main.rb
   db.rb
   scripts.rb
   lib/extra.rb
myprogram.rb done in 0.005 second(s)

That's all. I'd appreciate any comments to my proposition.

I'm new to this mailing-list, so I hope you don't find me too  
impudent :) I felt in love with Ruby few months ago, while programming  
regularly in C/C++ & PHP.
I've used PHP bytecode compilers for some of my past projects, and I  
miss this possibility in Ruby.

Best regards,
-- 
Adam Strzelecki |: nanoant.com :|