Re: Ruby and E.V.E. Paradox

Hey gga,

Thankyou for the reply and information.

gga wrote:
 > GD ha escrito:
 >
 >>This is what I needed to replace LUA:
 >>- Ability to load a script *from*memory*, not from a file (important).
 >
 >
 > Use rb_eval_string_protect().

Excellent, guessed that one at least. Maybe worth a mention in the docs
if that is the "official" solution; it wasn't too clear.

 >>- Hook a bunch of local functions (C++) so that the script can call them for
 >>information from the main code body.
 >
 >
 > Ideally, handle this using SWIG, as either a ruby extension or a pseudo
 > extension you link in statically.
 > Avoid doing the binding yourself manually.  Binding functions for most
 > basic C constructs (ints, chars, etc) is VERY easy.
 > Doing proper binding of classes, it is not as easy, as the GC requires
 > you to be careful.  Best thing for you to do is look at SWIG's output.
 > SWIG also deals properly now with inheritance of C++ classes into ruby,
 > including multiple inheritance.

I'm keeping things fairly simple; I mostly just bind functions/methods
and wrap pointers in opaque data structures (eg. using Data_Wrap_Struct
in the Ruby version). My plan was to store a pointer to the calling class
in an opaque structure and store it in the corresponding Ruby class, so
I can reference it when needed.

 >>- Call functions in the script both periodically and in response to events (eg.
 >>an update call every 1/60 of a second, in response to mouse clicks, etc).
 >
 >
 > This may be much trickier with the current ruby C interpreter,
 > depending on what you are looking for.   You can start a C++ thread to
 > do it, but the ruby C interpreter is not thread safe.  So, if you have
 > ruby code running in two different threads at the same time, bad things
 > WILL happen.
 > You can also try to rely on a single thread and let ruby's own light
 > threads handle this.  However, this may not be as efficient as actual
 > multi-threaded code.
 > Mind you, ruby is not the only language with issues here.  Python is
 > also pretty bad at this.
 > LUA and TCL are in a class of their own as far as multithreading
 > interpreters go.

Everything is single-threaded, or at least, the other threads don't even
know Ruby exists. Events and updates are handled in the one thread and
can be fed in in a safe and controlled manner. I'm not asking too much
of an embedded language, I'm keeping things fairly gentle. The nasty
stuff is handled in the C++ code.

 >>- Clean up everything entirely when the level is over.
 >
 >
 >
 >>This is what I have encountered:
 >>
 >>- There doesn't appear to be any way to indicate that Ruby should clean
 >>everything up and get to a state where I can start again. I have attempted to
 >>work around this by making each script contain a single unique class that I use,
 >>but this is a terrible hack.
 >
 >
 > ruby_cleanup(0);  SHOULD clean up the interpreter.   Problem is not so
 > much cleaning the interpreter but restarting it again.
 > I did find issues when I did this (back in 1.8.0).  I think now issues
 > are better, but I won't vouch for it being 100% solved.
 > If all fails, best thing you can do is what you did.  Encapsulate all
 > in its own class and clean that instead.  That will still leave the
 > class/module constants around, but it is better than nothing.

Indeed. Wasn't sure about ruby_cleanup, and I've seen a lot of examples
floating around, along with comments that things end up crashing shortly
afterward.

 >>- The documentation is flawed. It indicates calling rb_run(), but that can
 >>eventually call exit, which is the last thing you want from an embedded
 >>language. It mentions rb_apply() as having an argc parameter.
 >
 >
 > Again, rb_eval_string_protect() is what you probably want.

I have data structures in the C++ side, no names, so rb_eval_string and
company can't be used.

 >>- I am getting inexplicable crashes when calling the garbage collector.
 >
 > This is a probably a bug in your code.  Take a look at rubyMEL at
 > highend3d.com for a relatively simple embedding of MEL into a 3d
 > application (Maya).
 > Depending on platform (windows, mainly) you need other initialization
 > code besides ruby_init().

I won't be so arrogant to suggest my code has no bugs (haha) but we're
talking about code that has been run through days of heavy automated
testing each release with the current LUA setup being changed to use
Ruby instead, minimally reduced to using only two calls (ruby_init and
rb_gc) being called from multiple different locations and crashing
in different but consistent ways each time. It is entirely possible
that my code is interfering with it in some way, but it is manifesting
as predictable crashes in Ruby only, not in any of the other libraries
I am using. Short version: Looks more like a bug brought out by my
circumstances rather than because of the code itself.

I'm currently trying to get things going under Linux, so no tricky
init stuff yet.

 >>- The documentation is not at all clear how you protect something from the
 >>garbage collector, or indicate it is no longer in use.
 >
 >
 > You need to use a mark and a free function respectively, and use ruby's
 > mysterious Data_Wrap_Struct stuff.  This is the pretty hairy part of
 > exposing a C++ api.
 > Suggestion: look at SWIG's code.  It has a bunch of wrapper functions
 > to handle it nicely and it is now pretty mature.
 > For any questions regarding SWIG not clear from the SWIG manual, refer
 > to FXRuby's excellent wrapping of the Fox library.

I hadn't thought of going back into SWIG. Good idea, thankyou. I had a
play around with it a week or so ago, seemed to work well. No doubt
the generated code would grant some clues.

Should some of the tricks used by SWIG be mentioned on the official
Ruby site, perhaps? Seems odd that an offsite source is considered
(mildly) authoritative.

 >>- How does one call "Foo.new(args)" on a class from outside of Ruby, where the
 >>args have been generated from outside of Ruby (hence no name) meaning
 >
 >
 > You call it like any other method, with rb_funcall* like:
 >
 > VALUE cFoo;   // defined somewhere else...
 > int nargs = 1;
 > VALUE args[] = { INT2NUM( 30 ) };
 > ID new_id = rb_intern("new");
 > rb_funcall2( cFoo,  new_id, nargs, args );

Okay, so just like an object. Good news. Very elegant and Ruby-ish. :)
Any way to grab the class from the Ruby script? What I really want is
to replace your first line with something like:

VALUE cFoo = rb_grab_class_from_script("ClassDefinedInScript")

Each level defines its own class and I need to create an object of
the correct type. I can't find a call to lookup a class name and
return a VALUE. There is probably one, I just don't know what it is.

 >>- rb_protect() is a complete mess. Rather than being able to call an arbitrary
 >>function with a void pointer, you are stuck calling something that takes a
 >>single VALUE arg.
 >
 >
 > No, you are wrong.  It is extremely well thought out.
 > No need for ugly void*, thank god.
 > Think of VALUE as being a void*  (it is really an unsigned long).  Ruby
 > requires sizeof(unsigned long) == sizeof(void*) -- see ruby.h's check.
 > Just do a simple casting of your pointer to VALUE in your caller and do
 > the opposite cast from VALUE to whatever in your receiver function.
 > See how all the RCAST() and similar macros work in ruby.h.

The best way to illustrate would be with an example:

VALUE GlueFunc(void *v)
{
   Glue *glue = (Glue *)v;
   return rb_funcall(glue->game->getLevel(), glue->method, glue->arg0);
}

struct Glue
{
   Game *game;
   ID method;
   VALUE arg0;
};

void Game::runWithProtect1(VALUE level, ID method, VALUE arg0)
{
   Glue glue;
   glue.game = this;
   glue.method = method;
   glue.arg0 = arg0;
   VALUE rv = rb_protect_that_uses_void(GlueFunc, &glue, &status);
   if (status)
     reportError();
   return rv;
)


In main code:

ID id_update;
VALUE current_frame;

....

runWithProtect1(id_update, current_frame);

(NB: There would be a large number of calls to runWithProtect(), so I need
to keep the call format simple).

If the glue function must be a VALUE which is an array, what steps
would you suggest were used to ensure the relevant data, include a C++
pointer, were passed through the callback to reach the glue function
GlueFunc? If you're bored, code up a solution that works. I managed to
get something going, minus the "this" pointer, and it was messy. It
involved converting everything into Ruby objects and putting them into
an array, and deconstructing at the other end. Perhaps you have a better
solution, and I'm open to improved ideas.

 >>- The code runs fine for a while if I don't call the garbage collector.
 >>Unfortunately, I get stack depth errors and eventual segfaults if I make
 >>repeated calls to the Ruby script (via rb_apply()).
 >
 >
 > This likely means you wrapped your C++ classes incorrectly or you are
 > trying to run ruby in a multi-threaded environment.
 > If the first is true, look and learn from SWIG.  If the second is true,
 > remove the multi-threaded code, create a global mutex that all your
 > ruby functions should call before doing anything or use a different
 > scripting language at this time.
 > Ruby is not thread safe, unlike LUA.

Only using Ruby from a single thread. I am likely making mistakes
with creation and marking of Ruby objects/classes as I'm not familiar
enough with it nor does the documentation explain it. Thus manually
calling the GC, which has led to crashes even when all I do is init
and call the GC, hence my problems. I can't be making mistakes with
Ruby if I'm not actually loading any scripts or creating any objects
and classes.

 >>- I can't find one example that completely and correctly demonstrates the whole
 >>process involved in having C call Ruby then call C, return values through the
 >>chain neatly, and clean up. If there was such a reference, it would be 
outstanding.
 >
 >
 > There's no real trick to it.

I must disagree. If it is easy, there is a link _somewhere_ that
shows a concrete official example of how it should be done. Maybe
I have not found it. If it doesn't exist, if a Guru who intimately
understands the code could put one together and put it in the official
documentation, it would be an enormous help to people using the
embedded interface.

 > Ruby is fine as an embedded language, just as you remember it is not
 > safe as a multi-threaded embedded language.  YARV may eventually solve
 > this somewhere in the future.  The unofficial fork of Ruby that is
 > Sydney (and now rubinius) supposedly is multi-thread safe, but is
 > 32-bits only (and I won't vouch for it).
 > Python is just as bad as Ruby, btw, albeit it avoids some threading
 > issues by using a global lock every time two threads try to run some
 > code simultaneously, effectively running only a single thread at a
 > time.  In ruby, you can do the same, but you need to create this global
 > mutex yourself.
 > LUA and TCL are different and better in this aspect, as they both can
 > use different (and very light) interpreter instances to execute code.

I'm not using Ruby as a multithreaded language; only one thread will
ever talk to it and being MT safe is unimportant to me.

The Python docs look decent, I'd just rather not relearn Python again,
I chose Ruby years ago, I'm good with it, and I like it better than
Python! :) But I'd rather move on from LUA if I can...

LUA has some very nice elements; there is a call to get a context, a
call to tear it down, so forth. It works. The interface to the controlling
code, however, is less than ideal and poorly documented. Using it is
painful. Error-handling makes me want to gouge my eyes out. The language
is a mere flicker to the brilliance of Ruby. However, it is very well
suited for the task; its design has always been around being an embedded
language. The embedded interface works. It is a well-constructed embedded
language.

My knowledge of TCL is non-existent.

 >>Am I alone in thinking this? Should I be looking at other embeddable languages
 >>(Python, for example), or just go back and get the LUA code working again? Am I
 >>doing something wrong?
 >
 > Truth is... you are looking at the two best languages for embedding.
 > LUA is admittedly better to embed due its multi-threading aspects,
 > while Ruby is admittedly better for embedding due to its nice syntax.

Actually, I got LUA up and running in an hour or so. It was very basic,
but a good start that evolved into the current codebase. The documentation
was poor but I got it eventually. I'm not even using the MT stuff. I didn't
even know it handled it well.

I've spent close to two days on embedding Ruby and can't get the basic
things I've described working. The closest I've got is short runs followed
by crashes I can't diagnose. The embedding documentation is all over the
place, lacks examples, and has mistakes. When someone with twenty years
of development experience is having troubles this bad it does suggest there
might be a problem that needs addressing...

It is enormously frustrating as the Ruby language is significantly superior
to the LUA language, but the case is the exact reverse with the embedding
interface- and the LUA embedding interface could be much better! It breaks
my heart to have to struggle so hard to get a powerful language like Ruby
- that I use throughout my project extensively- to work reliably as an
embedded scripting language for the runtime itself. If I wasn't such a
huge fan of Ruby- and I promise you I am difficult to impress with a
language- then I would have given up after the first few frustrating
hours and said to myself "language is not ready yet", then moved on to
something else. I dearly want to be completely wrong. If I can get Ruby
going I can increase the significance of scripting in level design, and
add some really cool features I can't be bothered implementing in
LUA.

 >>Is there anything I can do to successfully and reliably add support for my
 >>favourite scripting language into my project? Or is it simply an unrealistic
 >>prospect?
 >
 > Follow the advices I gave you and you should be able to get ruby as an
 > embedded language.  Wrapping classes for the garbage collector is the
 > hairy part, but you can steal all of SWIG's source code for that now
 > (or use SWIG itself for doing it).
 > However, if your game engine does require heavy multi-threading in the
 > scripting language, you are indeed looking at the wrong language.
 > LUA is still the best language for this, unfortunately.

I'll dig around in the SWIG stuff, this may give me some ideas.

Thanks for the suggestions and input, much appreciated.

Garth