[ruby-core:18375] Stack "Overflow" Patch

In my spare time, I've been working on a OSX InputManager that =20
automatically injects a ruby console into every application (inspired =20=

by FScript). I noticed several crashes in ruby code, and I think one =20
of them might be a bug in ruby itself. There are a couple more in =20
rubycocoa, but that's a subject for another list. I should also =20
mention that I am not a ruby guru, so my understanding here may be =20
flawed. Please correct anything I get wrong :)

Ruby apparently integrates its own stack with the C stack, so it looks =20=

for overflows by measuring the C stack at certain points and making =20
sure it never gets too large. Unfortunately, when writing code that re-=20=

enters ruby, this mechanism can become confused. For instance, if I =20
have a 10k stack when I first call into ruby, rb_gc_stack_start is set =20=

to C_STACK_START - 10k if my stack grows down. Ruby's stack usage is =20
computed (in pseudocode again) by STACK_END - rb_gc_stack_start. If I =20=

then re-enter from a 1k stack in C, however, STACK_END - =20
rb_gc_stack_start becomes negative. Since pointers aren't signed, the =20=

stack size looks huge and ruby reports a stack overflow.

The least intrusive solution I could think of was to effectively take =20=

the absolute value of the stack length result. It isn't elegant =96 with =
=20
a hugely varying pre-entry stack, the same problem could occur =96 but =20=

it solves my immediate problem without adding any code to what seems =20
like a security sensitive part of ruby. This patch is =20
stack_check_1.patch, attached below.



A more proper solution would be to cut off the stack length  
computation at 0. It seems risky, and I'd love to hear from the more  
security-minded denizens of this list, but I can't really think of a  
situation where a "negative" stack length could be exploited. This is  
stack_check_2.patch.




Although it isn't really a bug in ruby, I wanted to include a third  
patch for IRB that adds an option in its @CONF to turn off the  
automatic SIGINT trapping it does. It isn't too difficult to work  
around, but it seems More Proper to have an actual configuration  
option for it.



All patches are against the 1.8 branch, revision 18781.


There is another problem when calling into ruby from multiple threads  
(bad, I know, but sometimes unavoidable, and it really ought to work  
within a mutex): each thread has its own stack, which predictably  
confuses the stack length computation. Before I make a patch that uses  
pthread thread-specific variables or stores ruby's stack in the heap,  
what would its chances be of actually being used (and which strategy  
would be preferable)?


Thanks,
Jonathan