Fwd: Is the SQLite3 gem known to have memory corruption?

   Reposting here after doing a bit more research (more details at  
the end)...

Begin forwarded message:

> From: "Timothy J. Wood" <[e-mail]>
> Date: February 5, 2006 9:50:19 AM PST
> To: [e-mail] (ruby-talk ML)
> Subject: Is the SQLite3 gem known to have memory corruption?
> Reply-To: [e-mail]
> X-Mailer: Apple Mail (2.746.2)
>
>
>   Running the following produces bizarre results (trimmed down from  
> an actual app):
>
> -----
> #!/opt/local/bin/ruby
>
> require 'rubygems'
> require_gem 'sqlite3-ruby'
>
> def each_batch(size=1)
>   until false
>     batch = []
>     (1..20).each {|i| batch << "x"}
>     yield batch
>   end
> end
>
> database = SQLite3::Database.new("/tmp/test.db")
> counter = 0
> each_batch {|batch|
>   database.transaction {
>     counter += batch.length
>     print "OK\n"
>   }
> }
> -----
>
>   For a while it prints "OK", but then eventually:
>
> ./test_bad_batch.rb:20: undefined method `length' for  
> 4085314:Fixnum (NoMethodError)
>         from /opt/local/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0/ 
> lib/sqlite3/database.rb:593:in `transaction'
>         from ./test_bad_batch.rb:19
>         from ./test_bad_batch.rb:11:in `each_batch'
>         from ./test_bad_batch.rb:11:in `each_batch'
>         from ./test_bad_batch.rb:18
>
>
>   That is the 'batch' is getting yielded as random pointer instead  
> of the array generated inside each_batch!
>
>   I've tried narrowing this further (particularly getting rid of  
> SQLite usage), but this is the simplest case I can come up with so  
> far.
>
>   I'm running on Mac OS X 10.4.4 with ruby 1.8.4 installed via  
> Darwin Ports (as well as rubygems installed from there and then  
> sqlite3 installed via that copy of gems).
>
> ----
> % /opt/local/bin/ruby -v
> ruby 1.8.4 (2005-12-24) [powerpc-darwin8.3.0]
>
> % /opt/local/bin/gem list sqlite3-ruby
>
> *** LOCAL GEMS ***
>
> sqlite3-ruby (1.1.0)
>     SQLite3/Ruby is a module to allow Ruby scripts to interface with a
>     SQLite3 database.
> ----
>
>   I've tried both the system supplied sqlite (v3.1.3) and the  
> latest version (installed via Darwin Ports), 3.3.3.
>
>   Any info would be helpful!
>
> -tim


   I've downloaded 1.8.4 and build with CFLAGS="-g -O0 -fno-inline- 
functions" and the bug still happens.  Breaking on rb_method_missing,  
I get a backtrace like:

#0  rb_method_missing (argc=1, argv=0xbfff4fc0, obj=36816053) at  
eval.c:5460
#1  0x00014524 in call_cfunc (func=0x12bb4 <rb_method_missing>,  
recv=36816053, len=-1, argc=1, argv=0xbfff4fc0) at eval.c:5553
#2  0x000137b8 in rb_call0 (klass=3366028, recv=36816053, id=3889,  
oid=3889, argc=1, argv=0xbfff4fc0, body=0x3340f8, flags=2) at eval.c: 
5692
#3  0x000151b4 in rb_call (klass=3366028, recv=36816053, mid=3889,  
argc=1, argv=0xbfff4fc0, scope=1) at eval.c:5920
#4  0x00015768 in rb_funcall2 (recv=36816053, mid=3889, argc=1,  
argv=0xbfff4fc0) at eval.c:6056
#5  0x000131f4 in method_missing (obj=36816053, id=4257, argc=0,  
argv=0x0, call_status=0) at eval.c:5533
#6  0x00015088 in rb_call (klass=3346628, recv=36816053, mid=4257,  
argc=0, argv=0x0, scope=0) at eval.c:5900
#7  0x0000be80 in rb_eval (self=3361128, n=0x3251e8) at eval.c:3381
#8  0x0000bd28 in rb_eval (self=3361128, n=0x3251ac) at eval.c:3376
#9  0x0000cfb4 in rb_eval (self=3361128, n=0x32510c) at eval.c:3551
#10 0x00010c50 in rb_yield_0 (val=6609412, self=3361128, klass=0,  
flags=0, avalue=0) at eval.c:4897
#11 0x0000b01c in rb_eval (self=6609412, n=0x643784) at eval.c:3185
#12 0x0000b0ec in rb_eval (self=6609412, n=0x6435b8) at eval.c:3196
#13 0x0000b38c in rb_eval (self=6609412, n=0x643568) at eval.c:3245
#14 0x00009b78 in rb_eval (self=6609412, n=0x64389c) at eval.c:2851
#15 0x00013fc4 in rb_call0 (klass=6437232, recv=6609412, id=10025,  
oid=10025, argc=0, argv=0x0, body=0x64389c, flags=0) at eval.c:5826
#16 0x000151b4 in rb_call (klass=6437232, recv=6609412, mid=10025,  
argc=0, argv=0x0, scope=0) at eval.c:5920
#17 0x0000be80 in rb_eval (self=3361128, n=0x32524c) at eval.c:3381
#18 0x0000aa94 in rb_eval (self=3361128, n=0x3250a8) at eval.c:3109
#19 0x00010c50 in rb_yield_0 (val=36816053, self=3361128, klass=0,  
flags=0, avalue=0) at eval.c:4897
#20 0x00011174 in rb_yield (val=36816053) at eval.c:4979
#21 0x00078cd0 in rb_hash_delete (hash=6311772, key=36816053) at  
hash.c:671
#22 0x00667900 in rb_dlmem_delete (ptr=0x118e25a) at ptr.c:30
#23 0x00667a94 in dlptr_free (data=0x118e210) at ptr.c:61
#24 0x0002a3f4 in obj_free (obj=3192568) at gc.c:1196
#25 0x00029b24 in gc_sweep () at gc.c:1072
#26 0x0002aa08 in garbage_collect () at gc.c:1411
#27 0x000277e0 in ruby_xmalloc (size=64) at gc.c:119
#28 0x0003218c in ary_new (klass=3337608, len=16) at array.c:127
#29 0x000321f4 in rb_ary_new2 (len=16) at array.c:137
#30 0x0003222c in rb_ary_new () at array.c:144
#31 0x0000d6d0 in rb_eval (self=3361128, n=0x325738) at eval.c:3711
#32 0x0000cfb4 in rb_eval (self=3361128, n=0x3255bc) at eval.c:3551
#33 0x0000a72c in rb_eval (self=3361128, n=0x3254b8) at eval.c:3068
#34 0x00013fc4 in rb_call0 (klass=3366168, recv=3361128, id=9969,  
oid=9969, argc=0, argv=0x0, body=0x3254b8, flags=2) at eval.c:5826
#35 0x000151b4 in rb_call (klass=3366168, recv=3361128, mid=9969,  
argc=0, argv=0x0, scope=1) at eval.c:5920
#36 0x0000c160 in rb_eval (self=3361128, n=0x324edc) at eval.c:3396
#37 0x0000aa94 in rb_eval (self=3361128, n=0x32583c) at eval.c:3109
#38 0x0000555c in eval_node (self=3361128, node=0x32583c) at eval.c:1368
#39 0x00005e58 in ruby_exec_internal () at eval.c:1543
#40 0x00005ee4 in ruby_exec () at eval.c:1563
#41 0x00005f3c in ruby_run () at eval.c:1573
#42 0x000028e0 in main (argc=4, argv=0xbfffeee4, envp=0xbfffeef8) at  
main.c:46

   I'm a total noob at the ruby internals, but the part that scares  
me are frames #20 through #23.

   It looks like ruby is doing a GC while in my 'each_block' method.   
As part of this, it is deleting some dl-related pointer (for the  
SQLite3 native driver, presumably).

   In frame #22, ptr.c we have:

static void
rb_dlmem_delete(void *ptr)
{
   rb_secure(4);
   rb_hash_delete(DLMemoryTable, DLLONG2NUM(ptr));
}

   and then in frame #21, hash.c we have:

VALUE
rb_hash_delete(hash, key)
     VALUE hash, key;
{
	...
     if (rb_block_given_p()) {
	return rb_yield(key);
     }
	...
}

   To my uneducated eye, it appears that the GC is invoking one of my  
blocks on a dl pointer int thingy when really the block is maybe only  
supposed to be invoked when user code is deleting items from a Hash.

   If I had to further speculate, I'd kind of think that maybe  
rb_dlmem_delete() should be doing something to prevent this (pushing  
some frame marker w/o a block, marking the table to not execute  
blocks, who knows what).

   Any suggestions for fixes welcome; I'll try to keep poking at  
this, but most of my poking will probably be flailing :)

-tim