[ruby-core:20783] Re: [PATCH] Promising C coding techniques to reduce MRI's memory use

Roger,

I just updated the patches at:

http://sites.google.com/site/brentsrubypatches

to fix the bug that was causing the drb test suite to segfault.

All the test suites now run to completion.

Responses to your questions:

R:  Is this the time to complete test-all?,  What patches for about 1.9?, 
Why slower?

B:  This is the time to complete the command:
        ruby runner.rb
     in the test subdir of the 1.8.7p72 directory.

I suspect that the unpatched interpreter is leaking throughout the execution
of the tests.
Process size just keep increasing.  With these patches is stabilizes about
1/3 the way through.
These techniques may work with v1.9 as my understanding is that the GC is
largely unchanged.

Apps that don't swap context much will be a few percent slower.  Those that
do should be faster.  There certainly is more that can be down to optimize
the stack clearing.  My initial goal was to plug the memory leaks so that
Ruby apps could run for long periods without swapping (or worse).  In
practice, once a Ruby process starts swapping to virtual memory, its
performance degrades much more than a few percent.


R:
> The patched version reports one additional failure:
>  2) Failure:
> test_should_propagate_signaled(TestBeginEndBlock)
> [./ruby/test_beginendblock.rb:81]:
> <""> expected to be =~
> </Interrupt$/>.

   Does it report this consistently?

B:  Funny you should ask that...
No, it does not fail consistently.   Any ideas what's happening here?
It does feel like the same problem you see with or mingw port.


R:  The install instructions mention using
-mpreferred-stack-boundary=2, though in the writeup it says it helps
only slightly--but you recommend it because it still helps?

B:  Yes, stack-boundary=2 helps keep the frames a little smaller.
For a multi-threaded app, this is probably worth the little performance hit.
For a single threaded app, it may be better to leave out the
-mpreferred-stack-boundary=2
We need more benchmarking to tell.
Ruby should no longer leak memory regardless.


R:  re:  MBARI2: gc sometimes segfaults: do you have any examples of how it
does this?  So these old frames are collected but not really--is that
what happens?

B:  Have a look at this post of mine dated 12/03/07
http://markmail.org/message/jjmqzsxenp7oaojm


R:  re:  MBARI3 is it possible to use memzero to forcefully overwrite local
variables [though as you pointed out, it would still leave
temporaries].  Are there any other culprits besides rb_eval [and
doesn't eval get called fairly rarely so this isn't a help for most
progs?]

B:  
I suspect memzero would be slower than the tight loop I have zeroing the
stack now.
In any case, the temporaries are critically important.  rb_eval is the 800
pound gorrilla :-)


R:
You mention that after this the callcc stuff should work--do you think
that only applying this one patch should be sufficient for that to
happen?

B:
I think so.  However, I'd recommend installing at least MBARI2 as well to
improve performance.


R:
why remove the dynamic malloc_limit?

B:
Because I believe the malloc_limit should be tuned for your target
environment.
In a target with 32MB DRAM, malloc_limit should not be 8MB and I certainly
don't want it to increase on its own.  Remember, once Ruby starts swapping,
performance goes into the toilet.
I probably won't be motivated enough to benchmark it.  A few percent run
time change does not matter much to me.  I want my app to run for months at
a time and to play nice with others.


R:
MBARI5 : ruby extends the stack when it needs to thread shift from a
"smaller stack" thread to a larger stack thread, is that right?  After
shifting to a smaller stack might be a good time to clean the stack...

B:
The MBARI3 patch updates the stack extent at a number of points, including
on every context switch, but it defers clearing it until the next
CHECKINTS(), when the stack is likely to be smaller still.  Even so,
optimizing this further is definitely possible.  I've considered only
clearing the stack after GC.increase rises to 75% of GC.limit, for instance.


R:
re: MBARI6 question: why are these included with 5 other gc patches?
[besides that they're cool and useful]?  Might be convenient to just
include the 1.9 style syntax by default [I might could come up with a
patch for it].:)

B:
MBARI6 probably should have been packaged separately.
My __line__ and __file__ patches predate the 1.9 stuff by about 5 years. 
See:
http://markmail.org/message/ybrbhvvzlhyv552y
I did think of redoing them in the 1.9 style, but I don't particularly like
the idea
of returning an array in this context, where numbered indices replace named
attributes.
In any case, I can emulate the 1.9 style methods with a tiny bit of Ruby
glue.


R:
I suppose my only wish list for these would be that it didn't clear
the stack but once per thread per GC.  I might could help out sometime
with it.

B:
That's on my wish list too.  I'd be very grateful for any help, even just
discussing ideas.

- brent



-- 
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p21125317.html
Sent from the ruby-core mailing list archive at Nabble.com.