[ruby-core:20784] Re: [PATCH] Promising C coding techniques to reduce MRI's memory use

Hey Brent,

Thanks for patches man. I am yet to dig deeper, but I benchmarked
rails against them:

Here is the Average request/response for patched version:

Requests per second:    234.77 [#/sec] (mean)
Time per request:       42.594 [ms] (mean)
Time per request:       4.259 [ms] (mean, across all concurrent requests)
Transfer rate:          108.82 [Kbytes/sec] received

Memory usage stayed around 30MB

For Stock Ruby version:

Requests per second:    138.48 [#/sec] (mean)
Time per request:       72.214 [ms] (mean)
Time per request:       7.221 [ms] (mean, across all concurrent requests)
Transfer rate:          64.21 [Kbytes/sec] received


Memory usage stayed around 53 MB

I compiled both ruby versions without "--disable-pthread" and was
wondering if your patches modify anything there.



On Mon, Dec 22, 2008 at 3:29 PM, Brent Roman <[e-mail]> wrote:
>
> Roger,
>
> I just updated the patches at:
>
> http://sites.google.com/site/brentsrubypatches
>
> to fix the bug that was causing the drb test suite to segfault.
>
> All the test suites now run to completion.
>
> Responses to your questions:
>
> R:  Is this the time to complete test-all?,  What patches for about 1.9?,
> Why slower?
>
> B:  This is the time to complete the command:
>        ruby runner.rb
>     in the test subdir of the 1.8.7p72 directory.
>
> I suspect that the unpatched interpreter is leaking throughout the execution
> of the tests.
> Process size just keep increasing.  With these patches is stabilizes about
> 1/3 the way through.
> These techniques may work with v1.9 as my understanding is that the GC is
> largely unchanged.
>
> Apps that don't swap context much will be a few percent slower.  Those that
> do should be faster.  There certainly is more that can be down to optimize
> the stack clearing.  My initial goal was to plug the memory leaks so that
> Ruby apps could run for long periods without swapping (or worse).  In
> practice, once a Ruby process starts swapping to virtual memory, its
> performance degrades much more than a few percent.
>
>
> R:
>> The patched version reports one additional failure:
>>  2) Failure:
>> test_should_propagate_signaled(TestBeginEndBlock)
>> [./ruby/test_beginendblock.rb:81]:
>> <""> expected to be =~
>> </Interrupt$/>.
>
>   Does it report this consistently?
>
> B:  Funny you should ask that...
> No, it does not fail consistently.   Any ideas what's happening here?
> It does feel like the same problem you see with or mingw port.
>
>
> R:  The install instructions mention using
> -mpreferred-stack-boundary=2, though in the writeup it says it helps
> only slightly--but you recommend it because it still helps?
>
> B:  Yes, stack-boundary=2 helps keep the frames a little smaller.
> For a multi-threaded app, this is probably worth the little performance hit.
> For a single threaded app, it may be better to leave out the
> -mpreferred-stack-boundary=2
> We need more benchmarking to tell.
> Ruby should no longer leak memory regardless.
>
>
> R:  re:  MBARI2: gc sometimes segfaults: do you have any examples of how it
> does this?  So these old frames are collected but not really--is that
> what happens?
>
> B:  Have a look at this post of mine dated 12/03/07
> http://markmail.org/message/jjmqzsxenp7oaojm
>
>
> R:  re:  MBARI3 is it possible to use memzero to forcefully overwrite local
> variables [though as you pointed out, it would still leave
> temporaries].  Are there any other culprits besides rb_eval [and
> doesn't eval get called fairly rarely so this isn't a help for most
> progs?]
>
> B:
> I suspect memzero would be slower than the tight loop I have zeroing the
> stack now.
> In any case, the temporaries are critically important.  rb_eval is the 800
> pound gorrilla :-)
>
>
> R:
> You mention that after this the callcc stuff should work--do you think
> that only applying this one patch should be sufficient for that to
> happen?
>
> B:
> I think so.  However, I'd recommend installing at least MBARI2 as well to
> improve performance.
>
>
> R:
> why remove the dynamic malloc_limit?
>
> B:
> Because I believe the malloc_limit should be tuned for your target
> environment.
> In a target with 32MB DRAM, malloc_limit should not be 8MB and I certainly
> don't want it to increase on its own.  Remember, once Ruby starts swapping,
> performance goes into the toilet.
> I probably won't be motivated enough to benchmark it.  A few percent run
> time change does not matter much to me.  I want my app to run for months at
> a time and to play nice with others.
>
>
> R:
> MBARI5 : ruby extends the stack when it needs to thread shift from a
> "smaller stack" thread to a larger stack thread, is that right?  After
> shifting to a smaller stack might be a good time to clean the stack...
>
> B:
> The MBARI3 patch updates the stack extent at a number of points, including
> on every context switch, but it defers clearing it until the next
> CHECKINTS(), when the stack is likely to be smaller still.  Even so,
> optimizing this further is definitely possible.  I've considered only
> clearing the stack after GC.increase rises to 75% of GC.limit, for instance.
>
>
> R:
> re: MBARI6 question: why are these included with 5 other gc patches?
> [besides that they're cool and useful]?  Might be convenient to just
> include the 1.9 style syntax by default [I might could come up with a
> patch for it].:)
>
> B:
> MBARI6 probably should have been packaged separately.
> My __line__ and __file__ patches predate the 1.9 stuff by about 5 years.
> See:
> http://markmail.org/message/ybrbhvvzlhyv552y
> I did think of redoing them in the 1.9 style, but I don't particularly like
> the idea
> of returning an array in this context, where numbered indices replace named
> attributes.
> In any case, I can emulate the 1.9 style methods with a tiny bit of Ruby
> glue.
>
>
> R:
> I suppose my only wish list for these would be that it didn't clear
> the stack but once per thread per GC.  I might could help out sometime
> with it.
>
> B:
> That's on my wish list too.  I'd be very grateful for any help, even just
> discussing ideas.
>
> - brent
>
>
>
> --
> View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p21125317.html
> Sent from the ruby-core mailing list archive at Nabble.com.
>
>
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org