[#3228] Core support for Gems, and namespace — "Luke A. Kanies" <luke@...>

Hi all,

21 messages 2004/07/27
[#3230] Re: Core support for Gems, and namespace — Austin Ziegler <halostatue@...> 2004/07/27

On Tue, 27 Jul 2004 11:39:08 +0900, Luke A. Kanies <luke@madstop.com> wrote:

[#3234] Re: Core support for Gems, and namespace — "Luke A. Kanies" <luke@...> 2004/07/27

On Tue, 27 Jul 2004, Austin Ziegler wrote:

[#3238] Re: Core support for Gems, and namespace — Austin Ziegler <halostatue@...> 2004/07/27

On Wed, 28 Jul 2004 00:14:29 +0900, Luke A. Kanies <luke@madstop.com> wrote:

cgi.rb possible memory leak

From: Kent Sibilev <ksibilev@...>
Date: 2004-07-28 07:05:07 UTC
List: ruby-core #3245
Hi all.

We are almost ready to deploy our commercial website completely written 
in Ruby and we are finishing up our testings. Our application is 
running in FastCGI environment using ruby-fcgi-0.8.4 binding.

Recently I've noticed that the memory usage of our Ruby processes goes 
very high after intensive usage and never goes down. I wrote a simple 
memory monitoring tool which uses ObjectSpace module to display the 
current heap content. I noticed that there were a lot of StringIO and 
Tempfile object hanging around and never got released by garbage 
collector. The only place where these objects can be created is inside 
the CGI class when the HTTP POST request has a multipart form. Having 
spent a couple of hours in debugger, I was able to identify the source 
of the problem. All these objects were marked by rb_mark_end_proc(), 
which was very alarming especially that 'end_procs' list was unusually 
huge.

The reason of all this was the following code snippet in cgi.rb 
initialize method:

2268:         initialize_query()  # set @params, @cookies
2269:         params  = @params.nil?  ? nil : @params.dup
2270:         cookies = @cookies.nil? ? nil : @cookies.dup
2271:         (class << self; self; end).class_eval do
2272:           const_set(:CGI_PARAMS,  params)
2273:           const_set(:CGI_COOKIES, cookies)
2274:         end
2275:         if defined?(MOD_RUBY) and (RUBY_VERSION < "1.4.3")
2276:           raise "Please, use ruby1.4.3 or later."
2277:         else
2278:           at_exit() do
2279:             if defined?(CGI_PARAMS)
2280:               CGI.class_eval do
2281:                 remove_const(:CGI_PARAMS)
2282:                 remove_const(:CGI_COOKIES)
2283:               end
2284:             end
2285:           end
2286:         end

If you look at it CGI class parses form query parameters and at the end 
calls at_exit function. As it turned out the block passed to at_exit 
function gets registered into Ruby's end_procs list and this block 
holds the references to all parsed query parameters. Ruby interpreter 
on the other hand cleans up this end_procs list only on the process 
exit. This means that all this parsed query parameters including in my 
case Tempfiles and StringIOs are sitting in memory doing nothing. And 
it happens for every HTTP request! Commenting out the at_exit call made 
the process memory bound to ~30M.

I don't know what was the reason to put this at_exit hook in there. 
Probably whoever wrote it (many thanks, BTW) assumed that CGI module 
will be used in the strictly CGI environment where process dies at the 
end of the request processing. But it is a real problem in the FastCGI 
environment!

BTW, I think I've found another small memory leak in ruby-fcgi-0.8.4 
extension library. Can somebody verify that? Here is the diff:

@@ -37,6 +37,12 @@
    rb_gc_mark(data->env);
  }

+static void fcgi_free(fcgi_data *data)
+{
+  xfree(data->req);
+  xfree(data);
+}
+
  static VALUE fcgi_s_accept(VALUE self)
  {
    int status;
@@ -64,7 +70,7 @@
      VALUE obj,key, value;
      char *pkey,*pvalue;

-    obj = Data_Make_Struct(self, fcgi_data, fcgi_mark, 0, data);
+    obj = Data_Make_Struct(self, fcgi_data, fcgi_mark, fcgi_free, 
data);
      data->req = req;
      data->in  = Data_Wrap_Struct(cFCGIStream, 0, 0, req->in);
      data->out = Data_Wrap_Struct(cFCGIStream, 0, 0, req->out);


Thanks for listening.
Cheers,
Kent.


In This Thread

Prev Next