From: Jason DiCioccio <jd@...>
Date: 2003-11-18T23:11:04+09:00
Subject: Re: Long-running daemon acquiring giant memory footprint

--On Tuesday, November 18, 2003 6:17 PM +0900 Robert Klemme 
<bob.news@gmx.net> wrote:

>
> "Jason DiCioccio" <jd@ods.org> schrieb im Newsbeitrag
> news:2147483647.1069109940@[10.0.2.2]...
>> Greetings,
>>
>> --On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme
>> <bob.news@gmx.net> wrote:
>>
>> >
>> > "Luke A. Kanies" <luke@madstop.com> schrieb im Newsbeitrag
>> > news:Pine.GSO.4.51.0311171048090.2656@pixie...
>> >> On Mon, 17 Nov 2003, Jason DiCioccio wrote:
>> >>
>> >> > Luke,
>> >> >   After a while of debugging I found a bunch of objects that were
>> > being
>> >> > created that apparently contain one of the primary keys in one of
> my
>> >> > database tables.  The thing is is the query should only return one
>> > result.
>> >> > This particular row is hardly ever referenced either.  So now I
> have
>> > found
>> >> > a line in my code that is something like you might have been
> referring
>> > to:
>> >> >
>> >> >       nsEntryId = nsEntryId[0][0]
>> >> >
>> >> >   That is called quite often, would that cause the object to stay
>> > around?
>> >
>> > Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.
>>
>> I've tried removing this line and it does indeed appear to be the
> problem.
>> There was no reference from nsEntryId[0][0] back to nsEntryId.  Is this
> a
>> bug?  I don't think it had this problem in ruby 1.6.x, so it appears
>> something changed in the handling of arrays between 1.6 and 1.8.
>
> Hm.  Since I don't know the data at hand, I'm not in a position to comment
> more specific.  I do wonder however, how this assignment should be
> responsible for growth of mem consumption if nsEntryId[0][0] does not
> backreference nsEntryId.
>
> I'd do a to_yaml of nsEntryId before and after the assignment to be able
> to analyse the object graph.  Maybe there's a backreference you
> overlooked.
>
> The other option seems to be a bug in the garbage collector but I'm not
> very inclined in believing this.
>
> If there is an extension involved that could hold on to the memory in ways
> not easy to determine.  That's another option that comes to my mind.
>
> Kind regards
>
>     robert
>
>

Robert,
  The object in question is just data from ruby dbi object.  You could be 
right about the back reference, however, the back reference was definitely 
not something that I am aware is there.  It could be a pointer that ruby 
DBI puts into all of its objects?

Here's a code snippet to illustrate what it contains:

nsEntryId, = Thread.current[:sql].query( "SELECT NSEntryId FROM NSEntry 
WHERE LOWER(Host) = LOWER(?) AND DomainId=?::bigint AND OwnerId=?::bigint 
AND UserId=?::bigint #{type} #{ target if target }", domain[0], 
Domain::id(domain[1]), Domain::ownerId(domain[1]), @aUser.id )

The Sql#query method is below as well:

  def query( sqlRequest, *vars )
    @conn.prepare( sqlRequest ) do |sqlVar|
      begin
        sqlVar.execute( *vars )
      rescue => detail
        if detail.message =~ /server closed the connection unexpectedly/
          log("Connection to DB unexpectedly closed")
          $sqlPool.reopen!( @conn )
        end

        Thread.current.raise NoncritError, "An error occurred: #{detail}, 
\"#{sqlRequest}\""
      end

      case sqlRequest.word( 0 )
        when /^INSERT/ then return
        when /^UPDATE/ then return
        when /^DELETE/ then return
      end

      result = []
      rcnt = 0
      while (row = sqlVar.fetch) != nil
        result.push row.dup
        rcnt = rcnt.next
      end

      return [nil, rcnt] if result.length == 0
      return [result, rcnt]
    end
  end

Any ideas where this reference might be coming from if it exists?

Regards,
-JD-