From: James Edward Gray II <james@...>
Date: 2009-12-24T12:41:54+09:00
Subject: FFI Memory Leak

I've found a memory leak in some FFI code.  I'll give example code below, but I can explain the issue easily enough first.

When calling a function provided by Tokyo Cabinet, it returns a value in the form of a pointer and a length to read at that address.  The documentation for the function (tchdbget() at http://1978th.net/tokyocabinet/spex-en.html#tchdbapi, if you are interested) warns that the returned region was allocated with malloc() and needs to be free()d.  I can't figure out how to free the value, so there is a memory leak.  What I would like to know is how to free the value at the end of the pointer.

Here's some code showing what I just explained.

#!/usr/bin/env ruby -wKU

require "rubygems"
require "ffi"

# map the C interface
module Lib
  extend FFI::Library
  ffi_lib(
    *Array(
      ENV.fetch(
        "TOKYO_CABINET_LIB",
        Dir["/{opt,usr}/{,local/}lib{,64}/libtokyocabinet.{dylib,so*}"]
      )
    )
  )
  
  attach_function :tchdbnew,   [ ],                                  :pointer
  attach_function :tchdbopen,  [:pointer, :string, :int],            :bool
  attach_function :tchdbput,   [:pointer, :pointer, :int, :pointer,
                                :int],                               :bool
  attach_function :tchdbget,   [:pointer, :pointer, :int, :pointer], :pointer
  attach_function :tchdbclose, [:pointer],                           :bool
end

# translate the interface to Ruby
class TokyoCabinet
  def self.open(*args)
    db = new(*args)
    yield db
  ensure
    db.close if db
  end
  
  def initialize(path)
    @db = Lib.tchdbnew
    Lib.tchdbopen(@db, path, (1 << 1) | (1 << 2))  # write create mode
  end
  
  def []=(key, value)
    k, v = key.to_s, value.to_s
    Lib.tchdbput(@db, k, k.size, v, v.size)
  end
  
  def [](key)
    k     = key.to_s
    size  = FFI::MemoryPointer.new(:int)
    value = Lib.tchdbget(@db, k, k.size, size)
    value.address.zero? ? nil : value.get_bytes(0, size.get_int(0))
  ensure
    size.free if size
    # FIXME:  How do I free value here?
  end
  
  def close
    Lib.tchdbclose(@db)
  end
end

# show the problem
def show_memory
  3.times { GC.start }  # try to clean up
  mem = `ps -o rss -p #{Process.pid}`[/\d+/]
  puts "Current memory:  #{mem}"
end

TokyoCabinet.open("leak.tch") do |db|
  db[:some_key] = "X" * 1024
  10.times do
    5000.times do
      db[:some_key]  # reading causes the memory leak
    end
    show_memory
  end
end

# Sample Run:
# 
# Current memory:  30324
# Current memory:  37828
# Current memory:  45364
# Current memory:  52896
# Current memory:  60428
# Current memory:  67960
# Current memory:  75488
# Current memory:  83020
# Current memory:  90552
# Current memory:  98080

__END__

Thanks in advance for any help provided.

James Edward Gray II