From: ara.t.howard@... Date: 2006-03-18T03:36:12+09:00 Subject: Re: Writing long-running daemons without memory leaks? On Sat, 18 Mar 2006, Toby DiPasquale wrote: > unknown wrote: >> harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep > [...] >> ahoward 13671 18.3 19.5 212736 200952 pts/26 R 10:14 1:12 ruby >> a.rb >> ahoward 13671 18.4 21.1 230824 216768 pts/26 R 10:14 1:12 ruby >> a.rb >> ahoward 13671 18.5 22.3 230816 229404 pts/26 S 10:14 1:12 ruby >> a.rb >> >> >> so memory builds to about 25 and, presumably when the gc kicks in, drops >> to 15 >> - but it certainly stays in this range. >> >> what os and ruby version? > > ruby 1.8.4 on Ubuntu Linux, kernel 2.6.12-10 on a P4 Thinkpad with 1GB > RAM. The other systems having this problem are all also running Linux > 2.6.x with ruby 1.8.4. > > One thing about the above: you do realize that's ~250MB and ~150MB, not > 25 and 15, right? yes - but it's definitely not leaking. i'm running harp:~ > ruby -v ruby 1.8.4 (2006-01-12) [i686-linux] harp:~ > cat /etc/redhat-release Red Hat Enterprise Linux WS release 3 (Taroon Update 6) harp:~ > uname -srm Linux 2.4.21-37.0.1.EL i686 btw. it's clear to me that your code is using tons of memory - it's not clear that it's leaking, however. if you looking for why your code is so big look to you handler: changing it to Thread.start(s){|s| loop{ s.read(8192) or break }} will result in runs that look like this: harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep 'ruby a.rb'|grep -v grep ;done USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ahoward 15535 4.1 0.4 6004 4556 pts/26 S 10:53 0:00 ruby a.rb ahoward 15535 7.1 0.7 8940 7520 pts/26 S 10:53 0:00 ruby a.rb ahoward 15535 9.4 0.2 4080 2704 pts/26 S 10:53 0:00 ruby a.rb ahoward 15535 10.7 0.5 6820 5432 pts/26 S 10:53 0:01 ruby a.rb ahoward 15535 13.0 0.7 9676 8212 pts/26 S 10:53 0:01 ruby a.rb ahoward 15535 13.2 0.3 4552 3176 pts/26 S 10:53 0:01 ruby a.rb ahoward 15535 15.0 0.5 7156 5712 pts/26 S 10:53 0:01 ruby a.rb ahoward 15535 15.9 0.8 9684 8300 pts/26 S 10:53 0:01 ruby a.rb ahoward 15535 17.1 0.2 4432 3056 pts/26 S 10:53 0:02 ruby a.rb ahoward 15535 19.2 0.5 6892 5392 pts/26 S 10:53 0:02 ruby a.rb ahoward 15535 19.5 0.7 9228 7796 pts/26 S 10:53 0:02 ruby a.rb ahoward 15535 21.1 0.2 3712 2332 pts/26 S 10:53 0:02 ruby a.rb ahoward 15535 21.6 0.4 5908 4484 pts/26 S 10:53 0:03 ruby a.rb so it's the handler/Buffer class that consumes (but does't leak) so much memory. in any case, i think you are making the code too hard : ruby already does massive internal buffering - you can do what you want with less code i think: harp:~ > cat netstring.rb class NetString < ::String def initialize arg = "" arg.respond_to?("read") ? read(arg) : super(arg) end def clear self.replace "" end def read port clear self.replace(port.read(port.read(4).unpack("N").first)) end def write port port << [size].pack("N") port << self end end harp:~ > cat a.rb require 'socket' require 'netstring' STDOUT.sync = true ss = TCPServer.new '127.0.0.1', 10001 loop do s = ss.accept puts "got connection" Thread.start(s) do |s| File.open("/dev/null", "w") do |f| ns = NetString.new s puts "received #{ ns.size } bytes" written = f.write ns puts "wrote #{ written } bytes" end end end harp:~ > cat b.rb require 'socket' require 'netstring' loop do TCPSocket.open('127.0.0.1', 10001) do |t| NetString::new("a" * 65536).write t end end harp:~ > ruby a.rb # terminal one harp:~ > ruby b.rb # terminal two harp:~ > ps aux|head -1;{ while true;do ps aux|grep 'ruby a.rb'|grep -v grep;sleep 1;done; }|head -42 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ahoward 17990 10.7 1.1 13480 11868 pts/26 S 11:30 0:27 ruby a.rb ahoward 17990 10.7 1.1 13164 11704 pts/26 S 11:30 0:27 ruby a.rb ahoward 17990 10.7 1.1 13660 12144 pts/26 S 11:30 0:27 ruby a.rb ahoward 17990 10.7 1.1 13424 11904 pts/26 S 11:30 0:27 ruby a.rb ahoward 17990 10.7 1.1 13328 11792 pts/26 S 11:30 0:27 ruby a.rb ahoward 17990 10.7 1.1 13784 12184 pts/26 S 11:30 0:27 ruby a.rb ahoward 17990 10.7 0.9 11040 9496 pts/26 S 11:30 0:27 ruby a.rb ahoward 17990 10.8 0.9 10904 9428 pts/26 S 11:30 0:28 ruby a.rb ahoward 17990 10.7 0.9 11968 10220 pts/26 S 11:30 0:28 ruby a.rb ahoward 17990 10.7 1.1 13464 12056 pts/26 S 11:30 0:28 ruby a.rb ahoward 17990 10.7 0.9 11000 9476 pts/26 S 11:30 0:28 ruby a.rb ahoward 17990 10.7 0.9 10992 9472 pts/26 S 11:30 0:28 ruby a.rb ahoward 17990 10.8 1.1 13364 11912 pts/26 S 11:30 0:28 ruby a.rb ahoward 17990 10.8 0.9 10976 9464 pts/26 S 11:30 0:28 ruby a.rb ahoward 17990 10.8 1.1 13356 11908 pts/26 S 11:30 0:28 ruby a.rb ahoward 17990 10.8 1.1 13796 12228 pts/26 S 11:30 0:29 ruby a.rb ahoward 17990 10.8 1.1 13572 12116 pts/26 S 11:30 0:29 ruby a.rb ahoward 17990 10.8 1.1 13836 12244 pts/26 S 11:30 0:29 ruby a.rb ahoward 17990 10.8 1.1 13872 12284 pts/26 S 11:30 0:29 ruby a.rb ahoward 17990 10.8 1.2 14112 12436 pts/26 S 11:30 0:29 ruby a.rb ahoward 17990 10.8 1.2 14224 12428 pts/26 S 11:30 0:29 ruby a.rb ahoward 17990 10.8 1.1 13888 12292 pts/26 S 11:30 0:29 ruby a.rb ahoward 17990 10.8 1.1 13896 12296 pts/26 S 11:30 0:29 ruby a.rb ahoward 17990 10.8 1.2 14200 12416 pts/26 S 11:30 0:30 ruby a.rb ahoward 17990 10.8 1.1 13856 12276 pts/26 S 11:30 0:30 ruby a.rb ahoward 17990 10.8 1.2 13960 12328 pts/26 S 11:30 0:30 ruby a.rb ahoward 17990 10.8 1.1 13624 12164 pts/26 S 11:30 0:30 ruby a.rb ahoward 17990 10.8 1.2 14048 12384 pts/26 S 11:30 0:30 ruby a.rb ahoward 17990 10.8 1.1 13984 12308 pts/26 S 11:30 0:30 ruby a.rb ahoward 17990 10.8 1.2 14088 12424 pts/26 S 11:30 0:30 ruby a.rb ahoward 17990 10.8 1.1 13744 12220 pts/26 S 11:30 0:30 ruby a.rb ahoward 17990 10.8 1.2 14040 12396 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.1 13496 12100 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.2 14056 12472 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.2 14080 12452 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.1 13520 12108 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.1 13656 12276 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.1 13624 12160 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.2 13824 12392 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.2 13912 12432 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.2 14152 12520 pts/26 S 11:30 0:31 ruby a.rb ahoward 17990 10.8 1.1 13768 12232 pts/26 S 11:30 0:32 ruby a.rb kind regards. -a -- share your knowledge. it's a way to achieve immortality. - h.h. the 14th dali lama