From: Daniel Berger Date: 2008-09-16T21:47:48+09:00 Subject: Optimization help - reading out of /proc on Solaris Hi all, Ruby 1.8.6 Solaris 10 I recently converted a C extension to get process table information on Solaris into a pure Ruby. I knew it would be slower, I just didn't realize how _much_ slower it would be. I was expecting the pure Ruby version to be about 1/10th as fast. Instead, it's about 1/70th as fast. Anticipating the, "Is it fast enough?" question, my answer is, "I'm not sure". Besides, tuning can be fun. :) Anyway, below is the code. I ran it through the profiler, but the top two most costly ops were Dir.foreach, which I don't see any way to optimize*, and the loop that gathers environment information, which I again see no way to optimize. # sunos.rb # # A pure Ruby version of sys-proctable for SunOS 5.8 or later #-- # Directories under /proc on Solaris 2.8+ # The Sys module serves as a namespace only. module Sys # The ProcTable class encapsulates process table information. class ProcTable # The version of the sys-proctable library VERSION = '0.8.0' private PRNODEV = -1 # non-existent device FIELDS = [ :flag, # process flags (deprecated) :nlwp, # number of active lwp's in the process :pid, # unique process id :ppid, # process id of parent :pgid, # pid of session leader :sid, # session id :uid, # real user id :euid, # effective user id :gid, # real group id :egid, # effective group id :addr, # address of the process :size, # size of process in kbytes :rssize, # resident set size in kbytes :ttydev, # tty device (or PRNODEV) :pctcpu, # % of recent cpu used by all lwp's :pctmem, # % of system memory used by process :start, # absolute process start time :time, # usr + sys cpu time for this process :ctime, # usr + sys cpu time for reaped children :fname, # name of the exec'd file :psargs, # initial characters argument list :wstat, # if a zombie, the wait status :argc, # initial argument count :argv, # address of initial argument vector :envp, # address of initial environment vector :dmodel, # data model of the process :taskid, # task id :projid, # project id :nzomb, # number of zombie lwp's in the process :poolid, # pool id :zoneid, # zone id :contract, # process contract :lwpid, # lwp id :wchan, # wait address for sleeping lwp :stype, # synchronization event type :state, # numeric lwp state :sname, # printable character for state :nice, # nice for cpu usage :syscall, # system call number (if in syscall) :pri, # priority :clname, # scheduling class name :name, # name of system lwp :onpro, # processor which last ran thsi lwp :bindpro, # processor to which lwp is bound :bindpset, # processor set to which lwp is bound :count, # number of contributing lwp's :tstamp, # current time stamp :create, # process/lwp creation time stamp :term, # process/lwp termination time stamp :rtime, # total lwp real (elapsed) time :utime, # user level cpu time :stime, # system call cpu time :ttime, # other system trap cpu time :tftime, # text page fault sleep time :dftime, # text page fault sleep time :kftime, # kernel page fault sleep time :ltime, # user lock wait sleep time :slptime, # all other sleep time :wtime, # wait-cpu (latency) time :stoptime, # stopped time :minf, # minor page faults :majf, # major page faults :nswap, # swaps :inblk, # input blocks :oublk, # output blocks :msnd, # messages sent :mrcv, # messages received :sigs, # signals received :vctx, # voluntary context switches :ictx, # involuntary context switches :sysc, # system calls :ioch, # chars read and written :path, # array of symbolic link paths from /proc// pid :contracts, # array symbolic link paths from /proc// contracts :fd, # array of used file descriptors :cmd_args, # array of command line arguments :environ # hash of environment associated with the process ] public ProcTableStruct = Struct.new("ProcTableStruct", *FIELDS) # In block form, yields a ProcTableStruct for each process entry that you # have rights to. This method returns an array of ProcTableStruct's in # non-block form. # # If a +pid+ is provided, then only a single ProcTableStruct is yielded or # returned, or nil if no process information is found for that +pid+. # # Example: # # # Iterate over all processes # ProcTable.ps do |proc_info| # p proc_info # end # # # Print process table information for only pid 1001 # p ProcTable.ps(1001) # def self.ps(pid = nil) array = block_given? ? nil : [] Dir.foreach("/proc") do |file| next if file =~ /\D/ # Skip non-numeric entries under / proc # Only return information for a given pid, if provided if pid next unless file.to_i == pid end # Skip over any entries we don't have permissions to read begin psinfo = IO.read("/proc/#{file}/psinfo") rescue StandardError, Errno::EACCES next end struct = ProcTableStruct.new struct.flag = psinfo[0,4].unpack("i")[0] # pr_flag struct.nlwp = psinfo[4,4].unpack("i")[0] # pr_nlwp struct.pid = psinfo[8,4].unpack("i")[0] # pr_pid struct.ppid = psinfo[12,4].unpack("i")[0] # pr_ppid struct.pgid = psinfo[16,4].unpack("i")[0] # pr_pgid struct.sid = psinfo[20,4].unpack("i")[0] # pr_sid struct.uid = psinfo[24,4].unpack("i")[0] # pr_uid struct.euid = psinfo[28,4].unpack("i")[0] # pr_euid struct.gid = psinfo[32,4].unpack("i")[0] # pr_gid struct.egid = psinfo[36,4].unpack("i")[0] # pr_egid struct.addr = psinfo[40,4].unpack("L")[0] # pr_addr struct.size = psinfo[44,4].unpack("L")[0] * 1024 # pr_size struct.rssize = psinfo[48,4].unpack("L")[0] * 1024 # pr_rssize # skip pr_pad1 # TODO: Convert this to a human readable string somehow struct.ttydev = psinfo[56,4].unpack("i")[0] # pr_ttydev # pr_pctcpu struct.pctcpu = (psinfo[60,2].unpack("S")[0] * 100).to_f / 0x8000 # pr_pctmem struct.pctmem = (psinfo[62,2].unpack("S")[0] * 100).to_f / 0x8000 struct.start = Time.at(psinfo[64,8].unpack("L")[0]) # pr_start struct.time = psinfo[72,8].unpack("L")[0] # pr_time struct.ctime = psinfo[80,8].unpack("L")[0] # pr_ctime struct.fname = psinfo[88,16].strip # pr_fname struct.psargs = psinfo[104,80].strip # pr_psargs struct.wstat = psinfo[184,4].unpack("i")[0] # pr_wstat struct.argc = psinfo[188,4].unpack("i")[0] # pr_argc struct.argv = psinfo[192,4].unpack("L")[0] # pr_argv struct.envp = psinfo[196,4].unpack("L")[0] # pr_envp struct.dmodel = psinfo[200,1].unpack("C")[0] # pr_dmodel # skip pr_pad2 struct.taskid = psinfo[204,4].unpack("i")[0] # pr_taskid struct.projid = psinfo[208,4].unpack("i")[0] # pr_projectid struct.nzomb = psinfo[212,4].unpack("i")[0] # pr_nzomb struct.poolid = psinfo[216,4].unpack("i")[0] # pr_poolid struct.zoneid = psinfo[220,4].unpack("i")[0] # pr_zoneid struct.contract = psinfo[224,4].unpack("i")[0] # pr_contract # skip pr_filler ### lwpsinfo struct info # skip pr_flag struct.lwpid = psinfo[236,4].unpack("i")[0] # pr_lwpid # skip pr_addr struct.wchan = psinfo[244,4].unpack("L")[0] # pr_wchan struct.stype = psinfo[248,1].unpack("C")[0] # pr_stype struct.state = psinfo[249,1].unpack("C")[0] # pr_state struct.sname = psinfo[250,1] # pr_sname struct.nice = psinfo[251,1].unpack("C")[0] # pr_nice struct.syscall = psinfo[252,2].unpack("S")[0] # pr_syscall # skip pr_oldpri # skip pr_cpu struct.pri = psinfo[256,4].unpack("i")[0] # pr_syscall # skip pr_pctcpu # skip pr_pad # skip pr_start # skip pr_time struct.clname = psinfo[280,8].strip # pr_clname struct.name = psinfo[288,16].strip # pr_name struct.onpro = psinfo[304,4].unpack("i")[0] # pr_onpro struct.bindpro = psinfo[308,4].unpack("i")[0] # pr_bindpro struct.bindpset = psinfo[308,4].unpack("i")[0] # pr_bindpset # Get the full command line out of /proc//as. begin fd = File.open("/proc/#{file}/as") fd.sysseek(struct.argv, IO::SEEK_SET) address = fd.sysread(struct.argc * 4).unpack("L")[0] struct.cmd_args = [] 0.upto(struct.argc - 1){ |i| fd.sysseek(address, IO::SEEK_SET) data = fd.sysread(128)[/^[^\0]*/] # Null strip struct.cmd_args << data address += data.length + 1 # Add 1 for the space } # Get the environment hash associated with the process. struct.environ = {} fd.sysseek(struct.envp, IO::SEEK_SET) env_address = fd.sysread(128).unpack("L")[0] loop do fd.sysseek(env_address, IO::SEEK_SET) data = fd.sysread(1024)[/^[^\0]*/] # Null strip break if data.empty? key, value = data.split('=') struct.environ[key] = value env_address += data.length + 1 # Add 1 for the space end rescue Errno::EACCES, Errno::EOVERFLOW, EOFError # Skip this if we don't have proper permissions, if there's # no associated environment, or if there's a largefile issue. ensure fd.close if fd end ### struct prusage begin prusage = 0.chr * 512 prusage = IO.read("/proc/#{file}/usage") # skip pr_lwpid struct.count = prusage[4,4].unpack("i")[0] # pr_count struct.tstamp = prusage[8,8].unpack("L")[0] # pr_tstamp struct.create = prusage[16,8].unpack("L")[0] # pr_create struct.term = prusage[24,8].unpack("L")[0] # pr_term struct.rtime = prusage[32,8].unpack("L")[0] # pr_rtime struct.utime = prusage[40,8].unpack("L")[0] # pr_utime struct.stime = prusage[48,8].unpack("L")[0] # pr_stime struct.ttime = prusage[56,8].unpack("L")[0] # pr_ttime struct.tftime = prusage[64,8].unpack("L")[0] # pr_tftime struct.dftime = prusage[72,8].unpack("L")[0] # pr_dftime struct.kftime = prusage[80,8].unpack("L")[0] # pr_kftime struct.ltime = prusage[88,8].unpack("L")[0] # pr_ltime struct.slptime = prusage[96,8].unpack("L")[0] # pr_slptime struct.wtime = prusage[104,8].unpack("L")[0] # pr_wtime struct.stoptime = prusage[112,8].unpack("L")[0] # pr_stoptime struct.minf = prusage[120,4].unpack("L")[0] # pr_minf struct.majf = prusage[124,4].unpack("L")[0] # pr_majf struct.nswap = prusage[128,4].unpack("L")[0] # pr_nswap struct.inblk = prusage[128,4].unpack("L")[0] # pr_inblk struct.oublk = prusage[128,4].unpack("L")[0] # pr_oublk struct.msnd = prusage[128,4].unpack("L")[0] # pr_msnd struct.mrcv = prusage[128,4].unpack("L")[0] # pr_mrcv struct.sigs = prusage[128,4].unpack("L")[0] # pr_sigs struct.vctx = prusage[128,4].unpack("L")[0] # pr_vctx struct.ictx = prusage[128,4].unpack("L")[0] # pr_ictx struct.sysc = prusage[128,4].unpack("L")[0] # pr_sysc struct.ioch = prusage[128,4].unpack("L")[0] # pr_ioch rescue Errno::EACCES # Do nothing if we lack permissions. Just move on. end # Information from /proc//path. This is represented as a hash, # with the symbolic link name as the key, and the file it links to # as the value, or nil if it cannot be found. #-- # Note that cwd information can be gathered from here, too. struct.path = {} Dir["/proc/#{file}/path/*"].each{ |entry| link = File.readlink(entry) rescue nil struct.path[File.basename(entry)] = link } # Information from /proc//contracts. This is represented as # a hash, with the symbolic link name as the key, and the file # it links to as the value. struct.contracts = {} Dir["/proc/#{file}/contracts/*"].each{ |entry| link = File.readlink(entry) rescue nil struct.contracts[File.basename(entry)] = link } # Information from /proc//fd. This returns an array of # numeric file descriptors used by the process. struct.fd = Dir["/proc/#{file}/fd/*"].map{ |f| File.basename(f).to_i } if block_given? yield struct else array << struct end end pid ? array[0] : array end end end Thanks, Dan * I tried tossing threads at it in a one-thread-per-directory approach, but they didn't get along with IO.read in MRI, and seemed to provide no real speed benefit with JRuby.