Optimization help - reading out of /proc on Solaris

Hi all,

Ruby 1.8.6
Solaris 10

I recently converted a C extension to get process table information on
Solaris into a pure Ruby. I knew it would be slower, I just didn't
realize how _much_ slower it would be. I was expecting the pure Ruby
version to be about 1/10th as fast. Instead, it's about 1/70th as
fast. Anticipating the, "Is it fast enough?" question, my answer is,
"I'm not sure". Besides, tuning can be fun. :)

Anyway, below is the code. I ran it through the profiler, but the top
two most costly ops were Dir.foreach, which I don't see any way to
optimize*, and the loop that gathers environment information, which I
again see no way to optimize.

# sunos.rb
#
# A pure Ruby version of sys-proctable for SunOS 5.8 or later
#--
# Directories under /proc on Solaris 2.8+

# The Sys module serves as a namespace only.
module Sys

   # The ProcTable class encapsulates process table information.
   class ProcTable

      # The version of the sys-proctable library
      VERSION = '0.8.0'

      private

      PRNODEV = -1 # non-existent device

      FIELDS = [
         :flag,      # process flags (deprecated)
         :nlwp,      # number of active lwp's in the process
         :pid,       # unique process id
         :ppid,      # process id of parent
         :pgid,      # pid of session leader
         :sid,       # session id
         :uid,       # real user id
         :euid,      # effective user id
         :gid,       # real group id
         :egid,      # effective group id
         :addr,      # address of the process
         :size,      # size of process in kbytes
         :rssize,    # resident set size in kbytes
         :ttydev,    # tty device (or PRNODEV)
         :pctcpu,    # % of recent cpu used by all lwp's
         :pctmem,    # % of system memory used by process
         :start,     # absolute process start time
         :time,      # usr + sys cpu time for this process
         :ctime,     # usr + sys cpu time for reaped children
         :fname,     # name of the exec'd file
         :psargs,    # initial characters argument list
         :wstat,     # if a zombie, the wait status
         :argc,      # initial argument count
         :argv,      # address of initial argument vector
         :envp,      # address of initial environment vector
         :dmodel,    # data model of the process
         :taskid,    # task id
         :projid,    # project id
         :nzomb,     # number of zombie lwp's in the process
         :poolid,    # pool id
         :zoneid,    # zone id
         :contract,  # process contract
         :lwpid,     # lwp id
         :wchan,     # wait address for sleeping lwp
         :stype,     # synchronization event type
         :state,     # numeric lwp state
         :sname,     # printable character for state
         :nice,      # nice for cpu usage
         :syscall,   # system call number (if in syscall)
         :pri,       # priority
         :clname,    # scheduling class name
         :name,      # name of system lwp
         :onpro,     # processor which last ran thsi lwp
         :bindpro,   # processor to which lwp is bound
         :bindpset,  # processor set to which lwp is bound
         :count,     # number of contributing lwp's
         :tstamp,    # current time stamp
         :create,    # process/lwp creation time stamp
         :term,      # process/lwp termination time stamp
         :rtime,     # total lwp real (elapsed) time
         :utime,     # user level cpu time
         :stime,     # system call cpu time
         :ttime,     # other system trap cpu time
         :tftime,    # text page fault sleep time
         :dftime,    # text page fault sleep time
         :kftime,    # kernel page fault sleep time
         :ltime,     # user lock wait sleep time
         :slptime,   # all other sleep time
         :wtime,     # wait-cpu (latency) time
         :stoptime,  # stopped time
         :minf,      # minor page faults
         :majf,      # major page faults
         :nswap,     # swaps
         :inblk,     # input blocks
         :oublk,     # output blocks
         :msnd,      # messages sent
         :mrcv,      # messages received
         :sigs,      # signals received
         :vctx,      # voluntary context switches
         :ictx,      # involuntary context switches
         :sysc,      # system calls
         :ioch,      # chars read and written
         :path,      # array of symbolic link paths from /proc/<pid>/
pid
         :contracts, # array symbolic link paths from /proc/<pid>/
contracts
         :fd,        # array of used file descriptors
         :cmd_args,  # array of command line arguments
         :environ    # hash of environment associated with the process
      ]

      public

      ProcTableStruct = Struct.new("ProcTableStruct", *FIELDS)

      # In block form, yields a ProcTableStruct for each process entry
that you
      # have rights to. This method returns an array of
ProcTableStruct's in
      # non-block form.
      #
      # If a +pid+ is provided, then only a single ProcTableStruct is
yielded or
      # returned, or nil if no process information is found for that
+pid+.
      #
      # Example:
      #
      #   # Iterate over all processes
      #   ProcTable.ps do |proc_info|
      #      p proc_info
      #   end
      #
      #   # Print process table information for only pid 1001
      #   p ProcTable.ps(1001)
      #
      def self.ps(pid = nil)
         array = block_given? ? nil : []

         Dir.foreach("/proc") do |file|
            next if file =~ /\D/ # Skip non-numeric entries under /
proc

            # Only return information for a given pid, if provided
            if pid
               next unless file.to_i == pid
            end

            # Skip over any entries we don't have permissions to read
            begin
               psinfo = IO.read("/proc/#{file}/psinfo")
            rescue StandardError, Errno::EACCES
               next
            end

            struct = ProcTableStruct.new

            struct.flag   = psinfo[0,4].unpack("i")[0]  # pr_flag
            struct.nlwp   = psinfo[4,4].unpack("i")[0]  # pr_nlwp
            struct.pid    = psinfo[8,4].unpack("i")[0]  # pr_pid
            struct.ppid   = psinfo[12,4].unpack("i")[0] # pr_ppid
            struct.pgid   = psinfo[16,4].unpack("i")[0] # pr_pgid
            struct.sid    = psinfo[20,4].unpack("i")[0] # pr_sid
            struct.uid    = psinfo[24,4].unpack("i")[0] # pr_uid
            struct.euid   = psinfo[28,4].unpack("i")[0] # pr_euid
            struct.gid    = psinfo[32,4].unpack("i")[0] # pr_gid
            struct.egid   = psinfo[36,4].unpack("i")[0] # pr_egid
            struct.addr   = psinfo[40,4].unpack("L")[0] # pr_addr

            struct.size   = psinfo[44,4].unpack("L")[0] * 1024 #
pr_size
            struct.rssize = psinfo[48,4].unpack("L")[0] * 1024 #
pr_rssize

            # skip pr_pad1

            # TODO: Convert this to a human readable string somehow
            struct.ttydev = psinfo[56,4].unpack("i")[0] # pr_ttydev

            # pr_pctcpu
            struct.pctcpu = (psinfo[60,2].unpack("S")[0] * 100).to_f /
0x8000

            # pr_pctmem
            struct.pctmem = (psinfo[62,2].unpack("S")[0] * 100).to_f /
0x8000

            struct.start = Time.at(psinfo[64,8].unpack("L")[0]) #
pr_start
            struct.time  = psinfo[72,8].unpack("L")[0]          #
pr_time
            struct.ctime = psinfo[80,8].unpack("L")[0]          #
pr_ctime

            struct.fname  = psinfo[88,16].strip          # pr_fname
            struct.psargs = psinfo[104,80].strip         # pr_psargs
            struct.wstat  = psinfo[184,4].unpack("i")[0] # pr_wstat
            struct.argc   = psinfo[188,4].unpack("i")[0] # pr_argc
            struct.argv   = psinfo[192,4].unpack("L")[0] # pr_argv
            struct.envp   = psinfo[196,4].unpack("L")[0] # pr_envp
            struct.dmodel = psinfo[200,1].unpack("C")[0] # pr_dmodel

            # skip pr_pad2

            struct.taskid   = psinfo[204,4].unpack("i")[0] # pr_taskid
            struct.projid   = psinfo[208,4].unpack("i")[0] #
pr_projectid
            struct.nzomb    = psinfo[212,4].unpack("i")[0] # pr_nzomb
            struct.poolid   = psinfo[216,4].unpack("i")[0] # pr_poolid
            struct.zoneid   = psinfo[220,4].unpack("i")[0] # pr_zoneid
            struct.contract = psinfo[224,4].unpack("i")[0] #
pr_contract

            # skip pr_filler

            ### lwpsinfo struct info

            # skip pr_flag

            struct.lwpid = psinfo[236,4].unpack("i")[0] # pr_lwpid

            # skip pr_addr

            struct.wchan   = psinfo[244,4].unpack("L")[0] # pr_wchan
            struct.stype   = psinfo[248,1].unpack("C")[0] # pr_stype
            struct.state   = psinfo[249,1].unpack("C")[0] # pr_state
            struct.sname   = psinfo[250,1]                # pr_sname
            struct.nice    = psinfo[251,1].unpack("C")[0] # pr_nice
            struct.syscall = psinfo[252,2].unpack("S")[0] # pr_syscall

            # skip pr_oldpri
            # skip pr_cpu

            struct.pri = psinfo[256,4].unpack("i")[0] # pr_syscall

            # skip pr_pctcpu
            # skip pr_pad
            # skip pr_start
            # skip pr_time

            struct.clname   = psinfo[280,8].strip          # pr_clname
            struct.name     = psinfo[288,16].strip         # pr_name
            struct.onpro    = psinfo[304,4].unpack("i")[0] # pr_onpro
            struct.bindpro  = psinfo[308,4].unpack("i")[0] #
pr_bindpro
            struct.bindpset = psinfo[308,4].unpack("i")[0] #
pr_bindpset

            # Get the full command line out of /proc/<pid>/as.
            begin
               fd = File.open("/proc/#{file}/as")

               fd.sysseek(struct.argv, IO::SEEK_SET)
               address = fd.sysread(struct.argc * 4).unpack("L")[0]

               struct.cmd_args = []

               0.upto(struct.argc - 1){ |i|
                  fd.sysseek(address, IO::SEEK_SET)
                  data = fd.sysread(128)[/^[^\0]*/] # Null strip
                  struct.cmd_args << data
                  address += data.length + 1 # Add 1 for the space
               }

               # Get the environment hash associated with the process.
               struct.environ = {}

               fd.sysseek(struct.envp, IO::SEEK_SET)

               env_address = fd.sysread(128).unpack("L")[0]

               loop do
                  fd.sysseek(env_address, IO::SEEK_SET)
                  data = fd.sysread(1024)[/^[^\0]*/] # Null strip
                  break if data.empty?
                  key, value = data.split('=')
                  struct.environ[key] = value
                  env_address += data.length + 1 # Add 1 for the space
               end
            rescue Errno::EACCES, Errno::EOVERFLOW, EOFError
               # Skip this if we don't have proper permissions, if
there's
               # no associated environment, or if there's a largefile
issue.
            ensure
               fd.close if fd
            end

            ### struct prusage

            begin
               prusage = 0.chr * 512
               prusage = IO.read("/proc/#{file}/usage")

               # skip pr_lwpid
               struct.count    = prusage[4,4].unpack("i")[0]  #
pr_count
               struct.tstamp   = prusage[8,8].unpack("L")[0]  #
pr_tstamp
               struct.create   = prusage[16,8].unpack("L")[0] #
pr_create
               struct.term     = prusage[24,8].unpack("L")[0] #
pr_term
               struct.rtime    = prusage[32,8].unpack("L")[0] #
pr_rtime
               struct.utime    = prusage[40,8].unpack("L")[0] #
pr_utime
               struct.stime    = prusage[48,8].unpack("L")[0] #
pr_stime
               struct.ttime    = prusage[56,8].unpack("L")[0] #
pr_ttime
               struct.tftime   = prusage[64,8].unpack("L")[0] #
pr_tftime
               struct.dftime   = prusage[72,8].unpack("L")[0] #
pr_dftime
               struct.kftime   = prusage[80,8].unpack("L")[0] #
pr_kftime
               struct.ltime    = prusage[88,8].unpack("L")[0] #
pr_ltime
               struct.slptime  = prusage[96,8].unpack("L")[0] #
pr_slptime
               struct.wtime    = prusage[104,8].unpack("L")[0] #
pr_wtime
               struct.stoptime = prusage[112,8].unpack("L")[0] #
pr_stoptime
               struct.minf     = prusage[120,4].unpack("L")[0] #
pr_minf
               struct.majf     = prusage[124,4].unpack("L")[0] #
pr_majf
               struct.nswap    = prusage[128,4].unpack("L")[0] #
pr_nswap
               struct.inblk    = prusage[128,4].unpack("L")[0] #
pr_inblk
               struct.oublk    = prusage[128,4].unpack("L")[0] #
pr_oublk
               struct.msnd     = prusage[128,4].unpack("L")[0] #
pr_msnd
               struct.mrcv     = prusage[128,4].unpack("L")[0] #
pr_mrcv
               struct.sigs     = prusage[128,4].unpack("L")[0] #
pr_sigs
               struct.vctx     = prusage[128,4].unpack("L")[0] #
pr_vctx
               struct.ictx     = prusage[128,4].unpack("L")[0] #
pr_ictx
               struct.sysc     = prusage[128,4].unpack("L")[0] #
pr_sysc
               struct.ioch     = prusage[128,4].unpack("L")[0] #
pr_ioch
            rescue Errno::EACCES
               # Do nothing if we lack permissions. Just move on.
            end

            # Information from /proc/<pid>/path. This is represented
as a hash,
            # with the symbolic link name as the key, and the file it
links to
            # as the value, or nil if it cannot be found.
            #--
            # Note that cwd information can be gathered from here,
too.
            struct.path = {}

            Dir["/proc/#{file}/path/*"].each{ |entry|
               link = File.readlink(entry) rescue nil
               struct.path[File.basename(entry)] = link
            }

            # Information from /proc/<pid>/contracts. This is
represented as
            # a hash, with the symbolic link name as the key, and the
file
            # it links to as the value.
            struct.contracts = {}

            Dir["/proc/#{file}/contracts/*"].each{ |entry|
               link = File.readlink(entry) rescue nil
               struct.contracts[File.basename(entry)] = link
            }

            # Information from /proc/<pid>/fd. This returns an array
of
            # numeric file descriptors used by the process.
            struct.fd = Dir["/proc/#{file}/fd/*"].map{ |f|
File.basename(f).to_i }

            if block_given?
               yield struct
            else
               array << struct
            end
         end

         pid ? array[0] : array
      end
   end
end

Thanks,

Dan

* I tried tossing threads at it in a one-thread-per-directory
approach, but they didn't get along with IO.read in MRI, and seemed to
provide no real speed benefit with JRuby.