From: Charles Oliver Nutter Date: 2011-03-05T16:09:34+09:00 Subject: Re: [ANN] posix-spawn 0.3.0 -- first public release (codename, "tigers blood") FWIW, https://github.com/headius/spoon On Sat, Mar 5, 2011 at 1:08 AM, Charles Oliver Nutter wrote: > Did you know about the "spoon" gem? It's a very simple binding of > posix_spawn via FFI that works fine in JRuby too. It would sure be > nice if this could be an FFI solution, so it would work without a C > extension. > > On Fri, Mar 4, 2011 at 9:49 PM, Ryan Tomayko wrote: >> >> $ gem install posix-spawn >> >> tmm1 and I are pleased to announce the initial release of posix-spawn, >> a small extension library that implements a subset of Ruby 1.9's new >> Process::spawn [1] in a way that takes advantage of fast process >> spawning (IEEE Std 1003.1 posix_spawn(2) systems interfaces [2]) where >> available and runs on all MRI Rubys >= 1.8.7. >> >>  - Fast, constant time process spawning across a variety of platforms >>  - A largish compatible subset of Ruby 1.9's Process::spawn interface >>   as well as 1.9 enhancements to Kernel#system, Kernel#`, etc. under >>   Ruby >= 1.8.7. >>  - High level and hopefully portable POSIX::Spawn::Child class for >>   quick and dirty (but correct!) non-streaming IPC scenarios. >> >> See the README for usage and graphs of benchmark results on Linux and >> Darwin, or run them yourself: >> >>    $ uname -a >>    Linux aux1 2.6.26-2-xen-amd64 #1 SMP Thu Aug 20 2009 x86_64 GNU/Linux >>    $ ruby --version >>    ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux] >>    $ gem install posix-spawn >>    $ posix-spawn-benchmark >>    benchmarking fork/exec vs. posix_spawn over 1000 runs at 100M res >>                                 user     system      total        real >>    fspawn (fork/exec):     0.080000  14.920000  38.040000 ( 39.029493) >>    pspawn (posix_spawn):   0.040000   0.010000   0.560000 (  0.939422) >> >> Work on the library started when tmm1 found, through the use of his >> brilliant rbtrace [3] program, a number of slow points in the GitHub >> codebase where fork/exec is used heavily to spawn processes. In some >> cases, a single fork() system call was using >30ms while in others >> using only ~1ms. Our testsuite fork()'d especially slowly. Hmmm. >> >> On Linux, fork(2) slows down as the parent process uses more memory >> due to the need to copy page tables for COW. In many common uses of >> fork(), where it is followed by one of the exec family of functions to >> spawn child processes (Kernel#system, IO::popen, Process::spawn, >> etc.), this overhead can be removed by using posix_spawn() or vfork() >> instead. >> >> After implementing a simple fast process spawner extension using >> posix_spawn() and gaining some familiarity with the posix_spawn family >> of C functions, we noticed that it could potentially be used to >> implement a large subset of features provided by Ruby 1.9's >> Process::spawn. >> >> We love Process::spawn. >> >> We love Process::spawn so much in fact that over the past few months, >> even before surfacing any of the issues with Linux fork() slowness, an >> effort had been underway at GitHub to move two key libraries (Grit, >> the Ruby interface to Git, and Albino, a Ruby wrapper around the >> excellent Pygments syntax highlighter) to use Process::spawn >> compatible method invocations (implemented with fork/exec under >> Ruby 1.8.7) so that we could take advantage of Process::spawn under >> Ruby 1.9. >> >> Once we had a basic Process::spawn interface implemented on top of >> posix_spawn(), we were able to take some higher level utility classes >> from this work on the Grit and Albino projects and include them in >> posix-spawn as a nice POSIX::Spawn::Child class. It is: >> >>  - Simple, requiring little code for simple stream input and capture >>  - Internally non-blocking (uses select(2)), so it handles all pipe >>   hang cases due to exceeding PIPE_BUF limits on one or more streams >>  - Potentially portable, due to the abstraction over lower-level >>   process and stream management APIs >> >> We hope to now remove large bodies of Ruby 1.8.7 spawn emulation code >> and replace it with posix-spawn. >> >> As the project continued to take shape, we noticed how much more >> feature-rich the Kernel#system, IO.popen, etc. methods were in Ruby >> 1.9. Having been built on the foundation of the new Process::spawn, >> they allow for setting up the child's environment, redirecting >> arbitrary fds, and all the other great stuff in Process::spawn. We >> were able to write Ruby 1.8.7 compatible subset implementations of >> those as well and put them under the POSIX::Spawn module. >> >> Now, about that subset. As of this initial release, we were able to >> implement the following arguments and options to spawn: >> >>> spawn([env,] command... [,options]) => pid >>> >>> env: hash >>>   name => val : set the environment variable >>>   name => nil : unset the environment variable >>> command...: >>>   command                     : /bin/sh -c 'command' >>>   cmdname, arg1, ...          : exec argv (no shell) >>>   [cmdname, argv0], arg1, ... : exec argv (no shell) >>> options: hash >>>   clearing environment variables: >>>     :unsetenv_others => true   : clear environment vars not in env >>>     :unsetenv_others => false  : don't clear (default) >>>   redirection: >>>     key: >>>       FD              : single fd in child process >>>       [FD, FD, ...]   : multiple fd in child process >>>     value: >>>       FD                        : redirect to fd in parent process >>>       :close                    : close the fd in child process >>>       string                    : redir w/ open(string, "r" or "w") >>>       [string]                  : redir w/ open(string, File::RDONLY) >>>       [string, open_mode]       : redir w/ open(string, open_mode, 0644) >>>       [string, open_mode, perm] : redir w/ open(string, open_mode, perm) >>>     FD is one of follows >>>       :in     : the fd 0 which is the standard input >>>       :out    : the fd 1 which is the standard output >>>       :err    : the fd 2 which is the standard error >>>       integer : the fd of specified the integer >>>       io      : the fd specified as io.fileno >>>   current directory: >>>     :chdir => str >> >> We have NOT yet implemented these options: >> >>> options: hash >>>   process group: >>>     :pgroup => true or 0 : make a new process group >>>     :pgroup => pgid      : join to specified process group >>>     :pgroup => nil       : don't change the process group (default) >>>   resource limit: resourcename is core, cpu, data, etc. >>>     :rlimit_resourcename => limit >>>     :rlimit_resourcename => [cur_limit, max_limit] >>>   umask: >>>     :umask => int >>>   redirection: >>>     value: >>>       [:child, FD]              : redirect to the redirected fd >>>   file descriptor inheritance: close non-redir non-standard fds > 3 >>>     :close_others => false : inherit fds (default for system and exec) >>>     :close_others => true  : no inherit (default for spawn and popen) >> >> We have ideas for some of these (:pgroup, :umask, [:child, FD]) and >> may implement them in future releases; others, like :rlimit, are not >> supported by posix_spawn() and have no clear implementations strategy >> outside of falling back to fork/exec when detected. >> >> [0] https://github.com/rtomayko/posix-spawn >> [1] http://www.ruby-doc.org/core-1.9/classes/Process.html#M002230 >> [2] http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html >> [3] https://github.com/tmm1/rbtrace >> >> Ryan Tomayko >> Aman Gupta >> >> >