Re: [ANN] posix-spawn 0.3.0 -- first public release (codename, "tigers blood")

FWIW, https://github.com/headius/spoon

On Sat, Mar 5, 2011 at 1:08 AM, Charles Oliver Nutter
<[e-mail]> wrote:
> Did you know about the "spoon" gem? It's a very simple binding of
> posix_spawn via FFI that works fine in JRuby too. It would sure be
> nice if this could be an FFI solution, so it would work without a C
> extension.
>
> On Fri, Mar 4, 2011 at 9:49 PM, Ryan Tomayko <[e-mail]> wrote:
>> <https://github.com/rtomayko/posix-spawn>
>> $ gem install posix-spawn
>>
>> tmm1 and I are pleased to announce the initial release of posix-spawn,
>> a small extension library that implements a subset of Ruby 1.9's new
>> Process::spawn [1] in a way that takes advantage of fast process
>> spawning (IEEE Std 1003.1 posix_spawn(2) systems interfaces [2]) where
>> available and runs on all MRI Rubys >=3D 1.8.7.
>>
>> =C2=A0- Fast, constant time process spawning across a variety of platfor=
ms
>> =C2=A0- A largish compatible subset of Ruby 1.9's Process::spawn interfa=
ce
>> =C2=A0 as well as 1.9 enhancements to Kernel#system, Kernel#`, etc. unde=
r
>> =C2=A0 Ruby >=3D 1.8.7.
>> =C2=A0- High level and hopefully portable POSIX::Spawn::Child class for
>> =C2=A0 quick and dirty (but correct!) non-streaming IPC scenarios.
>>
>> See the README for usage and graphs of benchmark results on Linux and
>> Darwin, or run them yourself:
>>
>> =C2=A0 =C2=A0$ uname -a
>> =C2=A0 =C2=A0Linux aux1 2.6.26-2-xen-amd64 #1 SMP Thu Aug 20 2009 x86_64=
 GNU/Linux
>> =C2=A0 =C2=A0$ ruby --version
>> =C2=A0 =C2=A0ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
>> =C2=A0 =C2=A0$ gem install posix-spawn
>> =C2=A0 =C2=A0$ posix-spawn-benchmark
>> =C2=A0 =C2=A0benchmarking fork/exec vs. posix_spawn over 1000 runs at 10=
0M res
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 user =C2=A0 =C2=A0 system =C2=A0 =
=C2=A0 =C2=A0total =C2=A0 =C2=A0 =C2=A0 =C2=A0real
>> =C2=A0 =C2=A0fspawn (fork/exec): =C2=A0 =C2=A0 0.080000 =C2=A014.920000 =
=C2=A038.040000 ( 39.029493)
>> =C2=A0 =C2=A0pspawn (posix_spawn): =C2=A0 0.040000 =C2=A0 0.010000 =C2=
=A0 0.560000 ( =C2=A00.939422)
>>
>> Work on the library started when tmm1 found, through the use of his
>> brilliant rbtrace [3] program, a number of slow points in the GitHub
>> codebase where fork/exec is used heavily to spawn processes. In some
>> cases, a single fork() system call was using >30ms while in others
>> using only ~1ms. Our testsuite fork()'d especially slowly. Hmmm.
>>
>> On Linux, fork(2) slows down as the parent process uses more memory
>> due to the need to copy page tables for COW. In many common uses of
>> fork(), where it is followed by one of the exec family of functions to
>> spawn child processes (Kernel#system, IO::popen, Process::spawn,
>> etc.), this overhead can be removed by using posix_spawn() or vfork()
>> instead.
>>
>> After implementing a simple fast process spawner extension using
>> posix_spawn() and gaining some familiarity with the posix_spawn family
>> of C functions, we noticed that it could potentially be used to
>> implement a large subset of features provided by Ruby 1.9's
>> Process::spawn.
>>
>> We love Process::spawn.
>>
>> We love Process::spawn so much in fact that over the past few months,
>> even before surfacing any of the issues with Linux fork() slowness, an
>> effort had been underway at GitHub to move two key libraries (Grit,
>> the Ruby interface to Git, and Albino, a Ruby wrapper around the
>> excellent Pygments syntax highlighter) to use Process::spawn
>> compatible method invocations (implemented with fork/exec under
>> Ruby 1.8.7) so that we could take advantage of Process::spawn under
>> Ruby 1.9.
>>
>> Once we had a basic Process::spawn interface implemented on top of
>> posix_spawn(), we were able to take some higher level utility classes
>> from this work on the Grit and Albino projects and include them in
>> posix-spawn as a nice POSIX::Spawn::Child class. It is:
>>
>> =C2=A0- Simple, requiring little code for simple stream input and captur=
e
>> =C2=A0- Internally non-blocking (uses select(2)), so it handles all pipe
>> =C2=A0 hang cases due to exceeding PIPE_BUF limits on one or more stream=
s
>> =C2=A0- Potentially portable, due to the abstraction over lower-level
>> =C2=A0 process and stream management APIs
>>
>> We hope to now remove large bodies of Ruby 1.8.7 spawn emulation code
>> and replace it with posix-spawn.
>>
>> As the project continued to take shape, we noticed how much more
>> feature-rich the Kernel#system, IO.popen, etc. methods were in Ruby
>> 1.9. Having been built on the foundation of the new Process::spawn,
>> they allow for setting up the child's environment, redirecting
>> arbitrary fds, and all the other great stuff in Process::spawn. We
>> were able to write Ruby 1.8.7 compatible subset implementations of
>> those as well and put them under the POSIX::Spawn module.
>>
>> Now, about that subset. As of this initial release, we were able to
>> implement the following arguments and options to spawn:
>>
>>> spawn([env,] command... [,options]) =3D> pid
>>>
>>> env: hash
>>> =C2=A0 name =3D> val : set the environment variable
>>> =C2=A0 name =3D> nil : unset the environment variable
>>> command...:
>>> =C2=A0 command =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 : /bin/sh -c 'command'
>>> =C2=A0 cmdname, arg1, ... =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: exec argv=
 (no shell)
>>> =C2=A0 [cmdname, argv0], arg1, ... : exec argv (no shell)
>>> options: hash
>>> =C2=A0 clearing environment variables:
>>> =C2=A0 =C2=A0 :unsetenv_others =3D> true =C2=A0 : clear environment var=
s not in env
>>> =C2=A0 =C2=A0 :unsetenv_others =3D> false =C2=A0: don't clear (default)
>>> =C2=A0 redirection:
>>> =C2=A0 =C2=A0 key:
>>> =C2=A0 =C2=A0 =C2=A0 FD =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0: single fd in child process
>>> =C2=A0 =C2=A0 =C2=A0 [FD, FD, ...] =C2=A0 : multiple fd in child proces=
s
>>> =C2=A0 =C2=A0 value:
>>> =C2=A0 =C2=A0 =C2=A0 FD =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: redirect to fd in parent process
>>> =C2=A0 =C2=A0 =C2=A0 :close =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0: close the fd in child process
>>> =C2=A0 =C2=A0 =C2=A0 string =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0: redir w/ open(string, "r" or "w")
>>> =C2=A0 =C2=A0 =C2=A0 [string] =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0: redir w/ open(string, File::RDONLY)
>>> =C2=A0 =C2=A0 =C2=A0 [string, open_mode] =C2=A0 =C2=A0 =C2=A0 : redir w=
/ open(string, open_mode, 0644)
>>> =C2=A0 =C2=A0 =C2=A0 [string, open_mode, perm] : redir w/ open(string, =
open_mode, perm)
>>> =C2=A0 =C2=A0 FD is one of follows
>>> =C2=A0 =C2=A0 =C2=A0 :in =C2=A0 =C2=A0 : the fd 0 which is the standard=
 input
>>> =C2=A0 =C2=A0 =C2=A0 :out =C2=A0 =C2=A0: the fd 1 which is the standard=
 output
>>> =C2=A0 =C2=A0 =C2=A0 :err =C2=A0 =C2=A0: the fd 2 which is the standard=
 error
>>> =C2=A0 =C2=A0 =C2=A0 integer : the fd of specified the integer
>>> =C2=A0 =C2=A0 =C2=A0 io =C2=A0 =C2=A0 =C2=A0: the fd specified as io.fi=
leno
>>> =C2=A0 current directory:
>>> =C2=A0 =C2=A0 :chdir =3D> str
>>
>> We have NOT yet implemented these options:
>>
>>> options: hash
>>> =C2=A0 process group:
>>> =C2=A0 =C2=A0 :pgroup =3D> true or 0 : make a new process group
>>> =C2=A0 =C2=A0 :pgroup =3D> pgid =C2=A0 =C2=A0 =C2=A0: join to specified=
 process group
>>> =C2=A0 =C2=A0 :pgroup =3D> nil =C2=A0 =C2=A0 =C2=A0 : don't change the =
process group (default)
>>> =C2=A0 resource limit: resourcename is core, cpu, data, etc.
>>> =C2=A0 =C2=A0 :rlimit_resourcename =3D> limit
>>> =C2=A0 =C2=A0 :rlimit_resourcename =3D> [cur_limit, max_limit]
>>> =C2=A0 umask:
>>> =C2=A0 =C2=A0 :umask =3D> int
>>> =C2=A0 redirection:
>>> =C2=A0 =C2=A0 value:
>>> =C2=A0 =C2=A0 =C2=A0 [:child, FD] =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0: redirect to the redirected fd
>>> =C2=A0 file descriptor inheritance: close non-redir non-standard fds > =
3
>>> =C2=A0 =C2=A0 :close_others =3D> false : inherit fds (default for syste=
m and exec)
>>> =C2=A0 =C2=A0 :close_others =3D> true =C2=A0: no inherit (default for s=
pawn and popen)
>>
>> We have ideas for some of these (:pgroup, :umask, [:child, FD]) and
>> may implement them in future releases; others, like :rlimit, are not
>> supported by posix_spawn() and have no clear implementations strategy
>> outside of falling back to fork/exec when detected.
>>
>> [0] https://github.com/rtomayko/posix-spawn
>> [1] http://www.ruby-doc.org/core-1.9/classes/Process.html#M002230
>> [2] http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn=
.html
>> [3] https://github.com/tmm1/rbtrace
>>
>> Ryan Tomayko
>> Aman Gupta
>>
>>
>