From: Pau Garcia i Quiles Date: 2005-12-27T18:01:51+09:00 Subject: Re: [ANN] ruby-dipus 0.1.2 Quoting Ilmari Heikkinen : Wow, that's exactly what I was looking for! Thank you! Just one "but": I think using a known path under /tmp/dipus may be a security risk. > DIPUS - Distributed IPC by Proxying UNIX Sockets > ================================================ > Sockets with user authentication, file permissions, encryption, > service discovery and network transparent addressing. > > Using services: > --- > host1$ dcat lame-svc foo.wav | dcat madplay.host3 > host1$ dcat sep://example.com/image/jpeg/to/image/png foo.jpeg | dcat > ssh://user@another.com/image/png/to/text/plain | less > > Creating services: > --- > host3$ dipus-local-pipe 'madplay -' madplay > host2$ dipus-export-pipe 'lame - -' lame-svc > > Some performance numbers: > --- > Local throughput around 50MBps (or 100MBps, depending whether you use > socat as a client or not), unencrypted network throughput 9.8MBps in a > 100Mbps LAN, CPU-limited when using SSH (my other computer is slow.) > > In what languages can I use and create services? > --- > Anything that supports a) UNIX domain sockets, and b) listing > filesystem directories. So, just about every language available. If > you install the helper scripts in bin, you don't even need those, just > the ability to read from stdin and write to stdout. > > Download: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2.tar.gz > RDocs: http://www.cs.helsinki.fi/u/irkheikk/dipus-rdoc/ > Darcs repo: http://dark.fhtr.org/repos/ruby-dipus > > Readme: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/readme.txt > Design: > http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-design.txt > Protocol spec: > http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-specification.txt > > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > > Hi, designed and implemented a distributed IPC system that takes > lessons from UNIX (keep it simple) and HTTP (keep it simple.) Opinions > on the design or anything else? I'm dying to get some feedback, > writing stuff in a bubble can take you only so far. > > And, now, with the first version of this December project out of the > way, I'm off to librend land~ > -- > Ilmari Heikkinen > > vvvvvvvvvvvvvvvvvvvvvvvvvv > > What is this for? > ----------------- > You have a program. You want to access it remotely with the minimum > hassle. It'd be nice to access it efficiently locally too. Without > rewriting your connection code. > > And it'd be nice to put some connections over SSH. And be notified > when a service dies or a new one is created. And not expose any > of the individual services to the network, or even to other users > on the machine running them. And it'd be cool to be able to have a > different name and protocol for each service without having to shoehorn > FTP over XML-RPC. Oh, also, display a real-time list of services > running on the local network and on explicitly defined machines that > aren't on the local network. And it'd be especially cool if all the > software could just use existing libraries like sockets and files. > > So, take UNIX sockets and sprinkle some sugar to make them usable for > distributed IPC. > > > Supported systems > ----------------- > Should work on Linux and *BSD, likely on OS X too. > No Windows support, although it's theoretically possible. > > > Required > -------- > Ruby version 1.8.2+ (one released in 2005) > cat > > > Recommended > ----------- > OpenSSH version > 3.9 (for connection sharing support (ControlPath)) > ssh-agent setup (for maximum convenience) > socat > More than one computer. > > > How does it work? > ----------------- > To create a service, make a listening socket in a subdir of > /tmp/dipus, so that the path is like: > /tmp/dipus/exported/audio/x-wav/to/audio/mpeg/lame.mybox..0 > > To access a service, open its socket. > > To make a service only accessible to members of group 'audio': > chgrp audio socket_name > chmod 770 socket_name > > Listing services that you can see: > tree /tmp/dipus > > DIPUS provides helper methods and daemons to manipulate the > sockets in /tmp/dipus, e.g. create proxies to remote sockets, > delete dead sockets, provide network access to exported > sockets, find sockets by their service name, hostname, or > protocol. > > > Quick usage example > ------------------- > # (optional) start dipusd to automatically discover and import services, > # export services, and delete dead sockets. > # > [kig@desktop] dipusd > [kig@encode] dipusd > > # Create local madplay service for playing MP3s on > # the computer connected to speakers. > # The first argument is the command to run for each incoming connection. > # The second argument is service_name/protocol, e.g. my_www/http. > # > # The following command creates a socket with the path > # /tmp/dipus/local/audio/mpeg/to/audio/out/madplay.stereo..0 > # and runs 'madplay -' for each incoming connection, setting the > # connection to madplay's stdin and stdout. > # > [kig@stereo] dipus-local-pipe 'madplay -' madplay/audio/mpeg/to/audio/out > > # Create an exported lame service for encoding wavs to MP3s. > # > [kig@encode] dipus-export-pipe 'lame - -' lame/audio/x-wav/to/audio/mpeg > > # Manually import the services from stereo. > # Import over SSH to gain access to all services and not just exported ones. > # > [kig@desktop] dipus-import ssh://stereo > > # Dipusd should've discovered and imported the lame service from encode, > # so we don't need to manually import that. But if that didn't happen, we > # can import it by doing `dipus-import sep://encode`, sep being > # Socket Export Protocol (see doc/dipus-specification.txt) > > # Now, let's pipe some wavs to the stereo. > # First encode the wav to MP3, using any service with a protocol that says > # that the service converts audio/x-wav to audio/mpeg. > # Play using a service named madplay running on stereo. > # > [kig@desktop] dcat audio/x-wav/to/audio/mpeg *.wav | dcat madplay.stereo > > # We could also access the services without importing them by giving the > # service URI as the service name. > # > # Here's the above piping example using service URIs. > # Dcat lists the services matching the URI and uses the first match. > # > [kig@desktop] dcat sep://encode/audio/x-wav/to/audio/mpeg *.wav | \ > dcat ssh://stereo/audio/mpeg/to/audio/out/madplay.stereo..0 > > > List of DIPUS commands > ---------------------- > Pass --help as an argument to get the full USAGE. > > * dls -- list services > dls sep://foobar/audio > dls ssh://barfoo > > * dcat -- pipe stdin and/or given files to a DIPUS socket > dcat lame foo.wav bar.wav > foobar.mp3 > dcat ssh://foobar/image/png/to/image/jpeg cat.png > cat.jpg > > * dipus-import -- imported all found service from an URI > dipus-import ssh://foobar/audio > dipus-import sep://barfoo > > * dipusd -- start exporter, importer, advertiser, discoverer, > dead socket monitor, and default services > > * dipus-exporter -- start exporter > > * dipus-discoverer -- discover exporters on the LAN and print > their addresses > > * dipus-service-monitor -- monitor for service changes on localhost, > used by SSH importers > > * dipus-ssh-importer -- import services over SSH to user's DIPUS > subdirectory (/tmp/dipus/username), can use > discovery to find exporter-running hosts on > localnet > > * dipus-local-pipe -- create a local DIPUS service that forks the > given command for the connection > dipus-local-pipe 'head' head/text > > * dipus-export-pipe -- create an exported DIPUS service like above > dipus-export-pipe 'madplay -' madplay/audio/mpeg/to/audio/out > > > Using socat to create and connect to DIPUS services > --------------------------------------------------- > # proxy a web server > mkdir -p /tmp/dipus/exported/http > socat UNIX-LISTEN:/tmp/dipus/exported/http/foo.foobar..0,fork \ > TCP4-CONNECT:foo.example.com:80 > rm /tmp/dipus/exported/http/foo.foobar..0 > > # using a lame service to encode a wav > socat foo.wav \ > UNIX-CONNECT:/tmp/dipus/local/audio/x-wav/to/audio/mpeg/lame.foobar..0 \ > > foo.mp3 > > # or by using dls to find the filename > socat foo.wav UNIX-CONNECT:`dls -1 -f lame` > foo.mp3 > > # proxy a DIPUS service to a TCP socket > socat TCP4-LISTEN:1234,fork EXEC:'dcat lame' > > > Possibilities > ------------- > Carry around a mobile phone with an Internet connection. Use it to > run a media player. The media player sees files on your home computer > and the files on the local network. Then it pipes the files in its > playlist to the closest encode server, and from there to the wanted > output (headphones, speakers, file, streaming server...) > > Send your 3D animation file over the Internet to a thousand idle > machines to render. > > Walk into an auditorium and display slides on the overhead projector, > streamed from a network mount, controlled from your mobile. > > Run a search engine, giving URIs to remote services, making automatable > the task of finding and utilizing a network service. >