From: Paulus Esterhazy Date: 2005-11-21T06:00:55+09:00 Subject: Re: Pipelined Processing --------------060306030303040200090101 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit >> this came up recently on IRC: question was how to chain processing so >> that each step runs concurrently to other steps. While I don't see real >> benefit as long as there are no native threads in Ruby I played around a >> bit and this is the result (attached). There's certainly rool for >> improvement. Do with this whatever you like. Hello, it's funny you should bring this up because I just cooked up something similar (albeit less sophisticated and robust). What I needed was several tasks being carried out in parallel. In this particular instance I download several pages using open-uri and output a chunk to webrick as soon as its processing is finished (as responsiveness is critical). For network I/O, ruby's pseudo-threads seem to work well, though I agree native threads would be much better. Attached the ParallelEnumerate class along with a trivial test case. Bear with me, it's my first ruby script. Cheers, Paulus --------------060306030303040200090101 Content-Type: application/x-ruby; name="parallel_enum.rb" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="parallel_enum.rb" cmVxdWlyZSAnZ2VuZXJhdG9yJwpyZXF1aXJlICd0aHJlYWQnCnJlcXVpcmUgJ3Rod2FpdCcK cmVxdWlyZSAndGVzdC91bml0JwoKIyBlbnVtZXJhdGVzIGluIHBhcmFsbGVsIG92ZXIgc2V2 ZXJhbCBFbnVtZXJhYmxlcwojICh1c2luZyB0aHJlYWRzKQoKY2xhc3MgUGFyYWxsZWxFbnVt ZXJhdGUKICBpbmNsdWRlIEVudW1lcmFibGUKCiAgRU5EX09GX1FVRVVFID0gbmlsCgogIGRl ZiBpbml0aWFsaXplKCplbnVtcykKICAgIEBlbnVtcyA9IGVudW1zCiAgZW5kCgogIGRlZiA8 PChlbnVtKQogICAgQGVudW1zIDw8IGVudW0KICBlbmQKICAKICBkZWYgZWFjaAogICAgcXVl dWUgPSBRdWV1ZS5uZXcKICAgIHRocmVhZHMgPSBbXQogICAgVGhyZWFkLmFib3J0X29uX2V4 Y2VwdGlvbiA9IHRydWUKICAgIGZvciBlbnVtIGluIEBlbnVtcwogICAgICB0aHJlYWRzIDw8 IFRocmVhZC5uZXcoZW51bSkgZG8KICAgICAgICBiZWdpbgogICAgICAgICAgZW51bS5lYWNo IHsgfGl0ZW18CiAgICAgICAgIAkgIHF1ZXVlIDw8IGl0ZW0KICAgICAgICAgIH0KICAgICAg ICBlbnN1cmUKICAgICAgICAgIHF1ZXVlIDw8IEVORF9PRl9RVUVVRSAjIHdha2UgdXAgYmxv Y2tlZCB0aHJlYWQKICAgICAgICBlbmQKICAgICAgZW5kCiAgICBlbmQKCiAgICB0aHJlYWRz X2FsaXZlID0gdGhyZWFkcy5zaXplCgogICAgd2hpbGUgdGhyZWFkc19hbGl2ZSA+IDAKICAg ICAgaXRlbSA9IHF1ZXVlLnBvcCAgICAgICAgIyBibG9jayB1bnRpbCBtZXNzYWdlIGFycml2 ZXMKICAgICAgaWYgaXRlbSA9PSBFTkRfT0ZfUVVFVUUKICAgICAgICB0aHJlYWRzX2FsaXZl IC09IDEKICAgICAgZWxzZQogICAgICAgIHlpZWxkIGl0ZW0KICAgICAgZW5kCiAgICBlbmQK CiAgICBUaHJlYWRzV2FpdC5hbGxfd2FpdHModGhyZWFkcykgdW5sZXNzIHRocmVhZHMuZW1w dHk/ICMgam9pbiB0aHJlYWRzIGFueXdheQogIGVuZAplbmQKCmNsYXNzIFRlc3RQYXJhbGxl bEVudW0gPCBUZXN0OjpVbml0OjpUZXN0Q2FzZQogIGRlZiB0ZXN0X3NpbXBsZQoJCWVudW0g PSBQYXJhbGxlbEVudW1lcmF0ZS5uZXcKCQllbnVtIDw8IFsxLDIsMyw0XQoJCWVudW0gPDwg WzUsNl0KCQlhc3NlcnRfZXF1YWwgNiwgZW51bS5lbnRyaWVzLnNpemUKICBlbmQKZW5kCg== --------------060306030303040200090101--