From: zuzu <sean.zuzu@...>
Date: 2004-08-27T03:16:39+09:00
Subject: python generators to ruby closures, help

can anyone help me decode the following python example using
generators into The Ruby Way using closures?

thanks,
-z

http://c2.com/cgi/wiki?GeneratorsInPython

You can emulate the unix-style shell piping with generators.

    from __future__ import generators

    class GenPipe?:
        def __init__(self, generator):
            self.gen = generator #sanity check omitted
        def __or__(self,nextPipe):
            self.gen=nextPipe._run(self.gen)
            return self
        def __iter__(self):
            return self.gen



    class PipeFilter?:
        def __init__(self):
            self._followingPipes=[]
        def __or__(self,nextPipe):
            self._followingPipes+=(nextPipe,)
            return self
        def _run(self,gen):
            gen=self.run(gen)
            while self._followingPipes:
                gen=self._followingPipes[0].run(gen)
                self._followingPipes.pop(0)
            return gen
        def run(self,gen):
            raise NotImplementedError?



    class Test(PipeFilter?):
        def __init__(self,test):
            PipeFilter?.__init__(self)
            self.test=test
        def run(self,gen):
            for x in gen:
                if self.test(x):
                    yield x



    class Quit(PipeFilter?):
        def __init__(self,test):
            PipeFilter?.__init__(self)
            self.test=test
        def run(self,gen):
            for x in gen:
                if self.test(x): 
                    raise StopIteration?
                else:
                    yield x



    class Count(PipeFilter?):
        def __init__(self,n):
            PipeFilter?.__init__(self)
            self.n=n
        def run(self,gen):
            for x in gen:
                yield x
                self.n -= 1
                if self.n == 0 :
                    break 



    def Ints():
        n = 0
        while 1:
            yield n
            n += 1



    def Files( start ):
        import os
        for file in os.listdir( start ):
            file = os.path.join( start, file )
            if os.path.isfile( file ): yield file
            elif os.path.isdir(file):
                for more in Files( file ):
                    yield more



    >>> odd = Test(lambda x:x % 2)
    >>> notDivBy3 = Test(lambda x: x % 3 )
    >>> oddNotDivBy3 = odd|notDivBy3
    >>> firstTen= Count(10)
    >>> result=GenPipe?(Ints())|firstTen|oddNotDivBy3
    >>> for each in result:
    >>>     print each



    >>> oddLowerThanTen=odd|Quit(lambda y:y>=10)
    >>> p3=GenPipe?(Ints())|oddLowerThanTen|notdiv3



    >>> isPyFile=Test(lambda s:s.split('.')[-1].lower()=='py')
    >>> tenPyFiles=GenPipe?(Files('/'))|isPyFile|Count(10)
    >>> for each in tenPyFiles:
    >>>     print each

-- JuneKim

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/21900

Subject: [ruby-talk:21900] Re: Python generators
From: Will Conant <WillC webmiles.com>
Date: Tue, 2 Oct 2001 08:18:10 +0900

Generators are a nifty feature for Python 2.2, but they are a solution to
Python problems... problems that Ruby doesn't have.

In Ruby, it is trivial to pass a closure that interacts with the scope that
created it to another function. Thus, callbacks in Ruby are sufficiently
useful to solve the problems that Python generators were designed to solve.

The example that I read somewhere on the Python site was the tokenizer
problem. The original approach to the tokenizer problem was to have a
function that took two parameters as input: the text source and a function
pointer to a callback that processed the tokens. The problem with this
approach was that the callback function, designed by the user, didn't have a
very good way of keeping state between calls. It could use globals, but that
would suck.

The other approach was to write a tokenizer object that kept its state and
had a "nextToken" function that looked for the next token in the stream and
returned it to the caller. This solution hoisted the state problem off on
the tokenizer instead of the user. Still a problem especially considering
the difficulty of unrolling very complex loops of logic.

For Python, where closures aren't that easy to use, the generator is the
ideal solution. This way, you write your tokenizer function as if were to
use a callback so you don't have to unroll your loop. However, your function
gets wrapped up in an iterator object, so the user uses it as if it were
implemented as a Tokenizer object with a "nextToken" function.

In Ruby, you could probably emulate this behavior directly, but you really
wouldn't need to. Instead, you would just write your tokenizer to use a
callback, and your callback method would be a local closure with local state
persistant between calls. Problem solved.

Anyway, that's the story. Generators are a whitespace friendly solution to a
few of the problems that Ruby's super-duper closure mechanism solves just
fine.

-Will Conant