[Pyrex] nagging differences / implementing generators

[David J. C. Beach]

Greetings,

I've been downloading the most recent releases of Pyrex, and have been
trying to keep track of the "unnecessary" differences between Pyrex and
Python.  As I understand it, the following Python language features are
slated for inclusion in the Pyrex language:

1) +=, -=, etc. operators
2) list comprehensions
3) tuples in python functions:   "def x(a, (b,c)): ..."

This are features that I miss in Pyrex, but see no inherent reason for
their absence.  Are these features planned for a 1.0 release of Pyrex? 
These features are important to me, and I'd like to see them included. 
Is there some way that I could help?

In addition, I wanted to ask about a few other differences:

1) having "cdef int x = 3" or "cdef double y = 2.2"
2) building generators / allowing yield

#1 would seem to make Pyrex just a little more friendly to program with.

The remainder of this email is about #2:

I'm proposing this because I believe it's possible, and could help with
the creation of certain extensions.  Naturally, this would only be legal
from within a Python "def" function, not a "cdef" one. Moreover, this
would make "generator expressions" possible from within Pyrex when they
become available in Python 2.4.

When a Python generator function is called, it yields a python
"generator" object which behaves as an "iterator".  In Pyrex, it would
be possible to generate a c-extension type which behaves as an iterator
to be returned whenever a Pyrex generator function is called.

The real question seems to be:  How do we store the state of the
iterator/generator object between "yield" statements?  My thought is to
have a special integer variable in the extension type which stores the
"position" of the execution within the next() function... all other
generator state should be stored as member variables of the generator
object (to preserve it between calls).  Upon entering the "next"
function, the C code should examine the value of the "position" variable
with a "switch" statement, and do a computed - goto to jump to the
appropriate place in the function's execution.

For example the code:

def mygenerator():
    yield 5
    for i in xrange(10):
        print "hello!"
        yield 7

Could be converted into a C-generator object like:

PyObject* Pyrex_mygenerator_next(PyObject* self)
{
    Pyrex_Generator* gen = (Pyrex_Generator*) self;
    switch(gen->position) {
        case 0:
            goto pyrex_position_0; /* logical begin of function */
        case 1:
            goto pyrex_position_1;
        case 2:
            goto pyrex_position_2;
        case 3:
            goto pyrex_position_3;
    }
    pyrex_position_0:
       /* first time in function */ 

    pyrex_position_1:
       gen->position = 2;
       return PyInt_New(5);

    pyrex_position_2:
       /* for i in xrange(10): -> goto pyrex_loop_end on loop exit */
       /* print "hello" */
       gen->position = 3;
       return PyInt_New(7);

    pyrex_position_3:
       goto /* for loop start */;

    pyrex_loop_end:
       /* raise StopIteration */
}

Such a strategy allows Pyrex to preserve the state and position of a
generator object between successive "yields", and, I believe, foots the
bill.

I realize that an implementation of this feature would be a fair amount
of work, and I'm not proposing that it necessarily be implemented, but I
wanted to point out that I believe such a strategy would be _possible_,
and might be worth consideration.  Generators which can be created and
utilized as C-speeds could be a serious boon to developing with Pyrex.

Please let me know what you think about this.

Sincerely,

Dave Beach

-- 
David J. C. Beach
<beach at verinet.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.copyleft.no/pipermail/pyrex/attachments/20040506/b339811f/attachment.bin