[Pyrex] Adding custom statements to pyrex ? (or passing through macros to C files)

[Lenard Lindstrom]

On Sat, 2 Oct 2004 23:29:47 +0100 (BST) Michael Sparks <zathras at thwackety.com> wrote:

> Hi,
> 
> 
> This is a question about which bits of pyrex I would have to touch in
> order to allow Duff's device type tricks in Pyrex. I've been playing with
> duff's device recently to allow translation of some python generators to C
> & C++ code, but from my perspective it'd be nice if I could hack support
> into Pyrex for this.
> 
> The approach I've taken for C is as follows:
>    * For each generator create a struct, a struct creator/initialiser and
>      a "next" function. (the struct holds function state, the next
>      function implements the logic)
> 
> In C I have to pass through the structure (obviously) as a first
> parameter to the next function, whereas with C++ I can just let C++ use
> an implicit this pointer.
> 
> To make the C code look clearer, I define a small bunch of macros for each
> bit of Duff's device. Making this a bit more concrete, the macros look
> like this:
> [NB, no error checking in all that follows!]
> 
> #define Resume switch (this->state) { default: this->state=__LINE__;
> #define Yield(x) this->state=__LINE__ ; return (x); case __LINE__ :
> #define Endresume this->state=__LINE__; case __LINE__: break; }
> 
> The actual next method (which maps directly to the generator), looks like
> this:
> 
> int Duffs_next(Duffs * this) {
>    Resume;
>       for(this->i=0; this->i<10; this->i++) {
>           Yield(0);
>           printf("%s: %d\n", this->tag, this->i);
>       }
>    Endresume ;
> }
> 
> The structure for capturing local state looks like this:
> typedef struct Duffs {
>    int state;
>    int i;
>    char * tag;
> } Duffs;
> 
> The initialiser looks like this:
> Duffs * Duffs_init(char *tag) {
>    Duffs * d;
>    d = (Duffs *)malloc(sizeof(Duffs));
>    d->state=0;
>    d->tag = tag; /* Nasty due to aliasing */
>    return d;
> }
> 
> Sample usage then looks like this:
> 
> int main(int argc, char *argv[]) {
>    Duffs * G;
>    Duffs * H;
>    int i;
> 
>    G=Duffs_init("G");
>    H=Duffs_init("H");
>    for(i=0; i<14; i++) {
>       Duffs_next(G);
>       if(i>0) { Duffs_next(H); }
>    }
>    printf("CORO.C\n");
>    return 0;
> }
> 
> Which whilst verbose does have a number of advantages (IMO over taking a
> state machine approach.
> 
> Essentially I'd like to be able to do the same thing in Pyrex. There's two
> possible routes AFAICT:
>    1 Simply make it so that macros like the 3 presented up top can be
>      passed through by pyrex to the underlying compiler without major
>      issue. (Perhaps something like 'cmacrodef NAME' to allow pass
>      through.) This would involve boilerplating some of these things,
>      but I can live with that.
> 
>    2 The alternative is to modify Pyrex to generate the boiler plate
>      itself when it sees a yield statement. This is probably a lot more
>      work, but perhaps a more satisying approach. (Macro passthrough
>      whilst more flexible probably opens pyrex up to all sorts of nasty
>      abuses... :-/ )
> 
> I suspect 1 is a lot simpler than 2, but lack of familiarity with the
> internals of Pyrex means I'm not sure where to start with this. I'm not
> expecting anyone else to do the work, but would appreciate any pointers
> from anyone who has hacked pyrex's internals as to which bits I should
> look at/where to start. (I'm currently going through the code
> familiarising myself with the structure :)
> 
> If I'm completely missing something, that'd be nice to hear too :)
> 
> Thanks in advance for any comments :)
> 
I cannot comment at what is involved for option 2, but one can
explore what happens when macros are inserted into Pyrex code
as suggested in option 1. By declaring the Resume, Yield, and EndResume
macros as function macros in a .h file they can be inserted into Pyrex
code as external C function calls:

File duff.h:

#define Resume() switch (this->state) { default: this->state=__LINE__;
#define Yield(x) this->state=__LINE__ ; return (x); case __LINE__ :
#define Endresume() this->state=__LINE__; case __LINE__: break; }


And now two Pyrex versions of Duffs_next using the above macros:

File duff.pyx:

cdef extern from "duff.pyx":
    void Resume()
    void Yield(int x)
    void Endresume()

cdef struct Duffs:
   int state
   int i
   char * tag

# Duffs_next that is all C
cdef int Duffs_next(Duffs *this):
    cdef int i
    Resume()
    for i from 0 <= i < 10:
        this.i = i
        Yield(0)
    Endresume()

# Duffs_next with some Python mixed in
cdef Py_Duffs_next(Duffs *this):
    cdef int i
    Resume()
    for i from 0 <= i < 10:
        this.i = i
        Yield(0)
    Endresume()

Finally the part of the Pyrex 0.9.3 output C file duff.c that defines the two
versions of Duffs_next:

static int __pyx_f_4duff_Duffs_next(struct __pyx_t_4duff_Duffs (*__pyx_v_this)) {
  int __pyx_v_i;
  int __pyx_r;

  /* "C:\user\projects\python\Duffs\duff.pyx":14 */
  Resume();

  /* "C:\user\projects\python\Duffs\duff.pyx":15 */
  for (__pyx_v_i = 0; __pyx_v_i < 10; ++__pyx_v_i) {

    /* "C:\user\projects\python\Duffs\duff.pyx":16 */
    __pyx_v_this->i = __pyx_v_i;

    /* "C:\user\projects\python\Duffs\duff.pyx":17 */
    Yield(0);
    __pyx_L2:;
  }
  __pyx_L3:;

  /* "C:\user\projects\python\Duffs\duff.pyx":18 */
  Endresume();

  __pyx_r = 0;
  goto __pyx_L0;
  __pyx_L1:;
  __Pyx_WriteUnraisable("duff.Duffs_next");
  __pyx_L0:;
  return __pyx_r;
}

static PyObject *__pyx_f_4duff_Py_Duffs_next(struct __pyx_t_4duff_Duffs (*__pyx_v_this)) {
  int __pyx_v_i;
  PyObject *__pyx_r;

  /* "C:\user\projects\python\Duffs\duff.pyx":23 */
  Resume();

  /* "C:\user\projects\python\Duffs\duff.pyx":24 */
  for (__pyx_v_i = 0; __pyx_v_i < 10; ++__pyx_v_i) {

    /* "C:\user\projects\python\Duffs\duff.pyx":25 */
    __pyx_v_this->i = __pyx_v_i;

    /* "C:\user\projects\python\Duffs\duff.pyx":26 */
    Yield(0);
    __pyx_L2:;
  }
  __pyx_L3:;

  /* "C:\user\projects\python\Duffs\duff.pyx":27 */
  Endresume();

  __pyx_r = Py_None; Py_INCREF(__pyx_r);
  goto __pyx_L0;
  __pyx_L1:;
  __Pyx_AddTraceback("duff.Py_Duffs_next");
  __pyx_r = 0;
  __pyx_L0:;
  return __pyx_r;
}


Pyrex does not know that Yield() is inserting a return in the middle
of the code. And Py_Duffs_next wants to return a Python object. The code
will not compile.

Lenard Lindstrom
<len-l at telus.net>