[Pyrex] [Cython-dev] Callbacks from threads and PyGILState_Ensure/PyGILState_Release

[Greg Ewing]

Stefan Behnel wrote:
> I was proposing the exact opposite: disallow "with GIL" in .pxd files and in
> function pre-declarations, and only allow them on the function implementation.

Yes, that would work, but it would mean that the
declaration could only be taken advantage of within
the one module. Not a severe limitation, but a
limitation nonetheless.

BTW, we seem to have a terminology clash here --
what you're calling "with GIL" is what I'm calling
"without GIL", since the signature is seen from the
point of view of the calling function.

I'm going to call it "nogil" from here on, because
it's shorter and avoids conflicting with the potential
control structures for acquiring and releasing the
GIL.

> There is no way Pyrex/Cython can
> always make sure it knows if the GIL is held by the calling code. It's C after
> all, things can get called from all over the place.

The programmer would be responsible for making the
correct declarations when dealing with external C
code, as always. But as long as those declarations
correctly reflect the behaviour of the external
code, Pyrex should be able to check that what the
programmer does with the GIL makes sense.

Perhaps I can give an example of how I see it working.
Suppose you have a C library with a call that does
some heavy work and takes a callback. You want to
release the GIL around calls to this function.

As a first attempt, the programmer writes

   # Version 1

   cdef extern from "heavy.h":
     ctypedef void (*callback)(void *)
     void do_something_heavy(callback)

   cdef void my_callback(void *data):
     print "Called back!"

   def heavy():
     without GIL:
     	do_something_heavy(my_callback)

Now Pyrex notices that do_something_heavy() is being
called with the GIL released, but hasn't been declared
as safe to do so, and complains.

So the programmer changes the declarations to:

   # Version 2

   cdef extern from "heavy.h":
     ctypedef void (*callback)(void *) nogil
     void do_something_heavy(callback) nogil

(At this point, the programmer needs to be smart enough
to realise that if he calls do_something_heavy() with
the GIL released, then the callback is going to get
called with the GIL released too, and add a nogil
declaration to it as well.)

Now he tries to compile it again, but this time Pyrex
complains that my_callback is being passed to something
that expects a different function signature. So he
amends it:

   # Version 3

   cdef void my_callback(void *data) nogil:
     print "Called back!"

The nogil declaration does two different things here:
it gives the function the appropriate signature, and it
also triggers the generation of code to acquire the
GIL if necessary around the body.

Now everything okay, and the code compiles.

We can take this further. Suppose instead of calling
the external function directly, the programmer wants to
do some of the calculation in Pyrex, using pure C
code, still with the GIL released:

   # Version 4

   cdef void very_heavy():
     cdef int i
     for i in 0 <= i < 10:
       do_something_heavy(my_callback)

   def heavy():
     without GIL:
       very_heavy()

Whoops - Pyrex complains again, because we haven't declared
very_heavy() safe to call without the GIL:

   # Version 5

   cdef void very_heavy() nogil:
     cdef int i
     for i in 0 <= i < 10:
       do_something_heavy(my_callback)

Now at this point, Pyrex sees that nothing in the function
body requires the GIL -- only C variables and operations
are used, and the only function called is declared safe
to call without the GIL. So it doesn't generate any code
to acquire the GIL in this function.

Looking back, the *only* thing the programmer had to
be trusted to get right was the two external declarations.
All the rest was checked automatically by the compiler,
and it was able to optimise away GIL acquiring code
when it wasn't needed.

I hope this gives a better idea of what I have in mind.

--
Greg