[Pyrex] [Cython-dev] Callbacks from threads and PyGILState_Ensure/PyGILState_Release

[Stefan Behnel]

Greg Ewing wrote:
> Stefan Behnel wrote:
>> There is no way Pyrex/Cython can
>> always make sure it knows if the GIL is held by the calling code. It's C after
>> all, things can get called from all over the place.
> 
> The programmer would be responsible for making the
> correct declarations when dealing with external C
> code, as always. But as long as those declarations
> correctly reflect the behaviour of the external
> code, Pyrex should be able to check that what the
> programmer does with the GIL makes sense.
> 
> Perhaps I can give an example of how I see it working.
> Suppose you have a C library with a call that does
> some heavy work and takes a callback. You want to
> release the GIL around calls to this function.
> 
> As a first attempt, the programmer writes
> 
>    # Version 1
> 
>    cdef extern from "heavy.h":
>      ctypedef void (*callback)(void *)
>      void do_something_heavy(callback)
> 
>    cdef void my_callback(void *data):
>      print "Called back!"
> 
>    def heavy():
>      without GIL:
>      	do_something_heavy(my_callback)
> 
> Now Pyrex notices that do_something_heavy() is being
> called with the GIL released, but hasn't been declared
> as safe to do so, and complains.

Ok, that's a simple example as you are passing the callback pointer *outside*
of the GIL context *directly* into the function. So you might succeed in
making Pyrex smart enough to complain about *some* problems. But how would
Pyrex deal with this:

   cdef class _Context:
       cdef someStruct* _struct
       def __init__(self):
           self._struct = extlib.newSomeStruct()
           self._struct.callback = callback
           self._struct.callbackData = <void*>self

       cdef int calcSomething(self):
           return True

   cdef int callback(void* context):
       return <_Context>context.calcSomething()

   def heavy():
       cdef _Context context
       context = _Context()
       without GIL:
           do_something_heavy(context._struct)

Back to my point that Pyrex can't know what's going on.


> We can take this further. Suppose instead of calling
> the external function directly, the programmer wants to
> do some of the calculation in Pyrex, using pure C
> code, still with the GIL released:
> 
>    # Version 4
> 
>    cdef void very_heavy():
>      cdef int i
>      for i in 0 <= i < 10:
>        do_something_heavy(my_callback)
> 
>    def heavy():
>      without GIL:
>        very_heavy()
> 
> Whoops - Pyrex complains again, because we haven't declared
> very_heavy() safe to call without the GIL:
> 
>    # Version 5
> 
>    cdef void very_heavy() nogil:
>      cdef int i
>      for i in 0 <= i < 10:
>        do_something_heavy(my_callback)
> 
> Now at this point, Pyrex sees that nothing in the function
> body requires the GIL -- only C variables and operations
> are used, and the only function called is declared safe
> to call without the GIL. So it doesn't generate any code
> to acquire the GIL in this function.
> 
> Looking back, the *only* thing the programmer had to
> be trusted to get right was the two external declarations.
> All the rest was checked automatically by the compiler,
> and it was able to optimise away GIL acquiring code
> when it wasn't needed.
> 
> I hope this gives a better idea of what I have in mind.

I like the idea, but you should not try to make it perfect. There will always
be code where Pyrex just can't know what's going on, so if you make it too
smart, it will start generating code that crashes and the programmer will have
a hard time to make it work. So we need at least both, "nogil" and "with GIL",
to override the default behaviour of Pyrex explicitly.

Stefan