[Pyrex] [Cython-dev] Callbacks from threads and PyGILState_Ensure/PyGILState_Release

[Gustavo Sverzut Barbieri]

On 9/15/07, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Greg Ewing wrote:
> > Stefan Behnel wrote:
> >> There is no way Pyrex/Cython can
> >> always make sure it knows if the GIL is held by the calling code. It's C after
> >> all, things can get called from all over the place.
> >
> > The programmer would be responsible for making the
> > correct declarations when dealing with external C
> > code, as always. But as long as those declarations
> > correctly reflect the behaviour of the external
> > code, Pyrex should be able to check that what the
> > programmer does with the GIL makes sense.
> >
> > Perhaps I can give an example of how I see it working.
> > Suppose you have a C library with a call that does
> > some heavy work and takes a callback. You want to
> > release the GIL around calls to this function.
> >
> > As a first attempt, the programmer writes
> >
> >    # Version 1
> >
> >    cdef extern from "heavy.h":
> >      ctypedef void (*callback)(void *)
> >      void do_something_heavy(callback)
> >
> >    cdef void my_callback(void *data):
> >      print "Called back!"
> >
> >    def heavy():
> >      without GIL:
> >       do_something_heavy(my_callback)
> >
> > Now Pyrex notices that do_something_heavy() is being
> > called with the GIL released, but hasn't been declared
> > as safe to do so, and complains.
>
> Ok, that's a simple example as you are passing the callback pointer *outside*
> of the GIL context *directly* into the function. So you might succeed in
> making Pyrex smart enough to complain about *some* problems. But how would
> Pyrex deal with this:
>
>    cdef class _Context:
>        cdef someStruct* _struct
>        def __init__(self):
>            self._struct = extlib.newSomeStruct()
>            self._struct.callback = callback
>            self._struct.callbackData = <void*>self
>
>        cdef int calcSomething(self):
>            return True
>
>    cdef int callback(void* context):
>        return <_Context>context.calcSomething()

At this point you're doing operations that are dependent on the
interpreter, so it would wrap with code to acquire to GIL.

Check Greg's example on the semantic required for the "nogil", that
you'd have to use for "callback" definition.


>    def heavy():
>        cdef _Context context
>        context = _Context()
>        without GIL:
>            do_something_heavy(context._struct)
>
> Back to my point that Pyrex can't know what's going on.

yes it can, just check Greg's example. Callback would need to be
defined with "nogil".


> > We can take this further. Suppose instead of calling
> > the external function directly, the programmer wants to
> > do some of the calculation in Pyrex, using pure C
> > code, still with the GIL released:
> >
> >    # Version 4
> >
> >    cdef void very_heavy():
> >      cdef int i
> >      for i in 0 <= i < 10:
> >        do_something_heavy(my_callback)
> >
> >    def heavy():
> >      without GIL:
> >        very_heavy()
> >
> > Whoops - Pyrex complains again, because we haven't declared
> > very_heavy() safe to call without the GIL:
> >
> >    # Version 5
> >
> >    cdef void very_heavy() nogil:
> >      cdef int i
> >      for i in 0 <= i < 10:
> >        do_something_heavy(my_callback)
> >
> > Now at this point, Pyrex sees that nothing in the function
> > body requires the GIL -- only C variables and operations
> > are used, and the only function called is declared safe
> > to call without the GIL. So it doesn't generate any code
> > to acquire the GIL in this function.
> >
> > Looking back, the *only* thing the programmer had to
> > be trusted to get right was the two external declarations.
> > All the rest was checked automatically by the compiler,
> > and it was able to optimise away GIL acquiring code
> > when it wasn't needed.
> >
> > I hope this gives a better idea of what I have in mind.
>
> I like the idea, but you should not try to make it perfect. There will always
> be code where Pyrex just can't know what's going on, so if you make it too
> smart, it will start generating code that crashes and the programmer will have
> a hard time to make it work. So we need at least both, "nogil" and "with GIL",
> to override the default behaviour of Pyrex explicitly.

I agree with this, we really should do the simple things first,
evaluate the usage and then try to improve common cases.

"with GIL" function signature or a block (following python 2.5) would
be great. Using the block approach would be more flexible and even
easier to use because it's clear it's about implementation, but the
signature approach is more "safe", since users can just mark their
callbacks and everything will be handled fine.

-- 
Gustavo Sverzut Barbieri
--------------------------------------
Jabber: barbieri at gmail.com
   MSN: barbieri at gmail.com
  ICQ#: 17249123
 Skype: gsbarbieri
Mobile: +55 (81) 9927 0010