[Pyrex] [Cython-dev] Callbacks from threads and PyGILState_Ensure/PyGILState_Release

[Ulisses Furquim]

Hi,

On 9/14/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> The programmer would be responsible for making the
> correct declarations when dealing with external C
> code, as always. But as long as those declarations
> correctly reflect the behaviour of the external
> code, Pyrex should be able to check that what the
> programmer does with the GIL makes sense.

Can we really check if what the programmer does with the GIL makes
sense all the times?

> Perhaps I can give an example of how I see it working.
> Suppose you have a C library with a call that does
> some heavy work and takes a callback. You want to
> release the GIL around calls to this function.
>
> As a first attempt, the programmer writes
>
>    # Version 1
>
>    cdef extern from "heavy.h":
>      ctypedef void (*callback)(void *)
>      void do_something_heavy(callback)
>
>    cdef void my_callback(void *data):
>      print "Called back!"
>
>    def heavy():
>      without GIL:
>         do_something_heavy(my_callback)
>
> Now Pyrex notices that do_something_heavy() is being
> called with the GIL released, but hasn't been declared
> as safe to do so, and complains.
>
> So the programmer changes the declarations to:
>
>    # Version 2
>
>    cdef extern from "heavy.h":
>      ctypedef void (*callback)(void *) nogil
>      void do_something_heavy(callback) nogil
>
> (At this point, the programmer needs to be smart enough
> to realise that if he calls do_something_heavy() with
> the GIL released, then the callback is going to get
> called with the GIL released too, and add a nogil
> declaration to it as well.)
>
> Now he tries to compile it again, but this time Pyrex
> complains that my_callback is being passed to something
> that expects a different function signature. So he
> amends it:
>
>    # Version 3
>
>    cdef void my_callback(void *data) nogil:
>      print "Called back!"
>
> The nogil declaration does two different things here:
> it gives the function the appropriate signature, and it
> also triggers the generation of code to acquire the
> GIL if necessary around the body.

Ok. I suppose that by "if necessary around the body" I take it that
you mean GIL will be grabbed if "my_callback" accesses any Python data
(or calls Python functions), right?

> Now everything okay, and the code compiles.
>
> We can take this further. Suppose instead of calling
> the external function directly, the programmer wants to
> do some of the calculation in Pyrex, using pure C
> code, still with the GIL released:
>
>    # Version 4
>
>    cdef void very_heavy():
>      cdef int i
>      for i in 0 <= i < 10:
>        do_something_heavy(my_callback)
>
>    def heavy():
>      without GIL:
>        very_heavy()
>
> Whoops - Pyrex complains again, because we haven't declared
> very_heavy() safe to call without the GIL:
>
>    # Version 5
>
>    cdef void very_heavy() nogil:
>      cdef int i
>      for i in 0 <= i < 10:
>        do_something_heavy(my_callback)
>
> Now at this point, Pyrex sees that nothing in the function
> body requires the GIL -- only C variables and operations
> are used, and the only function called is declared safe
> to call without the GIL. So it doesn't generate any code
> to acquire the GIL in this function.
>
> Looking back, the *only* thing the programmer had to
> be trusted to get right was the two external declarations.
> All the rest was checked automatically by the compiler,
> and it was able to optimise away GIL acquiring code
> when it wasn't needed.

Besides the two external declarations, the programmer still had to
call "do_something_heavy" under "without GIL". In this case it might
not be problematic, but you better call some blocking IO function or
some main loop entering function (like g_main_loop_run or other
equivalent) under "without GIL" if you want your multi-threaded
programs to work reliably.

> I hope this gives a better idea of what I have in mind.

Thanks for sharing what you had in mind. It does sound like an
interesting idea but I think it's a little bit confusing as it stands
right now.

Reading all this I thought we might just want something simpler to
start with like 'without GIL' and 'with GIL' block constructions (I
think Robert already said this before). The 'without GIL' block would
add "Py_BEGIN_ALLOW_THREADS" and "Py_END_ALLOW_THREADS" and the 'with
GIL' block would add calls to "PyGILState_Ensure" and
"PyGILState_Release".

The version 3 of your example would be:

    cdef extern from "heavy.h":
      ctypedef void (*callback)(void *)
      void do_something_heavy(callback)

    cdef void my_callback(void *data):
      with GIL:
        print "Called back!"

    def heavy():
      without GIL:
         do_something_heavy(my_callback)

It's everything explicit and if the programmer can be trusted to know
that calling "do_something_heavy" without the GIL will end up calling
the callback in the same context then we can be sure he'll add a 'with
GIL' block inside his callback if it's really necessary.

Maybe we can start adding those block constructions and improve upon
them later after we evaluate the best behavior for Pyrex/Cython?

Best regards,

-- Ulisses