[Pyrex] Callbacks from other threads (crash in PyGILState_Release)

[Stefan Behnel]

Hi Geoff,

Geoff Schmidt wrote:
> I'm trying to write a Pyrex extension that calls into a multithreaded  
> C library. I want to expose the ability to register a Python function  
> as a callback with the C library, but the C library creates its own  
> processing threads and delivers callbacks on them -- so I can't  
> guarantee that the callback will happen on the thread that registered  
> the callback, or that it will happen on a thread that Pyrex has even  
> seen before.
> 
> I figured that I just needed to wrap the callback function in  
> PyGILState_Ensure / PyGILState_Release.

That's not enough. That only makes sure you /have/ the GIL, it doesn't allow
you to use threads. You must wrap the sequence where you want to allow threads
with PyEval_SaveThread() and PyEval_RestoreThread() as written in the Python
docs. You then call GIL-ensure/release from the C callback function to get
back the GIL while it's running. Something like

cdef void callback():
    s = PyGILState_Ensure()
    call_python()
    PyGILState_Release(s)

cdef PyThreadState *state
state = PyEval_SaveThread()
function_that_calls_callback_in_thread(callback)
PyEval_RestoreThread(state)


I do that in lxml and it works nicely.

Stefan