[Pyrex] Callbacks from other threads (crash in PyGILState_Release)

[Geoff Schmidt]

Stefan,

thank you very much for your response. It's very encouraging to know  
that other people have been successful with this :)

> That's not enough. That only makes sure you /have/ the GIL, it  
> doesn't allow
> you to use threads. You must wrap the sequence where you want to  
> allow threads
> with PyEval_SaveThread() and PyEval_RestoreThread() as written in  
> the Python
> docs. You then call GIL-ensure/release from the C callback function  
> to get
> back the GIL while it's running.

My understanding of the docs was that PyEval_SaveThread effectively  
suspend the thread (from Python's perspective) so that another  
function had a chance to gain access to the interpreter by calling  
PyEval_RestoreThread or PyGILState_Ensure, and that PyGILState_Ensure/ 
Release was basically PyEval_Save/RestoreThread plus logic to create  
and destroy thread states if necessary. Is this understanding correct?

> cdef PyThreadState *state
> state = PyEval_SaveThread()
> function_that_calls_callback_in_thread(callback)
> PyEval_RestoreThread(state)

My function_that_calls_callback_in_thread() returns immediately, so I  
didn't think PyEval_Save/RestoreThread was necessary -- by the time  
the callback gets called, we're back in the interpreter, which (IIRC)  
is yielding to other threads with Save/Restore every few hundred  
bytecodes, giving the PyGILState_Ensure in my callback a chance to  
take the lock.

I tried adding the extra PyEval_Save/RestoreThread that you show in  
your example, but I still get the same segfault. The revised code is  
below. Could you take a look and tell me if I understood your advise?

--- cut ---
cdef extern from "stdio.h":
     int printf(char *str, ...)

cdef extern from "Python.h":
     ctypedef int PyGILState_STATE
     PyGILState_STATE PyGILState_Ensure()
     void PyGILState_Release(PyGILState_STATE gstate)
     struct PyThreadState:
         pass
     PyThreadState *PyEval_SaveThread()
     void PyEval_RestoreThread(PyThreadState *state)

cdef extern from "pthread.h":
     ctypedef void *pthread_t # it'll do
     int pthread_create(pthread_t *thread, void *attr,
                        void *(*start_routine)(void *), void *arg)

cdef extern void *func(void *x):
     printf("Entering func(%p)\n", x)

     cdef PyGILState_STATE st
     printf("PyGILState_Ensure\n")
     st = PyGILState_Ensure()
     printf("PyGILState_Release\n")
     PyGILState_Release(st)
     printf("Leaving func\n")

def callFuncDirectly():
     func(NULL)

def callFuncInThread():
     cdef PyThreadState *state
     state = PyEval_SaveThread()

     cdef pthread_t thr
     pthread_create(&thr, NULL, func, NULL);

     PyEval_RestoreThread(state)
--- cut ---

Sample output:
Python 2.4.3 (#1, Apr  7 2006, 10:54:33)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import foo
 >>> foo.callFuncInThread()
 >>> Entering func(0x0)
PyGILState_Ensure
PyGILState_Release
Bus error

FWIW, this shows that callFuncInThread() is in fact returning to the  
interpreter before the callback triggers.

thanks again,
Geoff Schmidt
gschmidt at gschmidt.org