[Pyrex] Callbacks from other threads (crash in PyGILState_Release)

[Geoff Schmidt]

Good morning fellow Pyrexians,

I'm trying to write a Pyrex extension that calls into a multithreaded  
C library. I want to expose the ability to register a Python function  
as a callback with the C library, but the C library creates its own  
processing threads and delivers callbacks on them -- so I can't  
guarantee that the callback will happen on the thread that registered  
the callback, or that it will happen on a thread that Pyrex has even  
seen before.

I figured that I just needed to wrap the callback function in  
PyGILState_Ensure / PyGILState_Release. When I do that, the Python  
code in the callback does run, but the process segfaults in the call  
to PyGILState_Release! It turns out that the minimal case of creating  
a thread with pthread_create and then calling PyGILState_Ensure /  
PyGILState_Release in the new thread is sufficient to get the crash  
(100% of the time.)

I took a look at the crash and as best I can tell the global  
head_mutex in pystate.c is getting corrupted somewhere in the process  
of PyThreadState_DeleteCurrent trying to take the tstate out of the  
linked list of tstates, but I don't know anything about this code and  
it's not obvious to me what's going on. I posted a little more detail  
to comp.lang.python.

Is this a bug in Python? Is there a way around it (short of leaving a  
Python-blessed thread blocking on a condition variable and passing  
Python functions to it to call?) Or am I just doing it wrong?

This is the official 2.4.3 build under OS.X 10.4.6 on PPC. Here's a  
test case. I run it as 'setup foo.pyx build_ext --inplace', 'python',  
'import foo', 'foo.callFuncInThread()'. This crashes. On the other  
hand, 'foo.callFuncDirectly()' does not. In the real program, the  
callback trampoline and the PyGILState_XXX are in pure C stub code  
and it still crashes, so it doesn't appear to be a problem with Pyrex  
emitting initialization code before the thread state is set up, which  
was the other mention I saw in the list archives.

--- snip (foo.pyx) ---
cdef extern from "stdio.h":
     int printf(char *str, ...)

cdef extern from "Python.h":
     ctypedef int PyGILState_STATE
     PyGILState_STATE PyGILState_Ensure()
     void PyGILState_Release(PyGILState_STATE gstate)

cdef extern from "pthread.h":
     ctypedef void *pthread_t # it'll do
     int pthread_create(pthread_t *thread, void *attr,
                        void *(*start_routine)(void *), void *arg)

cdef extern void *func(void *x):
     printf("Entering func(%p)\n", x)

     cdef PyGILState_STATE st
     printf("PyGILState_Ensure\n")
     st = PyGILState_Ensure()
     printf("PyGILState_Release\n")
     PyGILState_Release(st)
     printf("Leaving func\n")

def callFuncDirectly():
     func(NULL)

def callFuncInThread():
     cdef pthread_t thr
     pthread_create(&thr, NULL, func, NULL);

--- snip (setup.py) ---

from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext

setup(
     name = 'foo',
     ext_modules = [Extension("foo", ["foo.pyx"])],
     cmdclass = {'build_ext': build_ext},
     )

--- end ---

Thank you very much in advance for any help or pointers.

Geoff Schmidt
gschmidt at gschmidt.org