[Pyrex] [Cython-dev] Callbacks from threads and PyGILState_Ensure/PyGILState_Release

[Stefan Behnel]

Gustavo Sverzut Barbieri wrote:
> On 9/15/07, Stefan Behnel wrote:
>> making Pyrex smart enough to complain about *some* problems. But how would
>> Pyrex deal with this:
>>
>>    cdef class _Context:
>>        cdef someStruct* _struct
>>        def __init__(self):
>>            self._struct = extlib.newSomeStruct()
>>            self._struct.callback = callback
>>            self._struct.callbackData = <void*>self
>>
>>        cdef int calcSomething(self):
>>            return True
>>
>>    cdef int callback(void* context):
>>        return <_Context>context.calcSomething()
> 
> At this point you're doing operations that are dependent on the
> interpreter, so it would wrap with code to acquire to GIL.

I hope it wouldn't, because I never said this function was meant to be called
without GIL context, how should Pyrex know?

Greg uses words like "triggers the generation of code to acquire the
GIL if necessary around the body." The "if" is futile. It will always have to
do that, as it can't know who calls the function in what context.


> Check Greg's example on the semantic required for the "nogil", that
> you'd have to use for "callback" definition.
> 
>>    def heavy():
>>        cdef _Context context
>>        context = _Context()
>>        without GIL:
>>            do_something_heavy(context._struct)
>>
>> Back to my point that Pyrex can't know what's going on.
> 
> yes it can, just check Greg's example. Callback would need to be
> defined with "nogil".

I'm not questioning that I, as a programmer, can decide what I need to
annotate with "nogil" or "with GIL". I'm just questioning that Pyrex can tell
me where I need to do it and that it can find out in the example above that
the annotation is required. And even if I annotate it, Pyrex must still figure
out if it needs to acquire the GIL or not. We're fighting Turing here. And if
Pyrex can't figure it out, the benefit of making it smarter vanishes behind
the work required to make it "smart enough".

Stefan