[Pyrex] Callbacks from threads and PyGILState_Ensure/PyGILState_Release

[Robert Bradshaw]

On Sep 11, 2007, at 5:11 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>> The with GIL keyword causes the function to be generated such that  
>> it  acquires the GIL at the top of the function, and releases it  
>> just  before returning. It is the called, rather than the calling,  
>> function  that does the work, and hence one can call either or  
>> both and they'll  do the right thing.
>
> But all your the *other* functions, the ones that *aren't*
> declared as "with GIL", if they touch any Python data, will
> crash if they are ever called without the GIL. In particular,
> they will crash if used as one of these callbacks that we
> needed the GIL-acquiring function for in the first place.

OK, currently the only time one cares about about signature matching  
is overriding cdef class methods, and they all incref the parameter  
(currently), so that means the situation I was thinking of is mostly  
moot. However, I think it should not be compared for purposes of  
assignment (nor should exception values?). Other than that, is there  
any reason to not force them to match?

> Pyrex doesn't strictly *need* to check this, it can
> obviously be left up to the programmer. But the Pyrex
> philosophy is to make things as foolproof as possible
> when it comes to dealing with the Python/C API, and
> I'm trying to design this feature so that future
> implementations can build on it.

I agree 100% with this philosophy.

>
>> It could be wasteful to acquire the GIL for functions that don't  
>> need  it.
>
> That's exactly the point of the check. To do any better
> than just trusting the programmer, Pyrex needs to know
> when GIL acquisition is needed.

I think noticing a function is being used as a callback is inference  
that only the programer can decide.

>
> If you're only using GIL-free functions as callbacks
> passed to external C code, then Pyrex won't need to
> generate any GIL code outside of the function itself.
> But suppose we wanted to introduce a control structure
> such as
>
>    without GIL:
>      # Do some C operations in here that take
>      # a long time
>
> If you weren't careful and accidentally called some
> function you wrote in Pyrex that touched Python data
> in the body, bad things would happen. The check would
> prevent mistakes like that. It would also make it
> possible for Pyrex to automatically generate code to
> acquire the GIL around such calls.

If someone uses a (hypothetical) construction "without GIL" I would  
throw an error rather than wrap the offending code in that block.  
This is an interesting idea... I'm seeing how the "with GIL" flag  
fits into it though.

- Robert