[Pyrex] Namespaces, modules and includes - oh my!

[Gavin Baker]

Hello all,

I'm using Pyrex in an industrial processing application.  We're very
impressed with the ease in which we can write extension modules, and
mix C and Python.  Thanks to Greg & co for a great tool!

In my working with Pyrex, I have come up against a few difficulties,
and overcome most of them.  But the latest highlights something which
seems to be a problem with namespaces and type resolution.

In a nutshell, the problem is that types included via "include" in two
separate modules are not regarded as the same when one module calls
cdef methods that use these included types in the other.

I have written a test case based on the problem that has arisen in our
app, with no dependencies but showing the structure that highlights
the problem.  The full source is attached (tested against Pyrex 0.9.3
with Python 2.3).

The problem has a shape a little fairly similar to the Shrubbery
example showing shared extensions, but the problem only arises when
declaring types that are included by both modules.

So - say first that you have a C API for something you want to use in
Python (in this example we use stdio and FILE* as everyone has that). 
You first write a .pxi that declares all the stuff you want to use to
Pyrex.

cdef extern from "stdio.h":
    ctypedef struct FILE:
        int _fileno
    cdef FILE *fopen(char* fn, char* mode)

Then you write a pyrex extension that wrappers this into a nice
extension class, lets call it Buffer (with the appropriate .pxd and
.pyx).

include "stdio.pxi"

cdef class Buffer:
    cdef FILE *thing
    cdef FILE *get_thing(self)
    cdef void set_thing(self, FILE *fp)
    # ...

Now another extension class wants to create instances of this, but use
the C API to manipulate stuff directly.  So this module imports the C
API and the Buffer module, thus:


include "stdio.pxi"
cimport buf
import buf

cdef class BufferTestFails:
    cdef buf.Buffer buf
    def __init__(self):
        self.buf = buf.Buffer('/etc/passwd')

So far so good...  we can call the cdef'ed methods from the Buffer
class, provided we are careful to specify the module name:

   def test1(self):

        # Call normal method
        self.buf.get_fd()

        # Call C method
        cdef buf.FILE *p
        p=self.buf.get_thing()

But since we have included the .pxi we can use the regular names,
since we have them in our module's namespace:

    def test3(self):
        cdef FILE *p
        p = fopen('/etc/passwd','r')

But what happens if we try to mix the two?

    def test4(self):
        cdef FILE *p
        p = fopen('/etc/passwd','r')
        self.buf.set_thing(p)

die.pyx:39:27: Cannot assign type 'FILE (*)' to 'FILE (*)'

We know that the FILE* in our current module is the same as the FILE*
from the included extension.  They are identical in the C code that is
generated.  Yet Pyrex cannot resolve this, because the declarations
are enclosed by two different namespaces.

The implication of this seems to be that if you want to use C code in
Pyrex when sharing between multiple extension modules, you need to
"nominate" one Pyrex module to "own" the C interface, and only include
the declarations in that one module.  Any other modules wishing to use
the same C calls while interfacing with this owning module should not
include the .pxi declaring the C API but instead use the owning
module's namespace to refer to any of the C API calls or types.

I hope the above has made some sense, for those of you who made it
this far... Having dug around a bit in the code, it looks like C type
compatability is determined by identity ("is") between PyrexType
instances.

I suspect the above problem could be resolved by making CTypes
equivalent if the base typedef name is equal (by string comparing). 
Thus in the above example,

  assignable_from_resolved_type(buf.FILE*, FILE*) -> True

Does that make sense?  What do people think?

Thanks -

  :: Gavin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pyxmodshare.tar.gz
Type: application/unix-tar
Size: 900 bytes
Desc: not available
Url : http://lists.copyleft.no/pipermail/pyrex/attachments/20050114/40717b8f/pyxmodshare.tar.bin