[Pyrex] calling cdef'ed functions between pyrex modules

[Robby Dermody]

Hey guys,

I have a project where there is quite a bit of pyrex code. I've split
the code up into multiple .pyx files. Only one of the .pyx files has a
python interface (in the form of a python class). The rest of the pyrex
files have no python interface (i.e. a bunch of functions defined with
cdef, some module wide cdef'ed variables, .pxd files, no extension types
(cdef class)). For any given one of these pyrex modules, it needs to
make calls to cdef functions in any of the other pyrex modules. I bet
you guys can already see the problem here. :(

So for example, let's say foo1() in foo.pyx needs to call bla1() in
bla.pyx. I've tried the following:

---bla.pyx---
cdef public void bla1(int arg1):
   arg = arg + 1

---foo.pyx---
include "bla.pxi" (or just "cdef extern void bla1(int arg1)" )
cdef public void foo1(int arg1):
   arg = arg + 2
   bla1(arg)

I build this with distutils as two separate extensions and I've got it
building fine, creating two shared object libraries. However, foo can't
be imported at runtime, as the bla1 symbol in it is unresolved, of
course. 

I've also tried building a single extension module with all of the .pyx
files, but as I've read on this list, there are problems in using a
module name that is not the same as the pyrex file name (and as I am
trying to make one with multiple pyrex files it can only be the same as
one at best :). With this approach the thing will actually build, but
when I import it it complains about missing the init<modulename>
function that is generated. So I made one, and in that I called
initbla() and initfoo(), and that didn't work either...the python
interpreter said the module wasn't properly initialized. This method is
probably too much of a hack, I'm guessing.

Basically, I was wondering how I might get this to work. I really don't
want to stick everything in a single .pyx file...it would be a few
thousand lines of pyrex code in a single file...a mess. The optimum
solution would be like if I had a bunch of .c files. I would just
compile them separately into object files and link them up, and
functions in separate files could call each other to their hearts
content. However the hooks pyrex generates for dynamic importing into
python makes this a problem.

I'm guessing that I'll have to encapsulate this C functionality as a
pyrex extension type (in a cdef class construct) and instantiate as
necessary, so that I can call the external functions fine. But what
about the performance of this solution? Most, if not all of this needs
to remain straight C code running...very little python running in
anything but where I explicitly make use of it. What are the performance
implications of using an extension type made from one pyrex module from
within another pyrex module? Are there any other that could be
considered?

Thanks for any help,

Robby