[Pyrex] Why were .pxd files invented?

[Leif Strand]

Hi,

I've been working with Pyrex for some time now (even hacking the Pyrex 
source), but only just yesterday starting using .pxd files in a big way: 
to share extension types between modules -- as documented here:

"Sharing Extension Types"
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/sharing.html#SharingExtensionTypes

Already, I'm annoyed by .pxd files :-) First, I noticed a lot of 
redundancy between my .pxd files and my .pyx files (I'm defining many 
cdef functions). Then the thought occurred to me: "these .pxd files 
could be generated from the corresponding .pyx files". Then my next 
thought was, "Why generate anything? Pyrex's 'cimport' statement should 
read the .pyx files directly."

I can't figure out why .pxd files were invented. They obviously mimic 
the old C convention of splitting code into interface and implementation 
parts (.h and .c files). But C only does it that way for historical 
reasons: the preprocessor is completely disconnected from the compiler 
(and therefore the language). The preprocessor completely flattens 
whatever structure was defined by the #include directives, handing the 
result to the compiler as one big blob. So the C compiler has absolutely 
no context to work with; if it sees

   void foo() { ... }

it must assume it is a definition of 'foo', and therefore generate code 
for it.

But Pyrex has no historical baggage. So when it sees

    cimport Shrubbing

why not search for the Shrubbing.pyx file, parse it, and process it in 
"cimport mode" or "cimport context", in which it filters-out 
declarations that aren't relevant in that context? For example:

#--snip--

cdef class Shrubbery:

    cdef int width
    cdef int length

    def __new__(self, int w, int l):
        self.width = w
        self.length = l

    def trim(self):
        self.width = self.width - 1

    cdef void water(self):
        self.width = self.width + 1

#--snip--

In the context of a 'cimport' statement, Pyrex could ignore the 'def' 
methods and simply pick-out the 'cdef' attributes and methods that would 
be ordinarily placed in a .pxd file: 'width', 'length', and 'water'.

C/C++ programmers are bound to object that combining interface and 
implementation is bad. But the right way to do this is to introduce 
"interface" and "implementation" concepts into the language itself 
(defined in a single file, like Pascal units) -- not force the 
programmer to maintain two separate files. In any case, .pxd files 
weren't designed to give you an interface/implementation concept to 
begin with: all 'cdef' stuff is public (cdef declarations must appear in 
the "interface"), you can't have "private" cdef attributes or methods. 
(So perhaps I'm confusing the issue by even bringing it up.)

I always hated keeping my .h files in-sync when working in C/C++; I hate 
to see that old crufty way of doing things jump the gap into a fresh, 
new language like Pyrex...

--Leif