[Pyrex] Pyrex idioms and optimizations?

[Robert Bradshaw]

On Jul 11, 2007, at 7:35 PM, skip at pobox.com wrote:

>
> Way back in 2004 in a thread on optimization Greg wrote:
>
>     I'm quite happy to forego the dynamicity of builtins in order to
>     optimise this sort of thing. I'm planning to pre-declare some  
> of the
>     builtins as being equivalent to certain API calls, e.g. len()  
> would be
>     equivalent to PyObject_Size().
>
> Googling for "pyrex optimization" doesn't turn up much.  I'm wondering
> what's improved since 2004.  I did come across Paul Prescod's 2004  
> PyCon
> slides on the subject.

As part of the SAGE project, and due to the difficulty in getting  
some essential (to us) patches upstream, we have been keeping a  
separate branch of Pyrex named SageX. It's just Pyrex + (a bunch of  
patches), about many of which are optimizations (including the one  
mentioned above, which is a 10-fold speedup).

> Are there some standard idioms that represent the "best" way to recode
> certain common Python idioms?  For example, I know that list  
> comprehensions
> aren't supported, so I might need to use a loop:
>
>     mylist = []
>     for x in anotherlist:
>         mylist.append(x.someattr)
>
> That seems like a pretty common sort of thing to do.  Is there a  
> "best" way
> to do that in Pyrex?  Ideally, I'd like to avoid the method lookup  
> all the
> time.  Something like this?
>
>     mylist = []
>     for x in anotherlist:
>         PyList_Append(mylist, x.someattr)

SageX supports list comprehension fully (essentially expanding it to  
the above code). It also has special code to loop over lists  
(detected at runtime) quicker than using the iter() interface that  
Pyrex would use in the above examples.

> I guess I'm still thinking like a Python programmer and not quite  
> like a C
> programmer.  I keep thinking if I drop into the C API I might as  
> well just
> program in C.

There's refcounting and exception handling and a whole bunch of other  
magic that Pyrex takes care of for you. Many thanks to Greg for  
creating such an amazing tool--takes almost all the headaches out of  
writing extension types.

> Still, it seems that if I mylist is a local vrbl and I never
> rebind it in the loop Pyrex should be able to do the PyList_Append  
> trick for
> me under the covers.

That's our philosophy too. Another big one is indexing into tuples/ 
lists--one can do much better than a generic __getitem__ call. If the  
object is a tuple/list and the index is a cdef int (long, ...) within  
the correct range, then the PyList_GET_ITEM/PyTuple_GET_ITEM macros  
are called instead. This is a 7-fold speedup even with the overhead  
of checking the type at runtime (with only 1-3% slowdown for one of  
the generic cases).

Some other people have been using it as a standalone, and it works  
fine for that too. The latest version is here

http://sage.math.washington.edu/home/robertwb/sagex/

(to be included in SAGE 2.7). An spkg is just a bzipped tar. It's the  
only thing you need, extract with tar -xjvf sagex-20070710.spkg

- Robert