[Pyrex] Pyrex method 10x slower than Python, any hints why?

[John Machin]

On 1/06/2006 8:40 AM, David Chiang wrote:
> Hello all, when I compare this method
>    
>      def estimate(self, r):
>          sum = 0.0
>          for i in xrange(r.e.arity()+1):
>             cost = lookup_words(self.ngram, None, [r.e.getchunk(i)], 1, 0, self.mapdigits, self.ceiling) 
>              sum = sum + cost
>                                                                                   
>          return sum

I would have expected that it would not be worth the bother of rewriting 
that function in Pyrex -- e.g. cdef'ing i and cost doesn't help much as 
the values of i and cost have to be converted each time around the loop. 
Instead, I would experiment with tweaking up the Python somewhat, like 
below ... however again the time saved will probably be minute compared 
to the cost of just calling lookup_words() plus whatever is going on 
inside it.

def estimate(self, r):
     total = 0.0 # sum is a built-in function
     ngram = self.ngram
     getchunk = r.e.getchunk
     mapdigits = self.mapdigits
     ceiling = self.ceiling
     for i in xrange(r.e.arity()+1):
         total += lookup_words(
             ngram, None, [getchunk(i)], 1, 0, mapdigits, ceiling
             )
     return total
>    
>    against a Pyrex equivalent
>    
>        def estimate(self, rule.Rule r):
>            cdef float sum, cost
>            cdef int i
>            sum = 0.0
>            for i from 0 <= i <= r.e.arity():
>               cost = lookup_words(self.ngram, None, [r.e.getchunk(i)], 1, 0, self.mapdigits, self.ceiling) 
>                sum = sum + cost
>                                                                                   
>            return sum
>  
>  where the containing class is a user class and self.ngram, r, and r.e are instances of extension types, the Python version runs, it would seem, 10 times faster. I've looked a bit at the generated C code (attached) and didn't see anything obviously wrong; does anyone have any hints about what may be going on? I realize this is kind of a vague question but any pointers for further testing would be appreciated. Thanks in advance!
>    

I've had a quick look at the C code, but can't see anything obvious. You 
may wish to apply dis.dis() to your Python method and see what Python is 
doing. Another approach might be to start with a Pyrex method that is 
exactly the same as your Python method, measure it, then introduce Pyrex 
changes one at a time.

HTH,
John