[Pyrex] python dictionaries with unsigned long long keys and values

[Robert Bradshaw]

Depending on what you're doing, reading/writing dictionaries may not  
be your main slowdown. For instance, if I write the pyrex functions

def create_dict(n):
     dict = {}
     cdef unsigned long long i
     for i from 0 <= i < n:
         dict[i] = None
     return dict

def read_dict(dict, n, times):
     cdef unsigned long long i
     for _ from 0 <= _ < times:
         for i from 0 <= i < n:
             z = dict[i]

Then do

time dict = create_dict(1000000)
CPU time: 0.33 s,  Wall time: 0.35 s

time read_dict(dict, 1000000, 10)
CPU time: 1.36 s,  Wall time: 1.41 s

That's 3 million stores/sec, and 7 million reads/sec on my machine.  
Not too shabby for really clean pythonic code. I'm guessing somewhere  
around 20% of this time is spent in object creation (cdef int <-->  
python int), so if you left your ints as python objects for a bit  
where they were used multiple times, it could be even faster.

Of course, doing some custom/library c-level uint -> void map could  
be faster, it all depends on what the bottleneck of your code is.

- Robert


On Mar 14, 2007, at 4:05 PM, Greg Ewing wrote:

> Sam Tannous wrote:
>> From what
>> I can gather, any access to a dictionary (whether created in  
>> python or pyrex
>> code) slows everything down to pure python speeds.
>
> That's correct. You can gain some speed by using
> various PyDict_* functions directly, but all the
> keys and values still have to be Python objects,
> so there's a limit to what you can achieve.
>
> If you need a blazingly fast mapping from
> integers to integers, a Python dictionary just
> isn't going to give it. You'll have to implement
> your own C-level data structure.
>
> --
> Greg
>
> _______________________________________________
> Pyrex mailing list
> Pyrex at lists.copyleft.no
> http://lists.copyleft.no/mailman/listinfo/pyrex