[Pyrex] Can pyrex codes call python objects/methods such as String, re, datetime, os diretively

[Francesc Altet]

El dt 03 de 07 del 2007 a les 11:39 +1200, en/na Greg Ewing va escriure:
> Andreas Kostyrka wrote:
> 
> > Well, my experience shows that if one knows the exact Python types
> > returned, one can gain visible speedups (if not drastic) by using the
> > specifiy Python type functions, instead of the more generic ones that
> > Pyrex would generate.
> 
> Yes, you could try using things like PyList_GetItem and
> PyTuple_GetItem, but I wouldn't expect these to be much
> faster than using indexing notation, which already has
> a pure-C path to the underlying code.
> 
> In this particular case, I expect that almost all the
> slowness is due to the Oracle interface having to
> construct Python objects representing the results from
> the query. To get any spectacular speedup, you will
> have to find a way of bypassing all that.

Yes. This has been my experience as well. In that sense, I've bypassed
the cost of creating a Python object for each result from the query by
using NumPy containers.  NumPy let you create arbitrarily large
containers for heterogeneous data on a very fast way by using
numpy.empty() (just the cost of calling malloc() and populating the
header of the object; a few hundreds of us all in all).  The only
drawback is that you need to know in advance the number of results to
build the container, but most of databases should provide an API call
for this, so this shouldn't be an stopper at all.

Just as an example on the speed-up that can be achieved by using this
approach, look at the figure 5.6 of:

http://www.pytables.org/docs/manual-2.0rc2/ch05.html#indexedSearches

where it can be noticed that, for large result datasets, the advantage
of using NumPy containers is very important (up to 50x faster).

I expect that this interesting feature of NumPy would be soon discovered
by most of database wrapers developers and that they eventually start to
adopt it (let's hope they will do this more sooner than later).

Cheers,


-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth