[Pyrex] pickling pyrex classes

[William Stein]

On Monday 26 March 2007 12:43 pm, Robert Kern wrote:
> Brian Blais wrote:
> > Hello,
> >
> > I have a class defined in Pyrex (to get speed increases), and python
> > complains that is can't be pickled (class def included below).  I looked
> > the documentation, and see that I can define __setstate__ and
> > __getstate__ methods to pickle only certain methods/class variables. 
> > Unfortunately, I am not clear on which ones are causing the problem, so I
> > figured I'd ask on the list before doing trial-and-error.  I imagine it
> > might be the "double *" , and perhaps the cdef methods, but I am not
> > sure.
>
> There's no reliable way for the pickle machinery to introspect extension
> type instances like it does pure Python instances. Consequently, I believe
> that it simply throws up its hands unless if you have defined the
> appropriate protocol methods. No particular attribute is causing a problem,
> I don't think.

Here are some further remarks on how to define the appropriate protocol 
methods.

In the SAGE codebase (http://www.sagemath.org), which contains well over 
35,000 lines of Pyrex, we do a huge amount of pickling of Pyrex types.   Long 
ago, after many many hours of frustration, I learned that there are two key 
things to keep in mind when pickling types defined using Pyrex:

   (1) Use the cPickle module with protocol 2, e.g.,
                import cPickle
                cPickle.dumps(my_object, protocol=2)
       Using a different protocol, e.g., the default, is an exercise in 
       suffering and frustration.

   (2) Define the __reduce__ method (setstate and getstate are for pure 
       Python classes).  Here's a typical example (for gmp integers, by 
       the way):
    def __reduce__(self):
        # This is the *trick* needed to pickle pyrex extension types.
        # The trick is that you must put a pure Python function
        # as the first argument, and that function must return
        # the result of unpickling with the argument in the second
        # tuple as input. All kinds of problems happen
        # if we don't do this.
        return sage.rings.integer.make_integer, (self.str(32),)

In this case, make_integer is (you have to see the relevant files to know
what the Integer and PY_NEW things are, but they aren't really relevant):

        def make_integer(s):
           cdef Integer r
           r = PY_NEW(Integer)
           r._reduce_set(s)
           return r

 -- William