[Pyrex] FW: Newbie question & Error Found in Tutorial
Grant McDonald
gmcdonald at infocomp.com
Tue Nov 29 05:49:49 CET 2005
Vanitha,
Let me start by reposting the offending code:
cdef class Foo:
"""A foo, like any other.
Create one with Foo(s), where s is the name to assign to your
foo."""
cdef char *name
def __init__(self, name):
self.name = name
def__repr__(self):
return "A Foo named %s." % (self.name)
The problem with this code lies in the fact that self.name is a c variable
and as such doesn't appear to Python's garbage collector. First I'll give
you a bit of background: when you are assigning a python string to a char
pointer it actually sets the char pointer to the memory containing the
python string -> it does not create a copy of it. So if you have a char
pointer that references a Python string that has been garbage collected
you're pointing to invalid memory.
The above fact is demonstrated by what you are experiencing:
>>> import foo
>>> f = foo.Foo('Vanitha')
>>> f
A Foo named stdout.
>>> f
A Foo named <stdin>.
>>> f
A Foo named global.
>>>
Because the memory 'Vanitha' points to is garbage collected after line 2 the
memory it references is now occupied by something else. I'll explain what
happens on line 2 in more detail:
>>> f = foo.Foo('Vanitha')
What this does in Python is the following (briefly without ref count
information):
1) String constant 'Vanitha' is turned into a Python string object and is
passed to the Foo constructor
2) a Foo object is created
3) in the __init__ method the string object is bound to the local variable
'name'
4) self.name (a char pointer) is set to point to the memory occupied by the
Python string bound to 'name'
5) local binding to 'name' removed
5) back in the main statement, the foo object is bound to a local variable f
6) since there are no longer any references to the string 'Vanitha' the
python garbage collector will at some point free the Python object and the
memory it occupied will be reclaimed.
This has the effect that once the string 'Vanitha' is garbage collected the
char pointer points to something else. It is simply fortuitous that this
doesn't result in a seg fault. To further demonstrate this consider the
following code snippet:
>>> import foo
>>> name = 'Vanitha'
>>> f = foo.Foo(name)
>>> f
A Foo named Vanitha.
>>> f
A Foo named Vanitha.
>>> f
A Foo named Vanitha.
>>> f
A Foo named Vanitha.
>>>
In this instance we don't see the error because we are binding the 'Vanitha'
string to a local variable called name. Since there is still a reference to
it the char pointer doesn't become invalid. If we then type:
>>> del name
>>> f
A Foo named <stdin>.
>>>
We see the problem manifested again because 'del name' removes the reference
to name and it is then garbage collected. So if you are wanting to store
python string data in c variables persistently then you will need to
allocate enough memory to hold the string and then strcpy the python object.
The solution is as follows:
"""A simple extension module example"""
ctypedef long size_t
cdef extern from "Python.h":
void* PyMem_Malloc(size_t n)
void PyMem_Free(void* mem)
object PyErr_NoMemory()
cdef extern from "string.h":
void* memset(void* mem, int setting, size_t memsize)
char* strcpy (char* dest, char* src)
char* strcat (char* dest, char* src)
int strlen(char* str)
cdef class Foo:
"""A foo, like any other.
Create one with Foo(s), where s is the name to assign to your foo."""
cdef char *name
def __new__(self, name):
self.name = NULL
def __init__(self, name):
self.name = <char*>PyMem_Malloc(len(name)+1)
if not self.name:
raise MemoryError
memset(self.name, 0x00, len(name)+1)
strcpy(self.name,name)
def __repr__(self):
return "A Foo named %s." % (self.name)
def __dealloc__(self):
if self.name:
PyMem_Free(self.name)
In any respect the original tutorial author was incorrect in his use of
Pyrex. If any of this is unclear please ask, and it might be worth reading
the Python/C API documentation if you are thinking of playing around with
Pyrex (as well as the Pyrex doco ;)
Regards,
Grant M.
-----Original Message-----
From: vanitha at cs.wisc.edu [mailto:vanitha at cs.wisc.edu]
Sent: 29 November 2005 14:29
To: Grant McDonald
Subject: RE: [Pyrex] Newbie question
Hi Grant,
Here is the foo.c file that was generated!
Thanks,
- Vanitha
> Vanitha,
>
> I'm not sure why this is occurring, but looking at the generated foo.c
> should make things a bit clearer. Can you send me your foo.c file? (Pyrex
> generates this from your pyx file) It should be in the same directory as
> foo.pyx.
>
> Thanks,
>
> Grant
>
> -----Original Message-----
> From: vanitha at cs.wisc.edu [mailto:vanitha at cs.wisc.edu]
> Sent: 28 November 2005 23:57
> To: Grant McDonald
> Subject: RE: [Pyrex] Newbie question
>
>
> Hi Grant:
> Inserting the space took care of the compilation but I don't the results I
> am expecting.
> I did exactly what was given in the following website:
> http://cfl-x.uwindsor.ca/graham/pyrex/
>
> But here's what I get:
>
> Python 2.4.2 (#5, Oct 24 2005, 14:05:01)
> [GCC 3.4.4] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import foo
>>>> f = foo.Foo('Vanitha')
>>>> f.__class__
> <type 'foo.Foo'>
>>>> f
> A Foo named stdout.
>>>>
>
> I should really be getting "A Foo named Vanitha" as my output. I'm not
> sure why this is happening. Also, what is f.__class__ and what does it
> accomplish?
>
> Thanks,
>
> - Vanitha
>
>
>> Vanitha,
>>
>> It's actually a simple error, on line ten you forgot to add a space
>> after
>> the def:
>>
>> cdef class Foo:
>> """A foo, like any other.
>> Create one with Foo(s), where s is the name to assign to your
>> foo."""
>>
>> cdef char *name
>>
>> def __init__(self, name):
>> self.name = name
>>
>> def__repr__(self): # <-- this should be def
>> __repr__(self):
>> return "A Foo named %s." % (self.name)
>>
>> Everything should work then. Let me know if you have any further
>> questions
>> as I'm more than happy to help out.
>>
>> Grant
>>
>> -----Original Message-----
>> From: vanitha at cs.wisc.edu [mailto:vanitha at cs.wisc.edu]
>> Sent: 28 November 2005 13:57
>> To: Grant McDonald
>> Subject: RE: [Pyrex] Newbie question
>>
>>
>> Hi Grant,
>> I've attached the files. Your help is much appreciated!
>>
>> Thanks very much,
>>
>> - Vanitha
>>
>>> Vanitha,
>>>
>>> That's a strange error, there's nothing in the source code of the
>>> extension
>>> that should cause this problem. It's actually the Pyrex compiler
>>> complaining
>>> about something in your source code (maybe line endings???). Can you
>>> send
>>> me your exact source files as an email attachment? I might be able to
>>> figure
>>> it out for you.
>>>
>>> Regards,
>>>
>>> Grant
>>>
>>> -----Original Message-----
>>> From: vanitha at cs.wisc.edu [mailto:vanitha at cs.wisc.edu]
>>> Sent: 26 November 2005 22:57
>>> To: Grant McDonald
>>> Subject: RE: [Pyrex] Newbie question
>>>
>>>
>>> Hi Grant,
>>> Thanks! I did upgrade to 0.9.3.1 and I ran setup.py build_ext again and
>>> I
>>> get errors:
>>>
>>> Here is the simple example (off the web) that I tried to run:
>>>
>>> * foo.pyx - a straightforward extension class written in Pyrex.
>>>
>>> """A simple extension module example"""
>>>
>>> cdef class Foo:
>>> """A foo, like any other.
>>> Create one with Foo(s), where s is the name to assign to your
>>> foo."""
>>>
>>> cdef char *name
>>>
>>> def __init__(self, name):
>>> self.name = name
>>>
>>> def __repr__(self):
>>> return "A Foo named %s." % (self.name)
>>>
>>> * setup.py - note the import of Pyrex.Distutils.build_ext, its
>>> placement via the cmdclass argument, and the specification of .pyx
>>> files (not .c files) in the ext_modules parameter.
>>>
>>> from distutils.core import setup
>>> from distutils.extension import Extension
>>> from Pyrex.Distutils import build_ext
>>>
>>> setup(name='foo', ext_modules=[Extension("foo", ["foo.pyx"])],
>>> cmdclass = {'build_ext': build_ext}
>>> )
>>>
>>> And here is the error I get:
>>>
>>> running build_ext
>>> building 'foo' extension
>>> /afs/cs.wisc.edu/u/v/a/vanitha/private/CS838/PyrexExamples/foo.pyx:10:1:
>>> Executable statement not allowed here
>>>
>>> Any help will be much appreciated. I'm new to both Python and Pyrex :-(
>>>
>>> Thanks,
>>>
>>> - Vanitha
>>>
>>>
>>>
>>>> Vanitha,
>>>>
>>>> This bug is an incompatibility between Pyrex 0.9.3 build_ext.py and
>>>> Python
>>>> 2.4. distutils/build_ext.py For full details see:
>>>>
>>>> http://lists.copyleft.no/pipermail/pyrex/2004-December/001084.html
>>>>
>>>> This is fixed in 0.9.3.1 (or you could apply the patch yourself if you
>>>> didn't want any other updates)
>>>>
>>>> Grant M.
>>>>
>>>> -----Original Message-----
>>>> From: pyrex-bounces at lists.copyleft.no
>>>> [mailto:pyrex-bounces at lists.copyleft.no]On Behalf Of
>>>> vanitha at cs.wisc.edu
>>>> Sent: 26 November 2005 14:56
>>>> To: pyrex at lists.copyleft.no
>>>> Subject: Re: [Pyrex] Newbie question
>>>>
>>>>
>>>> Hi Grant:
>>>> I'm using Pyrex 0.9.3 along with Python 2.4 and gcc on Linux.
>>>> I think I'll have to upgrade to 0.9.3.1 to get the distutils patch.
>>>> Here is the whole error message anyways!
>>>>
>>>> python setup.py build_ext --inplace
>>>>> running build_ext
>>>>> building 'foo' extension
>>>>> Traceback (most recent call last):
>>>>> File "setup.py", line 6, in ?
>>>>> cmdclass = {'build_ext' : build_ext}
>>>>> File
>>>>> "/afs/cs.wisc.edu/u/v/a/vanitha/lib/python2.4/distutils/core.py",
>>>>> line 149, in setup
>>>>> dist.run_commands()
>>>>> File
>>>>> "/afs/cs.wisc.edu/u/v/a/vanitha/lib/python2.4/distutils/dist.py",
>>>>> line 946, in run_commands
>>>>> self.run_command(cmd)
>>>>> File
>>>>> "/afs/cs.wisc.edu/u/v/a/vanitha/lib/python2.4/distutils/dist.py",
>>>>> line 966, in run_command
>>>>> cmd_obj.run()
>>>>> File
>>>>>
>>>>
>>>
>>
>
"/afs/cs.wisc.edu/u/v/a/vanitha/lib/python2.4/distutils/command/build_ext.py
>>>> ",
>>>>> line 279, in run
>>>>> self.build_extensions()
>>>>> File
>>>>>
>>>>
>>>
>>
>
"/afs/cs.wisc.edu/u/v/a/vanitha/lib/python2.4/distutils/command/build_ext.py
>>>> ",
>>>>> line 405, in build_extensions
>>>>> self.build_extension(ext)
>>>>> File
>>>>>
>>>>
>>>
>>
>
"/afs/cs.wisc.edu/u/v/a/vanitha/lib/python2.4/distutils/command/build_ext.py
>>>> ",
>>>>> line 442, in build_extension
>>>>> sources = self.swig_sources(sources, ext)
>>>>> TypeError: swig_sources() takes exactly 2 arguments (3 given)
>>>>
>>>> Thanks,
>>>>
>>>> - Vanitha
>>>>
>>>> Original Message:
>>>>
>>>> Vanitha,
>>>>
>>>> What version of Pyrex are you using? I built the sample extension you
>>>> provided without encountering an error with Pyrex 0.9.3.1, Python
>>>> 2.4.1
>>>> and
>>>> MSVC 7.0. The stack trace that you provided seems to be lacking the
>>>> final
>>>> error message because the line it terminates on is:
>>>>
>>>> ### distutils/command/build_ext.py ###
>>>> def build_extensions(self):
>>>> # First, sanity-check the 'extensions' list
>>>> self.check_extensions_list(self.extensions)
>>>>
>>>> for ext in self.extensions:
>>>> self.build_extension(ext) # <-- this is where
>> your
>>>> stack trace terminates
>>>>
>>>> Are you able to repost your error and also some information about your
>>>> build
>>>> environment?
>>>>
>>>> Regards,
>>>>
>>>> Grant M.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pyrex mailing list
>>>> Pyrex at lists.copyleft.no
>>>> http://lists.copyleft.no/mailman/listinfo/pyrex
>>>>
>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.copyleft.no/pipermail/pyrex/attachments/20051129/f5461af1/attachment-0001.html
More information about the Pyrex
mailing list