[Pyrex] Python Strings and PyMem_Malloc/Free

[len-l at telus.net]

On 30 Jun 2005 at 16:06, Grant McDonald wrote:

> 
> Hi, 
> I fear I may have been operating under a false assumption with regards to creating Python 
> strings. This assumption goes like this:
> 1) Whatever memory you PyMem_Malloc you should always PyMem_Free.

Correct. These functions operate below Python's garbage collector.

> 2) For strings you malloc and need to turn into python strings assign them to a python variable in 
> Pyrex then PyMem_Free the pointer to the string before you return the python object.
> 3) This shouldn't present a problem because PyString_FromString copies the string to an internal 
> representation (this is my assumption).

Correct. PyString_FromString does copy the C string rather than just keep a pointer 
to it.

> I believe the last statement may be completely erroneous :). Here is some code based upon this 
> assumption: 
> cdef class Interface: 
[snip]
>  . 
>  def GetNote(self): 
>  cdef TRK_UINT bufferSize, remaining, max, titleSize, result 
>  cdef LPSTR buffer, title, temp_buffer 
>  
>  result = TrkGetNoteDataLength(self.trkNoteHandle, &bufferSize) 
>  if result != TRK_SUCCESS: 
>  raise ExtensionExceptions.TrackerError, result 
>  
>  buffer = <LPSTR>PyMem_Malloc(sizeof(char)*bufferSize) 
>  if buffer == NULL: 
>  raise Exception, "Could not malloc %d bytes!!" % bufferSize 
>  temp_buffer = <LPSTR>PyMem_Malloc(sizeof(char)*bufferSize) 
>  if temp_buffer == NULL: 
>  raise Exception, "Could not malloc %d bytes!!" % bufferSize 
>  memset(buffer, 0x00, bufferSize) 
>  memset(temp_buffer, 0x00, bufferSize) 
>  
>  if bufferSize > 10000: 
>  max = 10000 
>  else: 
>  max = bufferSize 
>  
>  remaining = bufferSize 
>  pyNote = '' 
>  
>  while remaining != 0: 
>  result = TrkGetNoteData(self.trkNoteHandle, max, temp_buffer, &remaining) 
>  if result != TRK_SUCCESS and result != TRK_E_DATA_TRUNCATED: 
>  PyMem_Free(buffer) 
>  PyMem_Free(temp_buffer) 
>  raise ExtensionExceptions.TrackerError, result 
>  strcat(buffer, temp_buffer) 
>  memset(temp_buffer, 0x00, bufferSize) 
>  
>  titleSize = 200 
>  
>  title = <LPSTR>PyMem_Malloc(sizeof(char)*titleSize) 
>  if title == NULL: 
>  raise Exception, "Could not malloc %d bytes!!" % titleSize 
>  memset(title, 0x00, titleSize) 
>  
>  result = TrkGetNoteTitle(self.trkNoteHandle, titleSize, title) 
>  if result != TRK_SUCCESS: 
>  PyMem_Free(buffer) 
>  PyMem_Free(temp_buffer) 
>  PyMem_Free(title) 
>  raise ExtensionExceptions.TrackerError, result 
>  
>  pyTitle = title 
>  pyNote = buffer 
>  
>  # it is this code that appears to be problematic 
>  PyMem_Free(buffer) 
>  PyMem_Free(temp_buffer) 
>  PyMem_Free(title) 
>  
>  return (pyTitle,pyNote) 
> As the code stands calling this function _sometimes_ crashes the Python interpreter (usually the 
> 3rd or 5th call) with an access violation (mem address 0x0000000 which would imply a null 
> pointer reference). If I comment out the final three calls to PyMem_Free then the problem does 
> not arise.
> If my assumption is wrong does the same hold true for structures? I have similar malloc/free 
> calls for structures which save data from them and then free the structure memory before 
> returning.

How are you declaring PyMem_Malloc / Free? It should be:

cdef extern from "python.h":
    void *PyMem_Malloc(unsigned int n)
    void PyMem_Free(void *p)

Anyway, if that is not the problem just try using malloc and free directly. Since you 
are managing the memory assigned to buffer, title and temp_buffer then it can be in 
a separate heap from that used by the interpreter.

A final suggestion. Wrap the body of GetNote within a try-finally statement with the 
PyMem_Free (or free) calls in the finally block. Then buffer, temp_buffer, and title 
are guaranteed to be freed when a value is returned or an exception is raised. Just 
make sure there are no PyMem_Free (free) calls in the try block.

def GetNote(self):
    ...
    cdef LPSTR buffer, title, temp_buffer
    buffer = NULL
    title = NULL
    temp_buffer = NULL
    try:
        ...
        if result != TRK_SUCCESS:
            raise ExtenionExceptions.TrackerError, result
        ...
        pyTitle = title
        pyNote = buffer
        return (pyTitle, pyNote)
    finally:
        # Garbage collection
        PyMem_Free(buffer) # or free(buffer)
        PyMem_Free(title)
        PyMem_Free(temp_buffer)

Lenard Lindstrom
<len-l at telus.net>