[Pyrex] problem with pyrex and cx_freeze (like py2exe)

[Robby Dermody]

Hey guys,

I have a very low level problem here that I'm a bit stumped about.

Basically, I have a very large pyrex library (under Linux). It's 
actually a bunch of pyrex files compiled together. (For instance, 
mainmod.pyx, sub1.pyx, sub2.pyx, etc. where mainmod.pyx calls 
initsub1(), initsub2() so that both C and python functionality work in 
sub1 and sub2.) For more clarification, I compile these files together 
into a single module though this kind of line in my setup.py file:

    Extension("mainmod",
        ["mainmod.pyx", "sub1.pyx", "sub2.pyx", ... ] )


Although I have been told that this kind of usage of Pyrex is not 
standard and not supported, the whole library works wonderfully. It's 
fast, and I can make C calls from file to file directly (and Python 
calls in any file work fine...it has been tested quite extensively).

FYI, I've written on this weird usage of pyrex before, see:
http://lists.copyleft.no/pipermail/pyrex/2005-February/001126.html

Alright, so this pyrex module works fine when I execute the python 
script that imports it directly with /usr/bin/python. However, when the 
library is frozen using cx_freeze (which is very much like py2exe, but 
works on Linux as well), I get a certain problem.

As a note of how cx_freeze runs python, it packages the python source 
tree in the executable, then uses zipimporter to get at it. 3rd party 
shared object libraries are simply dumped in the same directory as this 
executable, and sys.path is set up to contain the path of this 
executable, along with the executable itself.

So, what happens is that my python program gets converted into an 
executable. There is no change to the pyrex shared object library (it 
just is placed in the same directory of the frozen executable as I said 
earlier). When I run the frozen executable, it imports my pyrex library 
normally, and by doing low level traces I can see that the init() 
function for mainmod.pyx (initmainmod()) is called and completes fine. 
However, what goes wrong is that the init functions for each of the sub 
modules (sub1, sub2) fail at a certain point. These init functions are 
called from initmainmod() (but I have tried calling them from other 
places in mainmod.c too with the same results).

Here's the start of the pyrex-generated init function for one of these 
sub modules (happens to be called ar_voice). My comment for where in the 
code it fails is inline.

DL_EXPORT(void) initar_voice(void); /*proto*/
DL_EXPORT(void) initar_voice(void) {
  PyObject *__pyx_1 = 0;
  PyObject *__pyx_2 = 0;

  __pyx_m = Py_InitModule4("ar_voice", __pyx_methods, __pyx_mdoc, 0, 
PYTHON_API_VERSION);
  if (!__pyx_m) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1; goto 
__pyx_L1;};
  __pyx_b = PyImport_AddModule("__builtin__");
  if (!__pyx_b) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1; goto 
__pyx_L1;};
  if (PyObject_SetAttrString(__pyx_m, "__builtins__", __pyx_b) < 0) 
{__pyx_filename = __pyx_f[0]; __pyx_lineno = 1; goto __pyx_L1;};

  if (__Pyx_InternStrings(__pyx_intern_tab) < 0) {__pyx_filename = 
__pyx_f[0]; __pyx_lineno = 1; goto __pyx_L1;};
  if (__Pyx_InitStrings(__pyx_string_tab) < 0) {__pyx_filename = 
__pyx_f[0]; __pyx_lineno = 1; goto __pyx_L1;};
  __pyx_v_8ar_voice__pendingOperations = Py_None; 
Py_INCREF(__pyx_v_8ar_voice__pendingOperations);
  __pyx_type_8ar_voice_pendingOp.tp_free = _PyObject_GC_Del;
  if (PyType_Ready(&__pyx_type_8ar_voice_pendingOp) < 0) {__pyx_filename 
= __pyx_f[0]; __pyx_lineno = 37; goto __pyx_L1;}

^^^ fails at this PyType_Ready call, and the init function terminates 
early. Why this call is made is because I have a global level "cdef 
object pendingOp" declaration in the ar_voice module.

  if (PyObject_SetAttrString(__pyx_m, "pendingOp", (PyObject 
*)&__pyx_type_8ar_voice_pendingOp) < 0) {__pyx_filename = __pyx_f[0]; 
__pyx_lineno = 37; goto __pyx_L1;}
  __pyx_ptype_8ar_voice_pendingOp = &__pyx_type_8ar_voice_pendingOp;

[....]

As a result of this failure, whenever any function in this sub module 
tries to access any imported python library, it fails (obviously, 
because the Py_Import calls a made later in the init function, and it 
quits before that). What I'm wondering is why it might be failing at 
that PyType_Ready() call, and what I can do to fix that.

So I guess that things work fine if I'm executing through regular 
python, but through however cx_freeze executes, I get this type of 
failure. Maybe this is what Greg meant by "unsupported" and "wacky 
python behaviour", haha. :/

Here is some relevant C code for how cx freeze initializes python and 
starts my app. How this is different from how /usr/bin/python starts my 
app will probably contain the secret to why I am getting this problem 
(although with my limited knowledge I can't see it):

cx_freeze main:

int main(int argc, char **argv)
{
    const char *fileName;

    // initialize Python
    Py_NoSiteFlag = 1;
    Py_FrozenFlag = 1;
    Py_IgnoreEnvironmentFlag = 1;
    Py_SetPythonHome("");
    Py_SetProgramName(argv[0]);
    fileName = Py_GetProgramFullPath();
    Py_Initialize();
    PySys_SetArgv(argc, argv);

    // do the work
    if (ExecuteScript(fileName) < 0)
        return 1;

    Py_Finalize();
    return 0;
}

ExecuteScript looks like:

static int ExecuteScript(
    const char *fileName)               // name of file containing 
Python code
{
    PyObject *module, *importer, *code, *dict, *pathList, *temp;
    int result;
    // add the file to sys.path
    pathList = PySys_GetObject("path");
    if (!pathList)
        return FatalError("cannot acquire sys.path");
    temp = PyString_FromString(fileName);
    if (!temp)
        return FatalError("cannot create Python string");
    result = PyList_Insert(pathList, 0, temp);
    Py_DECREF(temp);
    if (result < 0)
        return FatalError("cannot insert file name in sys.path");

    // load and execute initscript
    module = PyImport_ImportModule("zipimport");
    if (!module)
        return FatalError("cannot import zipimport module");
    importer = PyObject_CallMethod(module, "zipimporter", "s", fileName);
    Py_DECREF(module);
    if (!importer)
        return FatalError("cannot get zipimporter instance");
    code = PyObject_CallMethod(importer, "get_code", "s", 
"cx_Freeze__init__");
    Py_DECREF(importer);
   if (!code)
        return FatalError("unable to locate initialization module");
    dict = PyDict_New();
    if (!dict)
        return FatalError("unable to create temporary dictionary");
    if (PyDict_SetItemString(dict, "__builtins__", PyEval_GetBuiltins()) 
< 0)
        return FatalError("unable to set builtins for initialization 
module");
    temp = PyEval_EvalCode( (PyCodeObject*) code, dict, dict);
    Py_DECREF(code);
    Py_DECREF(dict);
    if (!temp)
        return FatalScriptError();
    Py_DECREF(temp);

    return 0;
}



ANY ideas would be appreaciated!


Robby