[Pyrex] EasyExtend meets Pyrex

[kay schluehr]

Dear Pyrex list,

I'm the author of the EasyExtend[1] package which, among other things, 
is used to provide extension languages for Python.  My current research 
interest is in a test driven type-feedback system. The basic idea is to 
use a testsuite for a certain Python module Mod and recording dynamic 
type information ( filling a type profile ) during test execution. This 
type information can be fed back and merged with Mod. After merging Mod 
with the type profile TP to TPMod we have an almost-statically typed 
Python module. This methodology is open and non-prohibitive. If the user 
imagines a new usecase for M he might just write a new testcase and a 
new TP/TPMod will be derived.

While it is possible to use TPMod to perform dynamic type checks on 
input from modules / console interaction that uses TPMod it would be 
more challenging to use TPMod for performance optimization of M. A 
representation of TPMod is not uniquely determined by M so it might be 
represented in different domains. One of those domains might be Pyrex.

          Mod x TP -> TPMod(Pyrex)

Pyrex seems to be a quite natural choice because the mapping could be 
made smooth i.e. an incremental transformation of Mod to TPMod.

Some notes on EasyExtend and possible kinds of transformations. 
EasyExtend operates on parse trees generated from Pythons canonical 
LL(1) grammar and variations of it. The parse tree transformations are 
constructive: for each production rule of the original Python grammar 
there is a function used to represent the rule. So if e.g. the rule is

    funcdef: [decorators] 'def' NAME parameters ':' suite

the related function is

    def funcdef(*args):
        ...
        return [symbol.funcdef, [...]]   # symbol.funcdef is the type of 
the node represented as a nested list

where args will be checked on the correct node types:  
{symbol.decorators, token.NAME, symbol.parameters, symbol.suite} in that 
order with optional omission of decorators. The function set is 
complete: any Python parse-tree and therefore any Python program can be 
created using only this function set. The list representation is such 
that the Pythons stdlib parser module can be used for transformations:

    >>> import parser
    >>> parser.list2ast(csttree).compile()

where csttree ( concrete syntax tree ) is a parse-tree created 
constructively.

Back to Pyrex. It might be reasonable to merge Mod with TP by extending 
the function set of EasyExtend with functions that represent Pyrex 
grammar rules. This shall be possible even if Pyrex grammar is not LL(1) 
and there won't be a fiber / extension language representation of Pyrex 
in EasyExtend yet. Pyrex modules shall be constructed the same way as 
Python modules using EasyExtend. Unfortunately I don't know if there is 
an elegant way of compiling/transforming the list shaped nodes such that 
they Pyrex compiler can handle them. A back-transformation from a 
constructed Pyrex parse tree to source code is feasible but it seems to 
be excessive to me.

Since I know Pyrex only superficially and have even less time spent in 
understanding the Pyrex source, help from experienced Pyrexers is very 
welcome.

Thanks for attention, Kay

[1] http://www.fiber-space.de/EasyExtend/doc/EE.html