[Cython] Prototype patch for closures/inner functions

[Dag Sverre Seljebotn]

Stefan wrote:

> I agree with Robert. As long as Cython does not support closures, for
> example, it cannot come close enough to being a real option for speeding


I couldn't resist the challenge :-)

Attached is a prototype using transforms to add closure support to Cython.

NB! It's not ready for prime-time yet. Unfortunately I must leave it for 
some days now, so I post it it prototype state. Mainly so that others 
don't start on the same thing (though feel free to take over this, just 
give me a note).

I mainly write it to see how it would be like to write a "real" 
transform. Which was not too bad...

It is a bit hacky but I think the approach should give correct results.

Known fatal bugs:
- I don't know the first thing about reference counting, CPython etc.. 
At least on one occasion I messed up GC-ing. This is probably something 
that others will be much quicker at spotting.
- No name mangling, inner functions cannot collide in name with outer. 
Quick fix but don't have time now; also one might want a more generic 
"name mangler" support mechanism rather than just checking for 
collisions, not sure about how to do this.
- No consideration for anything nontrivial: The global keyword and 
accessing variables in modules comes to mind.

Conscious limitations:
- Only Python def's, not inner cdefs, and all bound vars are bound as 
Python objects. This restriction can be removed later (for 
optimization), but I think Python-only inner defs should work fine.

Strategy:
- First, run a transform that records used and assigned symbols within a 
function (don't know if this is already done anywhere, probably this 
might be redundant and done in the scope system? Suggestions?).
- Then, run a transform which lifts out inner functions, and replace 
with assignment to a method bound to a tuple containing the variables 
that should be bound (apparently this works. But one might create a 
specific type containing a tuple as a field as well.)
- The lifted-out functions gets an instruction added first to unpack the 
"self" tuple to the correct names (making the names shadowing the outer 
scope).

How to run (if you want to help me out or have a look, otherwise don't 
bother):

Apply patch.

$ cat <<END > test.pyx
def make_adder(n):
    def adder(x):
        return x + n
    return adder


def timesthree(n):
    def util(x):
        return x * 2
    return n + util(n)
END

$ cat <<END > test.sh
python cython.py \
  -Tafter_parse:Cython.Compiler.Transforms.InnerFunctions.FunctionSymbols \
  -Tafter_parse:Cython.Compiler.Transforms.InnerFunctions.InnerFunctions \
  
-Tafter_analyse_function:Cython.Compiler.Transforms.InnerFunctions.MethodTableIndex 
\
  test.pyx
gcc -Wall -shared -fPIC -I/usr/include/python2.5 -o test.so test.c
END

$ python
 >>> import test
 >>> a = test.make_adder(20)
 >>> b = test.make_adder(10)
 >>> a
<built-in method adder of tuple object at 0x2af646ca8910>
 >>> a(13)
33
 >>> b(13)
23
 >>> test.timesthree(100)
300

-- 
Dag Sverre

-------------- next part --------------
A non-text attachment was scrubbed...
Name: innerfuncs.diff
Type: text/x-patch
Size: 14964 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080409/0d8e7126/attachment-0001.bin