[Cython] Lisp inspired transforms

[Robert Bradshaw]

On Apr 8, 2008, at 9:09 AM, Stefan Behnel wrote:
> Martin C. Martin wrote:
>> Compile time metaprogramming doesn't exist in Python, so adding it to
>> Cython means extending Cython beyond what Python has.
>
> Cython has a couple of additional features that make sense because  
> it is a
> compiled language. I think what you call "metaprogramming" (and  
> generally
> most things that allow doing things at compile-time instead of run- 
> time)
> makes sense for Cython.
>
>> There are a couple options:
>>
>> 1. Add a way to generate C++ templates, and use that for
>> metaprogramming.  It keeps Cython as "writing both Python and C++  
>> with
>> an extended Python syntax."
>
> But that would be C++ specific and can't work with C.

I agree--I don't think we should make any Cython features dependent  
on C++, and don't know that the mataprogramming would even map nicely  
(let alone comprehensibly) from the one language to the other.

On a completely orthogonal note, I would like to make Cython template- 
aware (enough) to make it easy to wrap templated C++ libraries.

>> 3. Add a way to specify Python code for the transformation.  This
>> recognizes that metaprogramming is a valuable activity that Cython
>> developers will want to do; that the existing way to do it in C++ is
>> more-or-less not up to the task; and that it's better to provide a  
>> new,
>> cleaner mechanism using what we've learned in hindsight.
>
> I would say so. I would currently position it as a) an extension  
> mechanism
> for Cython itself and b) an advanced feature that most people won't  
> use
> (in the same way most people don't use metaclasses) - but as usual  
> with
> OSS, you never know what people will use it for.
>
>
>> Of course, metaprogramming in an imperative, stateful language,  
>> opens a
>> can of worms, e.g. it will be valuable to modify Python data at  
>> compile
>> time, and have that serialized once all transforming is done, then
>> loaded at the start of runtime.  I don't think any of these  
>> problems are
>> particularly difficult though.
>>
>> So, what do people this is the best way forward for Cython?
>
> I'll have to take a closer look at your proposal and compare it a  
> bit more
> to the other approaches we had so far (especially Dag's work),  
> before I
> make up my mind about it. Maybe others can already comment a bit  
> deeper on
> this.

I have thought a lot about this and I think the best way forward for  
Cython is to become as close to Python (from the end-users point of  
view) as possible without sacrificing speed and the ability to easily  
wrap existing libraries. Cython as a tool to for use by Python  
developers has a much greater potential then Cython as a new  
language. Unfortunately, when it comes to adding new features to the  
language, these two views of Cython are mutually- opposed, and I  
think we should stick with the first.

I do not want to devalue the benefits of metaprogramming (and  
probably several other features that will probably come up which  
Python is lacking) but if we decide to add such features I think  
there should be a clear distinction between this "Cython+" language  
and "normal Cython." (Not necessarily a separate compiler, but at  
least a separate file extension). Also, I think we should mostly  
focus on catching up to where Python is now before expending too much  
effort going beyond.

- Robert