[Cython] -dagss merge? + what next

[Robert Bradshaw]

On Jul 29, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote:

> I'm at a stage where I think we can consider a new merge of -dagss.  
> What
> I say below applies to revision 876:5df94de34ff7; the -dagss branch  
> will
> promptly go unstable again so make sure you pull this one.

Great. I was actually just wondering earlier today when a good stable  
pull would be.

> Buffers are basically working (I am reasonably confident it is working
> but I still have ideas for testcases that I haven't written so the
> testsuite is not exhaustive). In particular I'd like to put it out  
> as an
> "available beta component" in the next Cython release (I can write  
> some
> release notes for it).

Sure. I want to do a release again soon. This is safe for an alpha as  
it is a completely new feature with new (formerly illegal) syntax, so  
it shouldn't impact old working code, and hopefully we'll get a lot  
of feedback.

> What next? Comments welcome on this:
> - I'd like to leave buffers in the global scope or as attributes  
> out of
> it for now. It is really non-trivial and I think it would be wrong to
> prioritize it above all the other stuff I could do.

I agree. See my (very loose) priorities below.

> - Given that, I think these three tasks are important:
>     - Supporting structs (currently only ints and floats are  
> supported).
> Complex numbers are part of that.

3.

>     - Temp allocation gets quite hairy with buffers and I'd like to
> implement Greg Ewing's proposal to clean it up. But there is a  
> potential
> for cleaning it more than it is within the existing framework as well.

1.

>     - "final" functions in pxd file, in order to implement
> numpy.ndarray.__getbuffer__ in the pxd file. (I won't do any  
> inlining in
> Cython, it is just a way of having code automatically copied from the
> pxd to the pyx (and potentially skip vtable lookups if I get around  
> to it)).

2. (Though this should be really easy compared to the temps stuff).


> Those are the big tasks, but there are also a few smaller nice-haves I
> can go for (a "cython.shape" builtin that allows you to retrieve the
> shape information of a buffer, and so on).

I saw you mention shape (and other "magic" functions), and I'm  
curious. Could you elaborate?

>
> Notes:
>
> - Includes is now always included. (Though I am not any longer  
> using the
> __cython__ namespace, my test cases will like to cimport things).
>
> - Get/ReleaseBuffer is no longer defined when not needed

Excellent.

> - Temp allocation is rather nasty at the moment (using my prototype
> which we ended up rejecting!). I am thinking that I might rather spend
> half a day to a day implementing Greg Ewing's ideas and use those than
> fix it. (I'm thinking along the lines of first implementing
> "code.fork()" which forks the output stream, using a cStringIO  
> "tree" so
> that no buffer copying is needed).

Sounds good.

> But it is working, and even though I'll definitely remove it I don't
> think it is blocking a merge.
>
> - Overall strategy: I've modified FuncDefNode, NameNode and  
> IndexNode to
> do what I need to do. No new nodes. I ended up thinking this was much
> cleaner than inserting new nodes:
>    - IndexNode already supports two different concepts (C arrays and
> Python Get/SetItem), adding a third seemed to be consistent with this.
>    - NameNode: The way things are implemented, buffer auxiliary
> variables (a lot of them) are declared and connected to the local
> variable that is of the buffer type. So a typecasting-style node that
> primarily work through temporaries would have been less clean.  
> NameNode
> also already support many different kinds of operations and so adding
> another one seemed ok.

This sounds good and relatively clean, as well as consistent with  
what's there now (i.e. if someone wants to know how indexing works,  
there's one place to look).

> I'm ok with splitting into seperate buffer nodes later, but I think it
> should be done while also splitting the rest, i.e. transform IndexNode
> into CArrayIndexNode, PythonIndexNode and BufferIndexNode and so on.
> (But, seeing as that is only for abstract purity at the moment I
> definitely won't touch it.)

Yeah, sounds like another task for another day (though hopefully not  
too far off). Good work!

- Robert