[Cython] -dagss merge? + what next

[Robert Bradshaw]

On Jul 29, 2008, at 12:56 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Jul 29, 2008, at 3:57 AM, Dag Sverre Seljebotn wrote:
>>
>>> Robert Bradshaw wrote:
>>>> On Jul 29, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote:
>>>>
>>>>> Those are the big tasks, but there are also a few smaller nice-
>>>>> haves I
>>>>> can go for (a "cython.shape" builtin that allows you to  
>>>>> retrieve the
>>>>> shape information of a buffer, and so on).
>>>> I saw you mention shape (and other "magic" functions), and I'm
>>>> curious. Could you elaborate?
>>> When acquiring a buffer, you get hold of a lot of information  
>>> that is
>>> not currently exported -- shape is most important. For NumPy this
>>> is ok
>>> as ndarray exports its shape field for efficient access, but without
>>> another mechanism you can't write generic algorithms that work with
>>> all
>>> buffers without passing the shape explicitly. This even leads to  
>>> less
>>> clean test case code...
>>>
>>> I haven't thought much about the list (add as needed) but something
>>> like:
>>>
>>> - shape(buf) -- Returns a tuple containing the shape of the buffer.
>>> However, shape(buf)[0] should compile directly to
>>> __pyx_bshape0_buf. (I
>>> thought about shape(buf, 0) as well but decided against, how about
>>> you?)
>>>
>>> This would be the primary for-loop mechanism:
>>>
>>> from cython import shape
>>> ...
>>> cdef int i
>>> for i in range(shape(buf)[0]): buf[i] = 9
>>>
>>> - buffer(buf), strides(buf), suboffsets(buf) -- Could have these to
>>> allow low-level access (buffer gets a void* directly, strides and
>>> suboffsets probably). For contiguous arrays this could be more
>>> efficient
>>> (the proper high-level approach for such things are iterators). The
>>> idea
>>> is that even if you write low-level code, you could use the buffer
>>> syntax so that you don't need to worry about acquisition.
>>>
>>> - ndim(buf) -- If the ndim-at-compile-time-restriction is lifted
>>> (because you have low-level access or iterators you can use  
>>> instead of
>>> indexing) then this will be useful.
>>>
>>> Finally, if I get as far as iterators (I won't!), cython.ndenumerate
>>> could create very efficient iteration code. (Probably a NumPy- 
>>> specific
>>> iterator could be less work though...if I can't just copy the  
>>> approach
>>> of NumPy iterators directly...NumPy comes with a seperate n-
>>> dimensional
>>> iterator class that allows you to write ndim-generic code.)
>>
>> Ah, I see. These could be very useful, though I'm not sure we should
>> be augmenting the namespace to provide them (perhaps make them magic
>> methods of a buffer object instead?)
>
> What are the disadvantages of adding a cimportable cython namespace  
> for
> this? I don't think we can get around it personally -- in fact I was
> thinking of using it for the pragma code [1].

OK, for some reason I was missing the fact that you were cimporting  
them (I thought you were suggesting they be builtins like len). I  
think it should somehow be clear that these operate on buffers,  
either by placing them in cython.buffer or calling them buffer_shape,  
etc. (I prefer the former).

> Meanwhile, attributes can always collide. Especially having
> "object[int]" not do an attribute lookup for every attribute access
> would be a big loss I feel.
>
> What about adding syntax in "cdef extern class ndarray" to provide  
> these
> functions in addition, so that the class author is in control and can
> make the decision.
>
> That is in line with another feature I'd like feedback on, starting
> another thread.
>
> [1]:
>
> - cython.unchecked_getitem(buf, i, j) and cython.unchecked_cast(...)
> - with cython.noboundschecking: ...
> - @cython.noboundschecking
>    def f(): ...
>
> etc. Also it could be a primary vehicle (in a future far off) of  
> making
> Cython more namespace-compatible with Python:
>
> with cython.nogil:

Yeah, I like this general direction, though your first example is a  
bit verbose. Again, something to be explicit about the fact that  
these are buffer settings would be good.

- Robert