[Cython] Lists?

[Robert Bradshaw]

On Sep 1, 2008, at 10:41 AM, Hoyt Koepke wrote:

> Hi,
>
> This is actually something I am wondering about as well, so I hope
> it's okay to join the discussion.  My understanding is that when you
> declare a variable to have a list, tuple, or dict type, then cython
> optimizes it by using faster direct ways to accessing the elements
> when you use them. However, since these containers always store the
> elements as PyObject* pointers which hold any type, it's too difficult
> to use element type information in a robust way.  Is that correct?

Exactly.

> What is the most "cythonic" way to mimic the list functionality when
> you do have homogeneous list content?  I have run into cases in
> numerical calculations where I need to be append an arbitrary number
> of numbers to a list (e.g. accumulating data statistics while
> processing large amounts of data).  In such contexts, using a list is
> too slow. One thing I've done is use a numpy array with a significant
> amount of extra space, keep a separate counter to denote the end, and
> access it through the nice buffer interface.  This works pretty well,
> but seems quite clunky compared to how natural it is in python.  Doing
> slicing and concatenation stuff would also be nice.

The traditional way to do this is the same as in C, with malloc,  
realloc, etc. Then one can create actual int*, double*, etc. as in C.  
Of course then one has to worry about memory leaks and/or  
segfaults... The Numpy approach is a good one too, and there is a  
resize() method.

> Personally, what I think would be really nice is a unified wrapper to
> the C++ STL containers and some of the libraries with easy conversion
> functions between the parallel python containers.  I know that
> wrapping an STL vector is one of the examples given, so this shouldn't
> be too hard.  I don't have time for at least a few weeks, and there
> are several other important things on my to-do list, but I'll try to
> look into it.

That would be excellent! It would necessitate using C++, which is not  
going to be the default Cython output but would be very hand for  
people wanting to use it.

> That said, I am pretty new still to cython, so please correct me if my
> perspective on things is off or I'm just plain wrong.

Your assessment is entirely correct.

Another option would be something like http://wiki.cython.org/ 
enhancements/arraytypes, though this is probably further off in the  
future than wrapping several C++ containers.

- Robert