[Cython] Storing many, many instances

[Robert Bradshaw]

On May 4, 2009, at 2:43 PM, Stéfan van der Walt wrote:

> Hi all,
>
> I have code that generates about 40000 instances of a certain  
> cdef'd class.
>
> I would like to know:
>
> 1) If I define my cdef class in my_class.pyx, and then, in main.pyx  
> do:
>
> from my_class import MyClass
>
> [.. instantiate many, many MyClass objects ..]
>
> Does this happen in C, or am I making Python calls along the way?

You have to cimport to use C calls from another module. However, it  
will still call __init__ with Python semantics. The more arguments  
you pass (especially keyword arguments), the slower it'll get.

> 2) Is there a standard pattern for lowering the cost of instance  
> creation?
>
> I think I may have to switch to structs instead of classes, but then I
> no longer have convenient methods or array members.  Maybe I should
> rewrite my class to have many static methods that operate on a struct,
> and then collect the structs as data instead of the full instances.
>
> Any input much appreciated,

You could look into PY_NEW that we use in Sage. There was a thread on  
this earlier. There is inherently overhead in using Python objects,  
but it's usually worth the convenience. Another option would be to  
write a single class that provides access to a large set your  
objects. It all depends on your use case.

- Robert