[Cython] dynamic memory management

[Robert Bradshaw]

On Nov 24, 2008, at 3:56 PM, Michael Abshoff wrote:

> Stefan Behnel wrote:
>> Hi,
>
> Hi,
>
>> when re-reading an older thread about the struct syntax, I had a  
>> funny idea
>> I wanted to share.
>>
>> Robert Bradshaw wrote:
>>> I'm with Stefan that it is very
>>> dangerous to hide the malloc and expect the user to explicitly
>>> provide the free. If the user wants to manage their own memory, they
>>> can do so with malloc and friends (understanding the associated
>>> dangers), and any special memory-management stuff we add should  
>>> be as
>>> implicit and easy to use as Python's garbage collection (and  
>>> probably
>>> piggyback off of it).
>>
>> The "piggybacking" here makes me wonder if it would work to provide a
>> dedicated "Memory" object, that would be a Python object but could  
>> be used
>> as in
>>
>>   def do_stuff_with_dynamic_memory(Py_ssize_t size):
>>       cdef Memory mem
>>       cdef void* ptr
>>       mem = Memory(10*size)  # equiv to malloc()
>>       ptr = mem     # automagic coercion to a pointer to the  
>> memory buffer
>>       # do stuff with *ptr, e.g. hand it around in C code
>>       mem = None # => Py_DECREF(mem), equiv to free()
>>
>
> In general I consider this a great idea since people will be less  
> likely
> to shoot themselves in the foot and as you mentioned for small allocs
> this should really be a win. Having memory automatically freed on
> exceptions would also be great since that is tricky to write. The less
> the average user has to deal with memory allocs and deallocs in Cython
> code the better Cython will be.

I think this is a great idea too. Memory could have two C attributes  
like len and ptr (to avoid automagic coercion), and a realloc method  
(which may of course move ptr). Could probably be implemented on top  
of the str/bytes object with a new entry in the builtins table and  
perhaps some tree transformations to make creating one very fast.

> But I am concerned to some extend about the success of hunting memory
> leaks if this is the default allocator in some future version of  
> Cython
> and there is no way to use malloc() [currenly Sage provide  
> sagemalloc(),
> so that for example could be made to use  
> do_stuff_with_dynamic_memory()
> since we only need to change one define].  Looking for heap objects in
> Python that are leaked or not properly dereferenced is a giant pain
> right now and valgrind is next to useless here. muppy is starting to
> solve that problem, but there are at least in Sage issues with ipython
> allocating tens of thousands of heap objects at load time impacting
> performance of muppy badly, but those issues will be hopefully fixed.
> Anyway, no need to rant about python memory heap debugging here :)
>
> Since in Sage I have to deal with loads of small, but potentially  
> large
> cumulative leaks I would consider it very nice if the allocator could
> have some magic debug fallback to malloc(). That should certainly  
> be not
> the default behavior, just like --without-pymalloc is not the default
> for Python.

+1 to this idea.

- Robert