[Cython] [sage-devel] Re: Locally scoped dynamic memory (in SAGE and elsewhere)

[William Stein]

On Thu, Apr 10, 2008 at 10:04 AM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>
>  >  >  Lisandro Dalcin (author of mpi4py) came up with the following trick
>  >  >  that, while more complicated, prevents memory leaks:
>  >  >
>  >  >  cdef extern from "Python.h":
>  >  >     object PyString_FromStringAndSize(char*,Py_ssize_t)
>  >  >     char* PyString_AS_STRING(object)
>  >  >
>  >  >  cdef inline object pyalloc_i(int size, int **i):
>  >  >     if size < 0: size = 0
>  >  >     cdef Py_ssize_t n = size * sizeof(int)
>  >  >     cdef object ob = PyString_FromStringAndSize(NULL, n)
>  >  >     i[0] = <int*> PyString_AS_STRING(ob)
>  >  >     return ob
>  >  >
>  >  >  and now
>  >  >
>  >  >  def foo(sequence):
>  >  >     cdef int size = len(sequence),
>  >  >     cdef int *buf = NULL
>  >  >     cdef object tmp = pyalloc_i(size, &buf)
>  >  >
>  >  >  This could probably be adapted into a malloc-like function.  What do
>  >  >  people think?
>  >
>  >  Could you explain what the point is?  Is it that this is a trick so that
>  >  Cython will correctly garbage collect the allocated memory, even
>  >  if an exception occurs?
>
>  Yes, that is the idea.  By having a python object that knows about the
>  memory, the garbage collection should prevent memory leaks if an
>  exception occurs.  I am not sure if Lisandro has proved that this is
>  the case - he is just wondering how people typically handle this case.
>
>  Seems like we are not alone in thinking this is a problem though.
>

It's I think a general problem not only in Python but in programming
in general.  The authors of http://www.flintlib.org/ -- a pure C library --
spent a lot of time worrying about this, just to I think decide that it's
a really hard problem.

That said, there is definitely a significant class of problems in the context
of Cython that could be fixed if Lisandro's suggestion works.  Thanks
for posting it!

William