[lxml-dev] Leaking tracebacks

[Stefan Behnel]

Hi,

Christian Heimes wrote:
> A college run into a memory leak on one of our servers a couple of days 
> ago. With the help of Dowser - a CherryPy plugin - we were able to nail 
> down the leak to traceback and frame objects. Some code in Martijn 
> Faassen's oaipmh package kept references to an exception traceback.
> 
> Today I was able to track 'n nail down the issue. I highly suspect lxml 
> to be the culprit. Whenever the code reaches lxml with an exception in 
> sys.exc_info the exception leaks. My dowser patch shows a refcnt of 4 
> references to the root traceback object that can't be traced to objects 
> in the gc.get_objects() list.

There was a change in Cython regarding exception handling not so long ago.
Cython now mimics more the behaviour of Py3, i.e. exceptions disappear once
they were caught. If the problem was introduced at this point, it would
explain why this appears with later lxml releases only, as older releases
are based on older Cython versions.

I'm not sure what the problem is here, though, so this needs further
investigation. From a couple of quick tests, I can't seem to reproduce this
problem. Could you try to come up with some test code that shows this
behaviour? Is it only a problem with functionality that interacts with
libxml2 in both directions (such as XPath/XSLT, which uses function
callbacks back into lxml), or does it also happen in functions like
etree.iselement()?

Stefan