notes about memory-saving GCs =============================== XXX go into more detail, discuss which pypy GCs we want to try XXX [fijal] I think both approaches are worth trying. We need to see how they behave or try to think how much memory it'll be needed for a GC header. note that deferred refcounting without nursery is needed for deferred refcounting with nursery (just because it's easier to have this in steps). We also want to consider impacts of fragmentation and how underlaying C library handles stuff (ie do we use obmalloc or no. does it return memory to the OS or no for small objects etc.) * squeak-like mark-compact collector * deferred refcounting strategy * deferred refcounting with a nursery Some references =============== * Garbage collector without paging: http://www.cs.umass.edu/~emery/pubs/f034-hertz.pdf, some discussion: http://lambda-the-ultimate.org/node/2391 NOTE: this gc requires modest (600 LOC) support from operating system *very* unlikely we can make such assumptions * paper about measurments http://www-cs.canisius.edu/~hertzm/thesis.pdf FAQ === Why mark&sweep is consuming so much memory on some benchmarks? -------------------------------------------------------------- It seems that some allocation schemes (unfortunately ones happen to be found in pypy during normal operation) create very long free lists for small objects. Consider example: [(i,) for i in range(amount)] in CPython, any immediate object created during list construction is immediately discarded. On the other hand, while creating the same thing in RPython, for each allocated space, it gets stored in free list of obmalloc or C library until collection happens. This means that since collection happens not very often (due to high costs), this gets accumulated (also for different python-level operations, different freelists will be populated, if requested size is different enough). If we invoke gc.collect every couple iterations (say 100 or 1000), this effect is gone. On the other hand it's becoming very very slow as a side effect.