[Cython] Faster Pickle possible?

[Stefan Behnel]

Hi,

Eric Jonas wrote:
> I assume most of us are using Cython because normal python is too slow
> for our particular operation. I've been writing a compiler in python
> which generates large graphs with ~600k nodes, and I'd like to
> serialize/checkpoint these to disk for later stages of the compiler
> pipeline to use. However, at the moment, it takes ~60 seconds to
> serialize the resulting graph to disk with cPickle. 

You can implement your own pickle mechanism for a specific object. We do that
in lxml.objectify, where you can pickle an XML element into a serialised XML
byte stream. If you can find a way to efficiently walk your graph at the C
level and serialise that by hand step by step, you may end up being much
faster than pickle's introspection machinery.

Stefan