Python is a mature dynamic language whose interpreters can interact with all major computing platforms today.
execnet provides a share-nothing model with channel-send/receive communication for distributing execution across many Python interpreters across version, platform and network barriers. It has a minimal and fast API targetting the following uses:
- distribute tasks to (many) local or remote CPUs
- write and deploy hybrid multi-process applications
- write scripts to administer multiple environments
- automatic bootstrapping: no manual remote installation.
- safe and simple serialization of python builtin types for sending/receiving structured data messages. (New in 1.1) execnet offers a new dumps/loads API which allows cross-interpreter compatible serialization of Python builtin types.
- flexible communication: synchronous send/receive as well as callback/queue mechanisms supported
- easy creation, handling and termination of multiple processes
- well tested interactions between CPython 2.5-2.7, CPython-3.3, Jython 2.5.1 and PyPy interpreters.
- fully interoperable between Windows and Unix-ish systems.
- many tested examples
- py.test uses it for its distributed testing mechanism.
- quora uses it for connecting CPython and PyPy.
- Jacob Perkins uses it for his Distributed NTLK with execnet project to launch computation processes through ssh. He also compares disco and execnet in a subsequent post.
- Ronny Pfannschmidt uses it for his anyvc VCS-abstraction project to bridge the Python2/Python3 version gap.
- sysadmins and developers are using it for ad-hoc custom scripting
The current 1.0 series aims at basic API stabilization, improved tracing and robust termination.
The 1.1 series will target setting up permanent networks and offering unix-shell-like capabilities to spawn processes and applications.