execnet ad-hoc instantiates local and remote Python interpreters. Each interpreter is accessible through a Gateway which manages code and data communication. Channels allow to exchange data between the local and the remote end. Groups help to manage creation and termination of sub interpreters.
All Gateways are instantiated via a call to makegateway() passing it a gateway specification or URL.
create and configure a gateway to a Python interpreter.
The spec string encodes the target gateway type and configuration information. The general format is:
If you leave out the =value part a True value is assumed. Valid types: popen, ssh=hostname, socket=host:port. Valid configuration:
id=<string> specifies the gateway id python=<path> specifies which python interpreter to execute chdir=<path> specifies to which directory to change nice=<path> specifies process priority of new process
If no spec is given, self.defaultspec is used.
Here is an example which instantiates a simple Python subprocess:
>>> gateway = execnet.makegateway()
All gateways offer a simple method to execute source code in the connected interpreter:
It is allowed to pass a module object as source code in which case it’s source code will be obtained and get sent for remote execution. remote_exec returns a channel object whose symmetric counterpart channel is available to the remotely executing source.
A channel object allows to send and receive data between two asynchronously running programs.
- sends the given item to the other side of the channel, possibly blocking if the sender queue is full. The item must be a simple python type and will be copied to the other side by value. IOError is raised if the write pipe was prematurely closed.
- receive a data item that was sent from the other side. timeout: -1 [default] blocked waiting, but wake up periodically to let CTRL-C through. A positive number indicates the number of seconds after which a channel.TimeoutError exception will be raised if no item was received. Note that exceptions from the remotely executing code will be reraised as channel.RemoteError exceptions containing a textual representation of the remote traceback.
- Channel.setcallback(callback, endmarker=_NOENDMARKER)¶
set a callback function for receiving items.
All already queued items will immediately trigger the callback. Afterwards the callback will execute in the receiver thread for each received data item and calls to receive() will raise an error. If an endmarker is specified the callback will eventually be called with the endmarker when the channel closes.
- Channel.makefile(mode, proxyclose=False)¶
- return a file-like object. mode can be ‘w’ or ‘r’ for writeable/readable files. if proxyclose is true file.close() will also close the channel.
- close down this channel with an optional error message. Note that closing of a channel tied to remote_exec happens automatically at the end of execution and cannot be done explicitely.
- wait until this channel is closed (or the remote side otherwise signalled that no more data was being sent). The channel may still hold receiveable items, but not receive any more after waitclose() has returned. Exceptions from executing code on the other side are reraised as local channel.RemoteErrors. EOFError is raised if the reading-connection was prematurely closed, which often indicates a dying process. self.TimeoutError is raised after the specified number of seconds (default is None, i.e. wait indefinitely).
- Exception containing a stringified error from the other side.
- Exception indicating that a timeout was reached.
All created gateway instances are part of a group. If you call execnet.makegateway it actually is forwarded to the execnet.default_group. Group objects are container objects (see group examples) and manage the final termination procedure:
This method is implicitely called for each gateway group at process-exit, using a small timeout. This is fine for interactive sessions or random scripts which you rather like to error out than hang. If you start many processes then you often want to call group.terminate() yourself and specify a larger or not timeout.
All gateways offer a simple method to obtain some status information from the remote side.
Calling this method tells you e.g. how many execution tasks are queued, how many are executing and how many channels are active. Note that remote_status() works even if the other side is busy executing code and can thus be used to query status in a non-blocking manner.
execnet implements a simple efficient rsyncing protocol. Here is a basic example for using RSync:
rsync = execnet.RSync('/tmp/source') gw = execnet.makegateway() rsync.add_target(gw, '/tmp/dest') rsync.send()
And here is API info about the RSync class.
This class allows to send a directory structure (recursively) to one or multiple remote filesystems.
There is limited support for symlinks, which means that symlinks pointing to the sourcetree will be send “as is” while external symlinks will be just copied (regardless of existance of such a path on remote side).
By setting the environment variable EXECNET_DEBUG you can configure a tracing mechanism:
|write per-process trace-files to execnet-debug-PID|
|perform tracing to stderr (popen-gateway slaves will send this to their instantiator)|