basic local and remote communication¶
Execute source code in subprocess, communicate through a channel¶
You can instantiate a subprocess gateway, execute code in it and bidirectionally send messages:
>>> import execnet >>> gw = execnet.makegateway() >>> channel = gw.remote_exec("channel.send(channel.receive()+1)") >>> channel.send(1) >>> channel.receive() 2
The initiating and the remote execution happen concurrently.
channel.receive() operations return when input is available.
channel.send(data) operations return when the message could
be delivered to the IO system.
The initiating and the “other” process work use a share-nothing
channel.send|receive are means to pass basic data
messages between two processes.
remote-exec a function (avoiding inlined source part I)¶
You can send and remote execute parametrized pure functions like this:
import execnet def multiplier(channel, factor): while not channel.isclosed(): param = channel.receive() channel.send(param * factor) gw = execnet.makegateway() channel = gw.remote_exec(multiplier, factor=10) for i in range(5): channel.send(i) result = channel.receive() assert result == i * 10 gw.exit()
multiplier function executes remotely and establishes
a loop multipliying incoming data with a constant factor passed
in via keyword arguments to
- unfortunately, you can not type this example interactively because
inspect.getsource(func)fails for interactively defined functions.
- on Python2.6 and onwards you will get an explicit error if you try to execute non-pure functions, i.e. functions that access any global state (which will not be available remotely as we have a share-nothing model between the nodes).
remote-exec a module (avoiding inlined source part II)¶
You can pass a module object to
remote_exec in which case
its source code will be sent. No dependencies will be transferred
so the module must be self-contained or only use modules that are
installed on the “other” side. Module code can detect if it is
running in a remote_exec situation by checking for the special
# content of a module remote1.py if __name__ == '__channelexec__': channel.send('initialization complete')
You can now remote-execute the module like this:
>>> import execnet, remote1 >>> gw = execnet.makegateway() >>> ch = gw.remote_exec(remote1) >>> print (ch.receive()) initialization complete
which will print the ‘initialization complete’ string.
Compare current working directories¶
A local subprocess gateway has the same working directory as the instantiatior:
>>> import execnet, os >>> gw = execnet.makegateway() >>> ch = gw.remote_exec("import os; channel.send(os.getcwd())") >>> res = ch.receive() >>> assert res == os.getcwd()
“ssh” gateways default to the login home directory.
Get information from remote ssh account¶
Use simple execution to obtain information from remote environments:
>>> import execnet, os >>> gw = execnet.makegateway("ssh=codespeak.net") >>> channel = gw.remote_exec(""" ... import sys, os ... channel.send((sys.platform, tuple(sys.version_info), os.getpid())) ... """) >>> platform, version_info, remote_pid = channel.receive() >>> platform 'linux2' >>> version_info (2, 6, 6, 'final', 0)
Use a callback instead of receive() and wait for completion¶
Set a channel callback to immediately react on incoming data:
>>> import execnet >>> gw = execnet.makegateway() >>> channel = gw.remote_exec("for i in range(10): channel.send(i)") >>> l =  >>> channel.setcallback(l.append, endmarker=None) >>> channel.waitclose() # waits for closing, i.e. remote exec finish >>> l [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]
Note that the callback function will execute in the receiver thread so it should not block on IO or long to execute.
Sending channels over channels¶
You can create and transfer a channel over an existing channel and use it to transfer information:
>>> import execnet >>> gw = execnet.makegateway() >>> channel = gw.remote_exec(""" ... ch1, ch2 = channel.receive() ... ch2.send("world") ... ch1.send("hello") ... """) >>> c1 = gw.newchannel() # create new channel >>> c2 = gw.newchannel() # create another channel >>> channel.send((c1, c2)) # send them over >>> c1.receive() 'hello' >>> c2.receive() 'world'
a simple command loop pattern¶
If you want the remote side to serve a number of synchronous function calls into your module you can setup a serving loop and write a local protocol.
import os # contents of: remotecmd.py def simple(arg): return arg + 1 def listdir(path): return os.listdir(path) if __name__ == '__channelexec__': for item in channel: channel.send(eval(item))
Then on the local side you can do:
>>> import execnet, remotecmd >>> gw = execnet.makegateway() >>> ch = gw.remote_exec(remotecmd) >>> ch.send('simple(10)') # execute func-call remotely >>> ch.receive() 11
Our remotecmd module starts up remote serving
for item in channel loop which
will terminate when the channel closes. It evaluates
all incoming requests in the global name space and
sends back the results.
Instantiate gateways through sockets¶
In cases where you do not have SSH-access to a machine you need to download a small version-independent standalone socketserver.py script to provide a remote bootstrapping-point. You do not need to install the execnet package remotely. Simply run the script like this:
python socketserver.py :8888 # bind to all IPs, port 8888
You can then instruct execnet on your local machine to bootstrap itself into the remote socket endpoint:
import execnet gw = execnet.makegateway("socket=TARGET-IP:8888")
That’s it, you can now use the gateway object just like a popen- or ssh-based one.
Receive file contents from remote SSH account¶
Here is some small server code that you can use to retrieve contents of remote files:
# content of servefiles.py def servefiles(channel): for fn in channel: f = open(fn, 'rb') channel.send(f.read()) f.close() if __name__ == "__channelexec__": servefiles(channel)
And here is some code to use it to retrieve remote contents:
import execnet import servefiles gw = execnet.makegateway("ssh=codespeak.net") channel = gw.remote_exec(servefiles) for fn in ('/etc/passwd', '/etc/group'): channel.send(fn) content = channel.receive() print(fn) print(content)