You can instantiate a subprocess gateway, execute code in it and bidirectionally send messages:
>>> import execnet >>> gw = execnet.makegateway() >>> channel = gw.remote_exec("channel.send(channel.receive()+1)") >>> channel.send(1) >>> channel.receive() 2
The initiating and the remote execution happen concurrently. channel.receive() operations return when input is available. channel.send(data) operations return when the message could be delivered to the IO system.
The initiating and the “other” process work use a share-nothing model and channel.send|receive are means to pass basic data messages between two processes.
You can send and remote execute parametrized pure functions like this:
import execnet def multiplier(channel, factor): while not channel.isclosed(): param = channel.receive() channel.send(param * factor) gw = execnet.makegateway() channel = gw.remote_exec(multiplier, factor=10) for i in range(5): channel.send(i) result = channel.receive() assert result == i * 10 gw.exit()
The multiplier function executes remotely and establishes a loop multipliying incoming data with a constant factor passed in via keyword arguments to remote_exec.
You can pass a module object to remote_exec in which case its source code will be sent. No dependencies will be transferred so the module must be self-contained or only use modules that are installed on the “other” side. Module code can detect if it is running in a remote_exec situation by checking for the special __name__ attribute.
# content of a module remote1.py if __name__ == '__channelexec__': channel.send('initialization complete')
You can now remote-execute the module like this:
>>> import execnet, remote1 >>> gw = execnet.makegateway() >>> ch = gw.remote_exec(remote1) >>> print (ch.receive()) initialization complete
which will print the ‘initialization complete’ string.
A local subprocess gateway has the same working directory as the instantiatior:
>>> import execnet, os >>> gw = execnet.makegateway() >>> ch = gw.remote_exec("import os; channel.send(os.getcwd())") >>> res = ch.receive() >>> assert res == os.getcwd()
“ssh” gateways default to the login home directory.
Use simple execution to obtain information from remote environments:
>>> import execnet, os >>> gw = execnet.makegateway("ssh=codespeak.net") >>> channel = gw.remote_exec(""" ... import sys, os ... channel.send((sys.platform, sys.version_info, os.getpid())) ... """) >>> platform, version_info, remote_pid = channel.receive() >>> platform 'linux2' >>> version_info (2, 4, 2, 'final', 0)
Set a channel callback to immediately react on incoming data:
>>> import execnet >>> gw = execnet.makegateway() >>> channel = gw.remote_exec("for i in range(10): channel.send(i)") >>> l =  >>> channel.setcallback(l.append, endmarker=None) >>> channel.waitclose() # waits for closing, i.e. remote exec finish >>> l [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]
Note that the callback function will execute in the receiver thread so it should not block on IO or long to execute.
You can create and transfer a channel over an existing channel and use it to transfer information:
>>> import execnet >>> gw = execnet.makegateway() >>> channel = gw.remote_exec(""" ... ch1, ch2 = channel.receive() ... ch2.send("world") ... ch1.send("hello") ... """) >>> c1 = gw.newchannel() # create new channel >>> c2 = gw.newchannel() # create another channel >>> channel.send((c1, c2)) # send them over >>> c1.receive() 'hello' >>> c2.receive() 'world'
If you want the remote side to serve a number of synchronous function calls into your module you can setup a serving loop and write a local protocol.
import os # contents of: remotecmd.py def simple(arg): return arg + 1 def listdir(path): return os.listdir(path) if __name__ == '__channelexec__': for item in channel: channel.send(eval(item))
Then on the local side you can do:
>>> import execnet, remotecmd >>> gw = execnet.makegateway() >>> ch = gw.remote_exec(remotecmd) >>> ch.send('simple(10)') # execute func-call remotely >>> ch.receive() 11
Our remotecmd module starts up remote serving through the for item in channel loop which will terminate when the channel closes. It evaluates all incoming requests in the global name space and sends back the results.
In cases where you do not have SSH-access to a machine you need to download a small version-independent standalone socketserver.py script to provide a remote bootstrapping-point. You do not need to install the execnet package remotely. Simply run the script like this:
python socketserver.py :8888 # bind to all IPs, port 8888
You can then instruct execnet on your local machine to bootstrap itself into the remote socket endpoint:
import execnet gw = execnet.SocketGateway("TARGET-IP:8888")
That’s it, you can now use the gateway object just like a popen- or ssh-based one.
Here is some small server code that you can use to retrieve contents of remote files:
# content of servefiles.py def servefiles(channel): for fn in channel: f = open(fn, 'rb') channel.send(f.read()) f.close() if __name__ == "__channelexec__": servefiles(channel)
And here is some code to use it to retrieve remote contents:
import execnet import servefiles gw = execnet.makegateway("ssh=codespeak.net") channel = gw.remote_exec(servefiles) for fn in ('/etc/passwd', '/etc/group'): channel.send(fn) content = channel.receive() print(fn) print(content)