Managing multiple gateways and clusters¶
Usings Groups for managing multiple gateways¶
Use execnet.Group
to manage membership and lifetime of
of multiple gateways:
>>> import execnet
>>> group = execnet.Group(['popen'] * 2)
>>> len(group)
2
>>> group
<Group ['gw0', 'gw1']>
>>> list(group)
[<Gateway id='gw0' receive-live, thread model, 0 active channels>, <Gateway id='gw1' receive-live, thread model, 0 active channels>]
>>> 'gw0' in group and 'gw1' in group
True
>>> group['gw0'] == group[0]
True
>>> group['gw1'] == group[1]
True
>>> group.terminate() # exit all member gateways
>>> group
<Group []>
Assigning Gateway IDs¶
All gateways are created as part of a group and receive
a per-group unique id
after successful initialization.
Pass an id=MYNAME
part to group.makegateway
. Example:
>>> import execnet
>>> group = execnet.Group()
>>> gw = group.makegateway("popen//id=sub1")
>>> assert gw.id == "sub1"
>>> group['sub1']
<Gateway id='sub1' receive-live, thread model, 0 active channels>
Getting (auto) IDs before instantiation¶
Sometimes it’s useful to know the gateway ID ahead of instantiating it:
>>> import execnet
>>> group = execnet.Group()
>>> spec = execnet.XSpec("popen")
>>> group.allocate_id(spec)
>>> allocated_id = spec.id
>>> gw = group.makegateway(spec)
>>> assert gw.id == allocated_id
execnet.makegateway uses execnet.default_group¶
Each time you create a gateway with execnet.makegateway()
you actually use the execnet.default_group
:
>>> import execnet
>>> gw = execnet.makegateway()
>>> gw in execnet.default_group
True
>>> execnet.default_group.defaultspec # used for empty makegateway() calls
'popen'
Robust Termination of ssh/popen processes¶
Use group.terminate(timeout)
if you want to terminate
member gateways and ensure that no local sub processes remain
you can specify a timeout
after which an attempt at killing
the related process is made:
>>> import execnet
>>> group = execnet.Group()
>>> gw = group.makegateway("popen//id=sleeper")
>>> ch = gw.remote_exec("import time ; time.sleep(2.0)")
>>> group
<Group ['sleeper']>
>>> group.terminate(timeout=1.0)
>>> group
<Group []>
execnet aims to provide totally robust termination so if you have left-over processes or other termination issues please report them. thanks!
Using Groups to manage a certain type of gateway¶
Set group.defaultspec
to determine the default gateway
specification used by group.makegateway()
:
>>> import execnet
>>> group = execnet.Group()
>>> group.defaultspec = "ssh=localhost//chdir=mytmp//nice=20"
>>> gw = group.makegateway()
>>> ch = gw.remote_exec("""
... import os.path
... basename = os.path.basename(os.getcwd())
... channel.send(basename)
... """)
>>> ch.receive()
'mytmp'
This way a Group object becomes kind of a Gateway factory where the factory-caller does not need to know the setup.