execnet CHANGELOG

1.5.0

  • support shell escaping in python pathnames of popen.

    Eugene Ciurana discovered that execnet breaks if you use pathnames with spaces in a “python=” part of a spec. We now use shlex.split to split the string. There is a potential for regressions if you used quote or escape sequences as part of your python command.

  • Only insert importdir into sys.path if it is not already in the path.

    This prevents a bug when using enum34 with python 3.6 and pytest-xdist.

    The issue is that enum34 installs an ‘enum’ module in site-packages which is normally shadowed by the stdlib version of enum, however in gateway_bootstrap.py site-packages is added at the front the the search path. This means on the workers enum34 is hit for import enum which in turn causes import re to fail (as it makes use of the new enum features in 3.6).

  • fix #49 - use inspect.getfullargspec if possible to avoid deprecationwarnings

  • fix #56 - use partials in safe_terminate to avoid a bad carried binding

  • fix spec parsing on Windows due to path containing ‘’ characters.

1.4.1

  • fix a regression of the Serializer created by the implied opcode ordering which resulted in a incompatible opcode mapping

    warning stored serialized objects created with 1.4.0 are incompatible with previous versions and future versions additionally stored serialized objects containing complex objects will have a incompatible opcode when read with execnet < 1.4.0 and wont be loadable with execnet 1.4.0 either

    its strongly suggested to avoid using the Serializer of execnet 1.4.0 this affects devpi and the external pytest-cache plugin

1.4

  • de-vendor apipkg and use the pypi dependency instead (this also fixes the bpython interaction issues)
  • Fix issue38: provide ability to connect to Vagrant VMs easily using vagrant_ssh=defaut or vagrant_ssh=machinename this feature is experimental and will be refined in future releases. Thanks Christian Theune for the discussion and the initial pull request.
  • add support for serializing the “complex” type. Thanks Sebastian Koslowski.

1.3

  • fix issue33: index.txt to correctly mention MIT instead of GPL.
  • fix issue35: adapt some doctests, fix some channel tests for py3.
  • use subprocess32 when available for python < 3.
  • try to be a bit more careful when interpreter is shutting down to avoid random exceptions, thanks Alfredo Deza.
  • ignore errors on orphan file removal when rsyncing
  • fix issue34: limit use of import based bootstrap

1.2

  • fix issue22 – during interpreter shutdown don’t throw an exception when we can’t send a termination sequence anymore as we are about to die anyway.
  • fix issue24 – allow concurrent creation of gateways by guarding automatic id creation by a look. Thanks tlecomte.
  • majorly refactor internal thread and IO handling. execnet can now operate on different thread models, defaults to “thread” but allows for eventlet and gevent if it is installed.
  • gateway.remote_exec() will now execute in multiple threads on the other side by default. The previous neccessity of running “gateway.remote_init_threads()” to allow for such concurrency is gone. The latter method is now a no-op and will be removed in future versions of execnet.
  • fix issue20: prevent AttributError at interpreter shutdown by not trying to send close/last_message messages if the world around is half destroyed.
  • fix issue21: allow to create local gateways with sudo aka makegateway(“popen//python=sudo python”). Thanks Alfredo Deza for the PR.
  • streamline gateway termination and simplify proxy implementation. add more internal tracing.
  • if execution hangs in computation, we now try to send a SIGINT to ourselves on Unix platforms instead of just calling thread.interrupt_main()
  • change license from GPL to MIT
  • introduce execnet.dump/load variants of dumps/loads serializing/unserializing mechanism.
  • improve channel.receive() communication latency on python2 by changing the default timeout of the underlying Queue.get to a regular None instead of the previous default -1 which caused an internal positive timeout value (a hack probably introduced to allow CTRL-C to pass through for <python2.5 versions).
  • extended ssh-syntax to allow passing of command line args, e.g. “ssh= -p 50 hostname”. The options are passed to the underlying ssh client binary. Thanks tundish.
  • fix issue15: interoperability with inspect.getstack(). Thanks Peter Feiner.
  • fix issue10 : skip PYTHONDONTWRITEBYTECODE test if it we are running with PYTHONDONTWRITEBYTECODE set.
  • dont try the jython pid fixup on a RemoteIO
  • avoid accidentally setting exc_info() in gateway_base.py

1.1

  • introduce execnet.dumps/loads providing serialization between python interpreters.
  • group.remote_exec now supports kwargs as well
  • support per channel string coercion configuration
  • Popen2IO.read now reads correct amounts of bytes from nonblocking fd’s
  • added a dont_write_bytecode option to Popen gateways, this sets the sys.dont_write_bytecode flag on the spawned process, this only works on CPython 2.6 and higher. Thanks to Alex Gaynor.
  • added a pytest –broken-isp option to skip tests that assume DNS queries for unknown hosts actually are resolved as such (Thanks Alex Gaynor)
  • fix issue 1 - decouple string coercion of channels and gateway
  • fix issue #2 - properly reconfigure the channels string coercion for rsync, so it can send from python2 to python3
  • fix issue #9 - propperly terminate the worker threadpools in safe_terminate
  • fix issue #8 - no longer kill remote pids locally on jython ssh gateways
  • refactor socketserver, so it can be directly remote_exec’d for starting a socket gateway on a remote

1.0.9

  • add gw.reconfigure() to configure per gateway options. Currently supported: py2str_as_py3str and py3str_as_py2str to configure string deserialization
  • channel.makefile() objects now have a isatty() returning False
  • group.allocate_id(spec) allows to early-determine an (automatic) id
  • internal refactorings and cleanups (thanks Ronny Pfannschmidt): - refactor message types into received handler functions - refactor b(chr(opcode)) to bchr(opcode) - reorder Message ctor args, rename msgtype to msgcode - refactor gateway.send to take message’s init args instead of a message - inline and remove Message.writeto/readfrom - refactor collection loading to avoid the indirection over tuple - remove the unused NamedThreadPool

1.0.8

  • new gateway.remote_exec(func, **kwargs) style fo executing a pure function with parameters. The function on the remote side also needs to accept a channel which allows it to communicate back and forth. Thanks to Ronny Pfannschmidt for implementing it with special kudos to Maciej Fijalkowski for writing a “pure-function” checker so that on Python2.6 onwards non-pure functions will be rejected.
  • enhance rsyncing to also sync permissions (stat().st_mode) of directories and files. (should also resolve http://bitbucket.org/hpk42/py-trunk/issue/68/)
  • fix rsyncing of symlinks, thanks to Charles Solar (should also resolve http://bitbucket.org/hpk42/py-trunk/issue/70/)
  • update internal usage of apipkg to 1.0b6
  • remote_exec(module) now makes sure that the linecache is updated before reading and sending the source. thanks Ronny, Matt.
  • removed all trailing whitespace from source files

1.0.7

  • try to avoid a random KeyboardInterrupt Error when threads are ending.
  • extend xspec syntax to allow for one or multiple “env:NAME=value” environment variable settings which will be set on the remote side. (thanks Jakub Gustak)

1.0.6

  • fix jython/windows interactions
  • fix waitclose/callback-with-endmarker race condition
  • fix race condition where multiple threads sending data over channels would crash the serializer and process

1.0.5

  • more care during receiver-thread finalization during interp-shutdown, should get rid of annoying and meaningless exceptions
  • fix glitch in ssh-fileserver example
  • experimentally add “setup.py test” support - will run py.test

1.0.4

  • try to deal more cleanly with interpreter shutdown setting globals to None. this avoids (hopefully) some bogus tracebacks at process exit.

1.0.3

  • refine termination some more: CTRL-C and gateway.exit will now try harder to interrupt remote execution. this helps to avoid left-over ssh-processes.
  • fix read-on-non-blocking-files issue probably related to jython only: the low-level read on subprocess pipes may be non-blocking, returning less bytes than requested - so we now loop.
  • Windows/python2.4: fix bug that killing subprocesses would fail
  • make RemoteError and TimeoutError available directly on execnet namespace
  • fix some doc and test issues (thanks thm and ronny), add ssh_fileserver example
  • update internal copy of apipkg
  • always skip remote tests if no ssh specs given

1.0.2

  • generalize channel-over-channel sending: you can now have channels anywhere in a data structure (i.e. as an item of a container type). Add according examples.
  • automatically close a channel when a remote callback raises an exception, makes communication more robust because until now failing callbacks rendered the receiverthread unuseable leaving the remote side in-accessible.
  • internally split socket gateways, speeds up popen-gateways by 10% (now at <50 milliseconds per-gateway on a 1.5 GHZ machine)
  • fix bug in channel.receive() that would wrongly raise a TimeoutError after 1000 seconds (thanks Ronny Pfannschmidt)

1.0.1

  • revamp and better structure documentation
  • new method: gateway.hasreceiver() returns True if the gateway is still receive-active. remote_status now only carries information about remote execution status.
  • new: execnet.MultiChannel provides basic iteration/contain interface
  • new: execnet.Group can be indexed by integer
  • new: group.makegateway() uses group.default_spec if no spec is given and the execnet.default_group uses popen as a default spec.
  • have popen-gateways use imports instead of source-strings, also improves debugging/tracebacks, as a side effect popen-gateway startup can be substantially faster (>30%)
  • refine internal gateway exit/termination procedure and introduce group.terminate(timeout) which will attempt to kill all subprocesses that did not terminate within time.
  • EOFError on channel.receive/waitclose if the other side unexpectedly went away. When a gateway exits it now internally sends an explicit termination message instead of abruptly closing.
  • introduce a timeout parameter to channel.receive() and default to periodically internally wake up to let KeyboardInterrupts pass through.
  • EXECNET_DEBUG=2 will cause tracing to go to stderr, which with popen slave gateways will relay back tracing to the instantiator process.

1.0.0

  • introduce execnet.Group for managing gateway creation and termination. Introduce execnet.default_group through which all “global” calls are routed. cleanup gateway termination. All Gateways get an id through which they can be retrieved from a group object.
  • deprecate execnet.XYZGateway in favour of direct makegateway() calls.
  • refine socketserver-examples, experimentally introduce a way to indirectly setup a socket server (“installvia”) through a gateway url.
  • refine and automatically test documentation examples

1.0.0b3

  • fix EXECNET_DEBUG to work with win32
  • add support for serializing longs, sets and frozensets (thanks Benjamin Peterson)
  • introduce remote_status() method which on the low level gives information about the remote side of a gateway
  • disallow explicit close in remote_exec situation
  • perform some more detailed tracing with EXECNET_DEBUG

1.0.0b2

  • make internal protocols more robust against serialization failures
  • fix a seralization bug with nested tuples containing empty tuples (thanks to ronny for discovering it)
  • setting the environment variable EXECNET_DEBUG will generate per process trace-files for debugging

1.0.0b1

  • added new examples for NumPy, Jython, IronPython
  • improved documentation
  • include apipkg.py for lazy-importing
  • integrated new serializer code from Benjamin Peterson
  • improved support for Jython-2.5.1

1.0.0alpha2

  • improve documentation, new website
  • use sphinx for documentation, added boilerplate files and setup.py
  • fixes for standalone usage, adding boilerplate files
  • imported py/execnet and made it work standalone