tox: virtualenv-based automation of test activities

home |  install |  examples |  config |  support 

Table Of Contents

Using Tox with the Hudson Integration Server

Using Hudson multi-configuration jobs

The Hudson continous integration server allows to define “jobs” with “build steps” which can be test invocations. If you install tox on your default Python installation on each Hudson slave, you can easily create a Hudson multi-configuration job that will drive your tox runs from the CI-server side, using these steps:

  • create a “multi-configuration” job, give it a name of your choice

  • configure your repository so that Hudson can pull it

  • (optional) configure multiple nodes so that tox-runs are performed on multiple hosts

  • configure axes by using TOXENV as an axis name and as values provide space-separated test environment names you want Hudson/tox to execute.

  • add a Python-build step with this content (see also next example):

    import tox
    tox.cmdline() # environment is selected by ``TOXENV`` env variable
  • check Publish JUnit test result report and enter **/junit-*.xml as the pattern so that Hudson collects test results in the JUnit XML format.

The last point requires that your test command creates JunitXML files, for example with py.test it is done like this:

commands=py.test –junitxml=junit-{envname}.xml

See a real-life example in action with the pytest Hudson job

zero-installation for slaves

New in version 0.9.

If you manage many Hudson slaves and want to use the latest officially released tox (or latest development version) and want to skip manually installing tox then substitute the above Python build step code with this:

import urllib, os
url = ""
#os.environ['USETOXDEV']="1"  # use tox dev version
d = dict(__file__='')
exec urllib.urlopen(url).read() in d

The downloaded file downloads all neccessary files to install tox in a virtual sub environment. Notes:

  • uncomment the line containing USETOXDEV to use the latest development-release version of tox instead of the latest released version.
  • adapt the options in the last line as needed (the example code will cause tox to reinstall all virtual environments all the time which is often what one wants in CI server contexts)

Integrating “sphinx” documentation checks in a Hudson job

If you are using a multi-configuration Hudson job which collects JUnit Test results you will run into problems using the previous method of running the sphinx-build command because it will not generate JUnit results. To accomodate this issue one solution is to have py.test wrap the sphinx-checks and create a JUnit result file which wraps the result of calling sphinx-build. Here is an example:

  1. create a docs environment in your tox.ini file like this:

    changedir=doc # or whereever you keep your sphinx-docs
        py.test --tb=line -v --junitxml=junit-{envname}.xml
  2. create a doc/ file like this:

    import py
    import subprocess
    def test_linkcheck(tmpdir):
        doctrees = tmpdir.join("doctrees")
        htmldir = tmpdir.join("html")
            ["sphinx-build", "-W", "-blinkcheck",
              "-d", str(doctrees), ".", str(htmldir)])
    def test_build_docs(tmpdir):
        doctrees = tmpdir.join("doctrees")
        htmldir = tmpdir.join("html")
            "sphinx-build", "-W", "-bhtml",
              "-d", str(doctrees), ".", str(htmldir)])
  3. run tox -e docs and then you may integrate this environment along with your other environments into Hudson.

Note that py.test is only installed into the docs environment and does not need to be in use or installed with any other environment.

Access package artifacts between Hudson jobs

In an extension to Access package artifacts between multiple tox-runs you can also configure Hudson jobs to access each others artifacts. tox uses the distshare directory to access artifacts and in a Hudson context (detected via existence of the environment variable HUDSON_URL); it defaults to to {toxworkdir}/distshare.

This means that each workspace will have its own distshare directory and we need to configure Hudson to perform artifact copying. The recommend way to do this is to install the Hudson Copy Artifact plugin and for each job which “receives” artifacts you add a Copy artifacts from another project build step using roughly this configuration:

Project-name: name of the other (tox-managed) job you want the artifact from
Artifacts to copy: .tox/dist/*.zip   # where tox jobs create artifacts
Target directory: .tox/distshare     # where we want it to appear for us
Flatten Directories: CHECK           # create no subdir-structure

You also need to configure the “other” job to archive artifacts; This is done by checking Archive the artifacts and entering:

Files to archive: .tox/dist/*.zip

So our “other” job will create an sdist-package artifact and the “copy-artifacts” plugin will copy it to our distshare area. Now everything proceeds as Access package artifacts between multiple tox-runs shows it.

So if you are using defaults you can re-use and debug exactly the same tox.ini file and make use of automatical sharing of your artifacts between runs or Hudson jobs.