py.test: rapid testing with minimal effort

Author: Holger Krekel, merlinux GmbH
event:7.7.2008, EuroPython 2008, Vilnius

What is py.test?

Structure of Tutorial (I)

writing and running tests:
  • test functions
  • test classes
  • generative tests
  • test driven development
  • setup and teardown test state
  • skipping tests
  • doctests

Structure of Tutorial (II)

writing and using extensions:
  • conftest.py mechanism
  • test collection hooks
  • test running hooks
  • existing extensions

First: Install py.test

install "py.test" (see http://pylib.org):
  • via "easy_install py"
  • via download and "python setup.py install"
  • via svn and adding py/bin to system $PATH

Write and run a first test

let's write a "test_ospath1.py":

import os.path, py

def test_abspath():
    assert os.path.abspath("/a/b/c") == "/a/b/c"
    p = "a/b"
    cur = os.path.join(os.getcwd(), "a/b")
    assert os.path.abspath("a/b/c") == "/a/b/c"

def test_typeerror():
    assert py.test.raises(TypeError, "os.path.abspath()")
    XXX check why "os.path.abspath(3)" cannot display source lines correctly

then issue on the command line: py.test test_ospath1.py

Putting tests in Test Classes

test_ospath2.py (one test failing):

import os.path
class TestOSPath:
    def test_absolute(self):
        assert os.path.abspath("/a/b/c") == "/a/b/c"
    def test_relative(self):
        p = "a/b"
        cur = os.path.join(os.getcwd(), "a/b")
        assert os.path.abspath("a/b/c") == cur

observations and notes

Generative / Parametrized tests

test_ospath3.py:

class TestOSPath:
    def test_gen_absolute(self):
        def checkabssame(p1):
            print "checking on", repr(p1)
            assert os.path.abspath(p1) == p1
        for s in "/ /a /a/b c".split():
            yield checkabssame, s

    def test_gen_relative(self):
        for p in "a a/b".split():
            p = "a/b"
            expected = os.path.join(os.getcwd(), p)
            yield lambda x,y: x==y, p, expected

Useful Features

Test Driven Development - at its best!

meet py.test --looponfailing:
  • does a full test run
  • keeps a set of failing tests
  • checks for file changes and re-runs failing tests
  • if all failures are fixed, re-run the whole

great for refactoring!

Setup and Teardown of test state

hooks at module, class, instance level, example:

import py, os.path

def setup_module(mod):
    mod.tmpdir = py.test.ensuretemp(__name__)

class TestOS:
    def setup_method(self, method):
        self.tmpdir = tmpdir.mkdir(method.__name__)
        print "chdir() to ", self.tmpdir
        self.oldcwd = self.tmpdir.chdir()
    def teardown_method(self, method):
        self.oldcwd.chdir()

    def test_relative(self):
        p = "a/b"
        expected = os.path.join(str(self.tmpdir), p)
        assert os.path.abspath(p) == expected  + "xxx"

Skipping tests

the above tests do not make sense for win32, you can insert e.g.:

class TestOSPosix:
    def setup_class(cls):
        if sys.platform == "win32":
            py.test.skip("posix platform required")

you can also place such calls to "py.test.skip()" inside test functions or in other test setup functions.

Doctests

py.test automatically collects test_doc.txt files:

this is an example doc test for the py.test tutorial

>>> x = 3
>>> print x
3

Customizing py.test

write "confest.py" files at project or global level:

Important Internal Objects

Session: iterates over (custom) collection tree, executes tests and reports outcomes.

Collection Nodes ("py.test.collect.*"):

Directory: collect files

Module: collect test Classes and Functions

Class/Instance: collect test methods

DoctestFile: collect doctest of a .txt file

Generator: collect generative tests

*Function*: sets up and executes a python test function

The collection process

collection process forms a tree:
  • leaves: so called "items", e.g. py.test.collect.Function
  • nodes: so called "collectors" contain further collectors or items
  • collection tree is built iterativels

test collection usually starts from directories or files

dynamic lookup of Collector/Item class from conftest.py files

Example conftest.py: control traversal

a conftest.py to influence collection:

import py
mydir = py.magic.autopath().dirpath()
class Directory(py.test.collect.Directory):
    def run(self):
        if self.fspath == mydir:
            return ["x", "y", "z"]
        return super(Directory, self).run()

Example conftest.py: add ReST check support

look into py/doc/conftest.py:

Example conftest.py: html page generation

http://codespeak.net/svn/user/arigo/hack/misc/htmlconftest [courtesy Armin Rigo]

Example conftest.py: run Prolog tests

http://codespeak.net/svn/user/cfbolz/jitpl/dist/test/conftest.py [courtesy Armin Rigo, Carl Friedrich Bolz]

Example conftest.py: run Javascript tests

http://johnnydebris.net/svn/projects/jsbase [courtesy Guido Wesdorp]

Example conftest.py: cross-platform testing

[courtesy Holger Krekel]

Example conftest.py: integrate traditional unittest's

http://johnnydebris.net/svn/projects/py_unittest [courtesy Guido Wesdorp]

Summary extensions

note, however:

XXX refactor / do extra slides see what could stay / below

recap: main features

ad-hoc distribution of tests

py.test --dist sends tests to remote places or multiple processors

ad-hoc distribution of tests (2)

cross-platform testing

status cross-platform/distributed testing

todo:

collect info about functions (apigen)

doctests

Unifying Test Reporting

py.test contains two approaches for processing test results:

next:

py.test extensions (conftest)

Platform support

test networks

difference to "buildbot" ...

summary development topics

Goal

Maximize fun and efficiency of (test-driven) development