.. include:: <s5defs.txt>

=================================================================
py.test: rapid testing with minimal effort 
=================================================================

:author: Holger Krekel, merlinux GmbH 
:event: 7.7.2008, EuroPython 2008, Vilnius 

What is py.test? 
===================

- **external** testing tool 
- automatically collects and executes tests 
- **minimal boilerplate** approach 
- **per-project customization** 
- many mature features 

Structure of Tutorial (I)
==========================

writing and running tests:
   - test functions
   - test classes 
   - generative tests
   - test driven development 
   - setup and teardown test state 
   - skipping tests
   - doctests 

Structure of Tutorial (II)
==========================

writing and using extensions:
   - conftest.py mechanism 
   - test collection hooks 
   - test running hooks 
   - existing extensions 

First: Install py.test 
============================

install "py.test" (see http://pylib.org):
  - via "easy_install py"
  - via download and "python setup.py install" 
  - via svn and adding py/bin to system $PATH 


Write and run a first test 
===================================

let's write a "test_ospath1.py"::

    import os.path, py

    def test_abspath():
        assert os.path.abspath("/a/b/c") == "/a/b/c" 
        p = "a/b" 
        cur = os.path.join(os.getcwd(), "a/b")
        assert os.path.abspath("a/b/c") == "/a/b/c" 

    def test_typeerror():
        assert py.test.raises(TypeError, "os.path.abspath()")
        XXX check why "os.path.abspath(3)" cannot display source lines correctly 

then issue on the command line: ``py.test test_ospath1.py`` 


Putting tests in Test Classes
==================================

test_ospath2.py (one test failing)::

    import os.path
    class TestOSPath:
        def test_absolute(self):
            assert os.path.abspath("/a/b/c") == "/a/b/c" 
        def test_relative(self):
            p = "a/b" 
            cur = os.path.join(os.getcwd(), "a/b")
            assert os.path.abspath("a/b/c") == cur 
        

observations and notes
========================

- py.test automatically collects ``test_*`` functions 
- use ``py.test --collectonly`` to inspect collection 
- you can write tests at module global level 
- no need to subclass from py.test
- assertion failures provide detailed info 

Generative / Parametrized tests 
=================================

test_ospath3.py::

    class TestOSPath:
        def test_gen_absolute(self):
            def checkabssame(p1):
                print "checking on", repr(p1) 
                assert os.path.abspath(p1) == p1 
            for s in "/ /a /a/b c".split():
                yield checkabssame, s

        def test_gen_relative(self):
            for p in "a a/b".split():
                p = "a/b" 
                expected = os.path.join(os.getcwd(), p)
                yield lambda x,y: x==y, p, expected 


Useful Features 
============================

- stdout/stderr is captured per-test 
- ``-l | --showlocals``: show local variables in traceback 
- ``--pdb``: jump into pdb for failures 
- ``-k KEYWORD``: select tests to run
- ``-x | --exitfirst``: stop on first test failure


Test Driven Development - at its best!
=======================================

meet ``py.test --looponfailing``:
  - does a full test run 
  - keeps a set of failing tests 
  - checks for file changes and re-runs failing tests 
  - if all failures are fixed, re-run the whole 

great for refactoring!


Setup and Teardown of test state
==================================

hooks at module, class, instance level, example::

    import py, os.path

    def setup_module(mod):
        mod.tmpdir = py.test.ensuretemp(__name__)

    class TestOS:
        def setup_method(self, method):
            self.tmpdir = tmpdir.mkdir(method.__name__)
            print "chdir() to ", self.tmpdir
            self.oldcwd = self.tmpdir.chdir()
        def teardown_method(self, method):
            self.oldcwd.chdir()

        def test_relative(self):
            p = "a/b" 
            expected = os.path.join(str(self.tmpdir), p)
            assert os.path.abspath(p) == expected  + "xxx"


Skipping tests 
===================

the above tests do not make sense for win32,
you can insert e.g.::

    class TestOSPosix:
        def setup_class(cls):
            if sys.platform == "win32":
                py.test.skip("posix platform required")

you can also place such calls to "py.test.skip()" inside test functions
or in other test setup functions. 

Doctests 
=================

py.test automatically collects test_doc.txt files::

    this is an example doc test for the py.test tutorial 

    >>> x = 3
    >>> print x
    3


Customizing py.test 
===========================

write "confest.py" files at project or global level: 

- integrate new test "items" or "collectors" 
- add command line options 
- influence the collection process 
- conftest.py's are picked up "upwards" 
- to aid debugging use:: 

    py.test --collectonly 
    py.test --traceconfig 


Important Internal Objects
==================================

**Session**: iterates over (custom) collection tree, executes
tests and reports outcomes. 

Collection Nodes ("py.test.collect.*"): 

**Directory**: collect files 

**Module**: collect test Classes and Functions 

**Class**/**Instance**: collect test methods 

**DoctestFile**: collect doctest of a .txt file 

**Generator**: collect generative tests 

***Function***: sets up and executes a python test function 

The collection process 
===========================

collection process forms a tree:
  - leaves: so called "items", e.g. ``py.test.collect.Function``
  - nodes: so called "collectors" contain further collectors or items 
  - collection tree is built iterativels

test collection usually starts from directories or files 

**dynamic lookup of Collector/Item class from conftest.py files**

Example conftest.py: control traversal 
===================================================

a conftest.py to influence collection::

    import py
    mydir = py.magic.autopath().dirpath()
    class Directory(py.test.collect.Directory):
        def run(self):
            if self.fspath == mydir:
                return ["x", "y", "z"]
            return super(Directory, self).run()    


Example conftest.py: add ReST check support 
==============================================

look into py/doc/conftest.py:

- this produces non-python test items
- ``py.test --collectonly`` shows the custom tree
- produces syntax and link checks 
- drop the conftest.py file into your own project/doc directory 

Example conftest.py: html page generation 
===========================================

http://codespeak.net/svn/user/arigo/hack/misc/htmlconftest
[courtesy Armin Rigo]

- creates html output with "clickable" failures 
- creates a nice historic view on test failures 
- see http://wyvern.cs.uni-duesseldorf.de/pypytest/summary.html
- see README.sh for usage for your project 


Example conftest.py: run Prolog tests 
======================================

http://codespeak.net/svn/user/cfbolz/jitpl/dist/test/conftest.py
[courtesy Armin Rigo, Carl Friedrich Bolz]

- creates PrologTest items 
- collects items and runs them through Prolog interpreter 


Example conftest.py: run Javascript tests 
=========================================================
                                                                                                                   
http://johnnydebris.net/svn/projects/jsbase     
[courtesy Guido Wesdorp]

- collects Javascript tests 
- sends them for execution to Spidermonkey JS interpreter 


Example conftest.py: cross-platform testing
=============================================

[courtesy Holger Krekel]

- goal: remotely run tests on windows 
- check out py/misc/conftest-socketgatewayrun.py
- requires a small server script on the windows side 
- transfers your source code to windows, runs tests, 
  reports on your terminal 


Example conftest.py: integrate traditional unittest's 
=========================================================
                                                                                                               
http://johnnydebris.net/svn/projects/py_unittest                                                                   
[courtesy Guido Wesdorp]
                                                                                                                   
- additionally collect TestCase subclasses from 
  all test_*.py Module and execute them 

- allows to mix xUnit and py style tests 

- or: alternatively use ``py/tool/utestconvert.py`` to permanently 
  convert xUnit to py style tests

Summary extensions
========================

- you can extend and modify the collection process 
- you can add new (non-python) test items 
- conftest.py files are usually "drop in" plugins

note, however: 

- no easy customization of reporting (yet) 
- details will change for py lib 1.0 release, thus 
  if you use special conftest's be sure to subscribe
  http://codespeak.net/mailman/listinfo/py-dev


XXX refactor / do extra slides see what could stay / below 














recap: main features 
======================

- assertions by assert statement, nice debugging 
- unittests, generative tests, doctests, rest-tests
- automatic customize collection of tests 
- select tests by keyword 
- capture stdout/stderr per-test 

ad-hoc distribution of tests 
===================================

``py.test --dist`` sends tests to remote places
or multiple processors

- for each host: 

  - setup (ssh) connection 
  - sync local source code to remote place
  - trigger running of (isolated) tests 

ad-hoc distribution of tests (2)
=====================================

- synchronise source code "1 -> N" simultaneously 

- uses py.execnet: 

  - can connect e.g. via SSH
  - expects plain Python executable on remote side 
  - no need to pre-install other software remotely 

- generated AJAX application async displays test results 

- demo

cross-platform testing 
=============================

- test run on linux, test execution on windows 
- uses py.execnet (same technique as "--dist")
- works well with "--looponfailing" 

status cross-platform/distributed testing
==============================================

- should basically work for any project 
- gives full tracebacks, most options work 

todo: 

- make introspection/pdb work 
- use ``screen`` for access to failed tests 
- accellerate py.execnet setup 
- more unification among testing modes

collect info about functions (apigen) 
=============================================

- collect function signatures (via settrace) 
- track input values/types, stacktraces, return values 
- web page with rich automatically produced information 
- TODO: decoupling of collecting info and generating html 

doctests 
==============

- basically work (automatically collected from text files) 
- uses 2.5's doctest module 
- needs more fine-grained integration 

Unifying Test Reporting 
=================================

py.test contains two approaches for processing test results: 

- "old style": methods invoked on session object 
- "new style": reporting events are sent to Reporter object 

next: 

- use reporting event architecture pervasively 
- refactor existing reporting extensions 

py.test extensions (conftest)
=================================

- all conftest.py files are considered in traversed directories 
- can extend/modify testing process in arbitrary ways 
- however: 

  - often requires too much knowledge of internal architecture 
  - no convention for organising shared test support code 

- ergo: introduce mechanism to share test support code / plugins 


Platform support 
======================

- py lib works on linux, freebsd, windows and OSX, ...
- works on python 2.3, 2.4, 2.5 
- py lib provides doctest/optparse/... unified compat modules
- todo: improve and automate packaging/installation 

test networks
==================

- currently model for 1:1 process-to-process 
- "A <-> B and B <-> C" connections do not imply "A <-> C" 
- fix that! 
- then: manage (large) network of test hosts 
- select set of test deployment hosts by platform, load
- have modes for testing on several platforms at once 

difference to "buildbot" ...
==================================

- py.test generally works from developers WC 
- no need to first commit and wait for later (nightly) test run 
- developer's choice of test setting 
- more interactivity 

summary development topics 
=============================

- refactor/unify reporting mechanisms 
- introduce shared test support code (plugins) 
- improve py.execnet (towards networks) 
- interactive debugging / introspection everywhere 
- systematically persist test results 

Goal
=============

Maximize fun and efficiency of (test-driven) development