[py-dev] test selection/sets - request for comments

[holger krekel]

hi folks, 

recently py.test grew some "test selection" support as a
means to restrict the subset of tests one wants to run.  It's
preliminary and somewhat experimental, see 

    http://codespeak.net/py/current/doc/test.html#selecting-tests-by-keyword

Now i plan to change/extend it in the following way: 

    py.test -m PATTERN  

would select all tests where the "test path" matches the given 
pattern (python regexp).  You would use this to select out a 
specific test function or method that you are currently working
on.  Setup/teardown semantics continue to work as expected, of 
course. 

But for classifying and handling "sets of tests" it's nicer 
to have some explicit keyword-tagging and selection scheme. 

    py.test -k "keyword1 -keyword2" 

would select (it does currently) tests that have keyword1 but
not keyword2, think google syntax.  In PyPy we are already utilizing
the latter form by specifying keywords like 'core' for running
CPython regression tests tagged with the 'core' keyword.  See 

    http://codespeak.net/~hpk/pypy-testresult/

for what we make out of this (we distributedly checkin test 
reports which get evaluated at a central site). 

Currently, it's only possible to associate keywords with tests
by customizing the collection process in a conftest.py file
(which is probably neccessary at least for projects like PyPy
that deeply change the meaning of modules as they 
are loaded and run through the PyPy interpreter).  

However, i am very interested in suggestions on how keywords
should be associated with tests more directly.  My current
thinking (comments welcome!): 

    class TestClass: 
        keywords = "core" 

which would tag all test methods and/or 

    def test_method(): 
        ... 
    test_method.keywords = "core" 

which would just tag one method and/or: 

    keywords = "core network ..." 

which would obviously tag all tests in that module accordingly. 

One usage idea is that one could tag tests with e.g. a
"pendingbug" keyword and would not normally run that unless
you explicitely cared for pending bugs.  This also requires a
way to specify a default set of keywords so that you could set
the default set of to-be-run-keywords to e.g. "-pendingbug" and would
thus not run pending bug tests by default.  Of course, there is
nothing holding you from using bugnumbers like "bug123" that
relate to some issue tracking system. 

Obviously to nail down the exact semantics and coming
up with an easy-to-explain usage/user-level API is not 
completely trivial.  So please share your thoughts and 
usage-ideas (and don't worry about the implementation
too much :-) 

As a bonus task you could come up with a way to specify 
doctests and associating keywords with them :-) 

cheers & thanks, 

    holger

P.S.: all selection/tagging would first be done at 
      collection runtime but for the future we can think about 
      introducing cache file(s) that knows about 
      all test names and keywords so that selecting
      tests by PATTERN or keyword basically becomes 
      an O(1) operation.