[py-dev] keyword selection

[Ian Bicking]

holger krekel wrote:
> I agree that is quite unsatisfactory. Did you see
> 
>     http://codespeak.net/pipermail/py-dev/2005-May/000357.html
> 
> where i asked for comments about test selection support?  
> I would be most interested in your opinion.  The presented ideas 
> should not be too hard to implement but i'd like to get some 
> feedback before actually doing it. 

Ah, yes, I saw that go by but didn't have any opinions (I guess I wasn't 
in the right mindset at the time).

> Now i plan to change/extend it in the following way: 
> 
>     py.test -m PATTERN  
> 
> would select all tests where the "test path" matches the given 
> pattern (python regexp).  You would use this to select out a 
> specific test function or method that you are currently working
> on.  Setup/teardown semantics continue to work as expected, of 
> course. 

Generally I dislike regex matching, I'd rather see shell-style path 
matching, like */*upload* -- maybe as an extension ** could match 
multiple "segments", like **/*upload* matches every test with "upload" 
in its name.  This feels very natural to me, whereas the regex 
equivalents -- "[^/]+/[^/]*upload[^/]*", "upload[^/]*" -- hurt my eyes.

> But for classifying and handling "sets of tests" it's nicer 
> to have some explicit keyword-tagging and selection scheme. 
> 
>     py.test -k "keyword1 -keyword2" 
> 
> would select (it does currently) tests that have keyword1 but
> not keyword2, think google syntax.  

I think that's a good way to match.  I assume "keyword1 keyword2" means 
keyword1 AND keyword2?  I can imagine that multiple -k arguments could 
mean OR.
> 
> However, i am very interested in suggestions on how keywords
> should be associated with tests more directly.  My current
> thinking (comments welcome!): 
> 
>     class TestClass: 
>         keywords = "core" 
> 
> which would tag all test methods and/or 
> 
>     def test_method(): 
>         ... 
>     test_method.keywords = "core" 
> 
> which would just tag one method and/or: 
> 
>     keywords = "core network ..." 
> 
> which would obviously tag all tests in that module accordingly. 

I think these seem generally good enough.  But should names themselves 
be keywords?  Maybe only if the keywords weren't specified explicitly.

> As a bonus task you could come up with a way to specify 
> doctests and associating keywords with them :-) 

Well, one nice aspect of doctests is that they ignore a lot, so maybe 
you'd just search for ^\s+keywords\s*=(.*)$ in the docstring.  As to a 
means of indicating that something is a doctest, or that a module should 
be doctested, hrm.  Well, the presence of the variable __test__ is a 
good sign, as doctest uses that for testing (both at the module and 
class level).  But even that requires loading all the modules, and most 
doctestable modules don't have __test__ anyway.  It might be good enough 
to just do some enumeration, probably using wildcards again (e.g., 
examples/**/*.py, docs/*.txt, etc.)

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org