[py-dev] py.test and --session, re-run failures only

[Martin Blais]

On Apr 1, 2005 7:27 PM, holger krekel <hpk at trillke.net> wrote:
> Hi Martin,
> 
> nice to have you here!  I hope you had a good flight back to Canada.

thx
i drove back with mr docutils


> On Thu, Mar 31, 2005 at 21:23 -0500, Martin Blais wrote:
> > hi guys
> >
> > my first post on this list.  thanks for py.test, it's very nice, i'm
> > in the process of converting my cr*ppy unittest hacks into something a
> > little cleaner by using py.test.  in the past, i wrote some testing
> > infrastructure for a large C++ project in the past, i hope i can
> > contribute something to this project in the future.
> 
> sounds good!
> 
> > 1.
> > so i'm checking out py.test and i run across this:
> >
> >     --session          run a test session/rerun only failing tests.
> >
> > "rerun only failing tests" lets me think  that somehow the session is
> > recording somewhere (e.g. in a file) which tests have failed and only
> > reruns those tests.  in fact, fiddling in the code and running it, i
> > realized that it just runs a subprocess and monitors file changes (by
> > polling), and then automatically reruns the tests.  perhaps the help
> > string should be clarified.
> 
> Yes, although it's not exactly easy to put this in a short help text.
> If you have a suggestion please send it.
> 
> > in any case, it would be a cool feature to be able to record which
> > tests fail and then to be able to rerun just the ones which failed in
> > the past.  we used to have that in my previous job, and it was very
> > useful, we had LOTS of tests, and running them was very slow, so just
> > rerunniing the failures was a great timesaver.
> 
> Yes, recording failing tests is a good idea.  The problem is determining
> 
>     a) the location to store failing test information
>     b) the format in which "test addresses" should be stored
> 
> a) points to a more general problem (having per-user storage for the py
> lib and py.test in particular) that i'd like to solve in the near future.

how about a simple option for now?  this way a user can have multiple
session files if desired.  personally, when a cmdline gets too long, i
always stash it in a Makefile anyway so lots of options is not a
problem.  the makefile can determine whatever policy for selecting a
session file location.


> b) is more or less handlded in the "collect" branch which i will need to
> merge real soon now.  Basically a "test address" consists of
> a rootpath and a list of names.  It's important that such "addresses"
> will work nice for the upcoming doctest support.

i was thinking we could just use the funky extpy path syntax (like it
is seen with --collectonly).



> > 2.
> > i would like to be able to say "run just this one test", by specifying
> > the method, e.g. test_search.py:TestSearch.test_simple, or perhaps
> > some kind simpler pattern matching.  this is why i started fiddling in
> > the code.
> 
> Very valid use case and i have a plan how to satisfy it ...
> 
> > i would probably modify the driver to filter down the wanted methods
> > once the collector has passed.  any comments on how to go about this
> > welcome.
> 
> ... first of all, note that the collect branch contains the future version
> of py.test.  Second, i have added a chapter in the py.test documentation
> that describes future features (still open to discussion). Have a look at:
> 
> http://codespeak.net/py/current/doc/test.html#future-planned-features-of-py-test

oh, while we're taking a shot at the future, another feature that i
used to have was the ability to compare the test output to a fixed
file, we called this "expect files", because the test was expected to
generate specific output.  might not be useful now but you could note
that in the futures document as well, for some kinds of tests this can
be a quick-n-dirty way to write a test.  dump something on the output,
inspect it to make sure it's allright, dump that into an expect file. 
could be easy to implement since you already capture the output text. 
note that these kinds of tests fail on outputs sensitive to numerical
(im)precision.

and another one: for C++ code which can crash, a timeout option is
absolutely *essential*.

cheers,