[py-dev] running doctest-based tests

[holger krekel]

Hi Max, hi David, hi all, 

On Fri, Jun 10, 2005 at 13:07 +0300, Max Ischenko wrote:
> Holger,
> > > My situation requires a custom representation of a failure instead of
> plain
> > > traceback.
> > >
> > > A solution might be to abstract the way failures are reported by moving
> this
> > > responsibility from TerminalSession to the Failure class, for example.
> After
> > > all, this even seems logical - different kind of failures may require
> > > different representations and a Failure class is a natural place to hold
> > > this information.  OTOH, the session classes *are* responsible for an
> actual
> > > failure's representation: be it console output, graphical window or
> network
> > > socket.
> > >
> > > A method to the Failure class will be added that will return a suitable
> > > string representation of a failure. Failure class itself will provide
> > > default implementation that will yield formatted traceback.  This way I
> > > could subclass Failure that will yield doctest output instead of
> traceback.
> > >
> > > What would you say?
> > 
> > your analysis is pretty correct.  However, my idea was to just let
> > py.test.skip(someobj) internally take care that str(someobj) provides
> > the failure message representation.  Doing that by an internal
> > subclass is fine but an implementation detail that should not get
> > exposed (Failure classes are not visible directly).
> 
> Ah, now I got you, thanks for patience.
> 
> Yes, the trick with skip() works fine but, IMO, is no more then a temp hack.
> Errors in doctest-based tests are failures and should be reported as such. 

Oh sure, i am sorry.  Of course, i should have talked about 
py.test.fail() all along! skip() doesn't make any sense here. 

I also think that your idea of just copying and maybe slightly modifying 
the 2.4 doctest module into the py lib makes sense. The general idea is
that py.test and the py lib should work sanely on python 2.2, 2.3, 2.4 
and 2.5/cvs. 

If we integrate doctest support natively, then i now think that we 
should aim for offering simple conftest.py configuration items like: 

    test_files = ['test_foo.py', 'test_bar.py']
    extra_doctest_files = ['whatever.py', ...]

and test files should also be searched for doctests, btw. There would 
then be no need to write custom Collectors/Items for changing the order 
or the list of to-be-examined doctest modules.  The above example
file lists would only be associated with the directory containing
the conftest.py file and not any subdirectories.  On the implementation 
side, the given files should always be imported via the path.pyimport() 
method which imports modules with their fully qualified name. 
(http://codespeak.net/py/current/doc/test.html#package-name) 

David, are you interested in taking a stab at this?  It's a bit
involved, i am afraid, also because it involves changing some reporting
functionality.  If you or anyone else would like to do it, please make 
sure to write proper tests for any new features even though it's sometimes 
a bit involved to do that.  I sometimes spend half the time for writing 
tests for new py lib features but it really pays off. 

cheers, 

    holger