[py-dev] Integrating doctests

[Ian Bicking]

holger krekel wrote:
> Hi Ian, 
> 
> i have been away for a few days (and everyone seems to know this
> and mails me like crazy).   I have the impression you might have
> want to split your mail into multiple ones, but maybe i am just 
> missing the red line somehow. 
> 
> On Tue, May 10, 2005 at 18:04 -0500, Ian Bicking wrote:
> 
>>OK, so I really want to integrate doctests, along the lines I mention in
>>this post:
>>
>>http://blog.ianbicking.org/building-testability-into-web-applications.html
> 
> 
> Can't reach that server at the moment (not the first time, btw). 

Yeah, I know, it sucks :(

> 
>>So, here's what I have so far.  There's a couple parts.  Here's a
>>function to test unit tests:
> 
> 
> You started off with doctests, and now talk about unittests. Probably
> your weblog entry would help understand me your line of thinking better ... 

Sorry.  Doctests can be turned into unittest TestCases (TestSuites 
actaully, but I then turn those into a flat list of cases).  I was using 
that to integrate the doctests.  It's probably better to do it more 
flatly without the levels of wrappers, but the problems are essentially 
the same either way.  The architecture is fairly similar -- 
doctest.DocTest along with doctest.DocTestRunner are like 
unittest.TestCase and/or unittest.TestSuite, with a little 
unittest.TestResult mixed in.

The actual building of unittest test cases from doctests is a little 
crude, so it's probably not the best entry point, but it was expedient 
at the time.

>>def unittest_tester(test):
>>    result = test.defaultTestResult()
>>    test.run(result)
>>    for flavor, lst in [('FAIL', result.failures),
>>                        ('ERROR', result.errors)]:
>>        for test, err in lst:
>>            print '%s: %s' % (flavor, test)
>>            print err
>>    if result.failures or result.errors:
>>        py.test.fail('Errors occurred')
>>
>>This is a little lame, but I haven't gotten far enough into py.test to
>>figure out reporting.
> 
> 
> There is definitely a missing py.test customization hook which
> allows a user to customize reporting from e.g. a conftest.py
> file or from simply invoking py.test.fail()

That would be very useful, and probably solve most of the problems of 
integrating foreign test systems.

>>Actually, I still feel totally mystified by py.test, I just
>>wander around poking things until they stop breaking.  Is
>>there anyplace I should be starting at to figure out quite
>>what is going on?
> 
> 
> I at least wrote some documentation about the internal workings: 
> 
> http://codespeak.net/py/current/doc/test.html#collecting-and-running-tests-implementation-remarks
> 
> (sorry for the long URL).  Have you read that and deemed it unworthy
> or unhelpful? 

I read it a while ago, but I didn't look it up again.  It is helpful. 
There are some internal methods (like buildname2items) that would also 
be useful to have documented here, as they are useful for subclassing.

I was also confused about self.obj -- what is was and where it came 
from.  Especially since there's .obj(), .obj, and ._getobj().  The 
Function properties also confused me.

>>It's kind of that Ravioli Code kind of feel; a description
>>of the role of all the classes would be really useful.
>>Though personally I like descriptions that follow the code
>>path chronologically, it's a good compliment to the source
>>which tends not to have many chronological hints.
> 
> 
> I had the impression that i keep py-dev updated about refactorings
> and the intentions and even provide some detail of what's going 
> on.  But that can certainly be improved. 

I don't mean to complain particularly.  But diving into the code there's 
some parts that threw me.  If I had read the part about the collection 
process just before I did this, that probably would have helped 
considerably, just because I'd know what method to start reading from.

>>Anyway, I needed that function because I'm using doctest's
>>TestSuite-building capability, though it'd probably be just as easy to
>>use doctest.DocTest directly.  Anyway, I really think unittest testing
>>is a useful feature that belong in there somewhere, 
> 
> 
> You are mixing unittests and doctests freely here.  I admit i
> still haven't looked to closely into doctest's internal workings 
> so do you mean to imply that doctest and unittests are very
> uniformly handled with Python already? 
> 
> 
>>...  in addition to doctests, since I don't always have the
>>ability to change other projects' test structure.
> 
> 
> so here you mean that py.test should grok existing
> (unit/doc-)test structures?  I am not against the idea but it
> is a long way to follow that road i am afraid and i am not
> ready to follow that myself at the moment. (apart from the
> fact that we have certain ways of running unittests/doctests
> in PyPy with py.test mechanisms but it's code that i'd rather
> not like to explain at the moment, because it jumps through
> more hoops than neccessary: it additionally runs against 
> the PyPy interpreter instead of against CPython). 

It doesn't seem too hard -- collecting the tests seems easy from what I 
can tell; it's just a matter of doing some isinstance(obj, 
unittest.TestCase) tests, and then this small wrapper to turn test cases 
into py.test Items.  From there it would be nice to add configuration, 
e.g., pass through the verbosity setting, but that's not a core feature. 
  This would simulate most of what unittest.main() does, which isn't a 
whole lot.  To me it seems like it's almost there.  The reporting hook 
would be important.  The collectors we have here are close to good 
enough, at least as a first go.  Is there something I'm missing?  I 
don't think it has to duplicate everyone's custom loaders and whatnot; 
the only valuable part of current tests (IMHO) are the actual TestCases.

>>I don't think the support has to be very sophisticated; fix
>>the errors in that example and it might be good enough.  If
>>it doesn't integrate particularly well into other UIs, like
>>Tkinter, I don't think that's a big deal.
> 
> 
> I'd like Jan Balster's tkinter frontend to remain as much
> usable as possible and although i break Jan's code from
> time to time (sorry, Jan!) i'd like it very much to be
> an alternative frontend. 

By "not particularly well" I meant more "looks like the console embedded 
in a Tkinter window".  If that interface gains new fancy features -- 
which would be great, and I'm also very interested in -- then maybe 
unittest-based tests wouldn't benefit from those features.  But that's 
OK by me.  And maybe doctests would be left behind for a while too, 
which I'd be sad about, but we can fix that up as we go along.

> Right.  A better way would be to override the Directory collector 
> and produce two kinds of Modules, usual py.test ones and ones 
> with doctests with code like this (untested): 
> 
>     class Directory(py.test.collect.Directory): 
>         def buildname2items(self): 
>             # just let our base class do its building
>             d = super(Directory, self).buildname2items()
> 
>             # let's look for doctests ... 
>             for fn in self.fspath.listdir('*.py'): 
>                 if fn.basename in d or fn.basename == 'conftest.py': 
>                     continue
>                 # we have a candidate for doctests 
>                 if not isdoctestmodule(fn): # if possible to implement that
>                     continue
>                 d[fn.basename] = DoctestModule(fn, parent=self) 

I think if you run DocTestFinder on a module you'll get an error or 
empty list.  I suppose you could override DoctestModule.__init__ and 
pass the results from DocTestFinder in.

>     class DoctestModule(py.test.collect.Module): 
>         def run(self): 
>             module = self.fspath.pyimport()
>             # setup module appropriately 
>             try: 
>                 # run doctests from 'module'
>             finally: 
>                 # teardown module appropriately 
> 
> Like said earlier, this DoctestModule should have a way to "take over" 
> reporting somehow. It doesn't at the moment and be warned that there 
> might be other slight issues.   But basically the above should  
> work rather cleanly. 
> 
>>* I'd like a way to halt all tests.  With SQLObject your environment
>>must be configured properly, since you have to access a database.  It's
>>overwhelming when this isn't the case, because you get a huge list of
>>failing tests.  I'd like to raise an error when that happens so that
>>py.test will abort all future tests.  Hmm... I haven't tried SystemExit
>>or KeyboardInterrupt.  Maybe that'd do on its own.
> 
> 
> Have you tried py.test.exit(msg)? 

No I have not, but I will.

> P.S.: Please note that py.test is still under development and there are 
>       sometimes reasons why things are not done yet: often i prefer to 
>       refactor and consolidate the (not so small number of) features 
>       before i head off adding yet more features.

I understand.  I'm just trying to say that almost all the features are 
already there, it's just a couple small things.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org