[z3-five] Re: Functional doctests for Five

[Jim Fulton]

Tres Seaver wrote:

...

> I *am* worried, because I don't buy the "story" case as "best practice" 
> for unit testing (yes, I am aware that Jim disagrees with me). 

Yup. :)

 > I prefer
> *tiny* unit tests, which test exactly *one* set of initial conditions 
> and the expected outcome.  E.g.:
> 
>   def test_setValue_default_args(self):
> 
>       object = self._makeOne()
>       object.setValue()
>       self.assertEqual(object.getValue(), 'default')
> 
>   def test_setValue_explicit_bar(self):
> 
>       object = self._makeOne()
>       object.setValue('explicit')
>       self.assertEqual(object.getValue(), 'explicit')
> 
> I do *not* think that tests of "mutators" should interact with one 
> another, in particular;  the case gets even dodgier when module-level 
> globals are in play (e.g., registries).

I think that this is a legitimate but extreme view.  We followed this
sort of approach for a long time. We created thousands of tests.
The result was a bit of a disaster.  Tests like those above easily become
inscrutable.  I have seen no cases where the isolation shown above
improved the coverage of the tests.

AFAICT, you are afraid of the case that a test passes spuriously
because some other test code ran.  While I agree that this is a risk,
it is, in my experience a small one.  After all, we have many non-doctest
tests that test more than obe thing and I can't think of any cases where
this has been a problem, although I'm sure there are some.

Not using doctest-based tests raises the effort to create, understand,
and maintain tests to an unacceptable level.

> I *do* believe that the documentation value of the "story" is 
> worthwhile;  what I *don't* believe is that the "story" does as good a 
> job actually testing the underlying code.

I don't agree. I agree with your objection, but I consider the cost
of less-fine-grained tests to be much smaller than you do.  I find, OTOH
that story-based tests tend to provide better test coverage.  It's easier,
in my experience, to reason about the completeness of tests within the
context of a story than in isolation.

There is still a place for fine-grained tests deal with edge cases
that would detract from a story.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org