[z3-five] Re: Functional doctests for Five

[Philipp von Weitershausen]

Tres Seaver wrote:
> Philipp von Weitershausen wrote:
> 
>> Martijn Faassen wrote:
>>
>>> Philipp von Weitershausen wrote:
>>>
>>>> Martijn Faassen wrote:
>>>
>>>
>>>>>> Sidnei implemented functional doctests in ZopeTestCase;
>>>>>> Create a branch to convert Five tests to this.
>>>>>
>>>>>
>>>>> All of them?
>>>>
>>>>
>>>> Why not? :)
>>>
>>>
>>> Because I think some of the tests exercise python-level 
>>> functionality, though I think most don't and make sense as functional 
>>> tests.
>>
>>
>>
>> I am converting regular Five unit tests to unit doctests and 
>> functional tests to functional doctests. I am just converting between 
>> two formats of writing down tests. I'm not changing the metrics by 
>> which we test.
> 
> 
> Note that unless you convert them using the 
> "docstring-on-an-empty-method" spelling, you will be changing semantics 
> of the tests:  the "tell-a-story-in-a-text-file" model deliberated does 
> *not* isolate tests from one another.  This is my major beef with 
> doctests:  the "isolated" spelling doesn't have the win that the "story" 
> case does, but the "story" case doesn't (can't) do as good a job of 
> "test one thing only."

Yes you are right. However, most of the time that doesn't affect the 
single test because it tests something that depends on the test setUp(), 
not the stuff that other tests did before (because the other tests 
normally only check stuff, don't modify).

In case a test needs to be run by itself, experience says that it often 
is a larger test anyway, so it can be put in its own doctest.

> For functional tests, I still don't find the test paradigm very 
> attractive:  the raw, "converted-from-tcpwatch" files are *way* too 
> noisy,

Yes. You can get rid of a lot.

> and easy to mess up during editing.

And just as easy to fix :) (Btw, I never really had that problem)

> Once cleaned, they aren't *too* hard to work with, although I think
> we won't ever be able to show them to non-programmers

That wasn't the intention. But they can be shown to people who just know 
HTTP. I find them a lot more readable than regular functional tests 
because they treat Zope more or less like a blackbox. That way they're 
much closer to being real functional tests.

> (which was a goal of the INI-file format my framework uses).

Yes, as you say, different target audience. Functional tests, but 
doctests even more, are for developers who would like to see how the 
machinery works; for them, these tests are usually excellent examples 
(own experience!) because you see a typical use-cased being walked thru.

Philipp