[z3-five] pythonproducts monkey patches

[Philipp von Weitershausen]

yuppie wrote:
>>> I stumbled over this issue because I did run the Five tests together
>>> with tests that initialize Five.
>>
>> Well, you're actually not supposed to do that. Five tests will clean up
>> after them nicely. Stuff that does installProduct('Five') will not.
>> These two collide.
> 
> Sometimes I run tests I didn't write myself ;)

... like the whole friggin' Plone test suite which installProduct()s a
brazillion products?!?

> So far the only issue was that installProduct('Five') doesn't make sure
> the ZCML is loaded because it might have been cleaned up by other tests.
> If you are aware of that they don't collide. setupPythonProducts is
> initialized with initialize(), not with ZCML.

I consider installProduct() harmful in either way, because it happens at
test module import level, not at test setup or execution time. So, for
example, even if the test runner was only looking for test modules,
importing them to check whether they contain tests that match a certain
condition (e.g. regex), installProduct() would be executed.

installProduct() is evil.

>> The monkey patches, when tested, should clean up after themselves. ALL
>> code that operates on global registries or does monkey patching should
>> clean up after itself. If pythonproducts doesn't do that, it's a bug.
> 
> That's one part of the solution. AFAICS other parts of Five make also
> sure they don't apply monkey patches twice. Either using __five_method__
> or using some other bookkeeping. That makes things more robust.

Yes, but it's not enough. Monkey patches must be undoable (and the code
doing monkey patches must provides hooks for undoign them) for tests.
Let's say you run a test that exercises the monkey patches and then
later, within the same test run, a test is run that doesn't expect the
monkey patches in place. It's likely that the second test might fail.

Proper test isolation is hard, I realize that. But zope.testing
infrastructure makes it bearable, I think.


Philipp