Death to ZopeTestCase
=====================

There's a beast in Zope 2 that needs to die.  Its name is
ZopeTestCase.


Integration tests are good
--------------------------

I won't condemn ZopeTestCase entirely.  It actually serves a good
purpose: to test how a particular Zope 2 object (a Zope 2 meta type, a
CMF portal tool, a content object, etc.) behaves in a Zope 2-like
environment, which essentially means inside a folder hierarchy with
acquisition and all the bells and whistles.  Because of the way many
Zope 2 solutions were written, this is pretty much the only way you
would test them anyways, because they rely on having an acquisition
context, the ``REQUEST`` available all the time, etc.

Still, calling a ZopeTestCase test a *unit* test is pure blasphemy.
It's an *integration* test.


Unit tests are important, too
-----------------------------

The problem with ZopeTestCase is that it's so convenient and it lures
test authors into a type of test that they probably didn't want to
write and definitely shouldn't.  Sure, integration testing is
important and, as said above, ZopeTestCase actually serves that
purpose.

Unit tests are also important, though.  And as Zope 2 software is
being componentized, they're actually much easier to write.  The CMF,
for example, doesn't use ZopeTestCase (except for one test which
probably shouldn't use it, either).


ZopeTestCase's problem
----------------------

The problem with ZopeTestCase is really a technical one.  It is the
way it provides integration testing facilities:

* All test modules containing ZopeTestCase classes load a special file
  called ``framework.py``.  It starts up ZopeLite which is a small
  Zope setup with a ZODB DemoStorage and no HTTP servers.

* Test modules can make calls to ``installProduct`` at module level to
  have certain products initialized before tests are even found.

* ZopeTestCase instances have access to a folder hierarchy provided by
  the setup.  Given that ZopeLite and certain products are
  initialized, they can then pretty much test anything as if users
  were doing things to a live Zope instance.

The problem: there's no tear down and therefore no isolation.  If I
have unit tests and ZopeTestCase tests running together, the unit
tests will be executed in the context of ZopeLite and **all** the
initialized products that other tests wanted to have installed.

For this reason, for example, you're definitely asking for trouble
when doing ``installProducts('Five')`` because that loads all of
Five's ZCML, therefore registers things that your unit tests might not
expect to be there.


The new testrunner's solution
-----------------------------

Since Zope 3.2/2.9, we have a new testrunner in ``zope.testing``.  The
``test.py`` script is a small facade for it.  The new testrunner
supports a concept called *layers*.  Layers are levels of test setup
and all tests of the same layer are executed at once.  From
``zope.testing``'s ``testrunner.txt``:

  Most tests are unit tests.  They don't depend on other facilities, or
  set up whatever dependencies they have.  For larger applications,
  it's useful to specify common facilities that a large number of
  tests share.  Making each test set up and and tear down these
  facilities is both ineffecient and inconvenient.  For this reason,
  we've introduced the concept of layers, based on the idea of layered
  application architectures.  Software build for a layer should be
  able to depend on the facilities of lower layers already being set
  up.  (...)


How layers work
---------------

Layers are objects with ``setUp`` and ``tearDown`` methods.  When
using classes as layers, we must use class methods::

  class MyLayer(object):

      @classmethod
      def setUp(self):
          # do something here

      @classmethod
      def tearDown(self):
          # undo it here

You can nest layers using class inheritance::

  class MyExtendedLayer(MyLayer):

      @classmethod
      def setUp(self):
          # do additional stuff here
	  # don't call super

      @classmethod
      def tearDown(self):
          # undo it only the additional stuff here
	  # don't call super

The test runner will first set up ``MyLayer`` and run all tests of
that layer.  Then it will set up ``MyExtendedLayer`` and run all tests
of that layer.  When done, it will tear down ``MyExtendedLayer``, then
``MyLayer``.

As you can imagine, this allows very fine grained test setup.  For
example, there can be a very general layer that bootstraps common
things; specific tests can then supply their own test setup based on
that layer (by subclassing from the layer).


Using layers for integration tests
----------------------------------

In Zope 3, unit tests simply have no layer.  Integration tests (we
call them "functional tests") that rely on a full-blown ZCML setup
(starting at ``ftesting.zcml``) can use
``zope.app.testing.functional.ZCMLLayer``, for example.  Other tests
use their own layer.

I think ZopeTestCase, specifically ZopeLite, should be converted into
a test layer.  The trick is that it will also support tear down.  The
``installProduct`` functionality should also be moved into layers,
ones that obviously extend the ZopeLite layer.  Again, tear down will
be possible so that tests will be able to say which specific products
they need (right now, all tests run with all the products installed
that are somewhere specified as dependencies).

I think this might be a nice sprint task.  Paris, Switzerland,
EuroPython... anyone?