[z3-five] Proposal: use $INSTANCE_HOME/package-includes

[Philipp von Weitershausen]

==================================
Allow Zope3-style package includes
==================================

:Author: Philipp von Weitershausen
:Status: Proposal


Current Situation
-----------------

Zope 2 expects add-on packages, in Zope 2 terms called "products", to
be in a special directory of your instances, usually called
"Products".  It also allows these products to do some initialization
and configuration by calling their intialize() method which is defined
in the product's __init__.py file.

This imposes two basic requirements on Zope 2 addon packages:

   - location

   - implementation of at least initialize() in __init__.py

Zope 3, on the other hand, does not enforce packages to be in a
certain place as long as they can be lookuped in PYTHONPATH.  It
allows packages that want to be configured to drop in a small ZCML
file into a directory called 'package-includes'.  Configuration files
ending with -meta.zcml are called first, then the ones ending with
-configure.zcml.  This is an extremely flexible system, since add-on
packages can be enabled/disabled by simply adding or removing (or
renaming) one of these small ZCML files.


Problem
-------

Five allows us to almost write Zope3-compatible code which works in
Zope 2 (using Five).  One of the hurdles still is the package naming
which becomes a problem whenever absolute module paths (dotted names)
are used.  Being able to drop in packages anywhere and using global
package names like in Zope 3 should be possible.

Furthermore, Five increasingly depends on certain zope.app packages.
Some of these need to be configured through ZCML.  Instead of
including all their configuration files in five.zcml, the software
developer really should have easy control over what is configured and
what not.  Zope 3 gives the developer and administrator that control
using package-includes.


Proposal
--------

I propose to simulate the Zope3 package experience in Five.  There
will be a 'package-includes' directory in a Zope 2 instance, just like
in a Zope 3 instance.  It will contain a few default include files for
the packages that Five itself depends on.  A developer will then be
able to install an addon package by dropping a ZCML file in there,
just like in Zope3.

Five's __iniit__.py will grow an 'initialize()' method which will
lookup the 'package-includes' directory in the instance and execute
the ZCML files.  Like in Zope 3, *-meta.zcml files should be called
before *-configure.zcml files.

Note that developers who choose to drop their packages into 'Products'
will still be able to do that.  They can then choose whether to
provide their boilerplate initalize() function in __init__.py or
whether to drop a ZCML snippet into package-includes.  The latter
should be preferrable.


Risk factors
------------

Zope 2 supports the refreshing of products.  Upon a package refresh,
the Python module is reloaded and its initialize() function is called
again.

Package refresh will not work for packages outside of the Products
directory.  If refresh is enabled for Five, Five's initialize()
function will be re-called (unless we prevent it from being recalled
through some global reminder variable).  That might allow at least
ZCML configuration to be reprocessed, which might be preferable or
not, depending on what that configuration does.  However, it is more
likely that the ZCML machinery will detect the same files being
processed more than once and simply do nothing (that is what the ZCML
machinery does when encountering a file it has already processed).