[z3-five] Proposal: use $INSTANCE_HOME/package-includes

[Martijn Faassen]

Hey,

While the package-includes strategy has the benefit of being very much 
like Zope 3 and has various improved ways of doing things, I also see 
some difficulties and I'd like to discuss some of the background here.

Five is two sided by nature. As it evolves, more and more will be 
possible to do in the Zope 3 way, and less Zope 2 specific code will 
need to be imported. But Five's aim is to be a system to support the 
*evolution* of Zope 2 products into the Zope 3 direction, or differently 
put, to gain some of the benefits of Zope 3 inside Zope 2. Because we 
want to support evolution, we want to support in some cases the Zope 2 
way of doing things.

Take the case of a Plone or Silva. For Silva to follow a 
package-includes strategy we'd need to place the views and adapters in 
the package-includes direcory, and everything else remains in the Silva 
directory. The content objects are not going to be moved easily, as 
that'll break existing ZODBs rather badly. This means that we'll likely 
end up putting the views and adapters in the Silva product anyway, and 
the ZCML as well.

One feature that I think Silva could really use is using the ZCML 
override directive. I do not quite understand yet how this works in Zope 
3, but what I want for Silva is a way for a Silva extension to override 
certain view (or even adapter) registrations in the Silva core, to 
customize the behavior. In the Zope 2 way, you'd install an extension 
product to Silva that had this behavior.

In the Zope 3, I imagine you'd use the includeOverride directive. Does 
this depend on execution order? If it does, then package-includes (or 
initializing in the random product order in Zope 2) has a problem, as 
the order of execution would be fairly arbitrary, or at least not in the 
order of dependency. Or do you actually need to modify the existing 
package to make includeOverride work? That means includeOverride is 
pretty useless... Need to talk to zope3-dev.

Another advantage of the 'Products' way is that people are already 
familiar with it; supporting it makes us not having to worry about the 
Zope sysadmin getting confused; we have enough on our hands as 
developers. I'd like to keep the installation requirements for a someone 
who uses a system that *uses* Five, like Silva, as simple as possible. 
Zope 2 users are already used to installing products, so installing Five 
and ZopeThree4Two products is easy to understand.

So, can we really go the Zope 3 way here the whole hog? It looks to me 
we'll have to support the Product way for Five as well at the very least.

[snip]
> Zope 3, on the other hand, does not enforce packages to be in a
> certain place as long as they can be lookuped in PYTHONPATH.  It
> allows packages that want to be configured to drop in a small ZCML
> file into a directory called 'package-includes'.  

Do these get dropped in automatically? How does that mechanism work? Or 
does an actual sysadmin need to drop these in, instead? From what you 
say later, the latter; that's an extra burden on the sysadmin.

[snip]
> Problem
> -------
> 
> Five allows us to almost write Zope3-compatible code which works in
> Zope 2 (using Five).

As I argued before, the main goal of Five is to allow *existing* 
products to use Zope 3 code. The main goal is *not* to write Zope 3 
compatible code in Zope 2. That's only useful if you write new packages.

[snip]
> Furthermore, Five increasingly depends on certain zope.app packages.
> Some of these need to be configured through ZCML.  Instead of
> including all their configuration files in five.zcml, the software
> developer really should have easy control over what is configured and
> what not.  Zope 3 gives the developer and administrator that control
> using package-includes.

I highly doubt a developer will actually need to turn off particular 
features in Five. I think the flexibility to disable parts of Five is a 
YAGNI. The ZCML in Five is also highly particular to Five; it's not as 
if you can easily include .zcml from Zope 3 directly and expect things 
to work, so the utility is less than in Zope 3.

> Proposal
> --------
> 
> I propose to simulate the Zope3 package experience in Five.  There
> will be a 'package-includes' directory in a Zope 2 instance, just like
> in a Zope 3 instance.  It will contain a few default include files for
> the packages that Five itself depends on.  A developer will then be
> able to install an addon package by dropping a ZCML file in there,
> just like in Zope3.

This replaces one action by a sysadmin (install Product) by two
(install Python package, drop in ZCML). The main audience for 
installation isn't developers, but sysadmins.

> Five's __iniit__.py will grow an 'initialize()' method which will
> lookup the 'package-includes' directory in the instance and execute
> the ZCML files.  Like in Zope 3, *-meta.zcml files should be called
> before *-configure.zcml files.
> 
> Note that developers who choose to drop their packages into 'Products'
> will still be able to do that.  They can then choose whether to
> provide their boilerplate initalize() function in __init__.py or
> whether to drop a ZCML snippet into package-includes.  The latter
> should be preferrable.

I care about the additional burden on sysadmins. Basically you'll have 
to tell anyone who is installing a Five-based Plone or any of its 
extensions to do two things instead of one thing. From a Zope 2 
perspective, the former is actually preferable.

Anyway, in brief:

* Doing it the Zope 3 way is good.

* Evolution requirements block going the full Zope 3 way for everybody; 
we need to support the Zope 2 way and provide features for that audience.

* Supporting the Products way should not be in any way be deprecated and 
in fact requires our special attention. We need something that is 
similar to package-includes but actually doesn't require the moving 
around of zcml snippets into directories by a sysadmin. Perhaps we can 
make it so that installing a Product is equivalent to the dropping in of 
ZCML in package-includes, without actually doing so. After all, in Zope 
2, people install Products expecting to be able to use them.

* We need to figure out how to do overrides from extensions; a question 
for zope3-dev from me. :)

Regards,

Martijn