[z3-five] Proposal: use $INSTANCE_HOME/package-includes

[Philipp von Weitershausen]

Martijn Faassen wrote:
> Yes, I understand this. :)
> 
> But it means that the whole idea of having the directories be anywhere
> it not useful for projects which need to evolve from Zope 2 to Zope 3.
> It's only useful if you're writing a fresh Five project.

That is true and I totally understand that. But the idea of 
package-includes can be even useful to those products. Consider the 
following use-case:

Let's say that you want to store DublinCore metadata on SilvaDocuments. 
The machinery for that already exists in zope.app.dublincore, all you 
would have to do is make SilvaDocument annotatable in some way. So, you 
would want to enable zope.app.dublincore, which means you need to 
process its configuration.

You could use a hard-wired <include package="zope.app.dubincore /> in 
SilvaDocument's configure.zcml, but using the package-includes facility 
might have the following advantages:

- it is easy to see which "external" packages (external meaning external 
to Products) are loaded and configured, without having to dig into 
Product-specific configuration.

- it is easier to enable/disable this extra functionality on an on/off 
basis when that on/off switch is a simple file; it's harder when it's a 
line in some Product's configure.zcml

- as Zope 2 products emerge to use components more, add-on functionality 
can be plugged in with extra packages more easily. In case those are 
really *packages* (e.g. from zope.app) and not *Products*, they cannot 
be dropped into Products, which indeed is the most convenient thing for 
an administrator. So, the 2nd most convenient thing for packages (such 
like zope.app.dublincore) would be a drop-in into package-includes, I 
reckon.

- you can tell zpkg to count in certain dependencies, such as 
zope.app.dublincore (and probably zope.app.annotation). This will cause 
Silva packages generated with zpkg to include the zope.app.dublincore 
package and the drop-in configuration file(s) automatically. So, the 
automation is there, but it's not magic. I think we all agree that 
automation is good but Zope 2 mixes automation with too much magic.

>>> Another advantage of the 'Products' way is that people are already 
>>> familiar with it;
>>
>> Nothing prevents them from continuing to do so. Using package-includes 
>> will be an option, not a requirement. I think the proposal explicitly 
>> stated that.
> 
> Yes, but we should be very careful in making the option being only an
> option, not the "preferred way to do it".

It will unlikely become the preferred way of doing it for Products, but 
it shouldn't be impossible.

>>> So, can we really go the Zope 3 way here the whole hog? It looks to
>>> me we'll have to support the Product way for Five as well at the 
>>> very least.
>>
>> Right now, support the "Product way" is a simple boiler plate in 
>> whatever products:
>>
>> def initialize(context): zcml.process('configure.zcml', 
>> package=Products.FiveViewsDemo)
>>
>> People will still be able to do it. I personally would like to have 
>> the option for an automation. That's why I wrote the proposal.
> 
> My point is that the product way needs more consideration by itself. We
> haven't solved all issues yet.

That might very well be. My proposal touches the Product way on a 
tangent: how to configure 3rd party packages (see above).

When you say that we haven't solved all issues yet: what are the issues?

>>>> Zope 3, on the other hand, does not enforce packages to be in a 
>>>> certain place as long as they can be lookuped in PYTHONPATH.  It
>>>>  allows packages that want to be configured to drop in a small ZCML 
>>>> file into a directory called 'package-includes'.
> 
>>> Do these get dropped in automatically? How does that mechanism work? 
>>> Or does an actual sysadmin need to drop these in, instead? From what 
>>> you say later, the latter; that's an extra burden on the sysadmin.
>>
>> But it's pretty annoying that everything in Products automagically 
>> gets picked up. And yes, IIRC, zpkg drops in the file automatically.
> 
> It's very convenient that everything in Products gets automatically
> picked up. :) Like it or not, sysadmins are used to the Zope 2 way, and
> having a new class of Zope 2 products that works differently isn't going
> to charm them off their feet...

You are absolutely right. But, as I tried to point up above, the 
package-includes way would have some advantages for *extra/external* 
packages, I think.

>>>> Five allows us to almost write Zope3-compatible code which works in 
>>>> Zope 2 (using Five).
>>>
>>> As I argued before, the main goal of Five is to allow *existing* 
>>> products to use Zope 3 code. The main goal is *not* to write Zope 3
>>>  compatible code in Zope 2. That's only useful if you write new 
>>> packages.
>>
>> Your goal is the former, my goal is the latter. The idea of this 
>> proposal does not involve a conflict of the two interests.
> 
> I'm still happily going to defend the former if I think the latter is
> causing us to lose sights of that goal.
> 
> Being able to write Zope 3 compatible code in Zope 2 by itself is of
> limited utility; you could as well use Zope 3 and be done with it. Being 
> able to add Zope 3 abilities to Zope 2 code is of tremendous utility. 
> It'll allow existing products to become *closer* to compatibility to 
> Zope 3. Of course the one doesn't preclude the other, but we shouldn't 
> lose sight of the option with the tremendous utility. :)

You are right; and it is good that you can always remind me and others 
of what the goal really is.

One of the reasons why I would like this feature to work is because it 
would then allow us to see how well you can write a Five-based Product 
so that it would run in Zope 3 automatically. The other way around is 
important too: how much would you have to adjust a Zope3 product to make 
it work in Five?

I think finding answers to those questions would give us an estimate for 
the quality and quantity of the advancement we have made towards a 
converging of Zope 2 and Zope 3. We don't want to stop somewhere in the 
middle of the converging process because we fear breaking too much with 
the Zope 2 philosophy; what would then happen in the post-Five world? 
You and many others will port their Product to Five and expect Zope 3 
compatability to be near then. So, see my effort as a Quality Assurance 
  measurement for this goal...

>>> I highly doubt a developer will actually need to turn off particular 
>>> features in Five. I think the flexibility to disable parts of Five is 
>>> a YAGNI.
>>
>> I beg to differ. I see use cases where people don't need this or that
>>  or where they want to *customize* parts.
> 
> This sounds very dubious to me. If I install one Product that builds on
> Five and then another, I don't want conflicts between what I need to
> turn off and on, or override in Five. I'll call YAGNI until I see actual
> use cases that also describe how they work around this problem.

Ok, I don't feel strongly about it. I just wouldn't want us to introduce 
too many hard-wired dependencies where we could have done it 
differently, just to discover in the end, that it would have been more 
convenient if it was more flexible. Zope 2 has loads of examples there :)

Then again, we can always refactor later.

>>>> I propose to simulate the Zope3 package experience in Five. There
>>>> will be a 'package-includes' directory in a Zope 2 instance, just
>>>> like in a Zope 3 instance.  It will contain a few default include
>>>> files for the packages that Five itself depends on.  A developer
>>>> will then be able to install an addon package by dropping a ZCML
>>>> file in there, just like in Zope3.
>>>
>>>  
>>> This replaces one action by a sysadmin (install Product) by two 
>>> (install Python package, drop in ZCML). The main audience for 
>>> installation isn't developers, but sysadmins.
>>
>>
>> zpkg generates a setup.py that should do this automatically.
> 
> How does the setup.py know where the package-includes directory is?

Well, $INSTANCE_HOME/package-includes. So, it knows where to put it as 
soon as you make an instance and you tell it where to make that instance.

> This
> sounds like an impossible thing to solve unless there's only a single
> package-includes per system? At the very least you'll need some sysadmin 
> input about where to install.

Just like the sysadmin needs to know where his/her Zope instance is sitting.

>> Right. That's why I think the convention should be that everything 
>> that goes into Products doesn't require a package-includes snippet. 
>> But I would like to be able to use it.
> 
> Sounds good, as long as we don't deprecate Products.

I never wanted to do that. I think that's what Five is all about, isn't 
it? My package-includes proposal tries to solve the things that happen 
between those Products and the new Zope3 and non-Zope3 packages that 
need to be configure as well.

Philipp