Reunite Zope 2 and Zope 3 in the source code repository ======================================================= :Author: Philipp von Weitershausen, philikon@philikon.de :Status: IsProposal :Date: 2005-11-23 :Original: http://codespeak.net/svn/user/philikon/ReuniteZope2AndZope3.txt Current situation ----------------- Zope 2 lives at ``svn://svn.zope.org/repos/main/Zope``. Zope 3 lives at ``svn://svn.zope.org/repos/main/Zope3``. Right now, Zope 2 includes nearly everything from the Zope 3 repository, even though it doesn't use everything (but most of it) in a distribution. Both Zope 2 and Zope 3 also use the following extra technologies which are inserted via ``svn:externals``: * ZODB * ZConfig * zdaemon * zpkg * zope.testing (yes, this is an extra technology because it's managed within neither Zope repository tree) * docutils (external project, svn.zope.org isn't the primary repository; it's also not included as an external, instead Zope 2 and Zope 3 manage two different copies which are not necessarily in sync.) Problems -------- * Less than optimal use of synergies: Recent improvements to Zope 2 should have been forward-ported to Zope 3 as well. I'm thinking of the improved Windows service support and the file stream iterator, for example. Instead of having things implemented twice in the future, they should only be implemented once (preferably in below the ``zope`` namespace) and simply *used* twice. * Zope 2 needs Zope 3 to be more flexible: Zope 2 is using a lot of Zope 3 technology through Five, Zope 2.9 even started using some in the core (e.g. the test runner, events, i18n). Zope 2.10 will go even further and probably use things like ``zope.filereference`` for ZIP file import support, ``zope.pagetemplate`` for ZPTs and possibly other things. More use of Zope 3 technology in Zope 2 will inevitably mean some refactorings on the Zope 3 side. One part where this is already evident are the ZCML directive handlers; others may apply as we find out about them. * Zope 3 refactorings affect Zope 2: When Zope 3 is refactored, the parts that depend on it in Zope 2 (and Five) should also be refactored. This will avoid getting Zope 2 and Zope 3 out of sync regarding the use of Zope 3 technology. Small refactorings on the Zope 3 side have already broken tests in Zope 2 or Five in the past. * The ``zope`` namespace should be open for Zope 2 developers: It is generally agreed by now that top-level packages are a bad idea and that we should all use the ``zope`` package namespace. We already do for Zope 3, but we should start doing so for Zope 2 as well (``zope.testrecorder`` is some nice pioneer work in that respect). Having to go to the Zope 3 tree to put a package there and then later stitch it in via an ``svn:externals`` entry adds more development discomfort than necessary (e.g. for experimental stuff that would live on a branch, one would have to create two branches, etc.). In short, people will be discouraged to put their stuff in the ``zope`` package namespace. Instead they'll continue to create stuff in ``Products``, something which should be discouraged soon, if not now. Proposal -------- Manage Zope 2 and Zope 3 in the same SVN repository tree. My suggestion is to use the Zope 3 repository layout for the new tree (I prefer a ``src`` directory over ``lib/python``). For that we move all the Zope 2 stuff to the Zope 3 tree. Then we delete the ``Zope`` tree and rename the ``Zope3`` to ``Zope``. In pseudo shell commands:: $ svn mv Zope/lib/python/* Zope3/src $ svn rm Zope $ svn mv Zope3 Zope **Note:** Zope 2 and Zope 3 both contain separate copies of ``docutils`` and ``RestrictedPython``. The former should always be in sync because it's a vendor import. In the latter case, the package was copied from Zope 2 to Zope 3 in rev 26688 and has been edited in both branches, in the Zope 2 branch more than in the Zope 3 branch. Before this proposal is implemented, the changes from one branch should be merged to the other so that there's only one authorative ``RestrictedPython`` implementation. Your questions answered ----------------------- This is a change too huge for me! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I doubt that. Look at a checkout of Zope 2.8 or higher. It essentially is a checkout of both Zope 2 and Zope 3, except that the Zope 3 part is wired in via an ``svn:externals`` entry. Moving them together wouldn't change so much for the Zope 2 developer. Sure, the Zope 3 developer would see a couple more packages. Packages, though, that he would normally not care about, except when running tests before a check-in. Why do you want to pollute the clean Zope 3 codebase with Zope 2 rubbish? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I don't. I just want both codebases to live together in the same repository tree (which essentially means they'll be living together in a working copy). A compelling advantage of this is that Zope 2's evolution can benefit from the synergies of Zope 3 development. I even believe that sometimes it can happen the other around. In some minor things, Zope 2 has received more sophisticiated features than Zope 3. Windows service support and the file stream iterator are such examples. What happens when we tag a release? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We could tag both a Zope 2 and Zope 3 release with the same tag, but we don't have to. Either way, the release schedule for their major version is synchronized anyway. That means that having one tag for them, at least in the alpha, beta and RC phase, would actually reflect what we're trying to do in the release process. Does that mean Zope 2 and Zope 3 will be packaged as one? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ They already are! Since version 2.8 the Zope 2 distribution always contains the current Zope 3 stable release. Maybe in the future, with a different packaging strategy, we won't have to package them together anymore. Instead we could have Zope 2 require a suitable Zope 3 installation already in place (possibly throuh eggs). Having Zope 3 be distributed by itself won't change. Note: The way we organize a repository has very little to do with the way we make packages. The Zope 3 repository contains a lot more than what's packaged in a typical Zope 3 distribution. And since Zope 2 includes a full Zope 3 installation, even the parts it doesn't use (e.g. twisted) are in a Zope 2 checkout. They're not in a Zope 2 distribution, though. What happens to stuff that isn't in Python packages? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You're probably refering to the ``bin``, ``doc``, ``utilities``, ``skel``, ``zopeskel`` et.al. directories as well as the in-place build system. Some stuff would be conflicting when the locations are thrown together. Here are some suggestions on what to do: ``doc`` My suggestion is to keep separate ``doc`` directories, e.g. ``doc2`` and ``doc3`` (note that this doesn't affect the name of those directories in the tarball; it could still simply ``doc``). In the long term, one could think about merging the ``doc{2|3}`` directories into one, though that raises problems with standard texts which are available in both trees, e.g. ``CHANGES.txt``. Or maybe it doesn't because we'll only have one ``CHANGES.txt`` at some point. ``utilities`` The contents of this directory can simply be thrown together. Zope 2's ``mkzopeinstance.py`` and ``mkzeoinstance.py`` make an exception, they should be in the in-place instance's ``bin`` directory. ``setup.py``, ``configure``, ``Makefile``, and the in-place build system Since both Zope 2 and 3 now use zpkgutils for the in-place build, few changes if any would be necessary in ``setu.py``. For the ``Makefile`` Zope 3 uses a simple hard-wired one while Zope 2 generates it via a ``configure`` script. This is mostly for choosing the right Python interpreter executable and setting a view variables for the in-place instance creation. At least the in-place instance generation could be made unnecessary by providing a default in-place instance like in Zope 3 (see below). The (generated) Zope 2 ``Makefile`` does a bit more than its Zope 3 equivalent, which is mostly because it was packaged in a release tarball. Most of that isn't needed anymore, though, since Zope 2 is packaged with zpkgutils (some of the details here weren't figured out in Zope 2.9 at the time of this writing, though). ``bin``, ``skel``/``zopeskel``, and the in-place instance system Zope 3 comes with an in-place instance which resides in the root of the ``Zope3`` tree. Zope 2's ``Makefile`` can create an in-place instance. When living in the same repository tree, the two instances would conflict. I suggest to have two default in-place instances, one at ``zope2inst``, one at ``zope3inst``. They would hold whatever the in-place instance holds now, including the ``skel``/``zopeskel`` directories that are the skeleton for new instances as well as the ``mkzopeinstance``/``mkzeoinstance`` scripts. Probably the only top-level files that can remain would be related to the test runner. Do I have to throw away my Zope 2 and Zope 3 sandboxes? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ No, you don't. You will be able to switch them over to the new location. There might, however, be problems due ``svn:externals`` entries now appearing as real directories. In this case deleting the offending directory and re-initiating ``svn up`` should solve the problem. Risks ----- * There will be a bigger burden on the Zope developer in several ways: more tests to run, more code to touch when refactoring, etc. Of course, the flip side is that Zope 2 and Zope 3 will be much better kept in sync than before. * Developers of either Zope 2 or Zope 3 might find it confusing to be seeing the other projects code. This should really only affect Zope 3 developers since Zope 2 developers already see all the Zope 3 code in their checkout. * The distinction between the two projects will be blurred from the development point of view. This will inevitably have an affect on the development process. For example, there will most probably be only one checkins mailing list and soon we'd be thinking about throwing together the development lists and issue trackers as well. Yet another logical conclusion would be to present Zope 2 and 3 equally on the community web page. Personally I think that all this could very much be a good thing, but other people might not. Implementation progress ----------------------- None so far, still in discussion.