[z3-five] Five traversal and acquisition

[Walco van Loon]

Hi list,

I recently discovered that there are several issues with Five traversal
and acquisition.

The most obvious problem with Five traversal is that an (un)restricted
traverse (through a Five traversable) to a non-existing attribute is
done three times, by Zope 3 traversal, Five, and by Zope 2 traversal:

         try:
             return ITraverser(self).traverse(
                 path=[name], request=REQUEST).__of__(self)
         except (ComponentLookupError, NotFoundError,
                 AttributeError, KeyError, NotFound):
             pass

The NotFoundError raised by Zope 3 traversal is caught silently, then 
the Five's __bobo_traverse__ does an getattr and a getitem. That fails 
too, after which the Zope 2 traversal does the attribute lookup again. I 
suggest that the FiveTraversable should only do a view lookup and not 
call the traverse method of its superclass.

All tests still pass if the call to Zope 3's DefaultTraversable traverse 
method is removed, so I'd like to fix this in svn.

This fix also seems to solve a much less obvious problem with Five 
traversal which I'd like to discuss here - it could be that the fix 
above only fixes the symptoms of a more fundamental problem with Five 
and acquisition. As I'm not an expert on Zope acquisition, I'll try to 
describe the problem below.

The problem is that traversal time of Five traversables (having a 
monkey-patched __bobo_traverse_ method), with attributes that make use 
of implicit acquisition, can have near exponential characteristics if 
the last attribute in the path doesn't exist.

To demonstrate this phenomenon, consider the following url (somecontent 
is a Five traversable, someattr is an object that extends 
Acquisition.Implicit).

/somefolder/somecontent/someattr/somecontent/someattr/somecontent/someattr 
/somecontent/someattr/404

(and yes, variations on this kind of urls can be found in the wild. 
Don't ask ;-))

Zope 2 recognizes repeated attributes and objects in acquisition 
structures to avoid repeated attribute lookups, which explains why urls 
like this don't cause problems in 'pure Zope 2' traversal, but Five 
seems to frustrate this process.

By the time the last someattr on the path is searched for '404' (in Zope 
2's unrestrictedtraverse), the object returned by the Traverser adapter 
seems to have built up a huge and complex acquisition structure, because 
it's this getattr call in Zope 2's unrestrictedtraverse that takes 
increasingly more time with each segment that is added to the traversal 
path.

A reason for this, but I'm guessing here, could be that each time 
somecontent is traversed, its patched __bobo_traverse__ is called, and a 
wrapped traversal result is returned. The repeated wrapping of the 
traversal result by Five seems to frustrate Zope 2's acquisition short 
circuiting.

The problem may seem a bit theoretical, but since the attribute lookups 
are done in Acquisiton C code, which causes all other threads to wait 
for the interpreter lock, the result is that if urls like this get long 
enough, they can effectively 'hang' the entire Zope instance.

I've attached a test that demonstrates the problem.

Cheers,
Walco





-------------- next part --------------
##############################################################################
#
# Copyright (c) 2004, 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Test Five-traversable classes

$Id: test_traversable.py 14595 2005-07-12 21:26:12Z philikon $
"""
import os, sys
if __name__ == '__main__':
    execfile(os.path.join(sys.path[0], 'framework.py'))



def test_traversable():
    """
    Test the behaviour of Five-traversable classes.

      >>> import Products.Five
      >>> from Products.Five import zcml
      >>> zcml.load_config("configure.zcml", Products.Five)

    ``SimpleContent`` is a traversable class by default.  Its fallback
    traverser should raise NotFound when traversal fails.  (Note: If
    we return None in __fallback_traverse__, this test passes but for
    the wrong reason: None doesn't have a docstring so BaseRequest
    raises NotFoundError.)

      >>> from Products.Five.testing.simplecontent import manage_addSimpleContent
      >>> manage_addSimpleContent(self.folder, 'testoid', 'Testoid')
      >>> print http(r'''
      ... GET /test_folder_1_/testoid/doesntexist HTTP/1.1
      ... ''')
      HTTP/1.1 404 Not Found
      ...

    Now let's take class which already has a __bobo_traverse__ method.
    Five should correctly use that as a fallback.

      >>> configure_zcml = '''
      ... <configure xmlns="http://namespaces.zope.org/zope"
      ...            xmlns:browser="http://namespaces.zope.org/browser"
      ...             xmlns:five="http://namespaces.zope.org/five">
      ... 
      ... <!-- make the zope2.Public permission work -->
      ... <redefinePermission from="zope2.Public" to="zope.Public" />
      ... 
      ... <five:traversable
      ...     class="Products.Five.testing.fancycontent.FancyContent"
      ...     />
      ... 
      ... <browser:page
      ...     for="Products.Five.testing.fancycontent.IFancyContent"
      ...     class="Products.Five.browser.tests.pages.FancyView"
      ...     attribute="view"
      ...     name="fancy"
      ...     permission="zope2.Public"
      ...     />
      ... 
      ... </configure>'''
      >>> zcml.load_string(configure_zcml)

      >>> from Products.Five.testing.fancycontent import manage_addFancyContent
      >>> info = manage_addFancyContent(self.folder, 'fancy', '')

    In the following test we let the original __bobo_traverse__ method
    kick in:

      >>> print http(r'''
      ... GET /test_folder_1_/fancy/something-else HTTP/1.1
      ... ''')
      HTTP/1.1 200 OK
      ...
      something-else

    Of course we also need to make sure that Zope 3 style view lookup
    actually works:

      >>> print http(r'''
      ... GET /test_folder_1_/fancy/fancy HTTP/1.1
      ... ''')
      HTTP/1.1 200 OK
      ...
      Fancy, fancy

    Testing borked url traversal.

      >>> import Acquisition
      >>> class Acquiring(Acquisition.Implicit):
      ...    exist = 1

      >>> class NotAcquiring: 
      ...    def __init__(self): pass
      ...    def __getitem__(self, name): 
      ...        if name == '404': raise AttributeError, name
      ...        return self

      >>> setattr(self.folder.testoid, 'impl_acq', Acquiring())
      >>> segment_pairs = 15
      >>> url = '/test_folder_1_' + '/testoid/impl_acq' * segment_pairs + '/404'
      >>> self.folder.unrestrictedTraverse(url)
      Traceback (most recent call last):
      NotFound: 404

      >>> url = '/test_folder_1_' + '/testoid/impl_acq' * segment_pairs + '/exist'
      >>> self.folder.unrestrictedTraverse(url)
      1
      
      This case shows that the problem only occurs when acquisition is
      involved: 

      >>> setattr(self.folder.testoid, 'no_acq', NotAcquiring())
      >>> url = '/test_folder_1_' + '/testoid/no_acq' * segment_pairs + '/404'
      >>> self.folder.unrestrictedTraverse(url)
      Traceback (most recent call last):
      NotFound: 404

    Clean up:

      >>> from zope.app.tests.placelesssetup import tearDown
      >>> tearDown()
    """

def test_suite():
    from Testing.ZopeTestCase import FunctionalDocTestSuite
    return FunctionalDocTestSuite()

if __name__ == '__main__':
    framework()