[pypy-dev] builtin refactoring branch

[Armin Rigo]

Hello,

Here are some suggestions for mixing application-level and interpreter-level
functions. This starts from Holger's, from the builtinrefactor branch, and
from ideas discussed with Michael on #pypy.

The issue is related to how we want to control the wrapping of internal
interpreter objects. The relation becomes clear when you think about defining
a method of some PyPy class at the application-level: the 'self' argument
needs to be wrapped to be visible. For example,
PyBaseCode.app_decode_code_arguments() needs 'self' to access the 'co_xxx'
attributes of the PyBaseCode instance.

Here is a suggestion mixing properties and CPython's structmember.c (comments
following the code):

class PyByteCode(Wrappable):

    app_co_code     = appfield_str('co_code', readonly=True)
    app_co_argcount = appfield_int('co_argcount', readonly=True)

    def __init__(self):
        self.co_code = ''
        self.co_argcount = 0

    def app_decode(self, argtuple):   # app-level code, implicit wrapping
        assert len(argtuple) == self.co_argcount
        return argtuple
    decode = app2interp(app_decode) 

    def eval(self, space, w_args):
        w_args = self.decode(space, w_args)
        w_result = space.eval_frame(w_args)
        return w_result
    app_eval = appfunc(eval)

The base class 'Wrappable' contains the logic that allows the interpreter to
control the behavior of wrapped versions of its instances. More about it
below. The essential idea is that only the class attributes starting with
'app_' are visible to wrapped objects. To read or write the attribute xyz, a
wrapped Wrappable instance will look for a class attribute 'app_xyz'. It can
be:

  * an appfield_t instance, the equivalent of CPython's member list. Above,
the definition for 'app_co_code' means that the wrapped 'self' has a 'co_code'
attribute which can be read from the interpreter-level 'co_code' attribute and
wrapped. If the attribute were not read-only, it could be written to but would
only accept a string, which would be unwrapped before it is stored in the
interpreter-level 'co_code'.

  * app_decode() is a function, which is just executed at the
application-level as a method with the extra 'self' argument. The app_decode()
function is not supposed to be called directly from the interpreter-level, but
it can be called through the 'decode' name thanks to the bridge
'app2interp(app_decode)'.

  * conversely, eval() is just an interpreter-level method, but it can be
called from application-level thanks to the definition of app_eval(). This is
the equivalent of CPython's method table. It only works if eval() has a
"standard" signature: 'self', followed by 'space', followed by wrapped
arguments only. We should probably design a way to describe more about the
signature in the call to appfunc(), like specifying that some arguments are
strings or integers -- the declarative equivalent of PyArg_ParseTuple().

  * a way to define properties (the getset list of CPython) would be nice to
have too.

  * finally, for convenience, an app_xyz attribute could be just a simple
wrappable constant, e.g. "app_CO_VARARGS = CO_VARARGS" to make it visible at
application-level.

The trick of systematically adding an app_ prefix is useful to define special
attribute names (__xyz__) that would otherwise have another meaning at
interpreter-level. For example, a module definition could start:

class Builtin(BuiltinModule):
    app___name__ = '__builtin__'

This makes the above constant visible at application-level under the name
'__name__'.

Some of the above also applies to global helpers (as opposed to methods):
global variables called app_xyz are visible under the name xyz in the
application-level helpers. (It seems more regular this way, and generally
cleaner than making all global variables visible by default to helpers,
without requiring nor allowing the app_ prefix.)

Finally, as an extension of the idea, special method names 'app___xyz__' could
be used to control the details of the operations on internal classes, if
needed; for example, to define attribute reading without using appfield:

    def my_custom_getattr(self, space, w_attr):
        ....
    app___getattr__ = appfunc(my_custom_getattr)

or even defining getattr() itself as an application-level helper (!):

    def app___getattr__(self, attr):
        ...

About the implementation: the interpreter-level wrappable classes should
inherit from Wrappable, which can be checked for in object spaces wrap()
methods. This integrates with multimethods or whatever a particular object
space uses for dispatch with a trick similar to StdObjSpace's current
W_CPythonObject: we would have, say, a W_InternalObject class that works like
W_CPythonObject but just uses generic code from pypy.interpreter.something to
do the operations. I guess this could replace W_CPythonObject.


A bientot,

Armin.