[pyshell] remote code considerations

[holger krekel]

hi pyshellers, 

here are some more considerations regarding remote code execution. 

i think it's kind of agreed for now that having a UI process and a
separate backend process makes sense.  Also the basic idea of letting
them communicate by sending over source code and executing that in the
remote process seems easy and flexible.  I am not sure, though,  how
exactly to send back result objects and especially exceptions (including
tracebacks).  Here are some suggestions. 

We could define that the UI process does not know how to manipulate and
change representations of objects it receives from remote execution. 
Then remote objects need to be represented on the client side with
some remote-object-reference.  Let's say the remote side sends either 
one of three objects: 

    a Wrapped instance or
    a WrappedExcinfo instance or 
    a DirectValue instance 

All classes might conveniently exist in the same namespaces in both 
processes so that instances can be safely pickled to and fro.  

Thus if the UI process receives a WrappedExcinfo instance it might want
to compute a standard string representation of the contained traceback
object and therefore it sends something which executes the following
function and sends back the result:

    def get_tbstringlist(excinfo):
        import traceback
        return DirectValue(traceback.format_exception(*excinfo))

The return value will show up as a DirectValue instance (which just
has a '.value' attribute - a list of strings) on the client UI side.  
It can then be rendered in pygame. 

But an UI-side 'WrappedExcinfo' object will not carry a direct value but
instead just represent a backend object.  We would call into our little 
communication framework like e.g:

    remote_exec(get_tbstringlist, excinfo)

and our mechanism would send over the get_tbstringlist function (however
exactly) and the excinfo-handle. Note that it might *also* send over the 
code that finds the "real" backend object from the handle. 

One question is how we ever get rid of the handle-to-object mapping 
(otherwise our backend process will just grow all the time which
might be acceptable at the beginning).  That's actually a typical
problem for systems with remote references.  One concept is to create
the handle-to-object mapping in connection with some kind of session-id
so that if the session is finalized the mappings are destroyed.  Ah, and
we could of course use some kind of a 'shelve' module if we want to have
persistent backend objects .... :-) 

Note that we do *not* inherit the complexities of RMI-style systems: we
only have simple remote references but no real calling protocol apart
from some standard python-pickling.  This is where i expect that we
mostly avoid the CORBA/RMI complexity ... type mappings, argument/return
value specifications (IDL etc), Portable Adapters, component models and
all the nice "closure of RMI-object" problems, i.e. once you start with
a RMI object and you call methods on it you tend to get more RMI objects 
which you call methods again upon and so - before you notice - everything 
suddenly becomes a remote method invocation and needs to be specified. 
We shouldn't have such problems. 

Instead we have a 'create-your-own-protocol-on-the-fly' which can be
changed at will from the client side alone.  

Of course there also is the interesting topic of how to test such a
machinery ... but it shouldn't be too hard if we keep that in mind early
on, e.g.:

    def test_exception_traceback_string_extraction():
        def remote1():
            raise ValueError

        res = remote_exec(remote1)   
        check.isinstance(res, WrappedExcinfo)

        res = remote_exec(get_tbstring, res)
        check.isinstance(res, DirectValue)
        for x in res.value:
            check.isinstance(x, str)

and this implies that we are able to easily spawn a backend and client
process.  A simple (test or even interesting) setup might be to spawn 
a process and communicate via stdout/stdin with it ...

cheers,

    holger