=============================================================================== PyPy - Getting Started with the Translation Toolchain and Development Process =============================================================================== .. contents:: .. sectnum:: .. _`try out the translator`: Trying out the translator ------------------------- The translator is a tool based on the PyPy interpreter which can translate sufficiently static Python programs into low-level code (in particular it can be used to translate the `full Python interpreter`_). To be able to use it you need to (if you want to look at the flowgraphs, which you obviously should): * Download and install Pygame_. * Download and install `Dot Graphviz`_ (optional if you have an internet connection: the flowgraph viewer then connects to codespeak.net and lets it convert the flowgraph by a graphviz server). To start the interactive translator shell do:: cd pypy python bin/translatorshell.py Test snippets of translatable code are provided in the file ``pypy/translator/test/snippet.py``, which is imported under the name ``snippet``. For example:: >>> t = Translation(snippet.is_perfect_number) >>> t.view() After that, the graph viewer pops up, that lets you interactively inspect the flow graph. To move around, click on something that you want to inspect. To get help about how to use it, press 'H'. To close it again, press 'Q'. Trying out the type annotator +++++++++++++++++++++++++++++ We have a type annotator that can completely infer types for functions like ``is_perfect_number`` (as well as for much larger examples):: >>> t.annotate([int]) >>> t.view() Move the mouse over variable names (in red) to see their inferred types. Translating the flow graph to C code ++++++++++++++++++++++++++++++++++++ The graph can be turned into C code:: >>> t.rtype() >>> f = t.compile_c() The first command replaces the operations with other low level versions that only use low level types that are available in C (e.g. int). To try out the compiled version:: >>> f(5) False >>> f(6) True Translating the flow graph to CLI or JVM code +++++++++++++++++++++++++++++++++++++++++++++ PyPy also contains a `CLI backend`_ and JVM backend which can translate flow graphs into .NET executables or a JVM jar file respectively. Both are able to translate the entire interpreter. You can try out the CLI and JVM backends from the interactive translator shells as follows:: >>> def myfunc(a, b): return a+b ... >>> t = Translation(myfunc) >>> t.annotate([int, int]) >>> f = t.compile_cli() # or compile_jvm() >>> f(4, 5) 9 The object returned by ``compile_cli`` or ``compile_jvm`` is a wrapper around the real executable: the parameters are passed as command line arguments, and the returned value is read from the standard output. Once you have compiled the snippet, you can also try to launch the executable directly from the shell. You will find the executable in one of the ``/tmp/usession-*`` directories:: # For CLI: $ mono /tmp/usession-trunk-/main.exe 4 5 9 # For JVM: $ java -cp /tmp/usession-trunk-/pypy pypy.Main 4 5 9 To translate and run for the CLI you must have the SDK installed: Windows users need the `.NET Framework SDK 2.0`_, while Linux and Mac users can use Mono_. To translate and run for the JVM you must have a JDK installed (at least version 5) and ``java``/``javac`` on your path. A slightly larger example +++++++++++++++++++++++++ There is a small-to-medium demo showing the translator and the annotator:: cd demo ../pypy/translator/goal/translate.py --view --annotate bpnn.py This causes ``bpnn.py`` to display itself as a call graph and class hierarchy. Clicking on functions shows the flow graph of the particular function. Clicking on a class shows the attributes of its instances. All this information (call graph, local variables' types, attributes of instances) is computed by the annotator. To turn this example to C code (compiled to the executable ``bpnn-c``), type simply:: ../pypy/translator/goal/translate.py bpnn.py Translating Full Programs +++++++++++++++++++++++++ To translate full RPython programs, there is the script ``translate.py`` in ``translator/goal``. Examples for this are a slightly changed version of Pystone:: cd pypy/translator/goal python translate.py targetrpystonedalone This will produce the executable "targetrpystonedalone-c". The largest example of this process is to translate the `full Python interpreter`_. There is also an FAQ about how to set up this process for `your own interpreters`_. .. _`your own interpreters`: faq.html#how-do-i-compile-my-own-interpreters .. _`start reading sources`: Where to start reading the sources ---------------------------------- PyPy is made from parts that are relatively independent from each other. You should start looking at the part that attracts you most (all paths are relative to the PyPy top level directory). You may look at our `directory reference`_ or start off at one of the following points: * `pypy/interpreter`_ contains the bytecode interpreter: bytecode dispatcher in pyopcode.py_, frame and code objects in eval.py_ and pyframe.py_, function objects and argument passing in function.py_ and argument.py_, the object space interface definition in baseobjspace.py_, modules in module.py_ and mixedmodule.py_. Core types supporting the bytecode interpreter are defined in typedef.py_. * `pypy/interpreter/pyparser`_ contains a recursive descent parser, and input data files that allow it to parse both Python 2.3 and 2.4 syntax. Once the input data has been processed, the parser can be translated by the above machinery into efficient code. * `pypy/interpreter/astcompiler`_ contains the compiler. This contains a modified version of the compiler package from CPython that fixes some bugs and is translatable. That the compiler and parser are translatable is new in 0.8.0 and it makes using the resulting binary interactively much more pleasant. * `pypy/objspace/std`_ contains the `Standard object space`_. The main file is objspace.py_. For each type, the files ``xxxtype.py`` and ``xxxobject.py`` contain respectively the definition of the type and its (default) implementation. * `pypy/objspace`_ contains a few other object spaces: the thunk_, trace_ and flow_ object spaces. The latter is a relatively short piece of code that builds the control flow graphs when the bytecode interpreter runs in it. * `pypy/translator`_ contains the code analysis and generation stuff. Start reading from translator.py_, from which it should be easy to follow the pieces of code involved in the various translation phases. * `pypy/annotation`_ contains the data model for the type annotation that can be inferred about a graph. The graph "walker" that uses this is in `pypy/annotation/annrpython.py`_. * `pypy/rpython`_ contains the code of the RPython typer. The typer transforms annotated flow graphs in a way that makes them very similar to C code so that they can be easy translated. The graph transformations are controlled by the stuff in `pypy/rpython/rtyper.py`_. The object model that is used can be found in `pypy/rpython/lltypesystem/lltype.py`_. For each RPython type there is a file rxxxx.py that contains the low level functions needed for this type. * `pypy/rlib`_ contains the RPython standard library, things that you can use from rpython. .. _optionaltool: Running PyPy's unit tests ------------------------- The PyPy project uses test-driven-development. Right now, there are a couple of different categories of tests which you can run. To run all the unit tests:: cd pypy python test_all.py (this is not recommended, since it takes hours and uses huge amounts of RAM). Alternatively, you may run subtests by going to the correct subdirectory and running them individually:: python test_all.py interpreter/test/test_pyframe.py ``test_all.py`` is actually just a synonym for `py.test`_ which is our external testing tool. If you have installed that you can as well just issue ``py.test DIRECTORY_OR_FILE`` in order to perform test runs or simply start it without arguments to run all tests below the current directory. Finally, there are the CPython regression tests which you can run like this (this will take hours and hours and hours):: cd lib-python/2.5.2/test python ../../../pypy/test_all.py .. _`installed py.test`: https://codespeak.net/py/current/doc/download.html Special Introspection Features of the Untranslated Python Interpreter --------------------------------------------------------------------- If you are interested in the inner workings of the PyPy Python interpreter, there are some features of the untranslated Python interpreter that allow you to introspect its internals. Interpreter-level console +++++++++++++++++++++++++ If you start an untranslated Python interpreter via:: python pypy-svn/pypy/bin/py.py If you press on the console you enter the interpreter-level console, a usual CPython console. You can then access internal objects of PyPy (e.g. the `object space`_) and any variables you have created on the PyPy prompt with the prefix ``w_``:: >>>> a = 123 >>>> *** Entering interpreter-level console *** >>> w_a W_IntObject(123) The mechanism works in both directions. If you define a variable with the ``w_`` prefix on the interpreter-level, you will see it on the app-level:: >>> w_l = space.newlist([space.wrap(1), space.wrap("abc")]) >>> *** Leaving interpreter-level console *** KeyboardInterrupt >>>> l [1, 'abc'] .. _`object space`: objspace.html Note that the prompt of the interpreter-level console is only '>>>' since it runs on CPython level. If you want to return to PyPy, press (under Linux) or , (under Windows). You may be interested in reading more about the distinction between `interpreter-level and app-level`_. .. _`interpreter-level and app-level`: coding-guide.html#interpreter-level .. _`trace example`: Tracing bytecode and operations on objects ++++++++++++++++++++++++++++++++++++++++++ You can use the trace object space to monitor the interpretation of bytecodes in connection with object space operations. To enable it, set ``__pytrace__=1`` on the interactive PyPy console:: >>>> __pytrace__ = 1 Tracing enabled >>>> a = 1 + 2 |- <<<< enter a = 1 + 2 @ 1 >>>> |- 0 LOAD_CONST 0 (W_IntObject(1)) |- 3 LOAD_CONST 1 (W_IntObject(2)) |- 6 BINARY_ADD |- add(W_IntObject(1), W_IntObject(2)) -> W_IntObject(3) |- 7 STORE_NAME 0 (a) |- hash(W_StringObject('a')) -> W_IntObject(-468864544) |- int_w(W_IntObject(-468864544)) -> -468864544 |-10 LOAD_CONST 2 () |-13 RETURN_VALUE |- <<<< leave a = 1 + 2 @ 1 >>>> Demos ------- The `demo/`_ directory contains examples of various aspects of PyPy, ranging from running regular Python programs (that we used as compliance goals) over experimental distribution mechanisms to examples translating sufficiently static programs into low level code. Additional Tools for running (and hacking) PyPy ----------------------------------------------- We use some optional tools for developing PyPy. They are not required to run the basic tests or to get an interactive PyPy prompt but they help to understand and debug PyPy especially for the translation process. graphviz & pygame for flow graph viewing (highly recommended) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ graphviz and pygame are both necessary if you want to look at generated flow graphs: graphviz: http://www.graphviz.org/Download.php pygame: http://www.pygame.org/download.shtml CTypes on Python 2.4 ++++++++++++++++++++++++++++ `ctypes`_ is included in CPython 2.5 and higher. CPython 2.4 users needs to install it if they want to run low-level tests. See the `download page of ctypes`_. .. _`download page of ctypes`: http://sourceforge.net/project/showfiles.php?group_id=71702 .. _`ctypes`: http://starship.python.net/crew/theller/ctypes/ .. _`py.test`: py.test and the py lib +++++++++++++++++++++++ The `py library`_ is used for supporting PyPy development and running our tests against code and documentation as well as compliance tests. You don't need to install the py library because it ships with PyPy and `pypy/test_all.py`_ is an alias for ``py.test`` but if you want to have the ``py.test`` tool generally in your path, you might like to visit: http://codespeak.net/py/dist/download.html Getting involved ----------------- PyPy employs an open development process. You are invited to join our `pypy-dev mailing list`_ or look at the other `contact possibilities`_. Usually we give out commit rights fairly liberally, so if you want to do something with PyPy, you can become a committer. We are also doing coding Sprints which are separately announced and often happen around Python conferences such as EuroPython or Pycon. Upcoming events are usually announced on `the blog`_. .. _`full Python interpreter`: getting-started-python.html .. _`the blog`: http://morepypy.blogspot.com .. _`pypy-dev mailing list`: http://codespeak.net/mailman/listinfo/pypy-dev .. _`contact possibilities`: index.html .. _`py library`: http://codespeak.net/py .. _`Spidermonkey`: http://www.mozilla.org/js/spidermonkey/ .. _`.NET Framework SDK 2.0`: http://msdn.microsoft.com/netframework/downloads/updates/default.aspx .. _Mono: http://www.mono-project.com/Main_Page .. _`CLI backend`: cli-backend.html .. _clr: clr-module.html .. _`Dot Graphviz`: http://www.graphviz.org/ .. _Pygame: http://www.pygame.org/ .. _pyopcode.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/pyopcode.py .. _eval.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/eval.py .. _pyframe.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/pyframe.py .. _function.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/function.py .. _argument.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/argument.py .. _baseobjspace.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/baseobjspace.py .. _module.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/module.py .. _mixedmodule.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/mixedmodule.py .. _typedef.py: http://codespeak.net/svn/pypy/trunk/pypy/interpreter/typedef.py .. _Standard object space: objspace.html#the-standard-object-space .. _objspace.py: ../../pypy/objspace/std/objspace.py .. _thunk: ../../pypy/objspace/thunk.py .. _trace: ../../pypy/objspace/trace.py .. _flow: ../../pypy/objspace/flow/ .. _translator.py: ../../pypy/translator/translator.py .. _mailing lists: index.html .. _documentation: docindex.html .. _unit tests: coding-guide.html#test-design .. _`directory reference`: docindex.html#directory-reference .. include:: _ref.txt