Summary

Terminology:

• application level code - code written in full Python
• interpreter level code - code written in RPython, compiled to something else, say C, part of the interpreter.

PyPy's ctypes implementation in its current state proves the feasibility of implementing a module with the same interface and behavior for PyPy as ctypes for CPython.

PyPy's implementation internally uses libffi like CPython's ctypes. In our implementation as much as possible of the code is written in full Python, not RPython. In CPython's situation, the equivalent would be to write as little as possible code in C. We essentially favored rapid experimentation over worrying about speed for this first trial implementation. This allowed to provide a working implementation with a large part of ctypes features in 2 months real time.

We reused the ctypes package version 1.0.2 as-is from CPython. We implemented _ctypes which is a C module in CPython mostly in pure Python based on a lower-level layer extension module _rawffi.

Low-level part: _rawffi

This PyPy extension module (pypy/module/_rawffi) exposes a simple interface to create C objects (arrays and structures) and calling functions in dynamic libraries through libffi. Freeing objects in most cases and making sure that objects referring to each other are kept alive is responsibility of the higher levels.

This module uses bindings to libffi which are defined in pypy/rlib/libffi.py.

We tried to keep this module as small as possible. It is conceivable that other implementations (e.g. Jython) could use our ctypes implementation by writing their version of _rawffi.

High-level parts

The reused ctypes package lives in pypy/lib/ctypes. _ctypes implementing the same interface as _ctypes in CPython is in pypy/lib/_ctypes.

Discussion and limitations

Reimplementing ctypes features was in general possible. PyPy supports pluggable garbage collectors, some of them are moving collectors, this means that the strategy of passing direct references inside Python objects to an external library is not feasible (unless the GCs support pinning, which is not the case right now). The consequence of this is that sometimes copying instead of sharing is required, this may result in some semantics differences. C objects created with _rawffi itself are allocated outside of the GC heap, such that they can be passed to external functions without worries.

Porting the implementation to interpreter-level should likely improve its speed. Furthermore the current layering and the current _rawffi interface require more object allocations and copying than strictly necessary; this too could be improved.

The implementation was developed and has only been tested on x86-32 Linux.

Here is a list of the limitations and missing features of the current implementation:

• No support for PyXxx functions from libpython, for obvious reasons.
• We copy Python strings instead of having pointers to raw buffers
• Features we did not get to implement:
  • custom alignment and bit-fields
  • resizing (resize() function)
  • non-native byte-order objects
  • callbacks accepting by-value structures
  • slight semantic differences that ctypes makes between its primitive types and user subclasses of its primitive types

Getting the code and test suites

A stable revision of PyPy containing the ctypes implementation can be checked out with subversion from the tag:

http://codespeak.net/svn/pypy/tag/ctypes-stable

The various tests and later examples can be run on x86-32 Linux. We tried them on an up-to-date Ubuntu 7.10 x86-32 system.

If one goes inside the checkout it is possible to run _rawffi tests with:

$ cd pypy
$ python test_all.py module/_rawffi/

The ctypes implementation test suite is derived from the tests for ctypes 1.0.2, we have skipped some tests corresponding to not implemented features or implementation details, we have also added some tests.

To run the test suite a compiled pypy-c is required with the proper configuration. To build the required pypy-c one should inside the checkout:

$ cd pypy/translator/goal
$ ./translate.py --text --batch --gc=generation targetpypystandalone.py
  --withmod-_rawffi --allworkingmodules

this should produce a pypy-c executable in the goal directory.

To run the tests then:

$ cd ../.. # back to pypy/
$ ./translator/goal/pypy-c test_all.py lib/app_test/ctypes_tests

There should be 36 skipped tests and all other tests should pass.

Running application examples

pyglet is known to run. We had some success also with pygame-ctypes which is not maintained anymore and with a snapshot of the experimental pysqlite-ctypes. We will only describe how to run the pyglet examples.

$ cd pyglet/
$ PYTHONPATH=. ../ctypes-stable/pypy/translator/goal/pypy-c examples/opengl.py

ctypes configure

We also released ctypes-configure, which is an experimental package trying to approach the portability issues of ctypes-based code.