\chapter{Core Material\label{core-material}}

Everyone should know the material of every section of this chapter,
which is designed to be read from beginning to end.

The aim of this chapter is that after reading it (and possibly a small
amount of Python/C API reference documentation) you should be able to
write a procedural interface to more-or-less any third party library
with a C interface.  (Do I want to get into how to link to third party
C libraries?  It might be useful)

\section{Your First Python extension\label{your-first-extension}}

Here's a minimal yet complete Python extension in C:

\verbatiminput{firstmodule.c}

Short, huh?  Of course, this extension does almost nothing but it can
be used to illustrate a few points.

Foremost of these is the process of turning C source into a Python
extension.  Nowadays this is pretty simple: put the above source in a
file called 'firstmodule.c' and the following in a file called
'first-setup.py' in the same directory:

\verbatiminput{first-setup.py}

Running 

\begin{verbatim}
$ python first-setup.py build_ext -i
\end{verbatim} %$ <-- bow to font-lock  ;-(

from a shell or DOS box should result in a ``dynamic shared object''
(``first.so'' on Unix variants or ``first.pyd'' on Windows) in the
working directory.  Run Python from that directory and you should be
able to ``\code{import first}'', although the \module{first} module is
pretty dull.

You'll notice that the module name -- here ``first'' -- occurs in
three places: the name of the object file containing the extension
code, the name of the ``module init function'' and in the call to
Py_InitModule3.

When you execute \code{import first}, the interpreter finds the shared
object file, looks for a function named ``\code{initfirst}'' and
complains if executing that function fails to register a module called
``first''.  So this module really is minimal.

For obvious reasons, it is customary for the module's name to also be
part of the name of the file containing the C source.

I'll omit detailed instructions on how to compile other code samples
and assume that you can generalize this one.
% XXX unless things get more complex.

\file{Python.h} is usually the only Python specific file you need to
include to gain access to the Python/C API.  It is important that
\file{Python.h} is the first header you include.

The distutils package takes care of the details of compiling your file
so that this header is accessible.

Why \code{Py_InitModule\textbf{*3*}}?  Well, there are variants of
this function for historical reasons (which, yes, take 2 or 4
arguments), but there are no reasons to use them in new code.

The symbol \code{PyMODINIT_FUNC} is a preprocessor symbol provided by
Python that contains the necessary dynamic linker magic your platform
needs for the import of a dynamic extension to work.  It is strongly
advised that all other symbols in your extension are declared
\code{static}.

With this experience under our belt, we move on to examples that
actually do something.

\section{A Less Trivial Example\label{less-trivial}}

Here's the complete source to a module that exposes one function
called \code{func()} that takes two arguments and returns their sum:

\verbatiminput{secondmodule.c}

Although this is still fairly short, there's quite a bit to explain.

This module is essentially identical to the following Python module:

\begin{verbatim}
def func(a, b):
    return a + b
\end{verbatim}

Something it helps greatly to understand about writing C extensions is
the use and value of various conventions -- or perhaps ``patterns'' --
both in stylistic aspects of the source and in the design of the C
API.

A large fraction of C functions that are exposed to Python are defined
like this:

\begin{verbatim}
static PyObject *
<MODNAME>_<FUNCNAME>(PyObject *self, PyObject *args)
{
	...
}
\end{verbatim}

In this example the function ``\code{func()}'' is just a module-level
function and the \var{self} parameter will always be \code{NULL}.  The
\var{args} parameter will contain a tuple of the arguments the
function was called with.

There's not much good defining a function without allowing access to
it.  This is the purpose of the \code{second_methods} array: it lists
the functions exposed to Python by the module.

Each entry in the array is a struct of type \code{PyMethodDef} and has
four fields, in order:

\begin{tableii}{cl}{code}{field}{meaning}
  \lineii{ml_name}
         {The name of the function as seen from Python.}
  \lineii{ml_meth}
         {A pointer to the C implementation of the function.}
  \lineii{ml_flags}
         {A flag indicating the calling convention the function uses.
          \code{METH_VARARGS} is the usual choice here; the
          alternatives will be explained in a later section.}
  \lineii{ml_func}
         {The function's docstring or \NULL{} if you want to be evil
          and have a function with no docstring.}
\end{tableii}

The last entry in the array is indicated by a sentinel entry filled
with \code{NULL}s.

One of the conventions I mentioned earlier is that the name of the C
function implementing a function is of the form MODNAME_FUNCNAME.  As
you can from the above you can call it anything you like, but also
that it's sensible to have \emph{some} relation between the two names.

The \code{second_methods} array is passed as the second parameter to
\code{Py_InitModule3}.  There are other ways to arrange for functions
to be available at module level, but this is the easiest -- and
conventional -- way.

Having covered (lightly) exposing functions to Python, we now turn to
\code{second_func}'s implementation.

The first thing most functions should do is verify that the arguments
match expectations, both in number and type.  This function accepts
arguments of any type, so we use the API function
\code{PyArg_UnpackTuple}.  \code{PyArg_UnpackTuple} takes a variable
number of arguments:

\begin{enumerate}
\item \var{args}: An argument tuple.  A \code{METH_VARARGS} function
  knows its \var{args} parameter is a tuple, so there is no need to
  check.
\item \var{name}: The name of the function, which can appear in error
  messages.
\item \var{min}: The minimum number of arguments the function accepts.
\item \var{max}: The maximum number of arguments the function accepts.
\item \var{...}: \var{max} pointers to variables of type \code{PyObject *} 
\end{enumerate}

If there are a suitable number of arguments, the pointers passed to
\code{PyArg_UnpackTuple} will be filled out with the passed arguments
(if at least \var{min} but less than \var{max} arguments are passed,
the corresponding pointers are left alone, so it's best to initialize
them to \code{NULL} -- but this does not matter in our example as both
\var{min} and \var{max} are \code{2}).

There is a related function, \code{PyArg_ParseTuple} that can do more
sophisticated argument checking and processing, but
\code{PyArg_UnpackTuple} is more efficient in the cases where it
applies.

\code{PyArg_UnpackTuple} returns a true value on success and \code{0}
on failure.  This is the norm (???) for functions that return an
\code{int} as an error indicator.  Slightly confusingly, the
convention for functions that return a \emph{value} of type \code{int}
is to return \code{-1} in case of error.

The convention for C API functions that return \code{PyObject*}s is
much clearer: \code{NULL} indicates failure, non-\code{NULL} (and by
implication, a valid pointer) indicates success.  With a \emph{very}
small number of (not accidental or historical) exceptions, a
\code{NULL} return is indication that the called function set an
exception.  Conversely, a non-\code{NULL} return indicates that no
exception is set.

These conventions apply just as strongly to C functions you write and
getting them wrong is to be considered a severe bug in your code.

It is important to note (and tedious to live with) the fact that
essentially any API function can fail -- \code{MemoryError} in
particular is almost impossible to rule out.

The foregoing has two notable impacts on the code in the example.  The
first is that if \code{PyArg_UnpackTuple} returns \code{0}, we know it
has set an exception so we just return \code{NULL}.  The other is that
when we call \code{PyNumber_Add} we know that it will return
\code{NULL} or non-\code{NULL} exactly when we need to, so there is no
need for us to check its return value.  This is no accident: by and
large the calling conventions of the Python/C API are designed to make
programming with it convenient.

To return to explaining what the example function actually does...

In simple terms, it receives two arguments and returns their sum.
We've covered checking the number of arguments but not actually
discussed how Python objects are presented to C code that deals with
them.

All Python objects are allocated on the heap.  They can all be
referred to as pointers to a \code{PyObject}.  A \code{PyObject} just
contains a reference count and a pointer to a type object.  So when a
function receives arguments of arbitrary type, as the example does, it
will be dealing with pointers to \code{PyObject}.

The spec for the example function says that it returns the sum of the
arguments.  The Python/C API provides a function to do this:
\code{PyNumber_Add}.  This is part of what is sometimes termed the
``abstract objects layer'' of the API: it allows you to operate on
objects in a similar fashion to the way interpreted Python code does.
In this case \code{PyNumber_Add} is equivalent to the \code{+}
operator in Python, as good a definition of ``sum'' of two values as
any in Python.  You can probably guess the name of the functions that
subtract and multiply two values.

Something that is notable by its absence from the example function
above is explicit manipulation of reference counts.  This is another
example of the conventions of the Python/C API making programming
convenient.  It's not always so convenient.

\section{Refcounting Conventions\label{refcounting-conventions}}

As you may know, Python uses reference counting to keep track of
whether objects can be deallocated: if an object's refcount is greater
than zero, it is assumed that the object is still in use and when it
hits zero the object is assumed to be \dfn{garbage} and deallocated.

The terminology of reference counts is that of ownership.  If your
code has a reference to an object (i.e. a non-\code{NULL} pointer to a
\code{PyObject}) it is either \dfn{owned} by your code or merely
\dfn{borrowed}.  If you own a reference, you must either transfer the
ownership of or dispose of the reference before your code returns.  If
the reference is merely borrowed, however, you must \emph{not} do
either of these things.  Breaking either of these rules is a serious
bug in your code and will either lead to memory leaks or outright
crashes.

To dispose of a reference you use the \code{Py_DECREF} macro.  To
convert a borrowed reference into a owned reference you use the
\code{Py_INCREF} macro.

Most functions in the Python/C API that take \code{PyObject*}
parameters borrow the references you provide.  A very few \dfn{steal}
a reference to one of their arguments -- so you must own the reference
you pass, and if you wish to use the object after the call you should
call \code{Py_INCREF} first.

Again, most but not all API functions that return an object transfer
the ownership of the returned reference to their caller.  The
exceptions include (some?) type-specific accessors such as
\code{PyList_GetItem} and \code{PyDict_GetItem} which loan a reference
to one of their elements.

If that made sense, congratulations.  If not, lets see some examples.

First, lets consider the example from the last section:

\begin{verbatim}
static PyObject*
second_func(PyObject *self, PyObject *args)
{
	PyObject *a, *b;
	
	if (!PyArg_UnpackTuple(args, "func", 2, 2, &a, &b)) {
		return NULL;
	}

	return PyNumber_Add(a, b);
}
\end{verbatim}

References to arguments in C functions you write always start out
borrowed.  \code{PyNumber_Add} borrows references from and transfers
ownership of the reference to the result back to its caller.  Our code
then immediately transfers the ownership of this reference back to
\emph{its} caller -- so there is no need for explicit refcount
manipulation in this function.

To extend this example, lets consider a slightly different function:
we're going to write a function \code{func2} that takes one or two
arguments.  If given two, return their sum.  If given one, return it
unchanged.  To illustrate by example:

\begin{verbatim}
>>> func2(1, 2)
3
>>> func2(4)
4
\end{verbatim}

Here's one way to do it:

\begin{verbatim}
static PyObject*
second_func2(PyObject *self, PyObject *args)
{
	PyObject *a, *b = NULL;
	
	if (!PyArg_UnpackTuple(args, "func2", 1, 2, &a, &b)) {
		return NULL;
	}

        if (b == NULL) {
                Py_INCREF(a);
                return a;
        }
        else {
        	return PyNumber_Add(a, b);
        }
}
\end{verbatim}

The modification to \code{PyArg_UnpackTuple} should require no
explanation.  Because this function will not touch \code{b} if
\code{func2} is called with only one argument, we initialize that to
\code{NULL} and check whether it is still \code{NULL} after the call
to see if one or two arguments were passed.  There are other ways of
doing this, but this way is -- you guessed it -- conventional.

If only one argument was passed, we want to return a reference to
\code{a}.  But you must own the reference you return and we are only
loaned a reference to the argument so we convert the borrowed
reference to an owned reference with \code{Py_INCREF}.

If there are two arguments, the situation is as before.

There is a potential problem with borrowed references when calling
back into the Python interpreter.  If you've borrowed a reference to a
list item, say, and then execute code that may involve executing
Python code that Python code may invalidate the reference you've
borrowed.

This gets particularly subtle when you realize just how many innocuous
seeming calls may call back into the Python interpreter.
\code{PyNumber_Add} certainly may -- consider \code{__add__} methods.
Taking the length of a sequence or comparing two objects, also.  Most
subtle of all, however, are (perhaps indirect) calls to
\code{Py_DECREF} -- which may execute \code{__del__} methods.

A concrete example.  See if you can spot the problem with the
following code:

\begin{verbatim}
void
bug(PyObject *list)
{
	PyObject *item = PyList_GetItem(list, 0);

	PyList_SetItem(list, 1, PyInt_FromLong(0L));
	PyObject_Print(item, stdout, 0);
}
\end{verbatim}

Got it?  Remember that \code{PyList_GetItem} only loans a reference to
a list item to its caller.

The problem is that the call to \code{PyList_SetItem} overwrites the
previous value of \code{list[1]} -- so if the previous contents of
\code{list[1]} had a \code{__del__} method that overwrote
\code{list[0]} it's possible that \var{item} points to a
\code{PyObject} that has been deallocated.  The call to
\code{PyObject_Print} will then most likely crash.

The solution of course is to make sure we own the reference to
\var{item} during the call to \code{PyList_SetItem} (and
\code{PyObject_Print}, for that matter, as this function too can
execute arbitrary Python code):

\begin{verbatim}
void
no_bug(PyObject *list)
{
	PyObject *item = PyList_GetItem(list, 0);

	Py_INCREF(item);
	PyList_SetItem(list, 1, PyInt_FromLong(0L));
	PyObject_Print(item, stdout, 0);
	Py_DECREF(item);
}
\end{verbatim}

Bugs of this ilk can be immensely hard to detect.  There have been
quite a few found and squashed even in core Python in recent years,
and it would be optimistic to assume that none remain.

\section{Handling Different Types of Data\label{handling-different-types-of-data}}

Although we've covered a fair amount of ground already, we're still
not in a position to write an actually useful extension.  Part of the
reason for this is the fact that we've only handled raw
\code{PyObject}s.  One of the primary purposes of writing C extensions
is interfacing with third party APIs, and not many of them take
\code{PyObject*}s!


As was mentioned earlier, all Python objects are allocated on the
heap, and the first few bytes of every object can be interpreted as
being those of a \code{struct PyObject}.  In a release build, a
\code{PyObject} just consists of a reference count and a pointer to a
\dfn{type object}.  The reference count we have seen.  The type object
determines almost every aspect of the behavior of the object.

The problem of handling differing data-types is naturally two-sided:
one must be able to extract raw C-level data from Python objects and
assemble such data into Python objects.

\subsection{From Python to C}

In many extension functions, you can use the argument processing
function \code{PyArg_ParseTuple} to check the types of the arguments
passed to your function and extract values in a format more usual to C
programs.

For instance, here's a module that implements type-specific addition
functions (there's not much point to this, but never mind):

\verbatiminput{thirdmodule.c}

The main novelty here is the use of the \code{PyArg_ParseTuple}
function.  This function (like \code{PyArg_UnpackTuple}) takes a
variable number of arguments:

\begin{enumerate}
\item \var{args}: An argument tuple.
\item \var{format}: A format string, usually ending in a colon and the
  name of the function, which can appear in error messages.
\item \var{...}: A variable number of arguments, the type and number
  of which are determined by \var{format}.
\end{enumerate}

\code{PyArg_ParseTuple} is quite a complex function and no attempt to
describe every intricacy is made here.

In common usage, each character of the \var{format} string indicates
that the correspending varadic argument is a pointer to a given C type
and \code{PyArg_ParseTuple} will attempt to convert the corresponding
Python-level positional argument to that type and store this value
into the given pointer.

So, in the function \code{third_addi} in the above module, \code{"ll"}
declares that the final two arguments are pointers to C variables of
\code{long} type.  Attempting to pass a value that cannot be
implicitly converted to a Python integer -- a string or a file, say
-- will result in \code{PyArg_ParseTuple} setting an exception and
returning \code{0}.  Conversely, passing integers or instances of a
user defined class that defines an \code{__int__} method stores the
integer values obtained from these objects in the C variables
\code{a} and \code{b} (Python integers are implemented using C longs,
so long is the most natural type to use here).

Here's a list of a few of the simpler format codes:

\begin{tableii}{c|l}{textrm}{Code}{C type}
  \lineii{\code{"l"}}
         {\code{long}}
  \lineii{\code{"i"}}
         {\code{int}}
  \lineii{\code{"d"}}
         {\code{double}}
  \lineii{\code{"f"}}
         {\code{float}}
  \lineii{\code{"s"}}
         {\code{char *} (a C string)}
  \lineii{\code{"O"}}
         {\code{PyObject *}}
  \lineii{\code{"|"}}
         {indicates that the following arguments are optional.}
\end{tableii}

For more details, please see the fine documentataion XXX link!.

A final note is that parsing the format code on each invocation of the
function takes time.  If the call to \code{PyArg_ParseTuple} isn't
doing much -- if you aren't taking advantage of the type checking and
processing it provides -- a call to \code{PyArg_UnpackTuple} is
probably clearer and more efficient.

One could get the impression from the above that every function in an
extension module \emph{must} call one of \code{PyArg_UnpackTuple} or
\code{PyArg_ParseTuple}.  Nothing could further from the truth!  But,
like functions you define in Python, almost all extension functions
you write in fact take a fixed number of arguments, and using one of
the \code{PyArg_} API functions is a more convenient -- and
conventional -- way of checking the arguments for validity than doing
it by hand.

\subsection{From C to Python}

We've (partially) covered turning Python objects into C objects.  What
about the reverse?  A function that returns to Python must return a
pointer to a \code{PyObject}, not a raw C long or double.

Fortunately, converting simple C types such as integers and strings to
the corresponding Python types is pretty easy: as you can see from the
example module, \code{PyInt_FromLong} does the former job and the
predictably named \code{PyString_FromString} does the latter (for
\code{NULL}-terminated strings anyway; being aware of the
\code{PyString_FromStringAndSize} function is useful for when that's
not what you want).

For more complex situations, Python provides the function
\code{Py_BuildValue}:

\begin{verbatim}
PyAPI_FUNC(PyObject *) Py_BuildValue(char *format,  ...);
\end{verbatim}

The string \var{format} is similar to that passed to
\code{PyArg_ParseTuple}, with the difference that the following
arguments are not pointers to variables of appropriate types, but
values of these types.

The type of object returned by \code{Py_BuildValue} depends on the
number of ``format units''...

XXX need an example of using Py_BuildValue.

XXX describe Py_BuildValue, PyList_*, etc here?  Mention, at least?

\section{A Sketch of the Python/C API\label{api-sketch}}

There is no way a tutorial document like this can or should cover
every aspect of the Python/C API.  This section aims to give you some
pointers for what you should do when you find yourself needing to know
something not described in this document.

The two main sources of information are, perhaps predictably, the API
reference manual and the source of Python itself.

The Python/C API is divided into at least three layers: the ``very
high'', ``abstract objects'' and ``concrete objects'' layers, together
with a few other areas.

The very high level interface is mostly about executing Python source
code contained in files or C strings.  It is more useful when
embedding than when extending Python.

The abstract objects layer contains functions that interact with
objects irrespective of their type, or across a wide range of types
such as ``all sequence types''.  Most functions in this API perform
operations that also be executed from Python: if all your extension
does is call abstract layer functions, there's likely no reason for it
not being a Python module instead!  Such a C extension is not likely
to be very much quicker than the corresponding Python module, and
certainly a good deal more tedious to write.  It's good practice,
however, to use abstract objects layer functions to handle objects
that are passed in as arguments to your functions; for example, it's
friendlier to accept any sequence type as opposed to insisting on
lists.

The concrete layer deals with particular types of Python object --
dicts, ints, tuples and so on.  Which of these functions you are most
likely to need corresponds fairly closely to the types you use often
in Python, and also to those that most closely correspond to C types.
It's a rare extension module that doesn't use an integer-related
function; it's an even rarer one that explicitly manipulates
\code{PyGetSetDescrObject}s.  A frequent use of concrete layer
functions is assembling objects for returning to Python; if an object
has been created by a call to \code{PyList_New} you can be certain
that it's a list, and using a generic sequence function to fill it out
is just pointless.

Not all of the API fits into these layers; there are also functions
for importing modules, dealing with exceptions and the unpacking and
packing functions that have already been described.

Another part of the API consists not of functions but of structure
definitions, of which by far the most important 

\section{A Little More on Exceptions\label{more-on-exceptions}}

The stated goal of this chapter is to enable the reader to write an
extension providing a (procdedural) Pythonic interface to some C
library.  We're nearly there, but after perhaps reading the reference
documentation on format strings and some of the concrete types, the
big gap is the handling of errors.

We've touched on the matter of \emph{coping} with exceptions: that if
you detect that a function you've called has raised an exception, you
should pass this on to your caller without delay.  But what if the
error situation is in a C library you've called?  You could pass an
error code back to your Python caller, but that's hardly Pythonic.  We
need to be able to set an exception of our own.

Also, occasionally one needs to \emph{handle} exceptions raised by
functions you call, i.e. to write the C equivalent of:

\begin{verbatim}
try:
    func()
except KeyError:
    pass
\end{verbatim}

(the C equivalent of a bare \code{except:} statement is just as bad an
idea in C as it is in Python.  Or possibly even worse.)

Probably the most commonly used exception setting function is
\code{PyErr_SetString}:

\begin{verbatim}
PyAPI_FUNC(void) PyErr_SetString(PyObject *exception, 
				 const char *string);
\end{verbatim}

\var{exception} should be an exception class.  The standard Python
exception classes are all available to C code as global variables
named \code{PyExc_} and the Python name of the exception, so
e.g. \code{ArithmeticError} in Python is \code{PyExc_ArithmeticError}
in C.  \var{string} is the error message that will be converted to a
string object.

\section{Linking your Extension to other C Libraries}

Hmm, I'll have to do some reading to be able to write this section!

\section{A Note on Style}

The C examples in this document follow the coding conventions for the
Python project as described in PEP 7.  You're certainly not forced to
use this style for your extensions, but if you have hope for your work
to be sufficiently generally useful to one day be included as part of
Python, using the Python style can't hurt your chances of acceptance.

If you use Emacs...