[lxml-dev] install lxml 2.0.5 on Mac OS X Leopard - why is it so hard?

[Mike Meyer]

On Sun, 11 May 2008 09:01:01 +0200
Stefan Behnel <stefan_ml at behnel.de> wrote:

> you ask why this is so hard? Simple answer: because no-one has contributed a
> way so far to make it easier.

Gee, I had no trouble at all doing this last week (the release of
Oracle library bits for Intel OS-X means it's now desirable). I
installed macports, did a self-update, then installed py25-lxml.  It
installed python2.5.2 and the versions of libxml2 and libxslt that
were in macports as part of the process. Installing cx_Oracle after
that was more work.

> We had lots of reports about stuff not working and almost as many
> work-arounds, but no-one came up with a patch that would allow building lxml
> reliably at least on a subset of Mac-OS systems. And I just cannot believe
> that there is no-one amongst the Mac-OS-X users who knows how to use distutils
> to build a binary extension. Or at least someone who knows how to build C code
> statically against a C library.

I'm sorry, but my experience is that binary distributions make the
problems *worse*, not better - at least if you require multiple
different components to be installed. You have to make sure the
components all agree about the builds of any libraries they have in
common, and unless you have a coordinated build, that just doesn't
happen.

After all, I could build a binary distribution of lxml from macports,
but to use it, you'd have to have the macports versions of python,
libxml2 and libxslt. If you've got that, it's probably easier to
install the macports version than it is to download and install
whatever I might build. I could use ports to build a binary package
with all those things in it - is there anyone who really wants that?

I started working with lxml last year, when the latest version was
1.3.3.  Since updating the software after deployment would be a
traumatic operation (a single "instance" of the application uses about
40 cores spread across 10 systems and two SANs, and we typically run
three instances), I wanted the latest stable versions of everything. I
looked at five different systems, and on only two was getting that
combination sane: OSX using macports, and FreeBSD, mostly because they
had the latest versions in the ports system when I went to look. The
three GNU/Linux systems either had old versions of Python, of the xml
libraries, and lxml was either old or missing. So I wound up doing
initial development on OSX and FreeBSD while we dealt with the
GNU/Linux platform we were speced to run on.

Grabbing binaries only half worked. Replacing the installed system
tools on GNU/linux is a recipe for disaster, and our sysadmins
correctly refused to do so. Which means the LSB is no help at all. The
sysadmins found a binary build of python 2.5, and installed that. We
then grabbed the lxml rpm from PyPI, and installed that - only it
wouldn't run, because it had been built against a version of Python
that was compiled with a python shared library, and the version we had
hadn't been.

I eventually wound up building everything - Python, libxml2, libxslt,
lxml and cx_Oracle - by hand to run our installation on, and providing
a carefully tailored environment to run things in. Which is what I did
on OSX and FreeBSD, except their ports systems makes building from
sources trivial, and FreeBSD doesn't need the tailored
environment. Updating those two systems is trivial. Updating the
GNU/Linux systems is several days worth of work just to get to the
point where I have something to give to operations.

> From my POV, Mac-OS seems to lack three things that make this problem
> non-trivial. It doesn't have a standard package management system. Neither
> does it have something like the Linux Standard Base, which dictates where
> newly installed things belong. And it doesn't seem to support "rpath", which
> would allow a binary to say "I know where my dependencies come from". Or at
> least distutils don't support that on Mac. So everything I could try here on
> Linux to make it work better is bound to fail.

Providing a binary distribution for *any* system that includes
libraries that are "to old" in the base system is going to be a major
pain. Everyone runs into this on OSX, because they didn't update those
libraries in 10.5 for some unknown reason (I've filed bug ID #5926693
at adc about this). Those of us building for corporate environments -
where we always run on out of date platforms, because we can't get
corporate approval to use a new one before it becomes out of date -
run into it all the time.

    <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org