From stefan_ml at behnel.de Sun Nov 2 20:55:09 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Nov 2008 20:55:09 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <4909DAD3.4020103@colorstudy.com> References: <49075D83.2050100@colorstudy.com> <49095C1C.6090904@behnel.de> <4909DAD3.4020103@colorstudy.com> Message-ID: <490E059D.6060507@behnel.de> Hi Ian, Ian Bicking wrote: > Stefan Behnel wrote: > but I'd like it to work without buildout, and there's also several > buildout recipes and configurations out there and not one clear > canonical way to build lxml. I wouldn't mind to ship lxml with a buildout recipe. I think the current problem is that if we wait to find one that works well for as many people as possible, we'll wait forever. So I'm fine with adding any script that's somewhat tested and in use. Even a set of scripts that you can try is better than none if people can't manage to build lxml themselves. >>> Another option would be simply a different tarball that contains the >>> libxml2/libxslt source, and its setup.py would always build those. >>> It could be versioned like 2.1static or something, which should keep >>> it from being implicitly used by easy_install, etc. (since 2.1static >>> is considered an earlier version than 2.1). This might be more >>> reasonable? >> >> The problem with this (and with the static Windows builds) is that >> libxml2/libxslt both have their release cycles, which are independent of >> lxml's releases. If you want to upgrade your libxml2 in a static >> build, you'll have to copy it to the right place anyway. > > Another option is yet another environmental variable to set the > libxml2/libxslt versions, which are set to defaults. If you chose a > version that didn't exist in the tarball it'd download that version. What about providing a script that checks for the most recent version first, and then builds lxml with that? The "LATEST_IS_X.Y.Z" file in the libxml2 FTP directory will help here. Then it's fine to override the latest version with an env variable if people need to. >>> Maybe the two versions could be arranged with some svn:externals, >> >> You mean lxml and libxml2? That would mean you either build libxml2 >> from a tag >> (implying the same problems as with a shipped, ready-to-get-outdated >> version), >> or from the trunk, which is definitely not suitable for most users. > > No, I was just thinking about ways of structuring a full build (that > includes the libxml2 library) and the current build. Shipping with the latest libxml2/libxslt is not a big problem, setup.py could be changed to support a "--all-sources" option for sdist. There could be a separate tar.gz (or zip) file on PyPI that easy_install won't find at all. And when the build detects that the libxml2 and libxslt sources lie next to the src directory, it can switch to a static build automatically. It's not an unsolvable problem, these things just need to get implemented. > * Macs come with bad versions of libxml2 and libxslt (depending on the > version of the OS, you get either very bad, or not as bad, but bad > enough that you'll eventually get a segfault but not immediately, which > is actually much worse) > > * People keep getting the wrong runtime linking, and DYLD_LIBRARY_PATH > seems to be necessary. That makes a case for a static build, IMHO. > * There's several kinds of Python that people use on a Mac: at least the > system python, macports python, and fink python. I think there might be > another. They are all somewhat different. I'm not sure this makes a difference, distutils should handle this. The only problem is that this keeps us from preparing binaries for MacOS. > * It's not that obvious how to build libxml2, unless you are using > macports (which has a port for it). Not sure what you mean here. Isn't cmmi enough? > * Once you do build it, you have to be sure to get the right > xml2-config, which doesn't happen by default. Another reason for defaulting to a static build here. > I do now have ssh access to a Mac. Unfortunately it's behind our VPN, > so I'm not sure if I can get you access to it too, but I'll ask about that. I'm still waiting for the usual "we can pay you the hardware, so that you can do the testing yourself" offers. ;o) Stefan From {bymail{ignore} at bestley.co.uk Mon Nov 3 10:24:22 2008 From: {bymail{ignore} at bestley.co.uk (Mark Bestley) Date: Mon, 03 Nov 2008 09:24:22 -0000 Subject: [lxml-dev] lxml Mac installation idea References: <49075D83.2050100@colorstudy.com> <49095C1C.6090904@behnel.de> <4909DAD3.4020103@colorstudy.com> <490E059D.6060507@behnel.de> Message-ID: On Sun, 02 Nov 2008 19:55:09 -0000, Stefan Behnel wrote: > Hi Ian, > > Ian Bicking wrote: >> Stefan Behnel wrote: >> but I'd like it to work without buildout, and there's also several >> buildout recipes and configurations out there and not one clear >> canonical way to build lxml. > > I wouldn't mind to ship lxml with a buildout recipe. I think the current > problem is that if we wait to find one that works well for as many > people as > possible, we'll wait forever. So I'm fine with adding any script that's > somewhat tested and in use. Even a set of scripts that you can try is > better > than none if people can't manage to build lxml themselves. > > > >> * Macs come with bad versions of libxml2 and libxslt (depending on the >> version of the OS, you get either very bad, or not as bad, but bad >> enough that you'll eventually get a segfault but not immediately, which >> is actually much worse) >> >> * People keep getting the wrong runtime linking, and DYLD_LIBRARY_PATH >> seems to be necessary. > > That makes a case for a static build, IMHO. > Another way might be as done by Enthought and have lxml's own copy of the dynamic libraries ending up in site-packages/lxml-2.1.1.0001-py2.5-macosx-10.3-fat.egg/lxml/libxslt.dylib > >> * There's several kinds of Python that people use on a Mac: at least the >> system python, macports python, and fink python. I think there might be >> another. They are all somewhat different. > > I'm not sure this makes a difference, distutils should handle this. The > only > problem is that this keeps us from preparing binaries for MacOS. > I would suggest that for binaries you only need to consider the system python although that differs with each version of the OS. The reason I say this is that macports and Fink tend to have everything done through their own port environment. Macports does have lxml already There are also users that have built their own python to get the latest version e.g. 2.5.2 or 2.6 but they can use a compiler so a setup or buildout should be sufficient Even if I am wrong on this it is a start and I think would help those who know the least about the non Python parts of the Mac environment > >> * It's not that obvious how to build libxml2, unless you are using >> macports (which has a port for it). > > Not sure what you mean here. Isn't cmmi enough? > > >> * Once you do build it, you have to be sure to get the right >> xml2-config, which doesn't happen by default. > > Another reason for defaulting to a static build here. > > -- Mark From paul at agendaless.com Mon Nov 3 14:38:03 2008 From: paul at agendaless.com (Paul Everitt) Date: Mon, 3 Nov 2008 08:38:03 -0500 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <490E059D.6060507@behnel.de> References: <49075D83.2050100@colorstudy.com> <49095C1C.6090904@behnel.de> <4909DAD3.4020103@colorstudy.com> <490E059D.6060507@behnel.de> Message-ID: <52798438-2627-4411-BF2B-CFF18A60F6FE@agendaless.com> On Nov 2, 2008, at 2:55 PM, Stefan Behnel wrote: > Hi Ian, > > Ian Bicking wrote: >> Stefan Behnel wrote: >> but I'd like it to work without buildout, and there's also several >> buildout recipes and configurations out there and not one clear >> canonical way to build lxml. > > I wouldn't mind to ship lxml with a buildout recipe. I think the > current > problem is that if we wait to find one that works well for as many > people as > possible, we'll wait forever. So I'm fine with adding any script > that's > somewhat tested and in use. Even a set of scripts that you can try > is better > than none if people can't manage to build lxml themselves. FWIW, we have a very lean buildout that only focuses on the problem of building lxml. I could contribute it if you'd like. >>>> Another option would be simply a different tarball that contains >>>> the >>>> libxml2/libxslt source, and its setup.py would always build those. >>>> It could be versioned like 2.1static or something, which should >>>> keep >>>> it from being implicitly used by easy_install, etc. (since >>>> 2.1static >>>> is considered an earlier version than 2.1). This might be more >>>> reasonable? >>> >>> The problem with this (and with the static Windows builds) is that >>> libxml2/libxslt both have their release cycles, which are >>> independent of >>> lxml's releases. If you want to upgrade your libxml2 in a static >>> build, you'll have to copy it to the right place anyway. >> >> Another option is yet another environmental variable to set the >> libxml2/libxslt versions, which are set to defaults. If you chose a >> version that didn't exist in the tarball it'd download that version. > > What about providing a script that checks for the most recent > version first, > and then builds lxml with that? The "LATEST_IS_X.Y.Z" file in the > libxml2 FTP > directory will help here. Then it's fine to override the latest > version with > an env variable if people need to. > > >>>> Maybe the two versions could be arranged with some svn:externals, >>> >>> You mean lxml and libxml2? That would mean you either build libxml2 >>> from a tag >>> (implying the same problems as with a shipped, ready-to-get-outdated >>> version), >>> or from the trunk, which is definitely not suitable for most users. >> >> No, I was just thinking about ways of structuring a full build (that >> includes the libxml2 library) and the current build. > > Shipping with the latest libxml2/libxslt is not a big problem, > setup.py could > be changed to support a "--all-sources" option for sdist. There > could be a > separate tar.gz (or zip) file on PyPI that easy_install won't find > at all. And > when the build detects that the libxml2 and libxslt sources lie next > to the > src directory, it can switch to a static build automatically. That sounds pretty good. >> I do now have ssh access to a Mac. Unfortunately it's behind our >> VPN, >> so I'm not sure if I can get you access to it too, but I'll ask >> about that. > > I'm still waiting for the usual "we can pay you the hardware, so > that you can > do the testing yourself" offers. ;o) I'll chip in. I wonder if there are OS X hosting companies where we could rent a login. --Paul From sidnei at enfoldsystems.com Mon Nov 3 16:36:49 2008 From: sidnei at enfoldsystems.com (Sidnei da Silva) Date: Mon, 3 Nov 2008 13:36:49 -0200 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <52798438-2627-4411-BF2B-CFF18A60F6FE@agendaless.com> References: <49075D83.2050100@colorstudy.com> <49095C1C.6090904@behnel.de> <4909DAD3.4020103@colorstudy.com> <490E059D.6060507@behnel.de> <52798438-2627-4411-BF2B-CFF18A60F6FE@agendaless.com> Message-ID: On Mon, Nov 3, 2008 at 11:38 AM, Paul Everitt wrote: >>> I do now have ssh access to a Mac. Unfortunately it's behind our >>> VPN, >>> so I'm not sure if I can get you access to it too, but I'll ask >>> about that. >> >> I'm still waiting for the usual "we can pay you the hardware, so >> that you can >> do the testing yourself" offers. ;o) > > I'll chip in. I wonder if there are OS X hosting companies where we > could rent a login. Sourceforge.net Build Farm used to have OSX hosts. They discontinued the service somewhere in 2007 though. :( -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 Skype zopedc From ianb at colorstudy.com Mon Nov 3 20:35:06 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 03 Nov 2008 13:35:06 -0600 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <52798438-2627-4411-BF2B-CFF18A60F6FE@agendaless.com> References: <49075D83.2050100@colorstudy.com> <49095C1C.6090904@behnel.de> <4909DAD3.4020103@colorstudy.com> <490E059D.6060507@behnel.de> <52798438-2627-4411-BF2B-CFF18A60F6FE@agendaless.com> Message-ID: <490F526A.4060405@colorstudy.com> Paul Everitt wrote: >> Ian Bicking wrote: >>> Stefan Behnel wrote: >>> but I'd like it to work without buildout, and there's also several >>> buildout recipes and configurations out there and not one clear >>> canonical way to build lxml. >> >> I wouldn't mind to ship lxml with a buildout recipe. I think the current >> problem is that if we wait to find one that works well for as many >> people as >> possible, we'll wait forever. So I'm fine with adding any script that's >> somewhat tested and in use. Even a set of scripts that you can try is >> better >> than none if people can't manage to build lxml themselves. > > FWIW, we have a very lean buildout that only focuses on the problem of > building lxml. I could contribute it if you'd like. How does this compare to plone.recipe.lxml (http://pypi.python.org/pypi/plone.recipe.lxml)? I notice that package just had a release. Is this the config you've been using? https://svn.plone.org/svn/plone/PloneOrg/sandbox/xdv/new.plone.org/osx.cfg (if so, in your run-deliverance script, it'd be best to use exec to avoid the intermediate shell process and make it easier to kill the script) One problem with buildout recipes is that they aren't, AFAIK, very compatible with other systems. That is, they can build the egg, but the egg isn't on the path of any interpreter, so unless you use buildout entirely you'll have to do some further manipulation to make lxml available. I think I've also seen some reports that even using the macports lxml build, DYLD_LIBRARY_PATH can matter. I don't remember where I saw that now... maybe actually as part of another recipe? -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From stefan_ml at behnel.de Mon Nov 3 21:09:18 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 03 Nov 2008 21:09:18 +0100 Subject: [lxml-dev] [XML-SIG] Messages when installing xml In-Reply-To: <490F57D4.8020608@ncf.ca> References: <490F57D4.8020608@ncf.ca> Message-ID: <490F5A6E.5080907@behnel.de> Hi, first of all: is there any reason you are not using the latest binary packages of lxml 2.1? Also: the mailing list of lxml is a better place to ask these questions than the more general XML-SIG list. Colin J. Williams wrote: > The log of my install is below: > > Building lxml version 2.2.alpha1-59220. > Building with Cython 0.9.8.1.1. > ERROR: 'xslt-config' is not recognized as an internal or external command, > operable program or batch file. > > ** make sure the development packages of libxml2 and libxslt are installed ** You need to install libxml2 and libxslt and add the xslt-config script that ships with libxslt to your program search path when building lxml. > [...] > building 'lxml.etree' extension > C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\python25\include > -IC:\python25\PC -c src/lxml\lxml.etree.c -o > build\temp.win32-2.5\Release\src\lxml\lxml.etree.o -w It's generally recommended to build lxml statically against libxml2/libxslt on Windows. There are some build instructions on the web page. Stefan From paul at agendaless.com Tue Nov 4 15:13:22 2008 From: paul at agendaless.com (Paul Everitt) Date: Tue, 4 Nov 2008 09:13:22 -0500 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <490F526A.4060405@colorstudy.com> References: <49075D83.2050100@colorstudy.com> <49095C1C.6090904@behnel.de> <4909DAD3.4020103@colorstudy.com> <490E059D.6060507@behnel.de> <52798438-2627-4411-BF2B-CFF18A60F6FE@agendaless.com> <490F526A.4060405@colorstudy.com> Message-ID: <39E4BBD3-5FA6-4609-A727-AC6C9E792CB0@agendaless.com> On Nov 3, 2008, at 2:35 PM, Ian Bicking wrote: > Paul Everitt wrote: >>> Ian Bicking wrote: >>>> Stefan Behnel wrote: >>>> but I'd like it to work without buildout, and there's also several >>>> buildout recipes and configurations out there and not one clear >>>> canonical way to build lxml. >>> >>> I wouldn't mind to ship lxml with a buildout recipe. I think the >>> current >>> problem is that if we wait to find one that works well for as many >>> people as >>> possible, we'll wait forever. So I'm fine with adding any script >>> that's >>> somewhat tested and in use. Even a set of scripts that you can try >>> is better >>> than none if people can't manage to build lxml themselves. >> FWIW, we have a very lean buildout that only focuses on the problem >> of building lxml. I could contribute it if you'd like. > > How does this compare to plone.recipe.lxml (http://pypi.python.org/pypi/plone.recipe.lxml)? > I notice that package just had a release. Is this the config > you've been using? https://svn.plone.org/svn/plone/PloneOrg/sandbox/xdv/new.plone.org/osx.cfg > (if so, in your run-deliverance script, it'd be best to use exec to > avoid the intermediate shell process and make it easier to kill the > script) Here's the buildout.cfg that we've used: [libxml2] recipe = zc.recipe.cmmi url = http://somecustomer.agendaless.com/indexes/karl/cmmi/libxml2-2.6.32.tar.gz extra_options = --without-python [libxslt] recipe = zc.recipe.cmmi url = http://somecustomer.agendaless.com/indexes/karl/cmmi/libxslt-1.1.24.tar.gz extra_options = --with-libxml-prefix=${libxml2:location} --without-python [lxml-environment] XSLT_CONFIG=${buildout:directory}/parts/libxslt/bin/xslt-config XML2_CONFIG=${buildout:directory}/parts/libxml2/bin/xml2-config [lxml] recipe = zc.recipe.egg:custom egg = lxml include-dirs = ${libxml2:location}/include/libxml2 ${libxslt:location}/include library-dirs = ${libxml2:location}/lib ${libxslt:location}/lib rpath = ${libxml2:location}/lib ${libxslt:location}/lib environment = lxml-environment > One problem with buildout recipes is that they aren't, AFAIK, very > compatible with other systems. That is, they can build the egg, but > the egg isn't on the path of any interpreter, so unless you use > buildout entirely you'll have to do some further manipulation to > make lxml available. Right. On one hand, I'm reluctant to put all lxml's eggs in the buildout basket [wink] for that reason. lxml is much bigger than the world of Zope. Being confronted with something you didn't know (buildout) just to get to the thing you wanted (lxml) is a pain. OTOH, I wonder if the solution is to simply provide a number of options. We're really only talking about Mac users, so it isn't as big of a decision. And for them, perhaps the answer is to ensure there are a number of choices (MacPorts and fink kept up-to-date, easier to build by hand with Stefan's XML2_CONFIG support, etc.) > I think I've also seen some reports that even using the macports > lxml build, DYLD_LIBRARY_PATH can matter. I don't remember where I > saw that now... maybe actually as part of another recipe? I think I saw something in another recipe, perhaps Martin's Deliverance one. My guess is that some of these are false blips from the time before XML2_CONFIG. --Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20081104/e8faa4b8/attachment-0001.htm From ianb at colorstudy.com Tue Nov 4 23:01:24 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Nov 2008 16:01:24 -0600 Subject: [lxml-dev] setupinfo.py that downloads/builds libxml2/libxslt Message-ID: <4910C634.8020506@colorstudy.com> I've attached a version of setupinfo.py that will download and build libxml2 and libxslt. There are some problems with the script, but hopefully this is a start. If you give the build option --build-libxml2xslt then it will download the packages and build them, and set the options to use the xml2-config and xslt-config scripts. I also made it always use --static if you do this. I'm not sure if that's a good idea, but if they aren't built statically then I'm not sure where to install the libraries... though I guess they could be installed in some subdirectory of the lxml package? That would be fine, unless there were problems if those libraries were then moved around (as they are often during the installation process), it might mean that the linker won't find them...? Also adding files to the installation process means some more work. (It just occurred to me... do I have to run "make; make install" for those libraries if it is a static build, or would ./configure be sufficient?) The script will download the newest versions of these libraries, or you can give a specific version with --libxml2-version/--libxslt-version. Also BUILD_LIBXML2XSLT=true, LIBXML2_VERSION, and LIBXSLT_VERSION environmental variables work for these settings. Maybe there can also be a fallback to archives distributed directly with lxml? The tarballs are about 8Mb for the two of them. Or I suppose just some more flags to point to existing source code or tarballs. The biggest problem is that I'm not sure at all where to put this stuff in the setup.py build process. Right now the options to setup() include information about the built versions of these libraries, but simply calling "python setup.py" should not cause those libraries to be built. (Right now if you use --build-libxml2xslt it will build those libraries at setupinfo/setup.py import time.) This is the biggest problem I see with this at the moment. Lastly, I'm having problems building lxml trunk right now, with this error: > cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef readonly object localname > cdef readonly object namespace > def __init__(self, text_or_uri_or_element, tag=None): > if not _isString(text_or_uri_or_element): > if isinstance(text_or_uri_or_element, _Element): > text_or_uri_or_element = (<_Element>text_or_uri_or_element).tag > ^ > ------------------------------------------------------------ > > /home/ianb/src/lxml/src/lxml/lxml.etree.pyx:259:51: Declarator should be empty But I got this working with the last lxml release, so that's fine. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org -------------- next part -------------- A non-text attachment was scrubbed... Name: setupinfo.py Type: text/x-python Size: 16333 bytes Desc: not available Url : http://codespeak.net/pipermail/lxml-dev/attachments/20081104/567f120e/attachment.py From richard at indigo3.net Wed Nov 5 18:01:54 2008 From: richard at indigo3.net (Richard Smith) Date: Wed, 05 Nov 2008 18:01:54 +0100 Subject: [lxml-dev] lxml XSD Schema validation problem Message-ID: <4911D182.9060000@indigo3.net> Hi guys, I'm getting an error using the lxml objectification API and XSD Validation. Am using lxml and from distribution on standard hardy lib versions. Here's the boilerplate version info from the FAQ: lxml.etree: (2, 1, 2, 0) libxml used: (2, 6, 31) libxml compiled: (2, 6, 31) libxslt used: (1, 1, 22) libxslt compiled: (1, 1, 22) The code I'm testing with is as follows: ############################ >8 ############################ #!/usr/bin/python from lxml import objectify from lxml import etree SCHEMA_FILE = 'nvdcve.xsd' DATA_FILE = 'nvdcve-recent.xml' schema_file = open(SCHEMA_FILE) data_file = open(DATA_FILE) schema = etree.XMLSchema(file=schema_file) parser = objectify.makeparser(schema=schema) tree = objectify.parse(data_file, parser) ############################ 8< ############################ The XSD I'm validating against is the XSD provided by NIST for CVE posts and the XML doc is the recent NVDs: http://nvd.nist.gov/schema/nvdcve.xsd http://nvd.nist.gov/download/nvdcve-recent.xml The error I'm getting from lxml is: lxml.etree.XMLSyntaxError: Element '{http://nvd.nist.gov/feeds/cve/1.2}ref', attribute 'url': [facet 'pattern'] The value 'http://www.securitytracker.com/id?1021107' is not accepted by the pattern '(news|(ht|f)tp(s)?)://.+'. The problem is that on every implementation of regex patterns I can think of (Python, Perl, egrep etc) the regex validates... Indeed on my win32 xml editors (LiquidXML and OxygenXML) the data validates with no problems. $ egrep "(news|(ht|f)tp(s)?)://.+" nvdcve-recent.xml | wc -l 497 pcretest comes back fine too: $ pcretest PCRE version 7.4 2007-09-21 re> #(news|(ht|f)tp(s)?)://.+# data> http://www.securitytracker.com/id?1021107 0: http://www.securitytracker.com/id?1021107 1: http 2: ht Is this a libxml2 problem, my code or, dare I say, an evil bug? Thanks -- Richard From stefan_ml at behnel.de Wed Nov 5 21:32:42 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 05 Nov 2008 21:32:42 +0100 Subject: [lxml-dev] setupinfo.py that downloads/builds libxml2/libxslt In-Reply-To: <4910C634.8020506@colorstudy.com> References: <4910C634.8020506@colorstudy.com> Message-ID: <491202EA.7060001@behnel.de> Hi Ian, Ian Bicking wrote: > I've attached a version of setupinfo.py that will download and build > libxml2 and libxslt. There are some problems with the script, but > hopefully this is a start. thanks a lot for doing this. It does not yet build a static lxml for me (the static stuff was meant for Windows and it doesn't really work on other systems), but I'm fixing it up to do that. > If you give the build option --build-libxml2xslt then it will download > the packages and build them, and set the options to use the xml2-config > and xslt-config scripts. I also made it always use --static if you do > this. That's definitely the right thing to do. If you use a complete build, it's best to build statically against your libraries to make sure they are really used. > do I have to run "make; make install" for those libraries if it is > a static build, or would ./configure be sufficient?) "make" is sufficient, no installation required. libxslt even takes a "--with-libxml-src=DIR" option to build against an uninstalled libxml2. > Maybe there can also be a fallback to archives distributed directly with lxml? Yes, that would be great. If they are there, use them, otherwise, do a normal build. > The biggest problem is that I'm not sure at all where to put this stuff > in the setup.py build process. Right now the options to setup() include > information about the built versions of these libraries, but simply > calling "python setup.py" should not cause those libraries to be built. > (Right now if you use --build-libxml2xslt it will build those libraries > at setupinfo/setup.py import time.) This is the biggest problem I see > with this at the moment. I think an option for the download and a check if the library archives lie around somewhere to trigger the build is best. > Lastly, I'm having problems building lxml trunk right now, with this error: > >> cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> cdef readonly object localname >> cdef readonly object namespace >> def __init__(self, text_or_uri_or_element, tag=None): >> if not _isString(text_or_uri_or_element): >> if isinstance(text_or_uri_or_element, _Element): >> text_or_uri_or_element = >> (<_Element>text_or_uri_or_element).tag >> ^ >> ------------------------------------------------------------ >> >> /home/ianb/src/lxml/src/lxml/lxml.etree.pyx:259:51: Declarator should >> be empty Looks like your Cython is somehow broken. It works for me here. Thanks again, Stefan From jholg at gmx.de Thu Nov 6 15:15:58 2008 From: jholg at gmx.de (jholg at gmx.de) Date: Thu, 06 Nov 2008 15:15:58 +0100 Subject: [lxml-dev] lxml XSD Schema validation problem In-Reply-To: <4911D182.9060000@indigo3.net> References: <4911D182.9060000@indigo3.net> Message-ID: <20081106142202.159680@gmx.net> Hi, > The XSD I'm validating against is the XSD provided by NIST for CVE posts > and > the XML doc is the recent NVDs: > > http://nvd.nist.gov/schema/nvdcve.xsd > http://nvd.nist.gov/download/nvdcve-recent.xml > > The error I'm getting from lxml is: > > lxml.etree.XMLSyntaxError: Element > '{http://nvd.nist.gov/feeds/cve/1.2}ref', > attribute 'url': [facet 'pattern'] The value > 'http://www.securitytracker.com/id?1021107' is not accepted by the > pattern > '(news|(ht|f)tp(s)?)://.+'. > ?Using xmllint I also run into a whole bunch of these errors: ?nvdcve-recent.xml:18: element ref: Schemas validity error : Element '{http://nvd.nist.gov/feeds/cve/1.2}ref', attribute 'url': [facet 'pattern'] The value 'http://xforce.iss.net/xforce/xfdb/40864' is not accepted by the pattern '(news|(ht|f)tp(s)?)://.+'. ?$ xmllint --version xmllint: using libxml version 20627 ?Validates ok in oXygen for me.? ? > Is this a libxml2 problem, my code or, dare I say, an evil bug? ?libxml2, I'd say. ?Holger? -- "Feel free" - 5 GB Mailbox, 50 FreeSMS/Monat ... Jetzt GMX ProMail testen: http://www.gmx.net/de/go/promail -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20081106/817bbf2f/attachment-0001.htm From richard at indigo3.net Thu Nov 6 21:21:00 2008 From: richard at indigo3.net (Richard Smith) Date: Thu, 06 Nov 2008 21:21:00 +0100 Subject: [lxml-dev] lxml XSD Schema validation problem In-Reply-To: <4911D182.9060000@indigo3.net> References: <4911D182.9060000@indigo3.net> Message-ID: <491351AC.3050707@indigo3.net> Richard Smith wrote: > Is this a libxml2 problem, my code or, dare I say, an evil bug? Just an update... I've worked out this is a libxml2 problem so I'll bug them. Thanks -- Richard From stefan_ml at behnel.de Thu Nov 6 22:32:11 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 06 Nov 2008 22:32:11 +0100 Subject: [lxml-dev] setupinfo.py that downloads/builds libxml2/libxslt In-Reply-To: <491202EA.7060001@behnel.de> References: <4910C634.8020506@colorstudy.com> <491202EA.7060001@behnel.de> Message-ID: <4913625B.3030900@behnel.de> Hi, Stefan Behnel wrote: > Ian Bicking wrote: >> I've attached a version of setupinfo.py that will download and build >> libxml2 and libxslt. There are some problems with the script, but >> hopefully this is a start. > > thanks a lot for doing this. It does not yet build a static lxml for me (the > static stuff was meant for Windows and it doesn't really work on other > systems), but I'm fixing it up to do that. Ok, I reworked your (pretty complete and well done) implementation and renamed the option to "--static-deps" to make it clear that it triggers a static build (the name is still open to improvement). It's on the trunk now. Everyone who can give it a try: please install a recent Cython and do so! Fixes and bug reports are very welcome. Since the main incentive for this change was to make it work under MacOS-X, here's a big plea for Mac users to test this and suggest fixes! I enabled the static build for other platforms than Windows, so I hope that it still works under Windows. Another thing to test. >> do I have to run "make; make install" for those libraries if it is >> a static build, or would ./configure be sufficient?) > > "make" is sufficient, no installation required. libxslt even takes a > "--with-libxml-src=DIR" option to build against an uninstalled libxml2. I still left the "make install" in, as it makes the integration with the rest of the build process easier, which requires a working "xslt-config". That could potentially be stripped out. >> Maybe there can also be a fallback to archives distributed directly with lxml? > > Yes, that would be great. If they are there, use them, otherwise, do a normal > build. I changed the download directory to "libs" instead of "build/tmp" to allow users to put their libraries there. However, this does not currently trigger a static build without the "--static-deps" option. I'd like to see how people use this first before enabling even more smartness. Stefan From paul at agendaless.com Fri Nov 7 16:06:29 2008 From: paul at agendaless.com (Paul Everitt) Date: Fri, 7 Nov 2008 10:06:29 -0500 Subject: [lxml-dev] setupinfo.py that downloads/builds libxml2/libxslt In-Reply-To: <4913625B.3030900@behnel.de> References: <4910C634.8020506@colorstudy.com> <491202EA.7060001@behnel.de> <4913625B.3030900@behnel.de> Message-ID: On Nov 6, 2008, at 4:32 PM, Stefan Behnel wrote: > Hi, > > Stefan Behnel wrote: >> Ian Bicking wrote: >>> I've attached a version of setupinfo.py that will download and build >>> libxml2 and libxslt. There are some problems with the script, but >>> hopefully this is a start. >> >> thanks a lot for doing this. It does not yet build a static lxml >> for me (the >> static stuff was meant for Windows and it doesn't really work on >> other >> systems), but I'm fixing it up to do that. > > Ok, I reworked your (pretty complete and well done) implementation > and renamed > the option to "--static-deps" to make it clear that it triggers a > static build > (the name is still open to improvement). It's on the trunk now. > Everyone who > can give it a try: please install a recent Cython and do so! Fixes > and bug > reports are very welcome. > > Since the main incentive for this change was to make it work under > MacOS-X, > here's a big plea for Mac users to test this and suggest fixes! First, thanks Stefan and Ian for working on this. I really hope it works out. I just gave it a try with: MacIntel, OS X 10.5.5, Python 2.4.4 in a virtualenv. This resulted, after lots of downloading and compiling, with: cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef readonly object localname cdef readonly object namespace def __init__(self, text_or_uri_or_element, tag=None): if not _isString(text_or_uri_or_element): if isinstance(text_or_uri_or_element, _Element): text_or_uri_or_element = (<_Element>text_or_uri_or_element).tag ^ ------------------------------------------------------------ This is with Cython-0.9.8.1.1-py2.4-macosx-10.3-i386.egg --Paul From ianb at colorstudy.com Fri Nov 7 16:11:59 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 07 Nov 2008 09:11:59 -0600 Subject: [lxml-dev] setupinfo.py that downloads/builds libxml2/libxslt In-Reply-To: References: <4910C634.8020506@colorstudy.com> <491202EA.7060001@behnel.de> <4913625B.3030900@behnel.de> Message-ID: <49145ABF.9000508@colorstudy.com> Paul Everitt wrote: > This resulted, after lots of downloading and compiling, with: > > cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef readonly object localname > cdef readonly object namespace > def __init__(self, text_or_uri_or_element, tag=None): > if not _isString(text_or_uri_or_element): > if isinstance(text_or_uri_or_element, _Element): > text_or_uri_or_element = > (<_Element>text_or_uri_or_element).tag > ^ > ------------------------------------------------------------ > > This is with Cython-0.9.8.1.1-py2.4-macosx-10.3-i386.egg That's the same error I got. I wonder if it's a Python 2.4 issue? -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From stefan_ml at behnel.de Fri Nov 7 16:37:45 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 07 Nov 2008 16:37:45 +0100 Subject: [lxml-dev] setupinfo.py that downloads/builds libxml2/libxslt In-Reply-To: References: <4910C634.8020506@colorstudy.com> <491202EA.7060001@behnel.de> <4913625B.3030900@behnel.de> Message-ID: <491460C9.5050606@behnel.de> Hi, Paul Everitt wrote: > This resulted, after lots of downloading and compiling, with: > > cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef readonly object localname > cdef readonly object namespace > def __init__(self, text_or_uri_or_element, tag=None): > if not _isString(text_or_uri_or_element): > if isinstance(text_or_uri_or_element, _Element): > text_or_uri_or_element = > (<_Element>text_or_uri_or_element).tag > ^ > ------------------------------------------------------------ > > This is with Cython-0.9.8.1.1-py2.4-macosx-10.3-i386.egg Ah, I see what the problem is. I added that early cast to _Element recently, but didn't forward-declare the class before use. It looks like the latest Cython (0.10rc) handles this different that what you are using. Should be fixed on the trunk now. BTW, I'll switch to requiring Cython 0.10 for the trunk soon for a couple of other reasons, but it's still better to fix this now. Stefan From paul at agendaless.com Fri Nov 7 16:51:30 2008 From: paul at agendaless.com (Paul Everitt) Date: Fri, 7 Nov 2008 10:51:30 -0500 Subject: [lxml-dev] setupinfo.py that downloads/builds libxml2/libxslt In-Reply-To: <491460C9.5050606@behnel.de> References: <4910C634.8020506@colorstudy.com> <491202EA.7060001@behnel.de> <4913625B.3030900@behnel.de> <491460C9.5050606@behnel.de> Message-ID: <4B1724AF-1A39-4FB0-A022-FD52A1E61CB6@agendaless.com> On Nov 7, 2008, at 10:37 AM, Stefan Behnel wrote: > Hi, > > Paul Everitt wrote: >> This resulted, after lots of downloading and compiling, with: >> >> cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> cdef readonly object localname >> cdef readonly object namespace >> def __init__(self, text_or_uri_or_element, tag=None): >> if not _isString(text_or_uri_or_element): >> if isinstance(text_or_uri_or_element, _Element): >> text_or_uri_or_element = >> (<_Element>text_or_uri_or_element).tag >> ^ >> ------------------------------------------------------------ >> >> This is with Cython-0.9.8.1.1-py2.4-macosx-10.3-i386.egg > > Ah, I see what the problem is. I added that early cast to _Element > recently, > but didn't forward-declare the class before use. It looks like the > latest > Cython (0.10rc) handles this different that what you are using. > > Should be fixed on the trunk now. > > BTW, I'll switch to requiring Cython 0.10 for the trunk soon for a > couple of > other reasons, but it's still better to fix this now. Yep, that fixed it, for both Python 2.4 and 2.5...at least, for building. Running was a different story: >>> from lxml import etree Traceback (most recent call last): File "", line 1, in ImportError: dlopen(/Users/paul/venvs/lxml2.5.2/lib/python2.5/site- packages/lxml-2.2alpha1-py2.5-macosx-10.3-i386.egg/lxml/etree.so, 2): Symbol not found: _iconv Referenced from: /Users/paul/venvs/lxml2.5.2/lib/python2.5/site- packages/lxml-2.2alpha1-py2.5-macosx-10.3-i386.egg/lxml/etree.so Expected in: dynamic lookup --Paul From stefan_ml at behnel.de Fri Nov 7 16:55:23 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 07 Nov 2008 16:55:23 +0100 Subject: [lxml-dev] setupinfo.py that downloads/builds libxml2/libxslt In-Reply-To: <4B1724AF-1A39-4FB0-A022-FD52A1E61CB6@agendaless.com> References: <4910C634.8020506@colorstudy.com> <491202EA.7060001@behnel.de> <4913625B.3030900@behnel.de> <491460C9.5050606@behnel.de> <4B1724AF-1A39-4FB0-A022-FD52A1E61CB6@agendaless.com> Message-ID: <491464EB.8000807@behnel.de> Hi, Paul Everitt wrote: > Running was a different story: > >>>> from lxml import etree > Traceback (most recent call last): > File "", line 1, in > ImportError: > dlopen(/Users/paul/venvs/lxml2.5.2/lib/python2.5/site-packages/lxml-2.2alpha1-py2.5-macosx-10.3-i386.egg/lxml/etree.so, > 2): Symbol not found: _iconv In case the library is called 'iconv' on Macs, this *might* be fixed now. Stefan From stefan_ml at behnel.de Sun Nov 9 16:38:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Nov 2008 16:38:27 +0100 Subject: [lxml-dev] NotImplementedError for external functions with xsl:variable's In-Reply-To: <49084477.10102@gmail.com> References: <49084477.10102@gmail.com> Message-ID: <491703F3.3000404@behnel.de> Hi Dmitri, Dmitri Fedoruk wrote: > We have lxml 2.0.1 in production and 2.1 on the development machine. The > problem occurs, but in slightly different manner, in both situations. > 2.0.1 just performs nothing and returns an empty string; 2.1 raises > [...] > File "extensions.pxi", line 665, in > lxml.etree._extension_function_call (src/lxml/lxml.etree.c:82995) > File "extensions.pxi", line 513, in lxml.etree._unwrapXPathObject > (src/lxml/lxml.etree.c:81828) > NotImplementedError I was right in recalling that I had started working on this before, but according to the partial patch in my working directory, I never got around to finishing it up. I assume that your real stylesheet creates a result tree fragment for the variable that you pass into your function. Until now, lxml didn't handle result trees at all. I'm not exactly sure what a result tree fragment looks like in libxslt in all possible cases, but what I implemented on the trunk so far works for me. Please give it a try (after installing Cython 0.10). It would be nice if you could come up with a couple of additional test cases for test_xslt.py to make sure this works (and continues to work) as expected for different variable values or scenarios. Currently, there is only test_variable_result_tree_fragment(), which tests for result trees returned by . Have fun, Stefan From chris at simplistix.co.uk Sun Nov 9 20:35:56 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 09 Nov 2008 19:35:56 +0000 Subject: [lxml-dev] lxml Mac installation idea Message-ID: <49173B9C.6000308@simplistix.co.uk> Paul wrote: >>>>> Stefan Behnel wrote: >>>>> but I'd like it to work without buildout, and there's also several >>>>> buildout recipes and configurations out there and not one clear >>>>> canonical way to build lxml. Out of interest, what's the canonical way to build lxml when you're developing it? What's the canonical way to run the unit tests? I seem to be moving more and more to using buildout for each of the packages I maintain as it provides a simple answer to both these questions. > Here's the buildout.cfg that we've used: > > [libxml2] > recipe = zc.recipe.cmmi > url = http://somecustomer.agendaless.com/indexes/karl/cmmi/libxml2-2.6.32.tar.gz > extra_options = --without-python > > [libxslt] > recipe = zc.recipe.cmmi > url = http://somecustomer.agendaless.com/indexes/karl/cmmi/libxslt-1.1.24.tar.gz > extra_options = --with-libxml-prefix=${libxml2:location} > --without-python > > [lxml-environment] > XSLT_CONFIG=${buildout:directory}/parts/libxslt/bin/xslt-config > XML2_CONFIG=${buildout:directory}/parts/libxml2/bin/xml2-config > > [lxml] > recipe = zc.recipe.egg:custom > egg = lxml > include-dirs = ${libxml2:location}/include/libxml2 > ${libxslt:location}/include > library-dirs = ${libxml2:location}/lib > ${libxslt:location}/lib > rpath = ${libxml2:location}/lib > ${libxslt:location}/lib > environment = lxml-environment This looks quite similar to the buildout Martijn originally came up with back in 2005. Is it related to that? >> One problem with buildout recipes is that they aren't, AFAIK, very >> compatible with other systems. That is, they can build the egg, but >> the egg isn't on the path of any interpreter, so unless you use >> buildout entirely you'll have to do some further manipulation to >> make lxml available. Well, that's true, but surely lxml should or is ending up as a binary egg or few up on PyPI? http://pypi.python.org/pypi/lxml/2.1.2 ...would seem to suggest that this is the case. In which case, I guess buildout would help for building libxml2 and libxslt on environments other than Windows, but other than that? > Right. On one hand, I'm reluctant to put all lxml's eggs in the > buildout basket [wink] for that reason. ...I honestly don't think there's anything better in the python world. It's just a shame buildout's docs suck so much. Sometimes doctests have a *lot* to answer for! > lxml is much bigger than the > world of Zope. So is buildout ;-) > Being confronted with something you didn't know > (buildout) just to get to the thing you wanted (lxml) is a pain. Yes, I agree that simply for using lxml that shouldn't be the case. But, what about developing lxml? And what about if your project is using buildout already? > OTOH, I wonder if the solution is to simply provide a number of > options. We're really only talking about Mac users, so it isn't as > big of a decision. What's special about Mac users? (sorry for being dumb...) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Sun Nov 9 21:37:43 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 09 Nov 2008 20:37:43 +0000 Subject: [lxml-dev] specifying lxml as a dependency? Message-ID: <49174A17.4080607@simplistix.co.uk> Hi All, I'm just about to start moving Twiddler from ElementTree to lxml and I'm wondering how to specify lxml as a requirement in Twiddler's setup.py? Do I just do the normal setuptools dance or is there more to it than that? If I specify lxml as a requirement, how do lxml's requirements for libxml2 and libxlst get satisfied? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From ianb at colorstudy.com Sun Nov 9 21:49:42 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 09 Nov 2008 14:49:42 -0600 Subject: [lxml-dev] specifying lxml as a dependency? In-Reply-To: <49174A17.4080607@simplistix.co.uk> References: <49174A17.4080607@simplistix.co.uk> Message-ID: <49174CE6.60105@colorstudy.com> Chris Withers wrote: > Hi All, > > I'm just about to start moving Twiddler from ElementTree to lxml and I'm > wondering how to specify lxml as a requirement in Twiddler's setup.py? > > Do I just do the normal setuptools dance or is there more to it than that? > > If I specify lxml as a requirement, how do lxml's requirements for > libxml2 and libxlst get satisfied? Just as 'lxml'. On Windows it'll pull the binary egg. Maybe a Mac egg is doable now? I think there are still some difficulties to moving libraries around. Linux binary eggs are too problematic to be worth trying. The system will have to have libxml2/libxslt and the -dev packages (on Linux), and they could do "STATIC_DEPS=true easy_install Twiddler" to build a static lxml on Macs (with the trunk, which itself is a little harder to build, but hopefully there will be an easier release before too long). -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From stefan_ml at behnel.de Sun Nov 9 22:27:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Nov 2008 22:27:02 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <49173B9C.6000308@simplistix.co.uk> References: <49173B9C.6000308@simplistix.co.uk> Message-ID: <491755A6.6020809@behnel.de> Hi, Chris Withers wrote: > Paul wrote: >>>>>> Stefan Behnel wrote: >>>>>> but I'd like it to work without buildout, and there's also several >>>>>> buildout recipes and configurations out there and not one clear >>>>>> canonical way to build lxml. > > Out of interest, what's the canonical way to build lxml when you're > developing it? What's the canonical way to run the unit tests? Not sure I do it the "canonical" way, but I usually run "make test", sometimes passing PYTHON=..., sometimes setting XML2_CONFIG=... and XSLT_CONFIG=... > This looks quite similar to the buildout Martijn originally came up with > back in 2005. Is it related to that? I don't think there's so many things you could do different, even if you wrote it from scratch. >>> One problem with buildout recipes is that they aren't, AFAIK, very >>> compatible with other systems. That is, they can build the egg, but >>> the egg isn't on the path of any interpreter, so unless you use >>> buildout entirely you'll have to do some further manipulation to >>> make lxml available. > > Well, that's true, but surely lxml should or is ending up as a binary > egg or few up on PyPI? We have binary eggs for Windows, built statically for each release by Sidnei da Silva, as we figured that Windows users will have a hard time building lxml without a C compiler installed. We don't have a contributor for MacOS binaries. I don't see any reason to provide binaries for Linux, as lxml builds nicely with easy_install on almost any distribution. Holger tends to test the build on old Solaris systems from time to time. That's the systems I'm aware of. > ...would seem to suggest that this is the case. In which case, I guess > buildout would help for building libxml2 and libxslt on environments > other than Windows, but other than that? I happily accept binary builds for any system where users have problems in getting lxml built automatically. If buildout helps here, fine with me. >> OTOH, I wonder if the solution is to simply provide a number of >> options. We're really only talking about Mac users, so it isn't as >> big of a decision. > > What's special about Mac users? (sorry for being dumb...) The users are not the problem. The system itself is, and the (lack of) updates that Apple provides, which is only partly compensated by different package distributions that, instead of updating the system, install things separately but within reach. Stefan From mike at it-loops.com Sun Nov 9 22:40:17 2008 From: mike at it-loops.com (Guntsche Michael) Date: Sun, 9 Nov 2008 22:40:17 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <491755A6.6020809@behnel.de> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> Message-ID: <58E51230-844E-4379-A18B-92D0F0BC49C2@it-loops.com> On Nov 9, 2008, at 22:27, Stefan Behnel wrote: > The users are not the problem. The system itself is, and the (lack > of) updates > that Apple provides, which is only partly compensated by different > package > distributions that, instead of updating the system, install things > separately > but within reach. The reason for this is that nasty things may happen if you update the system internal files. The main idea is just to NOT do that ever. On the other hand I do not think that it will be a problem for the xml/xslt libs. But if a security fix for the packages comes out your files will be overwritten again, maybe without you noticing it. So best practice is to just leave the system internal files and install updated files separately. I switched from fink to macports and never had a problem with my lxml installations at all. I just make sure that xml-config and xslt-config are taken from the correct location. Well there is one thing though. Since the system python version AND the python version from python.org is build as Universal Binary I had to compile both libraries as Universal Binaries as well, but this is not a big problem with macports. Kind regards, Michael From stefan_ml at behnel.de Sun Nov 9 22:52:47 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Nov 2008 22:52:47 +0100 Subject: [lxml-dev] specifying lxml as a dependency? In-Reply-To: <49174CE6.60105@colorstudy.com> References: <49174A17.4080607@simplistix.co.uk> <49174CE6.60105@colorstudy.com> Message-ID: <49175BAF.2060802@behnel.de> Hi, Ian Bicking wrote: > Just as 'lxml'. On Windows it'll pull the binary egg. Maybe a Mac egg > is doable now? [...] > they could do "STATIC_DEPS=true easy_install > Twiddler" to build a static lxml on Macs we're not there yet. Paul tried on his Mac, which uses macports. The current trunk builds libxml2 and libxslt nicely and then builds lxml statically with them, but next the module fails to import due to problems with missing symbols in libiconv. My guess is that this is a problem with macports, but that needs further investigation. I keep being impressed by how broken a platform MacOS really is, and how little a package distribution can do to fix it that lives outside of the system itself. Maybe other Mac users who do not use macports can report their experience with the current trunk? > (with the trunk, which itself > is a little harder to build, but hopefully there will be an easier > release before too long). I was planning to release at least a 2.1.3 weeks ago, but there are still some open problems and I need to find the time to work on them. But I think the current trunk is good enough to become a real 2.2alpha1 soon. Stefan From stefan_ml at behnel.de Sun Nov 9 22:59:07 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Nov 2008 22:59:07 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <58E51230-844E-4379-A18B-92D0F0BC49C2@it-loops.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <58E51230-844E-4379-A18B-92D0F0BC49C2@it-loops.com> Message-ID: <49175D2B.6020907@behnel.de> Hi, Guntsche Michael wrote: > On Nov 9, 2008, at 22:27, Stefan Behnel wrote: > >> The users are not the problem. The system itself is, and the (lack >> of) updates >> that Apple provides, which is only partly compensated by different >> package >> distributions that, instead of updating the system, install things >> separately but within reach. > > The reason for this is that nasty things may happen if you update the > system internal files. > The main idea is just to NOT do that ever. On the other hand I do not > think that it will be a problem for the xml/xslt libs. But if a > security fix for the packages comes out your files will be overwritten > again, maybe without you noticing it. So best practice is to just > leave the system internal files and install updated files separately. On a normal system, that shouldn't pose a problem at all. But if lxml.etree reports that the runtime linker linked it with the correct, up-to-date library, and then it crashes due to a problem in the outdated system library, which should not even have been loaded, I just can't call that the normal behaviour of an "operating" system. > I switched from fink to macports and never had a problem with my lxml > installations at all. I just make sure that xml-config and xslt-config > are taken from the correct location. So you mean that you are building lxml manually and you do not use the macports build, right? Is there anything you have to do at runtime, or does it "just work" for you once it's built and installed? > Well there is one thing though. Since the system python version AND > the python version from python.org is build as Universal Binary I had > to compile both libraries as Universal Binaries as well, but this is > not a big problem with macports. That sounds perfectly reasonable to me. Stefan From mike at it-loops.com Mon Nov 10 07:41:22 2008 From: mike at it-loops.com (Michael Guntsche) Date: Mon, 10 Nov 2008 07:41:22 +0100 Subject: [lxml-dev] Fwd: Re: lxml Mac installation idea Message-ID: CC'ing lxml-dev -------- Original Message -------- Subject: Re: [lxml-dev] lxml Mac installation idea Date: Sun, 9 Nov 2008 23:12:32 +0100 From: Guntsche Michael To: Stefan Behnel On Nov 9, 2008, at 22:59, Stefan Behnel wrote: > Hi, > > > On a normal system, that shouldn't pose a problem at all. But if > lxml.etree > reports that the runtime linker linked it with the correct, up-to-date > library, and then it crashes due to a problem in the outdated system > library, > which should not even have been loaded, I just can't call that the > normal > behaviour of an "operating" system. > I never said that MacosX is normal. :) It is really botched in some places, especially in the library department. > So you mean that you are building lxml manually and you do not use the > macports build, right? Is there anything you have to do at runtime, > or does it > "just work" for you once it's built and installed? Exactly. I just made sure that every related library that comes from macports, in this case libxml, libexslt and libiconv are build as universal binary. The latest builds I have are running through a "make test" without any problems. They even survive the test-suite you posted a few months ago. I still do not understand why the linker picks up the internal libraries during runtime. Maybe if someone with a "broken" setup could give me more information about it (universal binaries, self-compiled from macports etc..) I can figure out what's going on. Kind regards, Michael From chris at simplistix.co.uk Mon Nov 10 09:43:43 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 10 Nov 2008 08:43:43 +0000 Subject: [lxml-dev] specifying lxml as a dependency? In-Reply-To: <49174CE6.60105@colorstudy.com> References: <49174A17.4080607@simplistix.co.uk> <49174CE6.60105@colorstudy.com> Message-ID: <4917F43F.30307@simplistix.co.uk> Ian Bicking wrote: > Chris Withers wrote: >> Hi All, >> >> I'm just about to start moving Twiddler from ElementTree to lxml and >> I'm wondering how to specify lxml as a requirement in Twiddler's >> setup.py? >> >> Do I just do the normal setuptools dance or is there more to it than >> that? >> >> If I specify lxml as a requirement, how do lxml's requirements for >> libxml2 and libxlst get satisfied? > > Just as 'lxml'. On Windows it'll pull the binary egg. ...and that binary egg includes libxml2 and libxlst? > Maybe a Mac egg > is doable now? I think there are still some difficulties to moving > libraries around. I haven't followed the lxml Mac story at all, so this is all a little lost on me... > Linux binary eggs are too problematic to be worth > trying. The system will have to have libxml2/libxslt and the -dev > packages (on Linux), Okay, so the user will have to get libxml2 and libxslt on the system "somehow" (ie: os package manager, source install, etc) *before* easy_install'ing lxml? What happens if they forget and easy_install lxml without libxml2 or libxslt? > and they could do "STATIC_DEPS=true easy_install > Twiddler" to build a static lxml on Macs Where would "STATIC_DEPS=true easy_install Twiddler" go? > (with the trunk, which itself > is a little harder to build, but hopefully there will be an easier > release before too long). Not sure what this means. What's the difference between the trunk and the last release on PyPI? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Mon Nov 10 09:48:23 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 10 Nov 2008 08:48:23 +0000 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <491755A6.6020809@behnel.de> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> Message-ID: <4917F557.1010605@simplistix.co.uk> Stefan Behnel wrote: >> Out of interest, what's the canonical way to build lxml when you're >> developing it? What's the canonical way to run the unit tests? > > Not sure I do it the "canonical" way, but I usually run "make test", What does "make test" do? > sometimes > passing PYTHON=..., sometimes setting XML2_CONFIG=... and XSLT_CONFIG=... This is to test with different versions of python, libxml2 and libxslt, right? > We have binary eggs for Windows, built statically for each release by Sidnei > da Silva, as we figured that Windows users will have a hard time building lxml > without a C compiler installed. *nods* > We don't have a contributor for MacOS binaries. Well, I have Macs, if there's a recipe to follow and no-one else is prepared to do it, I'd be more than happy to do so... > I don't see any reason to provide binaries for Linux, as lxml builds nicely > with easy_install on almost any distribution. ...provided you have libxml2 and libxslt along with their dev headers available, right? > I happily accept binary builds for any system where users have problems in > getting lxml built automatically. If buildout helps here, fine with me. Nope, I haven't heard anything that suggests buildout would help for projects that aren't already using buildout... It *is* a shame though that while, for Windows users, they just need to do "easy_install lxml", linux users have to make sure a bunch of packages are installed first :-/ > The users are not the problem. The system itself is, and the (lack of) updates > that Apple provides, which is only partly compensated by different package > distributions that, instead of updating the system, install things separately > but within reach. Ah :-( Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From stefan_ml at behnel.de Mon Nov 10 10:25:33 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 10 Nov 2008 10:25:33 +0100 (CET) Subject: [lxml-dev] specifying lxml as a dependency? In-Reply-To: <4917F43F.30307@simplistix.co.uk> References: <49174A17.4080607@simplistix.co.uk> <49174CE6.60105@colorstudy.com> <4917F43F.30307@simplistix.co.uk> Message-ID: <52065.213.61.181.86.1226309133.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Chris Withers wrote: > Ian Bicking wrote: >> Chris Withers wrote: >>> I'm just about to start moving Twiddler from ElementTree to lxml and >>> I'm wondering how to specify lxml as a requirement in Twiddler's >>> setup.py? >>> >>> Do I just do the normal setuptools dance or is there more to it than >>> that? >>> >>> If I specify lxml as a requirement, how do lxml's requirements for >>> libxml2 and libxlst get satisfied? >> >> Just as 'lxml'. On Windows it'll pull the binary egg. > > ...and that binary egg includes libxml2 and libxlst? As it says on PyPI and our web site, yes. https://codespeak.net/lxml/installation.html#ms-windows >> Linux binary eggs are too problematic to be worth >> trying. The system will have to have libxml2/libxslt and the -dev >> packages (on Linux), > > Okay, so the user will have to get libxml2 and libxslt on the system > "somehow" (ie: os package manager, source install, etc) *before* > easy_install'ing lxml? I wouldn't know any distribution that doesn't install libxml2/libxslt by default, and for most Linux users, installing the -dev packages for a library is not much harder than eating a warm pie (assuming you like the taste). https://codespeak.net/lxml/installation.html#installation (I just clarified to bit on MS Windows here, haven't uploaded it yet.) > What happens if they forget and easy_install lxml without libxml2 or > libxslt? You will get an error. AFAIR, it's even pretty specific about what's missing. >> and they could do "STATIC_DEPS=true easy_install >> Twiddler" to build a static lxml on Macs > > Where would "STATIC_DEPS=true easy_install Twiddler" go? That's the command that will install Twiddler on a Mac once Twiddler has a dependency on lxml and is uploaded to PyPI. >> (with the trunk, which itself >> is a little harder to build, but hopefully there will be an easier >> release before too long). > > Not sure what this means. What's the difference between the trunk and > the last release on PyPI? Well, development continues, so there are quite some internal differences. What's relevant here is that the setup.py in the current trunk knows how to download and build libxml2 and libxslt, and how to build lxml statically using the result. That's specifically made (but not only meant) for Mac users that run into crashes due to conflicting library versions. Stefan From stefan_ml at behnel.de Mon Nov 10 11:04:43 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 10 Nov 2008 11:04:43 +0100 (CET) Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <4917F557.1010605@simplistix.co.uk> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> Message-ID: <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Chris Withers wrote: > Stefan Behnel wrote: >>> Out of interest, what's the canonical way to build lxml when you're >>> developing it? What's the canonical way to run the unit tests? >> >> Not sure I do it the "canonical" way, but I usually run "make test", > > What does "make test" do? You asked for a way to run the unit tests, that's what "make test" does. You can find the respective commands in the Makefile, or read the web page. https://codespeak.net/lxml/build.html#running-the-tests-and-reporting-errors >> sometimes >> passing PYTHON=..., sometimes setting XML2_CONFIG=... and >> XSLT_CONFIG=... > > This is to test with different versions of python, libxml2 and libxslt, > right? Yes. If you want to make sure it takes the right library at runtime, call setup.py with --auto-rpath on Linux. >> We don't have a contributor for MacOS binaries. > > Well, I have Macs, if there's a recipe to follow and no-one else is > prepared to do it, I'd be more than happy to do so... As I said, this would be very much appreciated. The first thing I would ask you to do is check out the current trunk from SVN https://codespeak.net/lxml/build.html#subversion make sure you have an Internet connection to download libxml2/libxslt, and then run python setup.py build_ext -i --static-deps to download and build the libs and then statically build lxml with them. This will copy the modules into their package at the end, so you should be able to run python test.py -vv afterwards to see if it works as expected. Please report any errors you get on the way. >> I happily accept binary builds for any system where users have problems >> in >> getting lxml built automatically. If buildout helps here, fine with me. > > Nope, I haven't heard anything that suggests buildout would help for > projects that aren't already using buildout... You could use buildout to build a static egg, though. Anyway, if we get the built-in "--static-deps" way to work reliably, that's much better than a buildout. > It *is* a shame though that while, for Windows users, they just need to > do "easy_install lxml", linux users have to make sure a bunch of > packages are installed first :-/ I partly agree, although it's not so many packages at all (2 to be exact, or 3 if your Python comes with a -dev package also), and installing them once will usually instruct your system to keep them up-to-date for you. Windows is a mostly uniform target platform (e.g. x86 only) which lacks support for very basic things like package management and software compilation, but also (mostly) lacks a standard directory layout - you can basically install your software wherever you like. So, finding things that are already installed is actually hard on Windows, as is keeping the system updated (at least for things that do not come from MS). That makes a static, complete installer the best choice. It's even better to install dependencies 100 times in parallel, than to risk conflicts, or even rely on them being up-to-date. GNU/Linux is a very diverse but well standardised target platform (LSB) with very good support for package management, online software distribution, and software building. So it's much easier to target that platform with a source release than with a binary build. Stefan From tseaver at palladion.com Mon Nov 10 17:13:59 2008 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 10 Nov 2008 11:13:59 -0500 Subject: [lxml-dev] specifying lxml as a dependency? In-Reply-To: <52065.213.61.181.86.1226309133.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49174A17.4080607@simplistix.co.uk> <49174CE6.60105@colorstudy.com> <4917F43F.30307@simplistix.co.uk> <52065.213.61.181.86.1226309133.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49185DC7.3050108@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Stefan Behnel wrote: > Chris Withers wrote: >> Ian Bicking wrote: >>> Chris Withers wrote: >>>> I'm just about to start moving Twiddler from ElementTree to lxml and >>>> I'm wondering how to specify lxml as a requirement in Twiddler's >>>> setup.py? >>>> >>>> Do I just do the normal setuptools dance or is there more to it than >>>> that? >>>> >>>> If I specify lxml as a requirement, how do lxml's requirements for >>>> libxml2 and libxlst get satisfied? >>> Just as 'lxml'. On Windows it'll pull the binary egg. >> ...and that binary egg includes libxml2 and libxlst? > > As it says on PyPI and our web site, yes. > > https://codespeak.net/lxml/installation.html#ms-windows > > >>> Linux binary eggs are too problematic to be worth >>> trying. The system will have to have libxml2/libxslt and the -dev >>> packages (on Linux), >> Okay, so the user will have to get libxml2 and libxslt on the system >> "somehow" (ie: os package manager, source install, etc) *before* >> easy_install'ing lxml? > > I wouldn't know any distribution that doesn't install libxml2/libxslt by > default, and for most Linux users, installing the -dev packages for a > library is not much harder than eating a warm pie (assuming you like the > taste). > > https://codespeak.net/lxml/installation.html#installation > > (I just clarified to bit on MS Windows here, haven't uploaded it yet.) > > >> What happens if they forget and easy_install lxml without libxml2 or >> libxslt? > > You will get an error. AFAIR, it's even pretty specific about what's missing. > > >>> and they could do "STATIC_DEPS=true easy_install >>> Twiddler" to build a static lxml on Macs >> Where would "STATIC_DEPS=true easy_install Twiddler" go? > > That's the command that will install Twiddler on a Mac once Twiddler has a > dependency on lxml and is uploaded to PyPI. > > >>> (with the trunk, which itself >>> is a little harder to build, but hopefully there will be an easier >>> release before too long). >> Not sure what this means. What's the difference between the trunk and >> the last release on PyPI? > > Well, development continues, so there are quite some internal differences. > What's relevant here is that the setup.py in the current trunk knows how > to download and build libxml2 and libxslt, and how to build lxml > statically using the result. That's specifically made (but not only meant) > for Mac users that run into crashes due to conflicting library versions. It should also help users running on older Linux distrox (e.g., Centos4) whose system-installed libxml2 / libxslt is too old for installing a current lxml. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJGF3H+gerLs4ltQ4RArvnAKC5JU3joSrub0OQBKlxM+mZ6zPxiACgp/Su pjSh//ryuiqx6evoCr00hOg= =brxa -----END PGP SIGNATURE----- From dfedoruk at gmail.com Tue Nov 11 12:59:53 2008 From: dfedoruk at gmail.com (Dmitri Fedoruk) Date: Tue, 11 Nov 2008 14:59:53 +0300 Subject: [lxml-dev] NotImplementedError for external functions with xsl:variable's In-Reply-To: <491703F3.3000404@behnel.de> References: <49084477.10102@gmail.com> <491703F3.3000404@behnel.de> Message-ID: Hello everybody, > I assume that your real stylesheet creates a result tree > fragment for the variable that you pass into your function. To be frank - it's not like that. We used variables just as function parameters (e.g. a float number). But using a tree is a very powerful feature, too. > Please give it a try (after installing > Cython 0.10). lxml.etree: (2, 2, -199, 59835) libxml used: (2, 6, 30) libxml compiled: (2, 6, 30) libxslt used: (1, 1, 22) libxslt compiled: (1, 1, 22) I've attached the test source. In brief - it works :) I tried to test the variable as a number literal, as a string and as a tree. I got expected results in all cases. > It would be nice if you could come up with a couple of additional test > cases > for test_xslt.py to make sure this works (and continues to work) as > expected > for different variable values or scenarios. Currently, there > is > only test_variable_result_tree_fragment(), which tests for result trees > returned by . I'll try to - sorry, nothing has came up to my mind yet. Maybe we'll encounter some tricky cases in our templates, then I'll report them. Cheers, Dmitri -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20081111/6a254932/attachment.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: extern.py Type: application/octet-stream Size: 2673 bytes Desc: not available Url : http://codespeak.net/pipermail/lxml-dev/attachments/20081111/6a254932/attachment.obj From lxml{ at bestley.co.uk Tue Nov 11 18:52:45 2008 From: lxml{ at bestley.co.uk (Mark Bestley) Date: Tue, 11 Nov 2008 17:52:45 +0000 Subject: [lxml-dev] lxml Mac installation idea References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: I made an attempt to do a Mac OSX build and I think I can answer some of the questions raised in the various threads. I can't come up with a good general soultion. The easiest seems to me to use macports for all the C dependencies and things seem to just work as they do for Michael Guntsche. "Stefan Behnel" writes: > Chris Withers wrote: >> Stefan Behnel wrote: > > python setup.py build_ext -i --static-deps > > to download and build the libs and then statically build lxml with them. > This will copy the modules into their package at the end, so you should be > able to run > I attempted this on a Intel OSX 10.5 machine - with a new user so should have a clean environment I used the Apple installed python The steps I think I needed were 0) change setuptools to load to local library as per otherwise you need admin rights 1) easy_install Cython - I tried without Cython and it failed as no src/lxml/lxml.etree.c - which is a known step 2) The build failed with ld warning: in /Users/nomacports/Public/lxml/build/tmp/libxml2/lib/libexslt.a, file is not of required architecture Which is because python is a fat binary with PPC and i386 and the configure of libxml2 only provides a single archotecture. I could not get a simple fix as per as per to work. I edited setuptools.py adding a line after the test for darwin (at line 197) result.append( '-arch i386 -arch ppc' ) There is probably a better way to do this but I think we need a completed build first > python test.py -vv > > afterwards to see if it works as expected. Please report any errors you > get on the way. > > This failed with the iconv issue noted in other threads e.g. "setupinfo.py that downloads/builds libxml2/libxslt" I looked at the shared libraries with OSX toools e.g. otool and think that the issue is that iconv is called from libxml2 and the reference seems to have been stripped out in the static build of libxml2 So one answer is to have dynamic libxml etc but to tell configure to install these in the final target directory rather han the build directory or use something like py2app's macholib to rewrite the path in the dylib when copied to site-pakages (or use runtime environment variables like DYLD_LIBRARY_PATH ) Some alternatives might be i) download iconv as static as well ii) call iconv from etree to resolve the refererence iii) to make the etree link include all the objects in the libxml library As far as I understand Mac OSX dynamic libraries. They have the full path of the dynamic libraries they link to as well as their name embedded in the file. See the output of otool -L filename for details Macports thus works as all the libraries are in a fixed known place -- Mark From mike at it-loops.com Tue Nov 11 19:12:03 2008 From: mike at it-loops.com (Guntsche Michael) Date: Tue, 11 Nov 2008 19:12:03 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> On Nov 11, 2008, at 18:52, Mark Bestley wrote: > > I made an attempt to do a Mac OSX build and I think I can answer > some of > the questions raised in the various threads. I can't come up with a > good > general soultion. > The easiest seems to me to use macports for all the C dependencies and > things seem to just work as they do for Michael Guntsche. I had tossed around several E-Mails with Paul Everitt today to take a closer look at the iconv issue but you beat me to it, thanks. :) As for the static build, my feeling is that the majority of the issues here steam from the fact that while the python app is universal the self-compiled libraries are often i386 only. I had my shares of problems with this before so I compiled every related library as universal and now I no longer have problems. Some possible solutions I can add: * use macports to compile everything as Universal Binary * Explain in the README how to do this yourself, or provide a script etc. I do not really like the idea of static builds, since the static library has to be universal as well which means bigger files. Kind regards, Michael From lxml{ at bestley.co.uk Tue Nov 11 21:32:23 2008 From: lxml{ at bestley.co.uk (Mark Bestley) Date: Tue, 11 Nov 2008 20:32:23 +0000 Subject: [lxml-dev] lxml Mac installation idea References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> Message-ID: Guntsche Michael writes: > I had tossed around several E-Mails with Paul Everitt today to take a > closer look at the iconv issue but you beat me to it, thanks. :) > As for the static build, my feeling is that the majority of the issues > here steam from the fact that while the python app is universal the > self-compiled libraries are often i386 only. I had my shares of > problems with this before so I compiled every related library as > universal and now I no longer have problems. > As you said before we need to have some more information from someone with a broken setup. My macports setup is intel only and I use its python. If you want to use Apple Python or I think the python.org prebuilt you need the libraries to be universal. > Some possible solutions I can add: > > * use macports to compile everything as Universal Binary I think macports is not fully universal - if the compile is simple it will work but otherwise a maintainer has to alter the compile and that has not been done in all cases. From your experience it seems to be OK for lxml. My macports install is single architecture. If you are using macports then use its python much easier than mix and match. There is a lxml port and the easiest way to run is install that via macports e.g. # single architecture sudo port install py25-lxml # universal - Michael I assume this is what you did sudo port install py25-lxml +universal If you then need a newer version of lxml the the normal setup.py and not building libxml2 etc will work I think. > * Explain in the README how to do this yourself, or provide a script > etc. > > I do not really like the idea of static builds, since the static > library has to be universal as well which means bigger files. > I don't understand this point as the dynamic libraries have to be universal as well if python is universal. There are also some 64 bit architectures for OSX which implies that the build architectures should be derived from those used to build python. I think setup.py does this for the C code in the python package e.g. etree.so but the correct flags are not passed to libxml2 -- Mark From mike at it-loops.com Tue Nov 11 21:39:37 2008 From: mike at it-loops.com (Guntsche Michael) Date: Tue, 11 Nov 2008 21:39:37 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> Message-ID: On Nov 11, 2008, at 21:32, Mark Bestley wrote: >> I do not really like the idea of static builds, since the static >> library has to be universal as well which means bigger files. >> > I don't understand this point as the dynamic libraries have to be > universal as well if python is universal. What I ment was. If you build lxml yourself get universal builds of libiconv, libxml2 and libxslt from macports or compile them yourself. I do not use the lxml port myself, I just use the libxml,xslt and iconv libraries from there and build an egg. This works both with the system python version and the one form python.org. /Mike From stefan_ml at behnel.de Tue Nov 11 21:47:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Nov 2008 21:47:02 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> Message-ID: <4919EF46.50504@behnel.de> Hi, Guntsche Michael wrote: > What I ment was. If you build lxml yourself get universal builds of > libiconv, libxml2 and libxslt from macports or compile them yourself. Ok, two obvious questions: 1) how can setup.py figure out if it's running in a universally built Python? 2) how do you configure the build of libxml2/libxslt/libiconv to be universal? Mark wrote that "-arch i386 -arch ppc" will do the trick. Is that the right way of doing it? Stefan From ianb at colorstudy.com Tue Nov 11 21:53:10 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 11 Nov 2008 14:53:10 -0600 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> Message-ID: <4919F0B6.4020209@colorstudy.com> Guntsche Michael wrote: > On Nov 11, 2008, at 21:32, Mark Bestley wrote: > >>> I do not really like the idea of static builds, since the static >>> library has to be universal as well which means bigger files. >>> >> I don't understand this point as the dynamic libraries have to be >> universal as well if python is universal. > > What I ment was. If you build lxml yourself get universal builds of > libiconv, libxml2 and libxslt from macports or compile them yourself. > I do not use the lxml port myself, I just use the libxml,xslt and > iconv libraries from there and build an egg. This works both with the > system python version and the one form python.org. I cribbed the necessary options for libxml2 and libxslt from people's buildout.cfgs... what are the extra options for libiconv? Mostly, how do I get the lxml build to see it, and do the libxml2 and/or libxslt builds have to see libiconv (some ./configure option)? -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From lxml{ at bestley.co.uk Tue Nov 11 22:32:35 2008 From: lxml{ at bestley.co.uk (Mark Bestley) Date: Tue, 11 Nov 2008 21:32:35 +0000 Subject: [lxml-dev] lxml Mac installation idea References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919F0B6.4020209@colorstudy.com> Message-ID: Ian Bicking writes: > Guntsche Michael wrote: >> On Nov 11, 2008, at 21:32, Mark Bestley wrote: >> >>>> I do not really like the idea of static builds, since the static >>>> library has to be universal as well which means bigger files. >>>> >>> I don't understand this point as the dynamic libraries have to be >>> universal as well if python is universal. >> >> What I ment was. If you build lxml yourself get universal builds of >> libiconv, libxml2 and libxslt from macports or compile them yourself. >> I do not use the lxml port myself, I just use the libxml,xslt and >> iconv libraries from there and build an egg. This works both with the >> system python version and the one form python.org. > How do you build the egg as my understanding (very possibly incorrect) would be that the dynamic libraries would not necessarily have the correct paths. Unless you are leaving the dynamic libs in the place macports puts them. > I cribbed the necessary options for libxml2 and libxslt from people's > buildout.cfgs... what are the extra options for libiconv? Mostly, how > do I get the lxml build to see it, and do the libxml2 and/or libxslt > builds have to see libiconv (some ./configure option)? If they are like Paul Everitt's buildout I think they work as the libxml2 library is dynamic. But as noted in that message ''' One problem with buildout recipes is that they aren't, AFAIK, very compatible with other systems. That is, they can build the egg, but the egg isn't on the path of any interpreter, so unless you use buildout entirely you'll have to do some further manipulation to make lxml available. ''' What would be the command line to run that buildout (I have not learnt buildout yet) The current setup.py fails to get iconv as libxml etc are static. -- Mark From mike at it-loops.com Tue Nov 11 22:48:46 2008 From: mike at it-loops.com (Guntsche Michael) Date: Tue, 11 Nov 2008 22:48:46 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <4919EF46.50504@behnel.de> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> Message-ID: <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> On Nov 11, 2008, at 21:47, Stefan Behnel wrote: > Hi, > > Guntsche Michael wrote: >> What I ment was. If you build lxml yourself get universal builds of >> libiconv, libxml2 and libxslt from macports or compile them yourself. > > Ok, two obvious questions: > > 1) how can setup.py figure out if it's running in a universally > built Python? > If you use distutils or setuptools the flags are added automatically. I quickly looked at distutils and saw that you could check for a the UNIVERSALSDK config_var >>> from distutils import sysconfig >>> sysconfig.get_config_var('UNIVERSALSDK') '/Developer/SDKs/MacOSX10.4u.sdk' which should be empty on non-Universal Binary builds. > 2) how do you configure the build of libxml2/libxslt/libiconv to be > universal? > Mark wrote that "-arch i386 -arch ppc" will do the trick. Is that > the right > way of doing it? you need -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk in CFLAGS and LDFLAGS, isysroot is needed if you are building on a PPC machine. Furthermore you need to pass the "--disable-dependency-tracking" switch to ./configure, since this does not work with universal builds of the library. This way you can get a static and dynamic library in Universal Binary form. Example: ======== CFLAGS="-arch i386 -arch ppc" LDFLAGS="-arch i386 -arch ppc" ./ configure --without-python --disable-dependency-tracking --disable- shared for libxslt and static libs. hope that helps, Mike From ianb at colorstudy.com Wed Nov 12 03:11:46 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 11 Nov 2008 20:11:46 -0600 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919F0B6.4020209@colorstudy.com> Message-ID: <491A3B62.5000007@colorstudy.com> Mark Bestley wrote: > Ian Bicking writes: > >> Guntsche Michael wrote: >>> On Nov 11, 2008, at 21:32, Mark Bestley wrote: >>> >>>>> I do not really like the idea of static builds, since the static >>>>> library has to be universal as well which means bigger files. >>>>> >>>> I don't understand this point as the dynamic libraries have to be >>>> universal as well if python is universal. >>> What I ment was. If you build lxml yourself get universal builds of >>> libiconv, libxml2 and libxslt from macports or compile them yourself. >>> I do not use the lxml port myself, I just use the libxml,xslt and >>> iconv libraries from there and build an egg. This works both with the >>> system python version and the one form python.org. > > How do you build the egg as my understanding (very possibly incorrect) > would be that the dynamic libraries would not necessarily have the > correct paths. Unless you are leaving the dynamic libs in the place > macports puts them. My understanding is Enstaller has some code to fix up .so files when this happens, if we're really interested we could try to copy that functionality over. But I would really just prefer a static library for this reason. >> I cribbed the necessary options for libxml2 and libxslt from people's >> buildout.cfgs... what are the extra options for libiconv? Mostly, how >> do I get the lxml build to see it, and do the libxml2 and/or libxslt >> builds have to see libiconv (some ./configure option)? > > If they are like Paul Everitt's buildout I think they work as the > libxml2 library is dynamic. But as noted in that message > ''' > One problem with buildout recipes is that they aren't, AFAIK, very > compatible with other systems. That is, they can build the egg, but the > egg isn't on the path of any interpreter, so unless you use buildout > entirely you'll have to do some further manipulation to make lxml > available. > ''' > > What would be the command line to run that buildout (I have not learnt > buildout yet) You have to run bootstrap.py, which is usually shipped with the buildout.cfg, and that installs buildout. Then run bin/buildout, I think with no arguments, and it'll automatically build what's in buildout.cfg (or you can indicate other .cfg files with some options). > The current setup.py fails to get iconv as libxml etc are static. Right, and I'm assuming the buildout.cfg builds everything dynamically, which is easier for it because it has a layout that gives it a place to put the libxml libraries. But that's hard in a more general sense, so ideally iconv could be built static like everything else? Or is this going to cascade into a neverending set of libraries, all of which have to be static if any of them are static? -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From mantegazza at ill.fr Wed Nov 12 08:46:06 2008 From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=) Date: Wed, 12 Nov 2008 08:46:06 +0100 Subject: [lxml-dev] seg. fault using _Comment Message-ID: <200811120846.06230.mantegazza@ill.fr> Hello, I'm trying to use comments in my xml file. The goal is to be able to enable/disable some entries in a config. file. Parsing comments works fine; when getting a node, I check if it is a _Comment object. If so, I just parse the string contained in the comment. Now, I would like to do the reverse operation: save some entries as comments. But it fails. I instanciate a _Comment object, but as soon as I use it, the application crashed with seg. fault. Is it a known bug? I'm using the 1.1.1 version. Let me know if it can be solved by upgrading. Thanks, -- Fr?d?ric From stefan_ml at behnel.de Wed Nov 12 09:27:40 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 12 Nov 2008 09:27:40 +0100 (CET) Subject: [lxml-dev] seg. fault using _Comment In-Reply-To: <200811120846.06230.mantegazza@ill.fr> References: <200811120846.06230.mantegazza@ill.fr> Message-ID: <36915.213.61.181.86.1226478460.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Fr?d?ric Mantegazza wrote: > I'm trying to use comments in my xml file. The goal is to be able to > enable/disable some entries in a config. file. > > Parsing comments works fine; when getting a node, I check if it is a > _Comment object. If so, I just parse the string contained in the comment. I'd rather check for "el.tag is etree.Comment". "_Comment" starts with an underscore, which makes it a non-public class. > Now, I would like to do the reverse operation: save some entries as > comments. But it fails. I instanciate a _Comment object, but as soon as I > use it, the application crashed with seg. fault. Use the Comment factory. Stefan From mantegazza at ill.fr Wed Nov 12 09:34:35 2008 From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=) Date: Wed, 12 Nov 2008 09:34:35 +0100 Subject: [lxml-dev] seg. fault using _Comment In-Reply-To: <36915.213.61.181.86.1226478460.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <200811120846.06230.mantegazza@ill.fr> <36915.213.61.181.86.1226478460.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <200811120934.35982.mantegazza@ill.fr> Le mercredi 12 novembre 2008 09:27, Stefan Behnel a ?crit : > I'd rather check for "el.tag is etree.Comment". "_Comment" starts with an > underscore, which makes it a non-public class. Oups, I didn't see the Comment factory: sorry for the noise :o/ > Use the Comment factory. Ok, it works fine. Thanks :o) -- Fr?d?ric From lxml{ at bestley.co.uk Wed Nov 12 11:20:39 2008 From: lxml{ at bestley.co.uk (Mark Bestley) Date: Wed, 12 Nov 2008 10:20:39 +0000 Subject: [lxml-dev] lxml Mac installation idea References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919F0B6.4020209@colorstudy.com> <491A3B62.5000007@colorstudy.com> Message-ID: Ian Bicking writes: > Mark Bestley wrote: >> Ian Bicking writes: >> >>> Guntsche Michael wrote: >>>> On Nov 11, 2008, at 21:32, Mark Bestley wrote: >>>> > > My understanding is Enstaller has some code to fix up .so files when > this happens, if we're really interested we could try to copy that > functionality over. But I would really just prefer a static library for > this reason. > Yes and I now have a hacked build which seems to work for Apple's pythone. I will try it in a few more situations. But basically it changes the environment for the confurure for lbxml2 and libxslt. I suspect I might have got this earlier but removing the build directory does not seem to force setup.py to rebuild etree.so in all cases (and autoconf is frustrating) > > Right, and I'm assuming the buildout.cfg builds everything dynamically, > which is easier for it because it has a layout that gives it a place to > put the libxml libraries. But that's hard in a more general sense, so > ideally iconv could be built static like everything else? Or is this > going to cascade into a neverending set of libraries, all of which have > to be static if any of them are static? No I should be able to get the static to work -- Mark From lxml{ at bestley.co.uk Wed Nov 12 11:45:32 2008 From: lxml{ at bestley.co.uk (Mark Bestley) Date: Wed, 12 Nov 2008 10:45:32 +0000 Subject: [lxml-dev] lxml Mac installation idea References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> Message-ID: Guntsche Michael writes: > On Nov 11, 2008, at 21:47, Stefan Behnel wrote: > >> Hi, >> >> Guntsche Michael wrote: > If you use distutils or setuptools the flags are added automatically. > I quickly looked at distutils and saw that you could check for a > the UNIVERSALSDK config_var > > >>> from distutils import sysconfig > >>> sysconfig.get_config_var('UNIVERSALSDK') > '/Developer/SDKs/MacOSX10.4u.sdk' > > which should be empty on non-Universal Binary builds. > well it is set on my macports 2.5 single architecture, a 2.6 dual from python.org but not on Apple's dual system one ;( (+ we need Tiger's 2.3 etc ) > >> 2) how do you configure the build of libxml2/libxslt/libiconv to be >> universal? >> Mark wrote that "-arch i386 -arch ppc" will do the trick. Is that >> the right >> way of doing it? > you need > > -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk > > in CFLAGS and LDFLAGS, isysroot is needed if you are building on a PPC > machine. > Furthermore you need to pass the "--disable-dependency-tracking" > switch to ./configure, since this does not work with universal builds > of the library. > This way you can get a static and dynamic library in Universal Binary > form. > > Example: > ======== > > CFLAGS="-arch i386 -arch ppc" LDFLAGS="-arch i386 -arch ppc" ./ > configure --without-python --disable-dependency-tracking --disable- > shared > > for libxslt and static libs. > Seemes to for the command line you gave but not with the isysroot variable set. Apple's docs are not consistant says set isysroot but python specific docs say it is not So I'll leave for the time being. -- Mark From mike at it-loops.com Wed Nov 12 12:05:51 2008 From: mike at it-loops.com (Michael Guntsche) Date: Wed, 12 Nov 2008 12:05:51 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> Message-ID: <2bfae3f7acf6b2cd3da04f8806b13dc2@localhost> On Wed, 12 Nov 2008 10:45:32 +0000, Mark Bestley wrote: > Seemes to for the command line you gave but not with the isysroot > variable set. > Apple's docs are not consistant > > says set isysroot but python specific docs say it is not > > > So I'll leave for the time being. Please put it in, it is needed if you compile on a PPC machine. See http://developer.apple.com/technotes/tn2005/tn2137.html extract: Note: On an Intel-based Macintosh system the libraries are already universal, and support the Intel and PowerPC architectures, and you may specify only the -arch i386 -arch ppc options for CFLAGS; on a PowerPC-based Macintosh, you must use the MacOSX10.4u SDK. If you do not put it in you cannot build UBs on PPC. /Mike From stefan_ml at behnel.de Wed Nov 12 20:11:30 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 12 Nov 2008 20:11:30 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> Message-ID: <491B2A62.9080200@behnel.de> Hi, Guntsche Michael wrote: >> 2) how do you configure the build of libxml2/libxslt/libiconv to be >> universal? >> Mark wrote that "-arch i386 -arch ppc" will do the trick. Is that the >> right way of doing it? > you need > > -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk > > Example: > ======== > > CFLAGS="-arch i386 -arch ppc" LDFLAGS="-arch i386 -arch ppc" ./configure > --without-python --disable-dependency-tracking --disable-shared > > for libxslt and static libs. Ok, I've updated the trunk build script to reflect some of this. http://codespeak.net/svn/lxml/trunk/buildlibxml.py Please check if it works better. Patches are very welcome. Stefan From mike at it-loops.com Thu Nov 13 00:05:31 2008 From: mike at it-loops.com (Guntsche Michael) Date: Thu, 13 Nov 2008 00:05:31 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <491B2A62.9080200@behnel.de> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> Message-ID: <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> On Nov 12, 2008, at 20:11, Stefan Behnel wrote: > Ok, I've updated the trunk build script to reflect some of this. > > http://codespeak.net/svn/lxml/trunk/buildlibxml.py > > Please check if it works better. Patches are very welcome. > I spent far more time than I wanted on this one, but please check this.. Maybe there is a better way to add the environment variables, but I did not wnat to change call_subprocess that much. Kind regards, Michael Index: buildlibxml.py =================================================================== --- buildlibxml.py (revision 59893) +++ buildlibxml.py (working copy) @@ -165,7 +165,17 @@ prefix = os.path.join(os.path.abspath(build_dir), 'libxml2') safe_mkdir(prefix) - configure_cmd = ['./configure', '--without-python', + + # We just take the CFLAGS and LDFLAG from the python build environment + # so it does not matter if it was compiled Universal or not. + os.environ['CFLAGS'] = sysconfig.get_config_var('CFLAGS') + os.environ['LDFLAGS'] = sysconfig.get_config_var('LDFLAGS') + os.environ['MACOS_DEPLOYMENT_TARGET'] = "10.3" + + # We are loosing the link to iconv add it again + static_binaries.append('-liconv') + + configure_cmd = ['./configure', '--without-python' , '--disable-dependency-tracking', '--disable-shared', '--prefix=%s' % prefix] call_subprocess(configure_cmd, cwd=libxml2_dir) @@ -174,7 +184,7 @@ call_subprocess( ['make', 'install'], cwd=libxml2_dir) - libxslt_configure_cmd = configure_cmd + ['--with-libxml-prefix= %s' % prefix] + libxslt_configure_cmd = configure_cmd + ['--with-libxml-prefix= %s' % prefix, '--without-crypto'] call_subprocess(libxslt_configure_cmd, cwd=libxslt_dir) call_subprocess( ['make'], cwd=libxslt_dir) @@ -191,11 +201,6 @@ os.path.join(prefix, 'include', 'libexslt')]) static_library_dirs.append(lib_dir) - unisdk_dir = sysconfig.get_config_var('UNIVERSALSDK') - if unisdk_dir: - static_cflags.extend(['-isysroot', unisdk_dir, - '-arch', '-i386', '-arch', 'ppc']) - for filename in os.listdir(lib_dir): if [l for l in ['libxml2', 'libxslt', 'libexslt'] if l in filename]: if [ext for ext in ['.a'] if filename.endswith(ext)]: From mike at it-loops.com Thu Nov 13 11:33:46 2008 From: mike at it-loops.com (Michael Guntsche) Date: Thu, 13 Nov 2008 11:33:46 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> Message-ID: <349cf98f3614225d778caca46b530644@localhost> On Thu, 13 Nov 2008 00:05:31 +0100, Guntsche Michael wrote: > I spent far more time than I wanted on this one, but please check this.. > Maybe there is a better way to add the environment variables, but I > did not wnat to change call_subprocess that much. > Index: buildlibxml.py > + # We just take the CFLAGS and LDFLAG from the python build > environment > + # so it does not matter if it was compiled Universal or not. > + os.environ['CFLAGS'] = sysconfig.get_config_var('CFLAGS') > + os.environ['LDFLAGS'] = sysconfig.get_config_var('LDFLAGS') > + os.environ['MACOS_DEPLOYMENT_TARGET'] = "10.3" > + > + # We are loosing the link to iconv add it again > + static_binaries.append('-liconv') > + Replying to my own E-Mail here. My SMTP server crapped out so this is a little bit delayed. Maybe we should set back the CFLAGS and LDFLAGS after compiling libxml2 and libxslt. What happens is that the build of etree.so now has the flags twice. I is not doing any harm right now, because they are the same anyway but it looks "cleaner". Or we change call_subprocess so we can pass the env flags there. Anyway, I will try to test this today on my PPC machine with 10.4 (Tiger) still installed. Maybe others with a different setup could test this as well. Doing it this way links as against the system libraries just libxml2 and libxslt are taken from the static build. Kind regards, Michael From cfbearden at gmail.com Thu Nov 13 21:13:41 2008 From: cfbearden at gmail.com (Chuck Bearden) Date: Thu, 13 Nov 2008 14:13:41 -0600 Subject: [lxml-dev] Avoiding DTD default attributes in XSLT Message-ID: <433ebc870811131213u4886eb21ma5a7c9d7b36c295b@mail.gmail.com> xsltproc offers the '--nodtdattr' option to cause libxslt not to add default attributes from the DTD to the output document when they aren't present in the source document. How can I achieve the same effect when doing XSLT with lxml? Thanks in advance for your help! Chuck From mike at it-loops.com Thu Nov 13 21:59:49 2008 From: mike at it-loops.com (Guntsche Michael) Date: Thu, 13 Nov 2008 21:59:49 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <349cf98f3614225d778caca46b530644@localhost> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> Message-ID: On Nov 13, 2008, at 11:33, Michael Guntsche wrote: > > Maybe others with a different setup could test this as well. Doing > it this > way links as against the system libraries just libxml2 and libxslt are > taken from the static build. > Ah screw that. CFLAGS is not defined in every python build that's available so we cannot use that. Honestly, this is starting to piss me off. Python modules are always compiled correctly, but to get used FLAGS out of the different binaries is more difficult than I thought. What about ALWAYS compiling a Universal Binary on 10.4 and 10.5? Kind regards, Michael From mike at it-loops.com Fri Nov 14 01:27:24 2008 From: mike at it-loops.com (Guntsche Michael) Date: Fri, 14 Nov 2008 01:27:24 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> Message-ID: <2F4AA522-C797-474F-8B02-2FB55407B05B@it-loops.com> Ok, A new patch. We always create an universal binary if we are running on Macosx > 10.3. Furthermore the environment variables are removed before we compile lxml itself. Going to bed now will test this on my Tiger PPC machine tomorrow afternoon. Gnite, Mike Index: buildlibxml.py =================================================================== --- buildlibxml.py (revision 59905) +++ buildlibxml.py (working copy) @@ -165,7 +165,19 @@ prefix = os.path.join(os.path.abspath(build_dir), 'libxml2') safe_mkdir(prefix) - configure_cmd = ['./configure', '--without-python', + + # We compile Universal if we are on a machine > 10.4.0 + major_version = int(os.uname()[2].split('.')[0]) + + if major_version > 7: + os.environ['CFLAGS'] = "-arch ppc -arch i386 -isysroot / Developer/SDKs/MacOSX10.4u.sdk -O2" + os.environ['LDFLAGS'] = "-arch ppc -arch i386 -isysroot / Developer/SDKs/MacOSX10.4u.sdk" + os.environ['MACOSX_DEPLOYMENT_TARGET'] = "10.3" + + # We are loosing the link to iconv add it again + static_binaries.append('-liconv') + + configure_cmd = ['./configure', '--without-python' , '--disable-dependency-tracking', '--disable-shared', '--prefix=%s' % prefix] call_subprocess(configure_cmd, cwd=libxml2_dir) @@ -174,7 +186,7 @@ call_subprocess( ['make', 'install'], cwd=libxml2_dir) - libxslt_configure_cmd = configure_cmd + ['--with-libxml-prefix= %s' % prefix] + libxslt_configure_cmd = configure_cmd + ['--with-libxml-prefix= %s' % prefix, '--without-crypto'] call_subprocess(libxslt_configure_cmd, cwd=libxslt_dir) call_subprocess( ['make'], cwd=libxslt_dir) @@ -191,10 +203,9 @@ os.path.join(prefix, 'include', 'libexslt')]) static_library_dirs.append(lib_dir) - unisdk_dir = sysconfig.get_config_var('UNIVERSALSDK') - if unisdk_dir: - static_cflags.extend(['-isysroot', unisdk_dir, - '-arch', '-i386', '-arch', 'ppc']) + if major_version > 7: + for env in ('CFLAGS', 'LDFLAGS', 'MACOSX_DEPLOYMENT_TARGET'): + os.environ.pop(env) for filename in os.listdir(lib_dir): if [l for l in ['libxml2', 'libxslt', 'libexslt'] if l in filename]: From lxml{ at bestley.co.uk Fri Nov 14 11:36:11 2008 From: lxml{ at bestley.co.uk (Mark Bestley) Date: Fri, 14 Nov 2008 10:36:11 +0000 Subject: [lxml-dev] lxml Mac installation idea References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> Message-ID: Guntsche Michael writes: > On Nov 13, 2008, at 11:33, Michael Guntsche wrote: >> >> Maybe others with a different setup could test this as well. Doing >> it this >> way links as against the system libraries just libxml2 and libxslt are >> taken from the static build. >> > > Ah screw that. CFLAGS is not defined in every python build that's > available so we cannot use that. > Honestly, this is starting to piss me off. Python modules are always > compiled correctly, but to get used FLAGS out of the different > binaries is more difficult than I thought. > What about ALWAYS compiling a Universal Binary on 10.4 and 10.5? > Yep it is fun - UNIVERSALSDK is not always set either. (and on my build setting it was a problem) The cases I have are 1) Apple Python Leopard - nothing set but is universal 2) python.org which is universal and seems to have th flags set. 3) macports 2.5 which can be either but for this we should not build libxml2 etc so should work anyway (I suspect Fink would be the same) There are also 4) manual python builds that can be for 64bit 5) 10.4 versions 6) in theory 10.3.9 any more? Possibly the best way is that if CFLAGS and UNIVERSALSDK is set then use them otherwise compile Universal binary - From my fiddling I think this will work for 1&2 3 works now We need to investigate the others But 1 & 2 are the main cases I think we need to cover - although we ought to try 5. I'll try some more today. -- Mark From lists at cheimes.de Fri Nov 14 18:13:18 2008 From: lists at cheimes.de (Christian Heimes) Date: Fri, 14 Nov 2008 18:13:18 +0100 Subject: [lxml-dev] Leaking tracebacks Message-ID: <491DB1AE.9060809@cheimes.de> Hello! A college run into a memory leak on one of our servers a couple of days ago. With the help of Dowser - a CherryPy plugin - we were able to nail down the leak to traceback and frame objects. Some code in Martijn Faassen's oaipmh package kept references to an exception traceback. Today I was able to track 'n nail down the issue. I highly suspect lxml to be the culprit. Whenever the code reaches lxml with an exception in sys.exc_info the exception leaks. My dowser patch shows a refcnt of 4 references to the root traceback object that can't be traced to objects in the gc.get_objects() list. A minor change to oaipmh.server.XMLTreeServer gets rid of the leaking tracebacks: def handleException(self, exception): if isinstance(exception, error.ErrorBase): sys.exc_clear() ## <- REF LEAK FIX envelope = self._outputErrors( [(exception.oainame(), str(exception))]) return envelope # unhandled exception, so raise again raise We are using lxml 2.1.2 on Python 2.5.2. I think lxml 2.0.7 didn't have the problem. Christian From stefan_ml at behnel.de Sat Nov 15 15:00:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 15 Nov 2008 15:00:27 +0100 Subject: [lxml-dev] Leaking tracebacks In-Reply-To: <491DB1AE.9060809@cheimes.de> References: <491DB1AE.9060809@cheimes.de> Message-ID: <491ED5FB.3020203@behnel.de> Hi, Christian Heimes wrote: > A college run into a memory leak on one of our servers a couple of days > ago. With the help of Dowser - a CherryPy plugin - we were able to nail > down the leak to traceback and frame objects. Some code in Martijn > Faassen's oaipmh package kept references to an exception traceback. > > Today I was able to track 'n nail down the issue. I highly suspect lxml > to be the culprit. Whenever the code reaches lxml with an exception in > sys.exc_info the exception leaks. My dowser patch shows a refcnt of 4 > references to the root traceback object that can't be traced to objects > in the gc.get_objects() list. There was a change in Cython regarding exception handling not so long ago. Cython now mimics more the behaviour of Py3, i.e. exceptions disappear once they were caught. If the problem was introduced at this point, it would explain why this appears with later lxml releases only, as older releases are based on older Cython versions. I'm not sure what the problem is here, though, so this needs further investigation. From a couple of quick tests, I can't seem to reproduce this problem. Could you try to come up with some test code that shows this behaviour? Is it only a problem with functionality that interacts with libxml2 in both directions (such as XPath/XSLT, which uses function callbacks back into lxml), or does it also happen in functions like etree.iselement()? Stefan From stefan_ml at behnel.de Sat Nov 15 15:08:12 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 15 Nov 2008 15:08:12 +0100 Subject: [lxml-dev] Avoiding DTD default attributes in XSLT In-Reply-To: <433ebc870811131213u4886eb21ma5a7c9d7b36c295b@mail.gmail.com> References: <433ebc870811131213u4886eb21ma5a7c9d7b36c295b@mail.gmail.com> Message-ID: <491ED7CC.1020102@behnel.de> Hi, Chuck Bearden wrote: > xsltproc offers the '--nodtdattr' option to cause libxslt not to add > default attributes from the DTD to the output document when they > aren't present in the source document. >From a quick glance at xsltproc's source, it looks like this option does not impact the output document but the input document itself. So if you parse the input document without loading the DTD (which is actually the default), lxml should behave like xsltproc here. Stefan From mike at it-loops.com Sat Nov 15 23:47:24 2008 From: mike at it-loops.com (Guntsche Michael) Date: Sat, 15 Nov 2008 23:47:24 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> Message-ID: <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> On Nov 14, 2008, at 11:36, Mark Bestley wrote: > Yep it is fun - UNIVERSALSDK is not always set either. (and on my > build > setting it was a problem) Here the last incarnation of my patch. I got rid of setting the environment variables and deleting them afterwards. Should have done it like this in the first place. /Mike Index: buildlibxml.py =================================================================== --- buildlibxml.py (revision 59932) +++ buildlibxml.py (working copy) @@ -165,17 +165,34 @@ prefix = os.path.join(os.path.abspath(build_dir), 'libxml2') safe_mkdir(prefix) - configure_cmd = ['./configure', '--without-python', + + # We compile Universal if we are on a machine > 10.3 + major_version = int(os.uname()[2].split('.')[0]) + + if major_version > 7: + universal_env = { + 'CFLAGS' : "-arch ppc -arch i386 -isysroot /Developer/ SDKs/MacOSX10.4u.sdk -O2", + 'LDFLAGS' : "-arch ppc -arch i386 -isysroot /Developer/ SDKs/MacOSX10.4u.sdk", + 'MACOSX_DEPLOYMENT_TARGET' : "10.3" + } + else: universal_env = None + + # We are loosing the link to iconv add it again + static_binaries.append('-liconv') + + configure_cmd = ['./configure', '--without-python' , '--disable-dependency-tracking', '--disable-shared', '--prefix=%s' % prefix] - call_subprocess(configure_cmd, cwd=libxml2_dir) + call_subprocess(configure_cmd, cwd=libxml2_dir, env=universal_env) call_subprocess( ['make'], cwd=libxml2_dir) call_subprocess( ['make', 'install'], cwd=libxml2_dir) - libxslt_configure_cmd = configure_cmd + ['--with-libxml-prefix= %s' % prefix] - call_subprocess(libxslt_configure_cmd, cwd=libxslt_dir) + libxslt_configure_cmd = configure_cmd + ['--with-libxml-prefix= %s' % prefix, + '--without-crypto', + '--disable-shared'] + call_subprocess(libxslt_configure_cmd, cwd=libxslt_dir, env=universal_env) call_subprocess( ['make'], cwd=libxslt_dir) call_subprocess( @@ -191,11 +208,6 @@ os.path.join(prefix, 'include', 'libexslt')]) static_library_dirs.append(lib_dir) - unisdk_dir = sysconfig.get_config_var('UNIVERSALSDK') - if unisdk_dir: - static_cflags.extend(['-isysroot', unisdk_dir, - '-arch', '-i386', '-arch', 'ppc']) - for filename in os.listdir(lib_dir): if [l for l in ['libxml2', 'libxslt', 'libexslt'] if l in filename]: if [ext for ext in ['.a'] if filename.endswith(ext)]: From stefan_ml at behnel.de Sun Nov 16 09:22:19 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 16 Nov 2008 09:22:19 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> Message-ID: <491FD83B.8070803@behnel.de> Hi, Guntsche Michael wrote: > Here the last incarnation of my patch. I got rid of setting the > environment variables and deleting them afterwards. > Should have done it like this in the first place. Thanks a bunch. I rewrote the patch somewhat to restrict the changes to MacOS. If I didn't break it by doing that, the current trunk should work out-of-the-box now, if you pass "--static-deps" to setup.py, or set "STATIC_DEPS=true" in your env variables. Anyone wants to give it a try? Stefan From mike at it-loops.com Sun Nov 16 09:59:35 2008 From: mike at it-loops.com (Michael Guntsche) Date: Sun, 16 Nov 2008 09:59:35 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <491FD83B.8070803@behnel.de> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> <491FD83B.8070803@behnel.de> Message-ID: On Nov 16, 2008, at 9:22, Stefan Behnel wrote: > Thanks a bunch. I rewrote the patch somewhat to restrict the changes > to > MacOS. If I didn't break it by doing that, the current trunk should > work > out-of-the-box now, if you pass "--static-deps" to setup.py, or set > "STATIC_DEPS=true" in your env variables. > > Anyone wants to give it a try? This survives a "make test" here on my setup. I'll try to test it on my tiger machine later. I have not 10.3 (Panther) setup any more so maybe someone still running this can give it a try? Kind regards, Michael From lists at cheimes.de Mon Nov 17 01:50:54 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 17 Nov 2008 01:50:54 +0100 Subject: [lxml-dev] Leaking tracebacks In-Reply-To: <491ED5FB.3020203@behnel.de> References: <491DB1AE.9060809@cheimes.de> <491ED5FB.3020203@behnel.de> Message-ID: <4920BFEE.6090906@cheimes.de> Stefan Behnel wrote: > There was a change in Cython regarding exception handling not so long ago. > Cython now mimics more the behaviour of Py3, i.e. exceptions disappear once > they were caught. If the problem was introduced at this point, it would > explain why this appears with later lxml releases only, as older releases > are based on older Cython versions. I was able to come up with a minimal test case. It shows that lxml 2.0.7 is fine but 2.0.9 and 2.1.2 are leaking references somewhere. > I'm not sure what the problem is here, though, so this needs further > investigation. From a couple of quick tests, I can't seem to reproduce this > problem. Could you try to come up with some test code that shows this > behaviour? Is it only a problem with functionality that interacts with > libxml2 in both directions (such as XPath/XSLT, which uses function > callbacks back into lxml), or does it also happen in functions like > etree.iselement()? --- import sys import pkg_resources pkg_resources.require("lxml==%s" % sys.argv[1]) from lxml import etree class SampleException(Exception): pass def test(): print "lxml: %s, %s, %s" % (etree.LXML_VERSION, etree.LIBXML_VERSION, etree.LIBXSLT_VERSION) print "SampleException start: %i" % sys.getrefcount(SampleException) for i in range(1000): try: raise SampleException except Exception, err: el = etree.Element("test") if i == 999: print "exception: %i" % sys.getrefcount(err) print "SampleException stop: %i" % sys.getrefcount(SampleException) if __name__ == "__main__": test() --- Output: $ python2.5 lxml_bug.py 2.0.7 lxml: (2, 0, 7, 0), (2, 6, 31), (1, 1, 24) SampleException start: 4 exception: 4 SampleException stop: 7 $ python2.5 lxml_bug.py 2.0.9 lxml: (2, 0, 9, 0), (2, 6, 31), (1, 1, 24) SampleException start: 4 exception: 5 SampleException stop: 2006 $ python2.5 lxml_bug.py 2.1.2 lxml: (2, 1, 2, 0), (2, 6, 31), (1, 1, 24) SampleException start: 4 exception: 5 SampleException stop: 2006 Every loop leaks 2 references to the exeption class. Christian From lists at cheimes.de Mon Nov 17 02:16:15 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 17 Nov 2008 02:16:15 +0100 Subject: [lxml-dev] Leaking tracebacks In-Reply-To: <4920BFEE.6090906@cheimes.de> References: <491DB1AE.9060809@cheimes.de> <491ED5FB.3020203@behnel.de> <4920BFEE.6090906@cheimes.de> Message-ID: Christian Heimes wrote: > import sys > import pkg_resources > pkg_resources.require("lxml==%s" % sys.argv[1]) > > from lxml import etree > > class SampleException(Exception): > pass > > def test(): > print "lxml: %s, %s, %s" % (etree.LXML_VERSION, > etree.LIBXML_VERSION, > etree.LIBXSLT_VERSION) > print "SampleException start: %i" % sys.getrefcount(SampleException) > for i in range(1000): > try: > raise SampleException > except Exception, err: # this removes the ref leak sys.exc_clear() > el = etree.Element("test") > if i == 999: > print "exception: %i" % sys.getrefcount(err) > print "SampleException stop: %i" % sys.getrefcount(SampleException) > > if __name__ == "__main__": hristian From chris at simplistix.co.uk Mon Nov 17 13:54:17 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 17 Nov 2008 12:54:17 +0000 Subject: [lxml-dev] specifying lxml as a dependency? In-Reply-To: <52065.213.61.181.86.1226309133.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49174A17.4080607@simplistix.co.uk> <49174CE6.60105@colorstudy.com> <4917F43F.30307@simplistix.co.uk> <52065.213.61.181.86.1226309133.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49216979.9080401@simplistix.co.uk> Stefan Behnel wrote: >>> Linux binary eggs are too problematic to be worth >>> trying. The system will have to have libxml2/libxslt and the -dev >>> packages (on Linux), >> Okay, so the user will have to get libxml2 and libxslt on the system >> "somehow" (ie: os package manager, source install, etc) *before* >> easy_install'ing lxml? > > I wouldn't know any distribution that doesn't install libxml2/libxslt by > default, and for most Linux users, installing the -dev packages for a > library is not much harder than eating a warm pie (assuming you like the > taste). Indeed, but it still has to be done, while it doesn't on windows, which is interesting ;-) >>> and they could do "STATIC_DEPS=true easy_install >>> Twiddler" to build a static lxml on Macs >> Where would "STATIC_DEPS=true easy_install Twiddler" go? > > That's the command that will install Twiddler on a Mac once Twiddler has a > dependency on lxml and is uploaded to PyPI. Okay, why does Mac need this and linux doesn't? > Well, development continues, so there are quite some internal differences. > What's relevant here is that the setup.py in the current trunk knows how > to download and build libxml2 and libxslt, and how to build lxml > statically using the result. That's specifically made (but not only meant) > for Mac users that run into crashes due to conflicting library versions. Ah okay, interesting, would it work on linux too if libxml2 and libxslt aren't available? (of course, if you're following the discussions on the distutils-sig, setup.py's that do this amount of work are very very evil...) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Mon Nov 17 13:57:55 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 17 Nov 2008 12:57:55 +0000 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49216A53.5060004@simplistix.co.uk> Stefan Behnel wrote: > As I said, this would be very much appreciated. The first thing I would > ask you to do is check out the current trunk from SVN > > https://codespeak.net/lxml/build.html#subversion I'm currently about 6000 miles away from my Macs, but I believe someone else offerred to pick this up? If not, gimme a poke later this week and I'll have a go :-) > a static, complete installer the best choice. It's even better to install > dependencies 100 times in parallel, than to risk conflicts, or even rely > on them being up-to-date. Sometimes I feel this is the case on Linux too ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From paul at agendaless.com Mon Nov 17 16:13:06 2008 From: paul at agendaless.com (Paul Everitt) Date: Mon, 17 Nov 2008 10:13:06 -0500 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <49216A53.5060004@simplistix.co.uk> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49216A53.5060004@simplistix.co.uk> Message-ID: On Nov 17, 2008, at 7:57 AM, Chris Withers wrote: > Stefan Behnel wrote: >> As I said, this would be very much appreciated. The first thing I >> would >> ask you to do is check out the current trunk from SVN >> >> https://codespeak.net/lxml/build.html#subversion > > I'm currently about 6000 miles away from my Macs, but I believe > someone > else offerred to pick this up? If not, gimme a poke later this week > and > I'll have a go :-) I got: $ python setup.py build --static-deps [.........snip.......] gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -O2 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch - Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized - Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings - Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested- externs -Winline -Wredundant-decls -arch ppc -arch i386 -isysroot / Developer/SDKs/MacOSX10.4u.sdk -o xmllint xmllint.o ./.libs/libxml2.a -lpthread -lz /usr/lib/libiconv.dylib -lm ld: library not found for -lcrt1.10.5.o collect2: ld returned 1 exit status ld: library not found for -lcrt1.10.5.o collect2: ld returned 1 exit status lipo: can't open input file: /var/folders/i4/i4lSVTRWHsaTXG7UXMfGbE++ +TI/-Tmp-//ccDzwBUG.out (No such file or directory) make[2]: *** [xmllint] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 Traceback (most recent call last): File "setup.py", line 107, in ? STATIC_CFLAGS, STATIC_BINARIES), File "/Users/paul/sandboxes/lxml/trunk/setupinfo.py", line 46, in ext_modules libxslt_version=OPTION_LIBXSLT_VERSION) File "/Users/paul/sandboxes/lxml/trunk/buildlibxml.py", line 190, in build_libxml2xslt call_subprocess( File "/Users/paul/sandboxes/lxml/trunk/buildlibxml.py", line 151, in call_subprocess raise Exception('Command "%s" returned code %s' % (cmd_desc, returncode)) Exception: Command "make" returned code 2 --Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20081117/00f99e9f/attachment.htm From remy.roy at cgi.com Mon Nov 17 16:53:08 2008 From: remy.roy at cgi.com (Roy, Remy) Date: Mon, 17 Nov 2008 10:53:08 -0500 Subject: [lxml-dev] XmlTextReader support Message-ID: <6CC8E8868854DA4CB1C65DC49156F084E34A37@MTL-MSG-03.cgiclients.com> Hello, Is there any roadmap to provide support for the XmlTextReader interface as specified on this page http://www.xmlsoft.org/xmlreader.html in lxml? Regards, R?my Roy -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20081117/ffe1ceb1/attachment.htm From mike at it-loops.com Mon Nov 17 17:20:27 2008 From: mike at it-loops.com (Michael Guntsche) Date: Mon, 17 Nov 2008 17:20:27 +0100 Subject: [lxml-dev] Fwd: lxml Mac installation idea References: <3D8E108D-1E2D-4917-927B-9D4950B13ABF@it-loops.com> Message-ID: <9F695713-8E93-42B7-AD6D-889F0697835D@it-loops.com> CC'ing the list. Begin forwarded message: > From: Michael Guntsche > Date: November 17, 2008 17:08:48 GMT+01:00 > To: Paul Everitt > Subject: Re: [lxml-dev] lxml Mac installation idea > > > On Nov 17, 2008, at 16:13, Paul Everitt wrote: > >> >> I got: >> >> $ python setup.py build --static-deps >> [.........snip.......] >> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk - >> O2 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch >> -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized - >> Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings - >> Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested- >> externs -Winline -Wredundant-decls -arch ppc -arch i386 -isysroot / >> Developer/SDKs/MacOSX10.4u.sdk -o xmllint xmllint.o ./.libs/ >> libxml2.a -lpthread -lz /usr/lib/libiconv.dylib -lm >> ld: library not found for -lcrt1.10.5.o >> collect2: ld returned 1 exit status >> ld: library not found for -lcrt1.10.5.o >> collect2: ld returned 1 exit status >> lipo: can't open input file: /var/folders/i4/i4lSVTRWHsaTXG7UXMfGbE+ >> ++TI/-Tmp-//ccDzwBUG.out (No such file or directory) >> make[2]: *** [xmllint] Error 1 >> make[1]: *** [all-recursive] Error 1 >> make: *** [all] Error 2 >> Traceback (most recent call last): >> File "setup.py", line 107, in ? >> STATIC_CFLAGS, STATIC_BINARIES), >> File "/Users/paul/sandboxes/lxml/trunk/setupinfo.py", line 46, in >> ext_modules >> libxslt_version=OPTION_LIBXSLT_VERSION) >> File "/Users/paul/sandboxes/lxml/trunk/buildlibxml.py", line 190, >> in build_libxml2xslt >> call_subprocess( >> File "/Users/paul/sandboxes/lxml/trunk/buildlibxml.py", line 151, >> in call_subprocess >> raise Exception('Command "%s" returned code %s' % (cmd_desc, >> returncode)) >> Exception: Command "make" returned code 2 >> > > Hmm, is this with the latest version from trunk? From paul at agendaless.com Mon Nov 17 17:36:10 2008 From: paul at agendaless.com (Paul Everitt) Date: Mon, 17 Nov 2008 11:36:10 -0500 Subject: [lxml-dev] Fwd: lxml Mac installation idea In-Reply-To: <9F695713-8E93-42B7-AD6D-889F0697835D@it-loops.com> References: <3D8E108D-1E2D-4917-927B-9D4950B13ABF@it-loops.com> <9F695713-8E93-42B7-AD6D-889F0697835D@it-loops.com> Message-ID: On Nov 17, 2008, at 11:20 AM, Michael Guntsche wrote: > CC'ing the list. > > Begin forwarded message: > >> From: Michael Guntsche >> Date: November 17, 2008 17:08:48 GMT+01:00 >> To: Paul Everitt >> Subject: Re: [lxml-dev] lxml Mac installation idea >> >> >> On Nov 17, 2008, at 16:13, Paul Everitt wrote: >> >>> >>> I got: >>> >>> $ python setup.py build --static-deps >>> [.........snip.......] >>> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk - >>> O2 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch >>> -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized - >>> Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings - >>> Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested- >>> externs -Winline -Wredundant-decls -arch ppc -arch i386 -isysroot / >>> Developer/SDKs/MacOSX10.4u.sdk -o xmllint xmllint.o ./.libs/ >>> libxml2.a -lpthread -lz /usr/lib/libiconv.dylib -lm >>> ld: library not found for -lcrt1.10.5.o >>> collect2: ld returned 1 exit status >>> ld: library not found for -lcrt1.10.5.o >>> collect2: ld returned 1 exit status >>> lipo: can't open input file: /var/folders/i4/i4lSVTRWHsaTXG7UXMfGbE+ >>> ++TI/-Tmp-//ccDzwBUG.out (No such file or directory) >>> make[2]: *** [xmllint] Error 1 >>> make[1]: *** [all-recursive] Error 1 >>> make: *** [all] Error 2 >>> Traceback (most recent call last): >>> File "setup.py", line 107, in ? >>> STATIC_CFLAGS, STATIC_BINARIES), >>> File "/Users/paul/sandboxes/lxml/trunk/setupinfo.py", line 46, in >>> ext_modules >>> libxslt_version=OPTION_LIBXSLT_VERSION) >>> File "/Users/paul/sandboxes/lxml/trunk/buildlibxml.py", line 190, >>> in build_libxml2xslt >>> call_subprocess( >>> File "/Users/paul/sandboxes/lxml/trunk/buildlibxml.py", line 151, >>> in call_subprocess >>> raise Exception('Command "%s" returned code %s' % (cmd_desc, >>> returncode)) >>> Exception: Command "make" returned code 2 >>> >> >> Hmm, is this with the latest version from trunk? Yes: (lxmltrunk)bobo:trunk paul$ svn up At revision 59957. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20081117/6780205d/attachment.htm From mike at it-loops.com Mon Nov 17 17:36:31 2008 From: mike at it-loops.com (Michael Guntsche) Date: Mon, 17 Nov 2008 17:36:31 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49216A53.5060004@simplistix.co.uk> Message-ID: <5CF4E08E-B98C-4864-BDB9-C04D64BAA032@it-loops.com> Hello Paul, Please try with the attached patch applied.... I get the same problems if I build libxml on the command-line, do not understand why it is working when I am compiling with python and subprocess. Tell me if the attached patch works for you. Kind regards, Michael Index: buildlibxml.py =================================================================== --- buildlibxml.py (revision 59948) +++ buildlibxml.py (working copy) @@ -188,7 +188,7 @@ ] call_subprocess(configure_cmd, cwd=libxml2_dir, **call_setup) call_subprocess( - ['make'], cwd=libxml2_dir) + ['make'], cwd=libxml2_dir, **call_setup) call_subprocess( ['make', 'install'], cwd=libxml2_dir) @@ -202,7 +202,7 @@ call_subprocess(libxslt_configure_cmd, cwd=libxslt_dir, **call_setup) call_subprocess( - ['make'], cwd=libxslt_dir) + ['make'], cwd=libxslt_dir, **call_setup) call_subprocess( ['make', 'install'], cwd=libxslt_dir) From stefan_ml at behnel.de Mon Nov 17 17:58:53 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 17 Nov 2008 17:58:53 +0100 (CET) Subject: [lxml-dev] specifying lxml as a dependency? In-Reply-To: <49216979.9080401@simplistix.co.uk> References: <49174A17.4080607@simplistix.co.uk> <49174CE6.60105@colorstudy.com> <4917F43F.30307@simplistix.co.uk> <52065.213.61.181.86.1226309133.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49216979.9080401@simplistix.co.uk> Message-ID: <36304.213.61.181.86.1226941133.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Chris Withers wrote: >> What's relevant here is that the setup.py in the current trunk knows how >> to download and build libxml2 and libxslt, and how to build lxml >> statically using the result. That's specifically made (but not only >> meant) >> for Mac users that run into crashes due to conflicting library versions. > > Ah okay, interesting, would it work on linux too if libxml2 and libxslt > aren't available? It already does, I'm testing it on Linux. Stefan From stefan_ml at behnel.de Mon Nov 17 18:00:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 17 Nov 2008 18:00:27 +0100 (CET) Subject: [lxml-dev] XmlTextReader support In-Reply-To: <6CC8E8868854DA4CB1C65DC49156F084E34A37@MTL-MSG-03.cgiclients.com> References: <6CC8E8868854DA4CB1C65DC49156F084E34A37@MTL-MSG-03.cgiclients.com> Message-ID: <36978.213.61.181.86.1226941227.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Roy, Remy wrote: > Is there any roadmap to provide support for the XmlTextReader interface as > specified on this page http://www.xmlsoft.org/xmlreader.html in lxml? No, but I always take patches. May I ask why iterparse() doesn't suite your needs? Stefan From ianb at colorstudy.com Mon Nov 17 18:18:32 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 17 Nov 2008 11:18:32 -0600 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <491FD83B.8070803@behnel.de> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> <491FD83B.8070803@behnel.de> Message-ID: <4921A768.90602@colorstudy.com> Stefan Behnel wrote: > Guntsche Michael wrote: >> Here the last incarnation of my patch. I got rid of setting the >> environment variables and deleting them afterwards. >> Should have done it like this in the first place. > > Thanks a bunch. I rewrote the patch somewhat to restrict the changes to > MacOS. If I didn't break it by doing that, the current trunk should work > out-of-the-box now, if you pass "--static-deps" to setup.py, or set > "STATIC_DEPS=true" in your env variables. > > Anyone wants to give it a try? I installed it on 10.5, with the system Python: $ uname -a Darwin mouna.openplans.org 9.5.0 Darwin Kernel Version 9.5.0: Wed Sep 3 11:29:43 PDT 2008; root:xnu-1228.7.58~1/RELEASE_I386 i386 $ sw_vers ProductName: Mac OS X ProductVersion: 10.5.5 BuildVersion: 9F33 I had to run "make realclean" first to clean out the previous build (which had iconv problems). For some reason the tests won't run because it can't find the library (I installed it in a virtualenv), even though when I run python directly lxml imports just fine. selftest.py and selftest2.py run and work, though (I just noticed those, I'm not clear what they do?) -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From mike at it-loops.com Mon Nov 17 18:19:55 2008 From: mike at it-loops.com (Michael Guntsche) Date: Mon, 17 Nov 2008 18:19:55 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <5CF4E08E-B98C-4864-BDB9-C04D64BAA032@it-loops.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49216A53.5060004@simplistix.co.uk> <5CF4E08E-B98C-4864-BDB9-C04D64BAA032@it-loops.com> Message-ID: <08B84C77-4C60-476D-9998-61D05267826A@it-loops.com> On Nov 17, 2008, at 17:36, Michael Guntsche wrote: > > call_subprocess( > - ['make'], cwd=libxml2_dir) > + ['make'], cwd=libxml2_dir, **call_setup) > call_subprocess( > ['make', 'install'], cwd=libxml2_dir) > Hello Stefan, We need to set MACOSX_DEPLOYMENT_TARGET during "make" as well, otherwise we get linking errors. I am still trying to figure out why the errors do not happen on my system here. In the first patch I just added **call_setup to the call_subprocess invocation but this way the CFLAGS and LDFLAGS get passed as well once again. Do you have a better idea, should we just pass MACOSX_DEPLOYMENT_TARGEt alone? Kind regards, Michael From mike at it-loops.com Mon Nov 17 18:25:38 2008 From: mike at it-loops.com (Michael Guntsche) Date: Mon, 17 Nov 2008 18:25:38 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <0DDEF17F-CAB9-4ADA-ADFF-D2E5494D8C02@agendaless.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49216A53.5060004@simplistix.co.uk> <5CF4E08E-B98C-4864-BDB9-C04D64BAA032@it-loops.com> <0DDEF17F-CAB9-4ADA-ADFF-D2E5494D8C02@agendaless.com> Message-ID: <01C9EDF3-EC59-47CA-81DF-0DD54B652C79@it-loops.com> On Nov 17, 2008, at 18:18, Paul Everitt wrote: > > Unfortunately no change. > > --Paul > > On Nov 17, 2008, at 11:36 AM, Michael Guntsche wrote: > >> >> Hello Paul, >> >> Please try with the attached patch applied.... >> I get the same problems if I build libxml on the command-line, do not >> understand why it is working when I am compiling with python and >> subprocess. >> Tell me if the attached patch works for you. >> >> Kind regards, >> Michael >> >> >> Index: buildlibxml.py >> =================================================================== >> --- buildlibxml.py (revision 59948) >> +++ buildlibxml.py (working copy) >> @@ -188,7 +188,7 @@ >> ] >> call_subprocess(configure_cmd, cwd=libxml2_dir, **call_setup) >> call_subprocess( >> - ['make'], cwd=libxml2_dir) >> + ['make'], cwd=libxml2_dir, **call_setup) >> call_subprocess( >> ['make', 'install'], cwd=libxml2_dir) >> >> @@ -202,7 +202,7 @@ >> >> call_subprocess(libxslt_configure_cmd, cwd=libxslt_dir, >> **call_setup) >> call_subprocess( >> - ['make'], cwd=libxslt_dir) >> + ['make'], cwd=libxslt_dir, **call_setup) >> call_subprocess( >> ['make', 'install'], cwd=libxslt_dir) >> Paul, please completely clean up everything from the previous build (best just make a clean checkout) and try again. If this still does not work set MACOSX_DEPLOYMENT_TARGET=10.3 before starting the compilation. $ export MACOSX_DEPLOYMENT_TARGET=10.3 /Mike From stefan_ml at behnel.de Mon Nov 17 11:28:14 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 17 Nov 2008 11:28:14 +0100 Subject: [lxml-dev] Leaking tracebacks In-Reply-To: <4920BFEE.6090906@cheimes.de> References: <491DB1AE.9060809@cheimes.de> <491ED5FB.3020203@behnel.de> <4920BFEE.6090906@cheimes.de> Message-ID: <4921473E.5050302@behnel.de> Hi, Christian Heimes wrote: > I was able to come up with a minimal test case. It shows that lxml 2.0.7 > is fine but 2.0.9 and 2.1.2 are leaking references somewhere. Thanks, that made it really easy to track down. The modified exception handling code really leaks references to class, value and traceback when no exception is raised. I'll look into fixing it. Stefan From stefan_ml at behnel.de Mon Nov 17 19:32:33 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 17 Nov 2008 19:32:33 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <4921A768.90602@colorstudy.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> <491FD83B.8070803@behnel.de> <4921A768.90602@colorstudy.com> Message-ID: <4921B8C1.1010306@behnel.de> Hi, Ian Bicking wrote: > I installed it on 10.5, with the system Python: > > $ uname -a > Darwin mouna.openplans.org 9.5.0 Darwin Kernel Version 9.5.0: Wed Sep 3 > 11:29:43 PDT 2008; root:xnu-1228.7.58~1/RELEASE_I386 i386 > $ sw_vers > ProductName: Mac OS X > ProductVersion: 10.5.5 > BuildVersion: 9F33 > > I had to run "make realclean" first to clean out the previous build > (which had iconv problems). For some reason the tests won't run because > it can't find the library (I installed it in a virtualenv), even though > when I run python directly lxml imports just fine. I guess that's acceptable for now, unless it proves to be a general problem. > selftest.py and > selftest2.py run and work, though (I just noticed those, I'm not clear > what they do?) Those are copied from ElementTree for compatibility purposes. Not particularly well portable tests (e.g. tests for XML byte sequence equality), but they do mostly well in lxml. Stefan From paul at agendaless.com Mon Nov 17 19:33:40 2008 From: paul at agendaless.com (Paul Everitt) Date: Mon, 17 Nov 2008 13:33:40 -0500 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <01C9EDF3-EC59-47CA-81DF-0DD54B652C79@it-loops.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49216A53.5060004@simplistix.co.uk> <5CF4E08E-B98C-4864-BDB9-C04D64BAA032@it-loops.com> <0DDEF17F-CAB9-4ADA-ADFF-D2E5494D8C02@agendaless.com> <01C9EDF3-EC59-47CA-81DF-0DD54B652C79@it-loops.com> Message-ID: <5E553A58-7C59-496A-A15B-7EE560B39B48@agendaless.com> On Nov 17, 2008, at 12:25 PM, Michael Guntsche wrote: > > Paul, please completely clean up everything from the previous build > (best just make a clean checkout) and try again. I blew away my trunk directory, re-checked it out, and put the buildlibxml.py with your two changes from the patch back in place. No luck, same error. > If this still does not work set MACOSX_DEPLOYMENT_TARGET=10.3 before > starting the compilation. > > $ export MACOSX_DEPLOYMENT_TARGET=10.3 I got a new trunk, the patched buildlibxml.py, and very quickly got: $ python setup.py build --static-deps Traceback (most recent call last): File "setup.py", line 13, in ? import pkg_resources File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 657, in ? class Environment(object): File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 660, in Environment def __init__(self, search_path=None, platform=get_supported_platform(), python=PY_MAJOR): File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 55, in get_supported_platform plat = get_build_platform(); m = macosVersionString.match(plat) File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 187, in get_build_platform plat = get_platform() File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ util.py", line 78, in get_platform cfgvars = get_config_vars() File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ sysconfig.py", line 488, in get_config_vars func() File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ sysconfig.py", line 373, in _init_posix raise DistutilsPlatformError(my_msg) distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but "10.5" during configure --Paul From lists at cheimes.de Mon Nov 17 19:37:17 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 17 Nov 2008 19:37:17 +0100 Subject: [lxml-dev] Leaking tracebacks In-Reply-To: <4921473E.5050302@behnel.de> References: <491DB1AE.9060809@cheimes.de> <491ED5FB.3020203@behnel.de> <4920BFEE.6090906@cheimes.de> <4921473E.5050302@behnel.de> Message-ID: Stefan Behnel wrote: > Thanks, that made it really easy to track down. The modified exception > handling code really leaks references to class, value and traceback when no > exception is raised. > > I'll look into fixing it. Excellent! Can I expect lxml 2.0.10 and 2.1.3 anytime soon? Christian From mike at it-loops.com Mon Nov 17 19:39:41 2008 From: mike at it-loops.com (Michael Guntsche) Date: Mon, 17 Nov 2008 19:39:41 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <5E553A58-7C59-496A-A15B-7EE560B39B48@agendaless.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49216A53.5060004@simplistix.co.uk> <5CF4E08E-B98C-4864-BDB9-C04D64BAA032@it-loops.com> <0DDEF17F-CAB9-4ADA-ADFF-D2E5494D8C02@agendaless.com> <01C9EDF3-EC59-47CA-81DF-0DD54B652C79@it-loops.com> <5E553A58-7C59-496A-A15B-7EE560B39B48@agendaless.com> Message-ID: On Nov 17, 2008, at 19:33, Paul Everitt wrote: >> > > I got a new trunk, the patched buildlibxml.py, and very quickly got: > > $ python setup.py build --static-deps > Traceback (most recent call last): > File "setup.py", line 13, in ? > import pkg_resources > File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ > setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 657, in ? > class Environment(object): > File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ > setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 660, in Environment > def __init__(self, search_path=None, > platform=get_supported_platform(), python=PY_MAJOR): > File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ > setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 55, in > get_supported_platform > plat = get_build_platform(); m = macosVersionString.match(plat) > File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ > setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 187, in > get_build_platform > plat = get_platform() > File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ > util.py", line 78, in get_platform > cfgvars = get_config_vars() > File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ > sysconfig.py", line 488, in get_config_vars > func() > File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ > sysconfig.py", line 373, in _init_posix > raise DistutilsPlatformError(my_msg) > distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET > mismatch: now "10.3" but "10.5" during configure > Ah yes, great so the python you where using was build with a TARGET of 10.5 which is just great. Just for testing try to replace 10.3 with 10.5 and see if it compiles. BTW do you get the same error if you use the non-patched version? /Mike From stefan_ml at behnel.de Mon Nov 17 19:42:55 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 17 Nov 2008 19:42:55 +0100 Subject: [lxml-dev] Leaking tracebacks In-Reply-To: References: <491DB1AE.9060809@cheimes.de> <491ED5FB.3020203@behnel.de> <4920BFEE.6090906@cheimes.de> <4921473E.5050302@behnel.de> Message-ID: <4921BB2F.7080204@behnel.de> Hi, Christian Heimes wrote: > Stefan Behnel wrote: >> Thanks, that made it really easy to track down. The modified exception >> handling code really leaks references to class, value and traceback when no >> exception is raised. >> >> I'll look into fixing it. > > Excellent! Can I expect lxml 2.0.10 and 2.1.3 anytime soon? 2.1.3 was due weeks ago, so, yes, there will be bug fix releases pretty soon. Stefan From mike at it-loops.com Mon Nov 17 19:51:02 2008 From: mike at it-loops.com (Michael Guntsche) Date: Mon, 17 Nov 2008 19:51:02 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <5E553A58-7C59-496A-A15B-7EE560B39B48@agendaless.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49216A53.5060004@simplistix.co.uk> <5CF4E08E-B98C-4864-BDB9-C04D64BAA032@it-loops.com> <0DDEF17F-CAB9-4ADA-ADFF-D2E5494D8C02@agendaless.com> <01C9EDF3-EC59-47CA-81DF-0DD54B652C79@it-loops.com> <5E553A58-7C59-496A-A15B-7EE560B39B48@agendaless.com> Message-ID: On Nov 17, 2008, at 19:33, Paul Everitt wrote: > I got a new trunk, the patched buildlibxml.py, and very quickly got: > > $ python setup.py build --static-deps > Traceback (most recent call last): > File "setup.py", line 13, in ? > import pkg_resources > File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ > setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 657, in ? > class Environment(object): > File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ > setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 660, in Environment > def __init__(self, search_path=None, > platform=get_supported_platform(), python=PY_MAJOR): > File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ > setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 55, in > get_supported_platform > plat = get_build_platform(); m = macosVersionString.match(plat) > File "/Users/paul/venvs/lxmltrunk/lib/python2.4/site-packages/ > setuptools-0.6c9-py2.4.egg/pkg_resources.py", line 187, in > get_build_platform > plat = get_platform() > File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ > util.py", line 78, in get_platform > cfgvars = get_config_vars() > File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ > sysconfig.py", line 488, in get_config_vars > func() > File "/Users/paul/opt/Python-2.4.5/lib/python2.4/distutils/ > sysconfig.py", line 373, in _init_posix > raise DistutilsPlatformError(my_msg) > distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET > mismatch: now "10.3" but "10.5" during configure > Here a new patch that should fix the mismatch... Please try the following: 1.) Only apply the first part of the patch that changes the TARGET Try to compile that 2.) Apply the complete patch and try to compile it. Thank you very much for testing this. I really cannot reproduce this in my testing rig here.... Index: buildlibxml.py =================================================================== --- buildlibxml.py (revision 59948) +++ buildlibxml.py (working copy) @@ -174,7 +174,7 @@ call_setup['env'] = { 'CFLAGS' : "-arch ppc -arch i386 -isysroot / Developer/SDKs/MacOSX10.4u.sdk -O2", 'LDFLAGS' : "-arch ppc -arch i386 -isysroot / Developer/SDKs/MacOSX10.4u.sdk", - 'MACOSX_DEPLOYMENT_TARGET' : "10.3" + 'MACOSX_DEPLOYMENT_TARGET' : sysconfig.parse_makefile(sysconfig.get_makefile_filename()) ['MACOSX_DEPLOYMENT_TARGET'] } # We may loose the link to iconv, so make sure it's there @@ -188,7 +188,7 @@ ] call_subprocess(configure_cmd, cwd=libxml2_dir, **call_setup) call_subprocess( - ['make'], cwd=libxml2_dir) + ['make'], cwd=libxml2_dir, **call_setup) call_subprocess( ['make', 'install'], cwd=libxml2_dir) @@ -202,7 +202,7 @@ call_subprocess(libxslt_configure_cmd, cwd=libxslt_dir, **call_setup) call_subprocess( - ['make'], cwd=libxslt_dir) + ['make'], cwd=libxslt_dir, **call_setup) call_subprocess( ['make', 'install'], cwd=libxslt_dir) From remy.roy at cgi.com Mon Nov 17 19:52:54 2008 From: remy.roy at cgi.com (Roy, Remy) Date: Mon, 17 Nov 2008 13:52:54 -0500 Subject: [lxml-dev] XmlTextReader support In-Reply-To: <36978.213.61.181.86.1226941227.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <6CC8E8868854DA4CB1C65DC49156F084E34A37@MTL-MSG-03.cgiclients.com> <36978.213.61.181.86.1226941227.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <6CC8E8868854DA4CB1C65DC49156F084E6CFCB@MTL-MSG-03.cgiclients.com> > No, but I always take patches. May I ask why iterparse() doesn't suite > your needs? >From my understanding, iterparse does not work well with streams which is what I need to work with large xml content. The suggested solution with lxml for this problem seems to be the SAX-like events with a target parse on lxml.etree. The SAX interface could eventually be of use for my application but it would need many workarounds where as the XmlTextReader interface would naturally fit in. From stefan_ml at behnel.de Mon Nov 17 20:21:26 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 17 Nov 2008 20:21:26 +0100 Subject: [lxml-dev] XmlTextReader support In-Reply-To: <6CC8E8868854DA4CB1C65DC49156F084E6CFCB@MTL-MSG-03.cgiclients.com> References: <6CC8E8868854DA4CB1C65DC49156F084E34A37@MTL-MSG-03.cgiclients.com> <36978.213.61.181.86.1226941227.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <6CC8E8868854DA4CB1C65DC49156F084E6CFCB@MTL-MSG-03.cgiclients.com> Message-ID: <4921C436.4010709@behnel.de> Hi, Roy, Remy wrote: >> No, but I always take patches. May I ask why iterparse() doesn't suite >> your needs? > > From my understanding, iterparse does not work well with streams which > is what I need to work with large xml content. Who told you that? You can parse from any file-like object, you just have to care for the intermediate cleanup yourself. That's sort-of the opposite of the Reader interface, where you have to care about what you read or skip. iterparse() even gives you a real tree along the road. > The suggested solution with lxml for this problem seems to be the > SAX-like events with a target parse on lxml.etree. > The SAX interface could eventually be of use for my application but it > would need many workarounds where as the XmlTextReader interface would > naturally fit in. You should definitely try iterparse() before you go for a plain SAX solution. Stefan From lxml at bestley.co.uk Mon Nov 17 20:26:44 2008 From: lxml at bestley.co.uk (Mark Bestley) Date: Mon, 17 Nov 2008 19:26:44 +0000 Subject: [lxml-dev] lxml Mac installation idea References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> <491FD83B.8070803@behnel.de> Message-ID: Stefan Behnel writes: > Hi, > > Guntsche Michael wrote: >> Here the last incarnation of my patch. I got rid of setting the >> environment variables and deleting them afterwards. >> Should have done it like this in the first place. > > Thanks a bunch. I rewrote the patch somewhat to restrict the changes to > MacOS. If I didn't break it by doing that, the current trunk should work > out-of-the-box now, if you pass "--static-deps" to setup.py, or set > "STATIC_DEPS=true" in your env variables. > > Anyone wants to give it a try? > I have tried with mixed sucess Failed with Apple Python In the 'normal setup.py' bit gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -O2 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized -Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Winline -Wredundant-decls -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -o xmllint xmllint.o ./.libs/libxml2.a -lpthread -lz /usr/lib/libiconv.dylib -lm ld: library not found for -lcrt1.10.5.o collect2: ld returned 1 exit status Exception: Command "make" returned code 2 If the -isysroot /Developer/SDKs/MacOSX10.4u.sdk is removed then it worked worked with python.org 2.6 version dual architecture macports single arcitecture python I have also tested these two with no isysroot and that worked.q Now I don't understand how the parameters got passed through to the setup.py as Stefan's code only passes it to the configure builds. It seems like the environment is globally set Also should the '--disable-dependency-tracking' parameter to configure_cmd be only for darwin? -- Mark From stefan_ml at behnel.de Mon Nov 17 22:30:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 17 Nov 2008 22:30:57 +0100 Subject: [lxml-dev] lxml 2.0.10 and 2.1.3 released Message-ID: <4921E291.6030507@behnel.de> Hi, I just released lxml 2.0.10 and lxml 2.1.3. Both releases are pure bug-fix releases for the stable 2.1 series and the mature 2.0 series. The ChangeLog follows below. As there appears to be a problem with PyPI's upload currently, you can get them from the homepage for now: http://codespeak.net/lxml/lxml-2.1.3.tgz http://codespeak.net/lxml/2.0/lxml-2.0.10.tgz Have fun, Stefan 2.0.10 (2008-11-17) Bugs fixed * Ref-count leaks when lxml enters a try-except statement while an outside exception lives in sys.exc_*(). This was due to a problem in Cython, not lxml itself. 2.1.3 (2008-11-17) Bugs fixed * Ref-count leaks when lxml enters a try-except statement while an outside exception lives in sys.exc_*(). This was due to a problem in Cython, not lxml itself. * Parser Unicode decoding errors could get swallowed by other exceptions. * Name/import errors in some Python modules. * Internal DTD subsets that did not specify a system or public ID were not serialised and did not appear in the docinfo property of ElementTrees. * Fix a pre-Py3k warning when parsing from a gzip file in Py2.6. * Test suite fixes for libxml2 2.7. * Resolver.resolve_string() did not work for non-ASCII byte strings. * Resolver.resolve_file() was broken. * Overriding the parser encoding didn't work for many encodings. From mike at it-loops.com Mon Nov 17 22:35:47 2008 From: mike at it-loops.com (Michael Guntsche) Date: Mon, 17 Nov 2008 22:35:47 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> <491FD83B.8070803@behnel.de> Message-ID: <25D7F984-5453-4F53-9488-2294AF503C27@it-loops.com> Hello Mark and Paul, Ok, I admit defeat. So you have a deployment target of 10.3 or 10.5 the linker accepts -isysroot but in later versions needs -isystem. Final try then I give up. We only set isysroot/isystem in the beginning since this is needed to compile universal on PPC. If you use this switch you also need to set the M_D_T environment variable. Just forget it, only set the two archs and be done with it. This means compiling it like this won't work on PPC machines but heeeyyy you cannot have everything. :) Please apply this patch against the latest version in trunk, Stefan made an update in the meantime, and test it once more. Kind regards, Mike "I need a beer now" Index: buildlibxml.py =================================================================== --- buildlibxml.py (revision 59970) +++ buildlibxml.py (working copy) @@ -172,9 +172,7 @@ major_version = int(os.uname()[2].split('.')[0]) if major_version > 7: call_setup['env'] = { - 'CFLAGS' : "-arch ppc -arch i386 -isysroot /Developer/ SDKs/MacOSX10.4u.sdk -O2", - 'LDFLAGS' : "-arch ppc -arch i386 -isysroot / Developer/SDKs/MacOSX10.4u.sdk", - 'MACOSX_DEPLOYMENT_TARGET' : "10.3" + 'CFLAGS' : "-arch ppc -arch i386 -O2" } # We may loose the link to iconv, so make sure it's there From mike at it-loops.com Tue Nov 18 15:58:46 2008 From: mike at it-loops.com (Michael Guntsche) Date: Tue, 18 Nov 2008 15:58:46 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <25D7F984-5453-4F53-9488-2294AF503C27@it-loops.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> <491FD83B.8070803@behnel.de> <25D7F984-5453-4F53-9488-2294AF503C27@it-loops.com> Message-ID: <7E8C3AC1-2462-4554-BAC3-3AE4E1C3BF68@it-loops.com> On Nov 17, 2008, at 22:35, Michael Guntsche wrote: > Hello Mark and Paul, > > Ok, I admit defeat. So you have a deployment target of 10.3 or 10.5 > the linker accepts -isysroot but in later versions needs -isystem. Answering my own E-mail here. I tried to compile it again with the current trunk version * Stock python from Leopard (universal): OK * Python from python.org (universal): OK * Macports python (i386 only): OK So for me the current build script works without any problems with 3 different installations. Kind regards, Michael From stefan_ml at behnel.de Wed Nov 19 08:58:46 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 19 Nov 2008 08:58:46 +0100 Subject: [lxml-dev] lxml 2.0.10 and 2.1.3 released In-Reply-To: <4921E291.6030507@behnel.de> References: <4921E291.6030507@behnel.de> Message-ID: <4923C736.4090409@behnel.de> Stefan Behnel wrote: > As there appears to be a problem with PyPI's upload currently, you can get > them from the homepage for now PyPI is back, so you can get the source distributions from there now. Stefan From ross at kallisti.us Wed Nov 19 21:19:35 2008 From: ross at kallisti.us (Ross Vandegrift) Date: Wed, 19 Nov 2008 15:19:35 -0500 Subject: [lxml-dev] Parsing this document Message-ID: <20081119201935.GA6026@kallisti.us> Hi everyone, Sorry to as a -users question on a -dev mailing-list, but I don't see a -users resource for lxml. If one exists, I'd be happy to head over there. I have an XML document from a network device that I want to pull into my python code. The example document I'm working with is attached as vlan.xml. I find the results of this to be utterly perplexing: >>> import lxml.etree as ET >>> vtree = ET.parse("vlan.xml") >>> vtree.xpath("vlan") [] >>> vtree.xpath("//vlan") [] >>> vtree.xpath("rpc-reply") [] >>> vtree.xpath("/rpc-reply") [] >>> vtree.xpath("*") [] >>> vtree.xpath("vlan-information") [] Based on the examples in the lxml XPath documentation, I'd expect the first to give me a list of Elements, one for each ... in the file. Second should be the same based on XPath stuff I've read in Python and XML. My next two are shots in the dark: is the top-level element, but I can't retrieve it relatively or absolutely. Next is another shot in the dark, and somehow what I get the element one under the top-level. Huh? Also, I can't retrieve it by name. -- Ross Vandegrift ross at kallisti.us "If the fight gets hot, the songs get hotter. If the going gets tough, the songs get tougher." --Woody Guthrie -------------- next part -------------- A non-text attachment was scrubbed... Name: vlan.xml Type: application/xml Size: 9027 bytes Desc: not available Url : http://codespeak.net/pipermail/lxml-dev/attachments/20081119/fa3a8e68/attachment.xml From jkrukoff at ltgc.com Wed Nov 19 21:25:27 2008 From: jkrukoff at ltgc.com (John Krukoff) Date: Wed, 19 Nov 2008 13:25:27 -0700 Subject: [lxml-dev] Parsing this document In-Reply-To: <20081119201935.GA6026@kallisti.us> References: <20081119201935.GA6026@kallisti.us> Message-ID: <1227126327.4300.95.camel@jmk> On Wed, 2008-11-19 at 15:19 -0500, Ross Vandegrift wrote: > >>> vtree.xpath("*") > [] >From this result, it shows that at least your root element has a namespace. Take a look at this section of the documentation: http://codespeak.net/lxml/tutorial.html#namespaces -- John Krukoff Land Title Guarantee Company From stefan_ml at behnel.de Thu Nov 20 22:58:39 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 20 Nov 2008 22:58:39 +0100 Subject: [lxml-dev] [Lxml-checkins] r60038 - lxml/trunk - make static build the default In-Reply-To: <20081120192548.2A5D71684B2@codespeak.net> References: <20081120192548.2A5D71684B2@codespeak.net> Message-ID: <4925DD8F.4040800@behnel.de> Hi Ian, ianb wrote: > Modified: lxml/trunk/setupinfo.py > ============================================================================== > --- lxml/trunk/setupinfo.py (original) > +++ lxml/trunk/setupinfo.py Thu Nov 20 20:25:47 2008 > @@ -317,7 +317,7 @@ > OPTION_STATIC = has_option('static') > OPTION_DEBUG_GCC = has_option('debug-gcc') > OPTION_AUTO_RPATH = has_option('auto-rpath') > -OPTION_BUILD_LIBXML2XSLT = has_option('static-deps') > +OPTION_BUILD_LIBXML2XSLT = has_option('static-deps', 'true') > if OPTION_BUILD_LIBXML2XSLT: > OPTION_STATIC = True > OPTION_LIBXML2_VERSION = option_value('libxml2-version') I'm not sure this should become the default. A dynamic build is generally the right thing and works very well on most platforms, except for Windows and MacOS-X. I wouldn't mind defaulting it to 'true' explicitly on those two platforms, though. MacOS-X packagers can just pass the respective env var to disable it. Stefan From ianb at colorstudy.com Thu Nov 20 23:01:13 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 20 Nov 2008 16:01:13 -0600 Subject: [lxml-dev] [Lxml-checkins] r60038 - lxml/trunk - make static build the default In-Reply-To: <4925DD8F.4040800@behnel.de> References: <20081120192548.2A5D71684B2@codespeak.net> <4925DD8F.4040800@behnel.de> Message-ID: <4925DE29.1020608@colorstudy.com> Stefan Behnel wrote: > Hi Ian, > > ianb wrote: >> Modified: lxml/trunk/setupinfo.py >> ============================================================================== >> --- lxml/trunk/setupinfo.py (original) >> +++ lxml/trunk/setupinfo.py Thu Nov 20 20:25:47 2008 >> @@ -317,7 +317,7 @@ >> OPTION_STATIC = has_option('static') >> OPTION_DEBUG_GCC = has_option('debug-gcc') >> OPTION_AUTO_RPATH = has_option('auto-rpath') >> -OPTION_BUILD_LIBXML2XSLT = has_option('static-deps') >> +OPTION_BUILD_LIBXML2XSLT = has_option('static-deps', 'true') >> if OPTION_BUILD_LIBXML2XSLT: >> OPTION_STATIC = True >> OPTION_LIBXML2_VERSION = option_value('libxml2-version') > > I'm not sure this should become the default. A dynamic build is generally > the right thing and works very well on most platforms, except for Windows > and MacOS-X. > > I wouldn't mind defaulting it to 'true' explicitly on those two platforms, > though. MacOS-X packagers can just pass the respective env var to disable it. Sorry, I accidentally committed this along with the change to MANIFEST.in (which is what I intended to commit) when I was making a tarball with this defaulted on. I'll revert. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From stefan_ml at behnel.de Fri Nov 21 21:32:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 21 Nov 2008 21:32:36 +0100 Subject: [lxml-dev] lxml Mac installation idea In-Reply-To: <7E8C3AC1-2462-4554-BAC3-3AE4E1C3BF68@it-loops.com> References: <49173B9C.6000308@simplistix.co.uk> <491755A6.6020809@behnel.de> <4917F557.1010605@simplistix.co.uk> <46003.213.61.181.86.1226311483.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2D7CC2F0-A37A-4EE0-A808-CAA5F8086B96@it-loops.com> <4919EF46.50504@behnel.de> <016DD0B8-360A-44C4-9AF1-26E29BC2514C@it-loops.com> <491B2A62.9080200@behnel.de> <01AFC5CF-9F2A-44D1-AE31-1967176B0F45@it-loops.com> <349cf98f3614225d778caca46b530644@localhost> <0A2F2EAB-D627-45E6-8AD6-8C0E9850F634@it-loops.com> <491FD83B.8070803@behnel.de> <25D7F984-5453-4F53-9488-2294AF503C27@it-loops.com> <7E8C3AC1-2462-4554-BAC3-3AE4E1C3BF68@it-loops.com> Message-ID: <49271AE4.2070505@behnel.de> Hi, Michael Guntsche wrote: > I tried to compile it again with the current trunk version > > * Stock python from Leopard (universal): OK > * Python from python.org (universal): OK > * Macports python (i386 only): OK > > So for me the current build script works without any problems with 3 > different installations. Cool. Thanks a lot for working this out! Stefan From bkc at murkworks.com Sun Nov 23 02:28:11 2008 From: bkc at murkworks.com (Brad Clements) Date: Sat, 22 Nov 2008 20:28:11 -0500 Subject: [lxml-dev] lxml is loading dtd from w3.org but I don't want it to Message-ID: <4928B1AB.5000209@murkworks.com> I cannot seem to disable loading DTD from w3.org when transforming a file. I suppose I am doing something wrong, but I can't see what that could be. I am using lxml inside a wsgi application, and every network request causes my wsgi server to go back to w3 to get the dtd (I am not using local catalogs). I am on ubuntu 7.xx (can't recall, 2 versions old) x86_64, using python 2.5 >>> sys.version '2.5.1 (r251:54863, Mar 7 2008, 03:39:23) \n[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)]' >>> lxml.etree.__version__ u'2.1.3' >>> lxml.etree.LIBXSLT_COMPILED_VERSION (1, 1, 21) >>> lxml.etree.LIBXML_COMPILED_VERSION (2, 6, 30) my code looks like this (I added all the =False keywords to see if they helped, they did not): parser = etree.XMLParser(load_dtd=False, attribute_defaults=False, dtd_validation=False) parser.resolvers.add(Resolver(resolver=xml_src_object.resolve)) xml_doc = etree.fromstring(xml_src_object.get_source(), parser, base_url=document_uri) also the xml or stylesheet source might be loaded this way: parser = etree.XMLParser(load_dtd=False, attribute_defaults=False, dtd_validation=False, no_network=True) stylesheet_doc = etree.parse(xslt_src_object, parser) where xslt_src_object is a string containing the filepath to parse. I was running an older lxml previously, but it was also downloading catalogs. I don't know what version that was, sorry. I upgraded to 2.1.3 today. If I do the transform with xsltproc, it goes fast and does not download the dtd. I have been working on this for a couple of hours, so I'm likely to have made a mistake on this. However this is code I've been using for more than a year and I doubt very much that it's always been downloading the dtd. I think I did an apt-get update/upgrade some weeks back, I might have gotten a newer libxml2. Perhaps it's not honoring the load_dtd=False ? any ideas? -- Brad Clements, bkc at murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM: BKClements From stefan_ml at behnel.de Sun Nov 23 08:44:44 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 23 Nov 2008 08:44:44 +0100 Subject: [lxml-dev] lxml is loading dtd from w3.org but I don't want it to In-Reply-To: <4928B1AB.5000209@murkworks.com> References: <4928B1AB.5000209@murkworks.com> Message-ID: <492909EC.2050804@behnel.de> Hi, Brad Clements wrote: > I cannot seem to disable loading DTD from w3.org when transforming a file. > [...] > If I do the transform with xsltproc, it goes fast and does not download > the dtd. I checked what xsltproc is doing different. It sets the parse options on the XSLT transform context. lxml didn't do that so far. Could you check if the attached patch works for you? Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: set-xslt-parse-options.patch Type: text/x-patch Size: 1013 bytes Desc: not available Url : http://codespeak.net/pipermail/lxml-dev/attachments/20081123/90644b96/attachment-0001.bin From stefan_ml at behnel.de Sun Nov 23 20:54:53 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 23 Nov 2008 20:54:53 +0100 Subject: [lxml-dev] lxml 2.2alpha1 released Message-ID: <4929B50D.9030203@behnel.de> Hi all, I'm happy to announce the release of the first official alpha version of lxml 2.2. It brings a couple of new features that mainly round up the existing tool set. http://pypi.python.org/pypi/lxml/2.2alpha1 http://codespeak.net/lxml/dev/ I considered making this a beta release, but I wanted to permit some more testing even before a feature freeze. So, as usual, all kinds of feedback are appreciated. Have fun, Stefan 2.2alpha1 (2008-11-23) Features added * Support for XSLT result tree fragments in XPath/XSLT extension functions. * QName objects have new properties namespace and localname. * New options for exclusive C14N and C14N without comments. * Instantiating a custom Element classes creates a new Element. Bugs fixed * XSLT didn't inherit the parse options of the input document. * 0-bytes could slip through the API when used inside of Unicode strings. * With lxml.html.clean.autolink, links with balanced parenthesis, that end in a parenthesis, will be linked in their entirety (typical with Wikipedia links). From lxml at bestley.co.uk Tue Nov 25 00:42:46 2008 From: lxml at bestley.co.uk (Mark Bestley) Date: Mon, 24 Nov 2008 23:42:46 +0000 Subject: [lxml-dev] lxml 2.2alpha1 released References: <4929B50D.9030203@behnel.de> Message-ID: Stefan Behnel writes: > Hi all, > > I'm happy to announce the release of the first official alpha version of > lxml 2.2. It brings a couple of new features that mainly round up the > existing tool set. > > Have fun, > Stefan > Having fun with packaging :( Runiing python setup.py from svn works However easy_install xml==dev gives Building lxml version 2.2.alpha1-60098. Building with Cython 0.10. Using build configuration of libxslt 1.1.23 Building against libxml2/libxslt in the following directory: /opt/local/lib warning: no files found matching 'lxml.etree.c' under directory 'src/lxml' warning: no files found matching 'lxml.objectify.c' under directory 'src/lxml' warning: no files found matching 'lxml.etree.h' under directory 'src/lxml' warning: no files found matching 'lxml.etree_api.h' under directory 'src/lxml' warning: no files found matching '*.html' under directory 'doc' i686-apple-darwin9-gcc-4.0.1: src/lxml/lxml.etree.c: No such file or directory i686-apple-darwin9-gcc-4.0.1: no input files I thought this could be due to my fiddling with the OSX setup but I also got a similar set of errors with Ubuntu 8.10 -- Mark From ross at kallisti.us Tue Nov 25 17:55:14 2008 From: ross at kallisti.us (Ross Vandegrift) Date: Tue, 25 Nov 2008 11:55:14 -0500 Subject: [lxml-dev] Parsing this document In-Reply-To: <1227126327.4300.95.camel@jmk> References: <20081119201935.GA6026@kallisti.us> <1227126327.4300.95.camel@jmk> Message-ID: <20081125165514.GA19436@kallisti.us> On Wed, Nov 19, 2008 at 01:25:27PM -0700, John Krukoff wrote: > On Wed, 2008-11-19 at 15:19 -0500, Ross Vandegrift wrote: > > >>> vtree.xpath("*") > > [] > > From this result, it shows that at least your root element has a > namespace. Take a look at this section of the documentation: > http://codespeak.net/lxml/tutorial.html#namespaces Ahhhhh, until I read that, I don't think I actually understood namespaces at all. It's funny how I don't think I saw anywhere in the books I've read that namespaces can be nested and an internal namespace overrides ones already defined. Suddenly, things become quite a bit more clear :) Thanks a bunch, Ross > > -- > John Krukoff > Land Title Guarantee Company -- Ross Vandegrift ross at kallisti.us "If the fight gets hot, the songs get hotter. If the going gets tough, the songs get tougher." --Woody Guthrie From stefan_ml at behnel.de Thu Nov 27 11:28:08 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 27 Nov 2008 11:28:08 +0100 Subject: [lxml-dev] lxml 2.2alpha1 released In-Reply-To: References: <4929B50D.9030203@behnel.de> Message-ID: <492E7638.6070106@behnel.de> Hi, Mark Bestley wrote: > Runiing python setup.py from svn works > > However > easy_install xml==dev gives > > Building lxml version 2.2.alpha1-60098. > Building with Cython 0.10. > Using build configuration of libxslt 1.1.23 > Building against libxml2/libxslt in the following directory: /opt/local/lib > warning: no files found matching 'lxml.etree.c' under directory 'src/lxml' > warning: no files found matching 'lxml.objectify.c' under directory 'src/lxml' > warning: no files found matching 'lxml.etree.h' under directory 'src/lxml' > warning: no files found matching 'lxml.etree_api.h' under directory 'src/lxml' > warning: no files found matching '*.html' under directory 'doc' > i686-apple-darwin9-gcc-4.0.1: src/lxml/lxml.etree.c: No such file or directory > i686-apple-darwin9-gcc-4.0.1: no input files > > I thought this could be due to my fiddling with the OSX setup but I also > got a similar set of errors with Ubuntu 8.10 Hmm, is that the complete output? I can't see where the warnings come from, but I would suspect (especially seeing the "*.html") that it's easy_install that's packaging up stuff here (which would be fine since the files are not there yet). Having the C compiler fail afterwards makes me wonder why Cython wasn't run to generate the C code, though... Anyway, I recommend installing from a source release anyway, so it's better to use easy_install lxml==2.2alpha1 Stefan From lxml at bestley.co.uk Thu Nov 27 12:11:01 2008 From: lxml at bestley.co.uk (Mark Bestley) Date: Thu, 27 Nov 2008 11:11:01 +0000 Subject: [lxml-dev] lxml 2.2alpha1 released References: <4929B50D.9030203@behnel.de> <492E7638.6070106@behnel.de> Message-ID: Stefan Behnel writes: > Hi, > > Anyway, I recommend installing from a source release anyway, so it's better > to use > > easy_install lxml==2.2alpha1 > Yes that works -- Mark From olauzanne at gmail.com Thu Nov 27 18:32:02 2008 From: olauzanne at gmail.com (Olivier Lauzanne) Date: Thu, 27 Nov 2008 18:32:02 +0100 Subject: [lxml-dev] pyquery Message-ID: Hello, First thanks for lxml it's great. But I miss an interface on top of it. Something like jqueryor hpricot . Is there any work in progress to go toward something like that in python ? Missing a jquery like API in python, I started reproducing the jquery API in python by using lxml and released it a few days ago : pyquery If anyone is interested in contributing the code is clean and small. If anyone is working toward the same idea but in a slightly different direction I would be very interested. Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20081127/28766648/attachment.htm From stefan_ml at behnel.de Thu Nov 27 19:24:51 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 27 Nov 2008 19:24:51 +0100 Subject: [lxml-dev] pyquery In-Reply-To: References: Message-ID: <492EE5F3.9070005@behnel.de> Hi, Olivier Lauzanne wrote: > But I miss an interface on top of it. Something like > jqueryor > hpricot . > Is there any work in progress to go toward something like that in python ? Well, there's XPath and CSSSelect in lxml, and modifying XML isn't that hard either. But there's always space left for improvements, especially for special use cases. > Missing a jquery like API in python, I started reproducing the jquery API in > python by using lxml and released it a few days ago : > pyquery Thanks for sharing that, I added a link in the trunk docs (doc/FAQ.txt). Stefan From bgneal at gmail.com Sun Nov 30 07:03:29 2008 From: bgneal at gmail.com (Brian Neal) Date: Sun, 30 Nov 2008 00:03:29 -0600 Subject: [lxml-dev] Question on clean_html Message-ID: <1c1ea3180811292203m76aac375w8d654a6d107805df@mail.gmail.com> Hi, I would like to use lxml to remove all tags except 'a' tags. Is this possible? I don't seem to understand the arguments to the Cleaner class. What does allow_tags do? I tried this: >>> c = Cleaner(allow_tags=('a',), remove_unknown_tags=False) >>> print c.clean_html('Hi') Hi Do I instead have to list all the tags I don't want, except for 'a', in a remove_tags keyword argument? Any hints? Thank you.