Achille 2.0

From icon at fedoraproject.org Thu Jan 3 16:29:48 2008 From: icon at fedoraproject.org (Konstantin Ryabitsev) Date: Thu, 3 Jan 2008 10:29:48 -0500 Subject: [lxml-dev] Help with an error message Message-ID: Hi, everyone: I'm having trouble with the following case. One of my automatic import scripts takes data from one source and submits it to another as an XML feed. Recently, it started failing because one of the entries contains a null. The testcase is such: from lxml.etree import Element sourcestr = 'Contains a null: \x00' unistr = unicode(sourcestr, 'utf-8') elt = Element('foo').text = unistr Running it will cause the following error: Traceback (most recent call last): File "foo.py", line 6, in elt = Element('foo').text = unistr File "etree.pyx", line 741, in etree._Element.text.__set__ File "apihelpers.pxi", line 344, in etree._setNodeText File "apihelpers.pxi", line 648, in etree._utf8 AssertionError: All strings must be XML compatible, either Unicode or ASCII Can someone suggest the best way to deal with this? Kind regards, -- Konstantin Ryabitsev Montr?al, Qu?bec From stefan_ml at behnel.de Thu Jan 3 17:30:22 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 03 Jan 2008 17:30:22 +0100 Subject: [lxml-dev] Help with an error message In-Reply-To: References: Message-ID: <477D0D9E.3090205@behnel.de> Hi, Konstantin Ryabitsev wrote: > I'm having trouble with the following case. One of my automatic import > scripts takes data from one source and submits it to another as an XML > feed. Recently, it started failing because one of the entries contains > a null. The testcase is such: > > from lxml.etree import Element > sourcestr = 'Contains a null: \x00' > unistr = unicode(sourcestr, 'utf-8') > elt = Element('foo').text = unistr > > Running it will cause the following error: > > Traceback (most recent call last): > File "foo.py", line 6, in > elt = Element('foo').text = unistr > File "etree.pyx", line 741, in etree._Element.text.__set__ > File "apihelpers.pxi", line 344, in etree._setNodeText > File "apihelpers.pxi", line 648, in etree._utf8 > AssertionError: All strings must be XML compatible, either Unicode or ASCII > > Can someone suggest the best way to deal with this? My first question is: why do you need a '\x00' here? If you want to pass binary data in XML, the best way is to use a safe encoding such as uuencode or whatever. That should be part of your XML language spec/schema/... Stefan From azaroth at liverpool.ac.uk Thu Jan 3 17:33:33 2008 From: azaroth at liverpool.ac.uk (Rob Sanderson) Date: Thu, 03 Jan 2008 16:33:33 +0000 Subject: [lxml-dev] Help with an error message In-Reply-To: <477D0D9E.3090205@behnel.de> References: <477D0D9E.3090205@behnel.de> Message-ID: <1199378013.1868.18.camel@helmsdeep> The null character makes the XML non-well-formed anyway. The legal character ranges for XML (as per the spec, section 2.2): Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] Definitely no \x00! So ... I would base64 encode any offending data, as suggested by Stefan. Rob On Thu, 2008-01-03 at 17:30 +0100, Stefan Behnel wrote: > Konstantin Ryabitsev wrote: > > I'm having trouble with the following case. One of my automatic import > > scripts takes data from one source and submits it to another as an XML > > feed. Recently, it started failing because one of the entries contains > > a null. > My first question is: why do you need a '\x00' here? If you want to pass > binary data in XML, the best way is to use a safe encoding such as uuencode or > whatever. That should be part of your XML language spec/schema/... From stefan_ml at behnel.de Thu Jan 3 17:57:19 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 03 Jan 2008 17:57:19 +0100 Subject: [lxml-dev] Help with an error message In-Reply-To: <1199378013.1868.18.camel@helmsdeep> References: <477D0D9E.3090205@behnel.de> <1199378013.1868.18.camel@helmsdeep> Message-ID: <477D13EF.4040809@behnel.de> Hi, Rob Sanderson wrote: > The null character makes the XML non-well-formed anyway. > > The legal character ranges for XML (as per the spec, section 2.2): > > Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | > [#x10000-#x10FFFF] > > Definitely no \x00! that's true. While you could get away on the XML /generator/ side with adding an Entity (and lxml 2.0 will let you do that), this will just let you write out broken XML that the recipient will not be able to parse: >>> from lxml import etree as et >>> el = et.Element("test") >>> el.text = "mind the " >>> el.append(et.Entity("#0")) >>> xml = et.tostring(el) 'mind the �' >>> et.fromstring(xml) Traceback (most recent call last): lxml.etree.XMLSyntaxError: xmlParseCharRef: invalid xmlChar value 0, line 1, column 20 Maybe we should fix the Entity() factory here to prevent such misuse... Stefan From xkenneth at gmail.com Fri Jan 4 21:13:29 2008 From: xkenneth at gmail.com (Kenneth Miller) Date: Fri, 4 Jan 2008 14:13:29 -0600 Subject: [lxml-dev] XML Schemas (XSD) and Objectification Message-ID: <15529934-AE65-40B9-9302-A741CF512B0E@gmail.com> All, Is there any way to use an XSD file to generate an object in python using Objectify? Regards, Ken From jlovell at esd189.org Fri Jan 4 21:59:11 2008 From: jlovell at esd189.org (John Lovell) Date: Fri, 4 Jan 2008 12:59:11 -0800 Subject: [lxml-dev] XML Schemas (XSD) and Objectification In-Reply-To: <15529934-AE65-40B9-9302-A741CF512B0E@gmail.com> References: <15529934-AE65-40B9-9302-A741CF512B0E@gmail.com> Message-ID: <3A49C88789256B4AB33AC603DB6AF49B08E159@ZIRIA.esd189.org> Ken: While I do not know if you can do this using Objectify, you should look at generateDS as a backup. http://www.rexx.com/~dkuhlman/generateDS.html Good luck, John W. Lovell Web Applications Engineer Northwest Educational Service District 1601 R Avenue Anacortes, WA 98221 (360) 299-4086 jlovell at esd189.org www.esd189.org Together We Can ... -----Original Message----- From: lxml-dev-bounces at codespeak.net [mailto:lxml-dev-bounces at codespeak.net] On Behalf Of Kenneth Miller Sent: Friday, January 04, 2008 12:13 PM To: lxml-dev at codespeak.net Subject: [lxml-dev] XML Schemas (XSD) and Objectification All, Is there any way to use an XSD file to generate an object in python using Objectify? Regards, Ken _______________________________________________ lxml-dev mailing list lxml-dev at codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev From stefan_ml at behnel.de Fri Jan 4 22:45:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 04 Jan 2008 22:45:02 +0100 Subject: [lxml-dev] XML Schemas (XSD) and Objectification In-Reply-To: <15529934-AE65-40B9-9302-A741CF512B0E@gmail.com> References: <15529934-AE65-40B9-9302-A741CF512B0E@gmail.com> Message-ID: <477EA8DE.5090305@behnel.de> Hi, Kenneth Miller wrote: > Is there any way to use an XSD file to generate an object in > python using Objectify? Hmm, lxml.objectify actually works "as is", based on the XML document itself (i.e. an 'instance' of the schema), but without any schema interaction. What are you trying to achieve? Type enforcement based on schema types? Or do you mean 'generate an object' in the sense that you want to map an objectify object to a plain Python object? Stefan From stefan_ml at behnel.de Sat Jan 5 10:02:03 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 05 Jan 2008 10:02:03 +0100 Subject: [lxml-dev] XML Schemas (XSD) and Objectification In-Reply-To: References: <15529934-AE65-40B9-9302-A741CF512B0E@gmail.com> <477EA8DE.5090305@behnel.de> Message-ID: <477F478B.5090806@behnel.de> Hi, please reply also to the list. Kenneth Miller wrote: > On Jan 4, 2008, at 3:45 PM, Stefan Behnel wrote: >> Kenneth Miller wrote: >>> Is there any way to use an XSD file to generate an object in >>> python using Objectify? >> >> Hmm, lxml.objectify actually works "as is", based on the XML document >> itself >> (i.e. an 'instance' of the schema), but without any schema >> interaction. What >> are you trying to achieve? Type enforcement based on schema types? >> >> Or do you mean 'generate an object' in the sense that you want to map an >> objectify object to a plain Python object? >> > I'd like to be able to simply create the objects defined by the schema > as python objects. :) repeating an answer doesn't always help in understanding it. But I think what you mean is: you have a schema and you want to generate source code for Python objects *in advance* to represent its document instances. That's not how lxml.objectify works. What lxml.objectify does, is: you give it a document instance (no schema involved) and it will create Python objects for you *at runtime* to represent the document. That's a slight difference, and it's the reason why I asked back. It might or might not fit what you want to achieve with these objects. My bet is, if you validate your instance against the schema before you access the document tree, there shouldn't be a difference in behaviour. Also, if you want to tie specific objects to certain parts of the document, lxml will allow you to do that - just not with an /arbitrary/ Python object, as it requires inheritance of lxml's base objects. Have you looked at the examples on our web page? http://codespeak.net/lxml/dev/objectify.html#the-lxml-objectify-api http://codespeak.net/lxml/dev/objectify.html#python-data-types http://codespeak.net/lxml/dev/element_classes.html Stefan From stefan_ml at behnel.de Sat Jan 5 20:39:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 05 Jan 2008 20:39:02 +0100 Subject: [lxml-dev] lxml \ libxslt \ libxml2 leads to apache 2 crash on freebsd/amd64 In-Reply-To: <47769725.9050802@behnel.de> References: <4772AFBE.8020801@behnel.de> <47769725.9050802@behnel.de> Message-ID: <477FDCD6.9050904@behnel.de> Hi Dmitri, Stefan Behnel wrote: > The way XSLT is implemented in lxml is a bit tricky, as libxslt makes some > things hard to control that lxml uses in libxml2 for performance reasons. In > particular, lxml uses a thread-local hash table for constant strings, which is > much faster than a malloc() for each string that occurs in a document. > However, libxslt doesn't honour this dictionary and creates its own one based > on the stylesheet dictionary. The result is that the stylesheet can leak into > the result document through string references that now point into the hash > table of the stylesheet. > > There isn't a way in libxslt that would allow us to prevent this or to control > the allocation. That's why I decided to restrict the execution of XSL > transformations to threads that inherit the same hash table as the stylesheet, > this should normally prevent any problems. Here is a trivial patch (the one against xslt.pxi) that, instead of raising an exception, copies the stylesheet into the current thread context, and thus works around the current thread restrictions. It seems to work for me, any chance you could give it a try? In case it doesn't work reliably, could you additionally check the second change (in parser.pxi)? It should restrict 'acceptable' hash tables to the local thread, not including the main thread (as it did before). Stefan === src/lxml/xslt.pxi ================================================================== --- src/lxml/xslt.pxi (revision 3205) +++ src/lxml/xslt.pxi (local) @@ -373,7 +373,7 @@ cdef xmlDoc* c_doc if not _checkThreadDict(self._c_style.doc.dict): - raise RuntimeError, "stylesheet is not usable in this thread" + return self.__copy__()(_input, profile_run=profile_run, **_kw) input_doc = _documentOrRaise(_input) root_node = _rootNodeOrRaise(_input) === src/lxml/parser.pxi ================================================================== --- src/lxml/parser.pxi (revision 3205) +++ src/lxml/parser.pxi (local) @@ -132,8 +132,8 @@ """Check that c_dict is either the local thread dictionary or the global parent dictionary. """ - if __GLOBAL_PARSER_CONTEXT._c_dict is c_dict: - return 1 # main thread + #if __GLOBAL_PARSER_CONTEXT._c_dict is c_dict: + # return 1 # main thread if __GLOBAL_PARSER_CONTEXT._getThreadDict(NULL) is c_dict: return 1 # local thread dict return 0 From stefan_ml at behnel.de Mon Jan 7 11:14:32 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Jan 2008 11:14:32 +0100 Subject: [lxml-dev] cssselect and cssutils In-Reply-To: <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> Message-ID: <4781FB88.900@behnel.de> Hi Christof, H?ke, Christof wrote: > You are the main developer for lxml, right? Yep, but not the only one. :) > I was trying the CSSSelect > facility for a Python CSS library I am developing > (http://code.google.com/p/cssutils/) Cool. I knew about cssutils, felt that its field of application was related to cssselect (and lxml in general) but not with too much of an overlap - and always thought it would be nice to have it working with lxml in some way. > and I think there are some minor > problems with "*" or "*|*" (I need to check again and I'll put them on the > bug tracker then) but a question regarding support for pseudo selectors: > Would it be possible to support stuff like :first-letter (currently not > working is it not?) with Python XPath extension functions which should be > able to do what XPath cannot? Are you maybe even working on it? I guess > things like :first-line are problematic but other should be ok. I'm not the primary person to ask here. cssselect was developed by Ian Bicking, he knows best what works, what doesn't, and how to fix it. :) > If I get the time I would try some things out and report back, this was > just an idea that I had while playing with CSSSelector... Go ahead, this is open source. Any help, testing and ideas are always appreciated. > Lxml is really great stuff BTW, it was actually quite simple using lxml and > a CSSStyleSheet on a given HTML. (Not released yet but an example is in the > SVN). Great. In case there's anything we can do on lxml's side, please ask on the list. > Also the XPath extension facility is really great, I used Pyana until > some time ago but now use lxml for most projects. Competition is best when you win. :) Stefan From ianb at colorstudy.com Mon Jan 7 17:55:42 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 07 Jan 2008 10:55:42 -0600 Subject: [lxml-dev] cssselect and cssutils In-Reply-To: <4781FB88.900@behnel.de> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> Message-ID: <4782598E.3000803@colorstudy.com> Stefan Behnel wrote: > Hi Christof, > > H?ke, Christof wrote: >> You are the main developer for lxml, right? > > Yep, but not the only one. :) > > >> I was trying the CSSSelect >> facility for a Python CSS library I am developing >> (http://code.google.com/p/cssutils/) > > Cool. I knew about cssutils, felt that its field of application was related to > cssselect (and lxml in general) but not with too much of an overlap - and > always thought it would be nice to have it working with lxml in some way. Yeah, it would be cool to be able to take a stylesheet and turn it into style tags, to make the HTML relocatable without losing the style. That's just one idea that has occurred to me in the past. >> and I think there are some minor >> problems with "*" or "*|*" (I need to check again and I'll put them on the >> bug tracker then) but a question regarding support for pseudo selectors: >> Would it be possible to support stuff like :first-letter (currently not >> working is it not?) with Python XPath extension functions which should be >> able to do what XPath cannot? Are you maybe even working on it? I guess >> things like :first-line are problematic but other should be ok. ::first-letter is hard because it doesn't match any object in lxml. If it returned a string like "A" it would be very much out of context (e.g., no parent pointer), and it would be hard to do anything useful with it. To make it useful I think it would require some new stringish object that also looked nodeish (e.g., had a .getparent() method). Though maybe an object like that should exist; something similar would be needed for representing ranges. ::first-line, of course, depends on a rendering, so it's right out. I haven't been doing any work on selectors recently. There are a couple places where * doesn't work properly (and fixing it would probably require a Python XPath function), though they should give an exception. If not, then it's a bug of some sort. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From ianb at colorstudy.com Mon Jan 7 18:50:32 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 07 Jan 2008 11:50:32 -0600 Subject: [lxml-dev] cssselect and cssutils In-Reply-To: <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> <4782598E.3000803@colorstudy.com> <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> Message-ID: <47826668.90807@colorstudy.com> H?ke, Christof wrote: >> ::first-letter is hard because it doesn't match any object in lxml. >> If it returned a string like "A" it would be very much out of >> context (e.g., no parent pointer), and it would be hard to do >> anything useful with it. To make it useful I think it would >> require some new stringish object that also looked nodeish (e.g., >> had a .getparent() method). Though maybe an object like that should >> exist; something similar would be needed for representing ranges. > > What came to my mind was the DOM range spec stuff, but it is not > really finished, is it? I was reading about it some years (!) ago I > think in the Javascript Definitive Guide but I think it never went > anywhere really. It doesn't really matter too much, since lxml isn't that much like the DOM. But the same use cases for the DOM range can apply to lxml. > :first-letter should actually be element.text[0] I guess (which would > be a string in lxml currently?), I don't really know the lxml API but > would it be possible to define a subtype for element.text for this > case? But you are right, a more general approach would certainly be > better. element.text is just a unicode string. Maybe we could have a method like element.text_range(0, 1) that returns a subclass of unicode that also happens to know something about its location. E.g.: class ElementText(unicode): def __new__(cls, text, is_tail, range, parent): self = unicode.__new__(cls, text) self.is_tail = is_tail self.range = range self._parent = parent def getparent(self): return self._parent def enclose_in_tag(self, el): """ Enclose this text range in an element, like:: span = Element('span') el.text_range(0, 1).enclose_in_tag(span) """ parent = self.getparent() el.text = unicode(self) if self.is_tail: el.tail = parent.tail[self.range[1]:] parent.tail = parent.tail[:self.range[0]] index = parent.getparent().index(parent) parent.getparent().insert(index+1, el) else: el.tail = parent.text[self.range[1]:] parent.text = parent.text[:self.range[0]] parent.insert(0, el) self._parent = el self.range = (0, len(self)) self.is_tail = False All untested, of course. A real sense of a range would be a bit more difficult, as it involves lots of partial elements. But something like this would be necessary to do that work. Upon further thought, maybe subclassing unicode isn't the right thing -- perhaps it should really just wrap a string. Then perhaps you could just have do, say, el.text_range[:5], where e.text_range is a range object for all of its text, and you could slice range objects to further break them down. But dealing with changes to the element are tricky. It's all a bit tricky ;) -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From stefan_ml at behnel.de Mon Jan 7 19:30:54 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Jan 2008 19:30:54 +0100 Subject: [lxml-dev] cssselect and cssutils In-Reply-To: <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> <4782598E.3000803@colorstudy.com> <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> Message-ID: <47826FDE.9080703@behnel.de> Hi, H?ke, Christof wrote: >> Von: Ian Bicking [mailto:ianb at colorstudy.com] >> ::first-letter is hard >> because it doesn't match any object in lxml. If it returned a string >> like "A" it would be very much out of context (e.g., no parent pointer), >> and it would be hard to do anything useful with it. To make it useful I >> think it would require some new stringish object that also looked nodeish >> (e.g., had a .getparent() method). I considered that a while ago, as it would also be interesting for XPath in general. However, currently, we use fast Python string creation functions to serve the API level. At the time I deducted that changing that to the instantiation of a custom string object would almost certainly slow things down and complicate them, just to serve a rather special use case. Although maybe I might want to take another look at that today... > What came to my mind was the DOM range spec stuff, but it is not really > finished, is it? I was reading about it some years (!) ago I think in the > Javascript Definitive Guide but I think it never went anywhere really. I had a discussion about that starting over at the XML-SIG list last summer. http://permalink.gmane.org/gmane.comp.python.lxml.devel/2763 That was the first time I heard about DOM ranges and when I dug into that a little deeper, I almost ran away screaming. IMVHO, that's an insane and horribly complicated spec. > :first-letter should actually be element.text[0] I guess (which would be a > string in lxml currently?) Yes. > I don't really know the lxml API but would it > be possible to define a subtype for element.text for this case? But you are > right, a more general approach would certainly be better. You could define a special string (and unicode) subtype for the result of an XPath expression, which is determined independent of the Python API level. The freedom is right there. However, it would mean you have to search and (in the worst case) instantiate the parent Element to make sure it won't go away while the string result exists. That's some overhead compared to a simple string creation. As I said, I might reconsider that, but I'm not very confident. Stefan From stefan_ml at behnel.de Mon Jan 7 20:56:11 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Jan 2008 20:56:11 +0100 Subject: [lxml-dev] cssselect and cssutils In-Reply-To: <47826668.90807@colorstudy.com> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> <4782598E.3000803@colorstudy.com> <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> <47826668.90807@colorstudy.com> Message-ID: <478283DB.9090403@behnel.de> Hi Ian, Ian Bicking wrote: > element.text is just a unicode string. or a plain string. > Maybe we could have a method > like element.text_range(0, 1) that returns a subclass of unicode that > also happens to know something about its location. I prefer having the XPath string results be something like that. I think that's the only case where you can 'spuriously' end up with a text value and might want to know where it came from. > class ElementText(unicode): Maybe we should still keep up the str/unicode duality here. Although that will be history with Python 3, it isn't now, and it is an integral part of the current lxml API. > def __new__(cls, text, is_tail, range, parent): > self = unicode.__new__(cls, text) > self.is_tail = is_tail Right, 'is_tail' should be in. > self.range = range 'range' would be the substring indices? I would prefer calculating as much as possible on demand. Remember, most people will not use this object in any other way than a plain string. That's why I'm so hesitant about instantiating an Element object along the rode. > def enclose_in_tag(self, el): > """ > Enclose this text range in an element, like:: > > span = Element('span') > el.text_range(0, 1).enclose_in_tag(span) > """ Hmm, I'll have to think about that one. Not sure what the exact semantics should be. > Upon further thought, maybe subclassing unicode isn't the right thing -- > perhaps it should really just wrap a string. No, it must be a 'real' string to avoid having to check for another special case in the API (and likely in other places that we do not control). > It's all a bit tricky ;) I know. :) Stefan From ianb at colorstudy.com Mon Jan 7 22:15:59 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 07 Jan 2008 15:15:59 -0600 Subject: [lxml-dev] cssselect and cssutils In-Reply-To: <478283DB.9090403@behnel.de> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> <4782598E.3000803@colorstudy.com> <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> <47826668.90807@colorstudy.com> <478283DB.9090403@behnel.de> Message-ID: <4782968F.2000406@colorstudy.com> Stefan Behnel wrote: > Hi Ian, > > Ian Bicking wrote: >> element.text is just a unicode string. > > or a plain string. > > >> Maybe we could have a method >> like element.text_range(0, 1) that returns a subclass of unicode that >> also happens to know something about its location. > > I prefer having the XPath string results be something like that. I think > that's the only case where you can 'spuriously' end up with a text value and > might want to know where it came from. > > >> class ElementText(unicode): > > Maybe we should still keep up the str/unicode duality here. Although that will > be history with Python 3, it isn't now, and it is an integral part of the > current lxml API. ElementUnicodeText and ElementStrText? Not very pretty :-P > >> def __new__(cls, text, is_tail, range, parent): >> self = unicode.__new__(cls, text) >> self.is_tail = is_tail > > Right, 'is_tail' should be in. > > >> self.range = range > > 'range' would be the substring indices? I would prefer calculating as much as > possible on demand. Remember, most people will not use this object in any > other way than a plain string. That's why I'm so hesitant about instantiating > an Element object along the rode. Range is the slice that is selected, which is necessary for manipulation later (like enclose_in_tag). I'm not proposing this in any way replace text and tail. These don't feel quite like strings to me. Strings are interchangeable and simple. These are located in a specific place. If you call text.capitalize(), what does that do? Give you a capitalized view on the text? Give you a new string that loses all sense of place? It doesn't feel like a string at all, which is why I'm not sure it should even subclass from unicode/str. Or, for that matter, get used in lots of different contexts. Maybe having XPath return values makes it important to be fast. I'm not sure. It doesn't seem that important that XPath return values be particularly light. And I've found it problematic sometimes that non-node XPath return values are just strings (though that's been more an issue of attributes, where I'd like to know what attribute or element the value belonged to). OTOH, something like XPath's string() really is a string without any place. So it's all just kind of eclectic and awkward. > >> def enclose_in_tag(self, el): >> """ >> Enclose this text range in an element, like:: >> >> span = Element('span') >> el.text_range(0, 1).enclose_in_tag(span) >> """ > > Hmm, I'll have to think about that one. Not sure what the exact semantics > should be. It occurred to me thinking about how you could actually do something useful with ::first-letter, like: def apply_style(doc, selector, style): for item in selector(doc): if isinstance(item, ElementText): el = Element('span') item.enclose_in_tag(el) item = el item.set('style', item.get('style', '') + '; ' + style) This and the only other use case I currently have in my head for ranges (highlighting a selection of the document) would use something like enclose_in_tag. I can't remember what I was doing when I wanted XPath attributes, except I think it was matching something like @*, where the attribute name mattered but I didn't want to query on it. I think I ended up selecting elements and looping through the attributes instead. Maybe this was in some iteration of the HTML cleaning code. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From stefan_ml at behnel.de Tue Jan 8 12:45:33 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Jan 2008 12:45:33 +0100 Subject: [lxml-dev] special string subclasses for XPath string results In-Reply-To: <478333B6.6030906@behnel.de> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> <4782598E.3000803@colorstudy.com> <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> <47826668.90807@colorstudy.com> <478283DB.9090403@behnel.de> <4782968F.2000406@colorstudy.com> <478333B6.6030906@behnel.de> Message-ID: <4783625D.8010307@behnel.de> Hi again, Stefan Behnel wrote: > How do you instantiate a > custom unicode subclass from a UTF-8 char*? You can't use the normal C-API > functions, so I guess you'd have to instantiate a normal unicode object, then > determine its length, and then build the custom subclass for the result length > and copy the string over. That's ugly and it would certainly slow things down. Ok, I looked at the Python source and found that this is partially special cased already. All that is left to do is decode a unicode object from the char* and instantiate the subclass with it. The copying will be done internally. So here are some performance numbers for Py2.5.1. At first site, this looks like we have a clear winner: $ python -m timeit -s 'unicode("testtest")' 10000000 loops, best of 3: 0.0464 usec per loop $ python -m timeit -s 'class t(unicode): pass' 't("testtest")' 1000000 loops, best of 3: 1.67 usec per loop Now, a little more instantiating a subclass and copying the string: $ python -m timeit -s 'class t(unicode): pass' -s 's=unicode("test" * 20)' 't(s)' 1000000 loops, best of 3: 0.794 usec per loop $ python -m timeit -s 'class t(unicode): pass' -s 's=unicode("test" * 200)' 't(s)' 1000000 loops, best of 3: 1.09 usec per loop $ python -m timeit -s 'class t(unicode): pass' -s 's=unicode("test" * 2000)' 't(s)' 100000 loops, best of 3: 6.22 usec per loop Same for str: $ python -m timeit -s 'class t(str): pass' -s 's="test" * 200' 't(s)' 1000000 loops, best of 3: 1.27 usec per loop $ python -m timeit -s 'class t(str): pass' -s 's="test" * 2000' 't(s)' 100000 loops, best of 3: 7.03 usec per loop Funny enough, this is actually slower than unicode on my machine. As the following numbers show, however, the task at hand is clearly dominated by decoding: $ python -m timeit -s 'class t(unicode): pass' -s 's="test" * 200' 't(unicode(s, "utf-8"))' 100000 loops, best of 3: 5.23 usec per loop $ python -m timeit -s 'class t(unicode): pass' -s 's="test" * 2000' 't(unicode(s, "utf-8"))' 10000 loops, best of 3: 41.2 usec per loop Decoding by itself: $ python -m timeit -s 's="test" * 2000' 'unicode(s, "utf-8")' 10000 loops, best of 3: 34.9 usec per loop Even going straight through the C-API doesn't help much - 'decode' is a little test module written in Cython for that purpose: $ python -m timeit -s 'from decode import decode' -s 's="test" * 2000' 'decode(s)' 10000 loops, best of 3: 34.4 usec per loop $ python -m timeit -s 'class t(unicode): pass' -s 'from decode import decode' -s 's="test" * 2000' 't(decode(s))' 10000 loops, best of 3: 40.7 usec per loop We also shouldn't forget that we are talking microseconds here, so, performance-wise, there is no reason why we shouldn't use a subclass, especially after having just given the XPath engine a run. I'll give it a try. Maybe this can even still go in for 2.0. Stefan From stefan_ml at behnel.de Tue Jan 8 09:26:30 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Jan 2008 09:26:30 +0100 Subject: [lxml-dev] special string subclasses for XPath string results In-Reply-To: <4782968F.2000406@colorstudy.com> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> <4782598E.3000803@colorstudy.com> <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> <47826668.90807@colorstudy.com> <478283DB.9090403@behnel.de> <4782968F.2000406@colorstudy.com> Message-ID: <478333B6.6030906@behnel.de> Hi Ian, Ian Bicking wrote: > Stefan Behnel wrote: >> Maybe we should still keep up the str/unicode duality here. Although >> that will >> be history with Python 3, it isn't now, and it is an integral part of the >> current lxml API. > > ElementUnicodeText and ElementStrText? Not very pretty :-P Well, fine, but that's how it is. Users will not have to deal with the classes anyway, they will just be used in the background. isinstance(result, unicode) will work just like before, as will isinstance(result, basestring). The difference is just what happens when you call str() on them, or when you pass them into Python's API, or... Some APIs are ignorant regarding str/unicode, others are not. We shouldn't deliberately break those that are not, that would just slow down *everything*. > Maybe having XPath return values makes it important to be fast. Not necessarily fast, but it shouldn't slow things down unnecessarily for stuff that most people won't use. I imagine that the tricky part is the case where it actually is a (non-ASCII) Unicode value. How do you instantiate a custom unicode subclass from a UTF-8 char*? You can't use the normal C-API functions, so I guess you'd have to instantiate a normal unicode object, then determine its length, and then build the custom subclass for the result length and copy the string over. That's ugly and it would certainly slow things down. > It doesn't seem that important that XPath return values be > particularly light. And I've found it problematic sometimes that > non-node XPath return values are just strings (though that's been more > an issue of attributes, where I'd like to know what attribute or element > the value belonged to). OTOH, something like XPath's string() really is > a string without any place. So it's all just kind of eclectic and awkward. There's only so much we can do anyway. If the libxml2 result is a string, there is no way we can figure out where it came from. So the result of string() will always be a normal string instance. The only way where we could change something would be the case where the expression selects a text node or an attribute (text() and @...). I don't even think we can support ranges here. They would normally result from the substring() function, right? I doubt that would return anything but a plain string value. So to handle the result properly, you could do if isinstance(result, basestring): if hasattr(result, 'getparent'): print result.getparent().tag print result.is_text print result.is_tail print result.is_attribute else: print result BTW, I would also add "is_text" in that case. It would seem weird to check "is_attribute" and "is_tail" before you can determine that it actually is the common case of a normal ".text" value. (Although "is_text" sounds a bit more general than what it means here...) I actually think we are talking about lxml 2.1 stuff here. I'll try to get 2.0 out of the door and then we can see how we could implement these things. Stefan From dfedoruk at gmail.com Wed Jan 9 18:55:23 2008 From: dfedoruk at gmail.com (Dmitri Fedoruk) Date: Wed, 9 Jan 2008 20:55:23 +0300 Subject: [lxml-dev] lxml \ libxslt \ libxml2 leads to apache 2 crash on freebsd/amd64 In-Reply-To: <477FDCD6.9050904@behnel.de> References: <4772AFBE.8020801@behnel.de> <47769725.9050802@behnel.de> <477FDCD6.9050904@behnel.de> Message-ID: Hello, > Here is a trivial patch (the one against xslt.pxi) > It seems to work for me, any chance you could give it a try? Thank you for the patches, I'll apply them and see what happens next. The thing is that such an exception occurs very seldom and I can not reproduce it. Nevertheless, coming back to the thread subject. As we have managed to find out, it is indeed the deallocation problem. I've played around with the variable taht caused the trouble, tried to make it global, for example - this changed only the positon of the crash, but not the reason. When the memory has to be free'd, the crash happens. Unfortunatelly I managed to reproduce this bug on 3 versions of FreeBSD 6.2 and on the i386 architecture too, which had never happened in 6 months of development. But i386 is capable of running valgrind. So I got this errors: ==77394== Invalid free() / delete / delete[] ==77394== at 0x3C03867F: free (in /usr/local/lib/valgrind/vgpreload_memcheck.so) ==77394== by 0x3CF97668: xmlFreeNodeList (in /usr/X11R6/lib/libxml2.so.5) ==77394== by 0x3CF974F0: xmlFreeProp (in /usr/X11R6/lib/libxml2.so.5) ==77394== by 0x3CF9754F: xmlFreePropList (in /usr/X11R6/lib/libxml2.so.5) ==77394== Address 0x3C9C5E8B is 743 bytes inside a block of size 1024 alloc'd ==77394== at 0x3C038183: malloc (in /usr/local/lib/valgrind/vgpreload_memcheck.so) ==77394== by 0x3D02B4EE: xmlDictAddString (in /usr/X11R6/lib/libxml2.so.5) ==77394== by 0x3D02BBEB: xmlDictLookup (in /usr/X11R6/lib/libxml2.so.5) ==77394== by 0x3CF80425: xmlDetectSAX2 (in /usr/X11R6/lib/libxml2.so.5) More than that, this messages were preceeded by a bunch of errors from the libpython itself: ==77390== Use of uninitialised value of size 4 ==77390== at 0x3C6D2AC9: PyObject_Realloc (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C735BA4: _PyObject_GC_Resize (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C6BE525: PyFrame_New (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C711382: PyEval_EvalFrameEx (in /usr/X11R6/lib/libpython2.5.so) ==77390== ==77390== Invalid read of size 4 ==77390== at 0x3C6D2AAF: PyObject_Realloc (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C735BA4: _PyObject_GC_Resize (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C6BE525: PyFrame_New (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C711382: PyEval_EvalFrameEx (in /usr/X11R6/lib/libpython2.5.so) ==77390== Conditional jump or move depends on uninitialised value(s) ==77390== at 0x3C6D2AB8: PyObject_Realloc (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C735BA4: _PyObject_GC_Resize (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C6BE525: PyFrame_New (in /usr/X11R6/lib/libpython2.5.so) ==77390== by 0x3C711382: PyEval_EvalFrameEx (in /usr/X11R6/lib/libpython2.5.so) (repeated many times during the apache thread initialisation). So, this is not the lxml problem really... But maybe somebody has any idea? Right now I'm thinking of opportunity to replace mod_python with mod_fastcgi . Thanks for attention so far! Dmitri From dfedoruk at gmail.com Wed Jan 9 19:05:02 2008 From: dfedoruk at gmail.com (Dmitri Fedoruk) Date: Wed, 9 Jan 2008 21:05:02 +0300 Subject: [lxml-dev] lxml \ libxslt \ libxml2 leads to apache 2 crash on freebsd/amd64 In-Reply-To: References: <4772AFBE.8020801@behnel.de> <47769725.9050802@behnel.de> <477FDCD6.9050904@behnel.de> Message-ID: Greetings once more, Update of the previous message: nevermind the different PID's in valgrind output, I just copied the wrong ones. Error messages from the libpython2.5.so in the process with invalid free\delete are the same as I have posted before. > ==77394== Invalid free() / delete / delete[] [omitted] > More than that, this messages were preceeded by a bunch of errors from > the libpython itself: > ==77390== Use of uninitialised value of size 4 [omitted] Cheers, Dmitri From ianb at colorstudy.com Wed Jan 9 19:19:02 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 09 Jan 2008 12:19:02 -0600 Subject: [lxml-dev] special string subclasses for XPath string results In-Reply-To: <478333B6.6030906@behnel.de> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> <4782598E.3000803@colorstudy.com> <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> <47826668.90807@colorstudy.com> <478283DB.9090403@behnel.de> <4782968F.2000406@colorstudy.com> <478333B6.6030906@behnel.de> Message-ID: <47851016.6050905@colorstudy.com> Stefan Behnel wrote: >> It doesn't seem that important that XPath return values be >> particularly light. And I've found it problematic sometimes that >> non-node XPath return values are just strings (though that's been more >> an issue of attributes, where I'd like to know what attribute or element >> the value belonged to). OTOH, something like XPath's string() really is >> a string without any place. So it's all just kind of eclectic and awkward. > > There's only so much we can do anyway. If the libxml2 result is a string, > there is no way we can figure out where it came from. So the result of > string() will always be a normal string instance. The only way where we could > change something would be the case where the expression selects a text node or > an attribute (text() and @...). I don't even think we can support ranges here. > They would normally result from the substring() function, right? I doubt that > would return anything but a plain string value. That's kind of why I think performance doesn't matter, because it won't even come into play most of the time. In relation to XPath, the one thing I would like is some representation of attributes. There's a backward compatible issue, but the underlying engine returns attributes as something different than normal text anyway, right? I think attributes are mostly a different use case than text ranges. For something like ::first-letter, I didn't really expect it to be possible to compile that to XPath. Instead it would have to be something like: def first_letter_selector(xpath_expr): def selector(el): result = xpath_expr(el) return result.text_range(0, 1) return selector For representing ranges I'd also like some text range (I don't have any immediate needs, so I'm personally in no rush here). But it's not something that would have to replace the current text/tail attributes. It's just that in some code it can be nice to have something similar to the DOM TextNode, and this kind of provides that (except more nicely I think, as it would be more like a view). Then the range might just be like: class TextRange(object): def __init__(self, el, range, is_text): self.el = el assert range[0] >= 0 self.range = range # (start, end) tuple self.is_text = is_text def __unicode__(self): start, end = self.range if end == 0: return '' if self.is_text: t = self.el.text else: t = self.el.tail if t is None: raise ValueError(...) if not isinstance(t, unicode): t = unicode(t, 'utf8') #? if range[1] > len(t): raise ValueError( "TextRange %r is invalid (element has been changed?)" % self) return t[start:end] def __repr__(self): if self.is_text: meth = 'text_range' else: meth = 'tail_range' return '%r.%s%s' % (self.el, meth, range) def getparent(self): return self.el # and other stuff that might be convenient... No place where I am currently using text/tail do I really want this kind of behavior. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From stefan_ml at behnel.de Wed Jan 9 19:43:26 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 09 Jan 2008 19:43:26 +0100 Subject: [lxml-dev] lxml \ libxslt \ libxml2 leads to apache 2 crash on freebsd/amd64 In-Reply-To: References: <4772AFBE.8020801@behnel.de> <47769725.9050802@behnel.de> <477FDCD6.9050904@behnel.de> Message-ID: <478515CE.5050101@behnel.de> Hi, Dmitri Fedoruk wrote: >> Here is a trivial patch (the one against xslt.pxi) >> It seems to work for me, any chance you could give it a try? > Thank you for the patches, I'll apply them and see what happens next. Thanks. > The thing is that such an exception occurs very seldom and I can not > reproduce it. You will still notice if it doesn't work. :) > Nevertheless, coming back to the thread subject. As we have managed to > find out, it is indeed the deallocation problem. I've played around > with the variable taht caused the trouble, tried to make it global, > for example - this changed only the positon of the crash, but not the > reason. When the memory has to be free'd, the crash happens. > > Unfortunatelly I managed to reproduce this bug on 3 versions of > FreeBSD 6.2 and on the i386 architecture too, which had never happened > in 6 months of development. It shouldn't be machine dependent. If it's there, it's in the code. Garbage collection and threading might work different on different architectures, but that won't remove the actual problem. > But i386 is capable of running valgrind. So I got this errors: > ==77394== Invalid free() / delete / delete[] > ==77394== at 0x3C03867F: free (in > /usr/local/lib/valgrind/vgpreload_memcheck.so) > ==77394== by 0x3CF97668: xmlFreeNodeList (in /usr/X11R6/lib/libxml2.so.5) > ==77394== by 0x3CF974F0: xmlFreeProp (in /usr/X11R6/lib/libxml2.so.5) > ==77394== by 0x3CF9754F: xmlFreePropList (in /usr/X11R6/lib/libxml2.so.5) > ==77394== Address 0x3C9C5E8B is 743 bytes inside a block of size 1024 alloc'd > ==77394== at 0x3C038183: malloc (in > /usr/local/lib/valgrind/vgpreload_memcheck.so) > ==77394== by 0x3D02B4EE: xmlDictAddString (in /usr/X11R6/lib/libxml2.so.5) > ==77394== by 0x3D02BBEB: xmlDictLookup (in /usr/X11R6/lib/libxml2.so.5) > ==77394== by 0x3CF80425: xmlDetectSAX2 (in /usr/X11R6/lib/libxml2.so.5) Funny place for a malloc. Anyway, this is only a symptom. The problem is that the document or an XML node gets freed either while it's still in use, or by two independent parties (i.e. Python element proxies that refer to it). With a bug that occurs this seldom and a setup as complex as mod_python, it's really hard to narrow down the test case, so I don't how far you could get here. However, I'm currently chasing a (pretty old) bug myself. Maybe it's related already. Could you check with the current SVN trunk if that works better for you? Although the stack trace above gives me doubts... > More than that, this messages were preceeded by a bunch of errors from > the libpython itself: > ==77390== Use of uninitialised value of size 4 > ==77390== at 0x3C6D2AC9: PyObject_Realloc (in /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C735BA4: _PyObject_GC_Resize (in > /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C6BE525: PyFrame_New (in /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C711382: PyEval_EvalFrameEx (in > /usr/X11R6/lib/libpython2.5.so) > ==77390== > ==77390== Invalid read of size 4 > ==77390== at 0x3C6D2AAF: PyObject_Realloc (in /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C735BA4: _PyObject_GC_Resize (in > /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C6BE525: PyFrame_New (in /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C711382: PyEval_EvalFrameEx (in > /usr/X11R6/lib/libpython2.5.so) > ==77390== Conditional jump or move depends on uninitialised value(s) > ==77390== at 0x3C6D2AB8: PyObject_Realloc (in /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C735BA4: _PyObject_GC_Resize (in > /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C6BE525: PyFrame_New (in /usr/X11R6/lib/libpython2.5.so) > ==77390== by 0x3C711382: PyEval_EvalFrameEx (in > /usr/X11R6/lib/libpython2.5.so) > (repeated many times during the apache thread initialisation). Hmmm, not sure what this means. Might be entirely unrelated. Valgrind uses a suppression file that drops a lot of false positives, maybe those are just false positives of mod_python. > So, this is not the lxml problem really... But maybe somebody has any idea? > Right now I'm thinking of opportunity to replace mod_python with mod_fastcgi . I have neither experience with mod_python nor with mod_fastcgi, sorry. Stefan From stefan_ml at behnel.de Thu Jan 10 00:33:12 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 10 Jan 2008 00:33:12 +0100 Subject: [lxml-dev] special string subclasses for XPath string results In-Reply-To: <47851016.6050905@colorstudy.com> References: <57254BE3C7FAD845A0E402A369B0D0DD992E8E@gt1mail1.gt.e-7.local> <4781EDAF.1010604@behnel.de> <57254BE3C7FAD845A0E402A369B0D0DD992F92@gt1mail1.gt.e-7.local> <4781FB88.900@behnel.de> <4782598E.3000803@colorstudy.com> <57254BE3C7FAD845A0E402A369B0D0DD993058@gt1mail1.gt.e-7.local> <47826668.90807@colorstudy.com> <478283DB.9090403@behnel.de> <4782968F.2000406@colorstudy.com> <478333B6.6030906@behnel.de> <47851016.6050905@colorstudy.com> Message-ID: <478559B8.5020703@behnel.de> Hi Ian, I now implemented the basic behaviour on the trunk. It first crashed because of a bug in Cython 0.9.6.10b, hope my patch will find its way into a release soon. Attributes and text nodes are now handled. String results (such as returned by the string() function) will remain plain strings - no way to recover here. Ian Bicking wrote: > For something like ::first-letter, I didn't really expect it to be > possible to compile that to XPath. Instead it would have to be > something like: > > def first_letter_selector(xpath_expr): > def selector(el): > result = xpath_expr(el) > return result.text_range(0, 1) > return selector Something like that, yes. You could do for element in xpath_expr(el): for text in element.itertext() if text: return text[0] return None # or raise or ... or wouldn't we have to return a list here? As in return [ text[0] for element in xpath_expr(el) for text in element.itertext() if text ] > For representing ranges I'd also like some text range (I don't have any > immediate needs, so I'm personally in no rush here). But it's not > something that would have to replace the current text/tail attributes. > It's just that in some code it can be nice to have something similar to > the DOM TextNode, and this kind of provides that (except more nicely I > think, as it would be more like a view). > > Then the range might just be like: > > class TextRange(object): [...] You could file this as a feature request in the bug tracker so that we won't forget about it. It's definitely not clear enough for 2.0. Stefan From stefan_ml at behnel.de Fri Jan 11 16:42:06 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Jan 2008 16:42:06 +0100 Subject: [lxml-dev] lxml 2.0 beta1 released Message-ID: <47878E4E.5080800@behnel.de> Hi all, I finally managed to push lxml 2.0beta1 over to PyPI. This release marks the end of the four month alpha cycle of lxml 2.0. The last stable release series, lxml 1.3, saw the light of day more than six months ago. http://codespeak.net/lxml/dev/ http://pypi.python.org/pypi/lxml/2.0beta1 The complete changelog for beta1 and the 2.0 alpha series follows below. Apart from a number of important fixes and enhancements, this beta release also finalises the major API changes that make the difference between 1.x and 2.x. Incompatible changes after this release will require a very good motivation. As usual, compatible enhancements will always be embraced - as will be updates, clarifications and fixes for the documentation! Asking back helps. I expect beta1 to also be the last beta release before lxml 2.0 final (hopefully not in the sense that alpha4/5/6 were), so please test as much as you can to spot any remaining bugs and problems. Note that this release depends on a bug fix in Cython that will hopefully be released as Cython 0.9.6.11 in a couple of days. I attached the necessary patch for those who want work on the sources. Another thing: there was a security advisory on the libxml2 mailing list. To prevent DoS attacks, systems that parse XML from untrusted sources should be updated to libxml2 2.6.31 (or should apply the patch that is referenced in Daniel's post below). http://mail.gnome.org/archives/xml/2008-January/msg00036.html Sidnei, when you build the Windows binaries, could you please wait for libxml2 2.6.31 to become available as binaries as well? Hopefully, that won't take too long... Have fun, Stefan 2.0beta1 (2008-01-11) ===================== Features added -------------- * Parse-time XML schema validation (``schema`` parser keyword). * XPath string results of the ``text()`` function and attribute selection make their Element container accessible through a ``getparent()`` method. As a side-effect, they are now always unicode objects (even ASCII strings). * ``XSLT`` objects are usable in any thread - at the cost of a deep copy if they were not created in that thread. * Invalid entity names and character references will be rejected by the ``Entity()`` factory. * ``entity.text`` returns the textual representation of the entity, e.g. ``&``. Bugs fixed ---------- * XPath on ElementTrees could crash when selecting the virtual root node of the ElementTree. * Compilation ``--without-threading`` was buggy in alpha5/6. Other changes ------------- * Minor performance tweaks for Element instantiation and subelement creation 2.0alpha6 (2007-12-19) ====================== Features added -------------- * New properties ``position`` and ``code`` on ParseError exception (as in ET 1.3) Bugs fixed ---------- * Memory leak in the ``parse()`` function. * Minor bugs in XSLT error message formatting. * Result document memory leak in target parser. Other changes ------------- * Various places in the XPath, XSLT and iteration APIs now require keyword-only arguments. * The argument order in ``element.itersiblings()`` was changed to match the order used in all other iteration methods. The second argument ('preceding') is now a keyword-only argument. * The ``getiterator()`` method on Elements and ElementTrees was reverted to return an iterator as it did in lxml 1.x. The ET API specification allows it to return either a sequence or an iterator, and it traditionally returned a sequence in ET and an iterator in lxml. However, it is now deprecated in favour of the ``iter()`` method, which should be used in new code wherever possible. * The 'pretty printed' serialisation of ElementTree objects now inserts newlines at the root level between processing instructions, comments and the root tag. * A 'pretty printed' serialisation is now terminated with a newline. * Second argument to ``lxml.etree.Extension()`` helper is no longer required, third argument is now a keyword-only argument ``ns``. * ``lxml.html.tostring`` takes an ``encoding`` argument. 2.0alpha5 (2007-11-24) ====================== Features added -------------- * Rich comparison of ``element.attrib`` proxies. * ElementTree compatible TreeBuilder class. * Use default prefixes for some common XML namespaces. * ``lxml.html.clean.Cleaner`` now allows for a ``host_whitelist``, and two overridable methods: ``allow_embedded_url(el, url)`` and the more general ``allow_element(el)``. * Extended slicing of Elements as in ``element[1:-1:2]``, both in etree and in objectify * Resolvers can now provide a ``base_url`` keyword argument when resolving a document as string data. * When using ``lxml.doctestcompare`` you can give the doctest option ``NOPARSE_MARKUP`` (like ``# doctest: +NOPARSE_MARKUP``) to suppress the special checking for one test. Bugs fixed ---------- * Target parser failed to report comments. * In the ``lxml.html`` ``iter_links`` method, links in ``