From matt.barto at gmail.com Thu Jul 1 22:01:39 2010 From: matt.barto at gmail.com (matt barto) Date: Thu, 1 Jul 2010 13:01:39 -0700 Subject: [lxml-dev] html.fromstring returning encoded string from a non unicoded string source Message-ID: Hello, I am trying to obtain a title from a website which has a Unicode tm and register mark, but the Unicode behavior is not what I expect. tree = html.fromstring("

Apple® - iPad™ with Wi-Fi - 16GB

" print tree.text_content() --------> print(tree.text_content()) Apple? - iPad? with Wi-Fi - 16GB I would expect the output from the print is "Apple® - iPad™ with Wi-Fi - 16GB", but it seems some encoding occurred during the tree creation. The browser is able to handle these html characters correctly using "iso-8856-1" char set (even though the document says "utf-8"). Can you provide some insight how these two html tags are handled and what I can do to have the expected behavior? My lxml version is 2.2.2. Thanks in advanced. Best, Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20100701/15bfe9b9/attachment.htm From eugene.vandenbulke at gmail.com Fri Jul 2 10:49:45 2010 From: eugene.vandenbulke at gmail.com (Eugene Van den Bulke) Date: Fri, 2 Jul 2010 18:49:45 +1000 Subject: [lxml-dev] regexp Message-ID: Hi, I am experimenting with web scraping using lxml. I have played a little with BeautifulSoup in the past and scrapy recently. I am recoding something I did with scrapy with lxml but encounter a problem I am not sure how to iron out. With scrapy, hxs is an xpath selector which has a select and re method types = hxs.select('.//a[@href]/@href').re(r'type=([A-Z]*)') Which will return a list of the matches in href. How would I do the same thing with lxml? types = doc.xpath('.//a[@href]/@href') ... Thanks a lot, -- EuGeNe -- I lend my books on COlivri http://www.colivri.org/user/eugene, do you? From stefan_ml at behnel.de Fri Jul 2 11:42:29 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 11:42:29 +0200 Subject: [lxml-dev] regexp In-Reply-To: References: Message-ID: <4C2DB485.2080504@behnel.de> Eugene Van den Bulke, 02.07.2010 10:49: > I am experimenting with web scraping using lxml. > > I have played a little with BeautifulSoup in the past and scrapy recently. > > I am recoding something I did with scrapy with lxml but encounter a > problem I am not sure how to iron out. > > With scrapy, hxs is an xpath selector which has a select and re method > > types = hxs.select('.//a[@href]/@href').re(r'type=([A-Z]*)') > > Which will return a list of the matches in href. > > How would I do the same thing with lxml? > > types = doc.xpath('.//a[@href]/@href') ... http://lmgtfy.com/?q=lxml+regular+expressions&l=1 ;-) Stefan From stefan_ml at behnel.de Fri Jul 2 12:13:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 12:13:56 +0200 Subject: [lxml-dev] regexp In-Reply-To: References: <4C2DB485.2080504@behnel.de> Message-ID: <4C2DBBE4.8080807@behnel.de> Eugene Van den Bulke, 02.07.2010 11:53: > Stefan Behnel, 02.07.2010 11:42: >> Eugene Van den Bulke, 02.07.2010 10:49: >>> I am experimenting with web scraping using lxml. >>> >>> I have played a little with BeautifulSoup in the past and scrapy >>> recently. >>> >>> I am recoding something I did with scrapy with lxml but encounter a >>> problem I am not sure how to iron out. >>> >>> With scrapy, hxs is an xpath selector which has a select and re method >>> >>> types = hxs.select('.//a[@href]/@href').re(r'type=([A-Z]*)') >>> >>> Which will return a list of the matches in href. >>> >>> How would I do the same thing with lxml? >>> >>> types = doc.xpath('.//a[@href]/@href') ... Note that this is redundant, './/a/@href' is enough. >> http://lmgtfy.com/?q=lxml+regular+expressions&l=1 > > I did read the doc before I took the liberty to post ... I am afraid I > just don't get it. Personally, I wouldn't even use XPath regular expressions here. I'd rather do something like this: from lxml import html import re parse_type_value = re.compile(r'type=([A-Z]*)').findall root = html.parse(the_file).getroot() for el, attr, link, pos in root.iterlinks(): if 'type=' in link: print el.tag, parse_type_value(link) Note that this will give you all links, not only those in href's. If you really only want those, the XPath expression above will do just fine. Stefan From eugene.vandenbulke at gmail.com Fri Jul 2 12:18:38 2010 From: eugene.vandenbulke at gmail.com (Eugene Van den Bulke) Date: Fri, 2 Jul 2010 20:18:38 +1000 Subject: [lxml-dev] regexp In-Reply-To: <4C2DBBE4.8080807@behnel.de> References: <4C2DB485.2080504@behnel.de> <4C2DBBE4.8080807@behnel.de> Message-ID: On Fri, Jul 2, 2010 at 8:13 PM, Stefan Behnel wrote: >>>> types = doc.xpath('.//a[@href]/@href') ... > > Note that this is redundant, './/a/@href' is enough. I am discovering XPath as well as you can tell :P > Personally, I wouldn't even use XPath regular expressions here. I'd rather > do something like this: > > ? ?from lxml import html > ? ?import re > > ? ?parse_type_value = re.compile(r'type=([A-Z]*)').findall > > ? ?root = html.parse(the_file).getroot() > > ? ?for el, attr, link, pos in root.iterlinks(): > ? ? ? ?if 'type=' in link: > ? ? ? ? ? ? print el.tag, parse_type_value(link) > > Note that this will give you all links, not only those in href's. If you > really only want those, the XPath expression above will do just fine. > > Stefan Thanks ! -- EuGeNe -- I lend my books on COlivri http://www.colivri.org/user/eugene, do you? From paul.girard at sciences-po.fr Fri Jul 2 12:20:06 2010 From: paul.girard at sciences-po.fr (Paul Girard) Date: Fri, 02 Jul 2010 12:20:06 +0200 Subject: [lxml-dev] mac os x installation process Message-ID: <4C2DBD56.9020800@sciences-po.fr> Hi dear people of lxml, I wrote a small lib in Python using lxml to generate graph file in a specific format gexf. This little thing is called pygexf : http://packages.python.org/pygexf/ http://github.com/paulgirard/pygexf First thanks for your great work on lxml, I am loving it ! Second I am experiencing rude probs with installing lxml on my mac os x. I read that this hasn't been completed ported yet but still some workarounds with static libs seems to have worked for some of us. I will only focus on the method : STATIC_DEPS=true sudo easy_install lxml I am having probs with gcc. I have 3 different versions of gcc : gcc-4 : coming from fink install gcc42 gcc-4.0 gcc-4.2 : bot coming from Xcode mac os x tools note: I am changing the link gcc to the different versions to test all of them Now here are the different errors I have with the various gcc versions : gcc-4 $ls -la /usr/bin/gcc lrwxr-xr-x 1 root wheel 13 2 jul 11:43 /usr/bin/gcc -> /sw/bin/gcc-4 $STATIC_DEPS=true sudo easy_install lxml searching for lxml [...] Building against libxml2/libxslt in the following directory: /usr/lib gcc: unrecognized option '-no-cpp-precomp' cc1: erreur: option "-mno-fused-madd" de la ligne de commande non reconnue cc1: erreur: option "-arch" de la ligne de commande non reconnue cc1: erreur: option "-arch" de la ligne de commande non reconnue cc1: erreur: option "-Wno-long-double" de la ligne de commande non reconnue error: Setup script exited with error: command 'gcc' failed with exit status 1 gcc-4.2 $ STATIC_DEPS=true sudo easy_install lxml Searching for lxml [...] Using build configuration of libxslt 1.1.12 Building against libxml2/libxslt in the following directory: /usr/lib cc1: error: unrecognized command line option "-Wno-long-double" cc1: error: unrecognized command line option "-Wno-long-double" lipo: can't open input file: /var/tmp//ccDn5F16.out (No such file or directory) error: Setup script exited with error: command 'gcc' failed with exit status 1 The error list of gcc-4.0 is huge. I'll not post it here but I could if necessary. So here I am facing building problems. I am not a expert into that kind of thing. I am usually developing on linux (ubuntu) but many users of my small lib (including me) are mac users. I don't really want to change my code to use another xml lib so I hope i'll finally find a way.. If anyone can help on this issue it'd be more than great, thanks for reading me Paul ps: I couldn't try the darwin port method I couldn't understand how to use it... -- Paul Girard responsable num?rique m?dialab paul.girard at sciences-po.fr 01 45 49 63 58 m?dialab | Sciences Po medialab.sciences-po.fr 13 rue de l'universit? 75007 PARIS -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20100702/81d7d8c0/attachment-0001.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: Sc-Po-Medialab-Gris.jpg Type: image/jpeg Size: 7276 bytes Desc: not available Url : http://codespeak.net/pipermail/lxml-dev/attachments/20100702/81d7d8c0/attachment-0001.jpg From ab at rdprojekt.pl Fri Jul 2 17:47:22 2010 From: ab at rdprojekt.pl (=?UTF-8?B?QWRhbSBCaWVsYcWEc2tp?=) Date: Fri, 02 Jul 2010 17:47:22 +0200 Subject: [lxml-dev] Fwd: iterparse() doesn't do dtd_validation Message-ID: <4C2E0A0A.40009@rdprojekt.pl> Hi, Can anyone help me with validating XML file against internal DTD with iterparse()? I just can't make iterparse() use dtd_validation flag. I ended up with two execution paths, one that uses etree.XMLParser and finds all errors in validated file and another one which uses iterparse and just prints all elements from XML file. Below is the code I used, and attached are sample XML and DTD files I used. Just to make it clear: I'm using lxml in version 2.2.4 on Python 2.6.5. The XmlWithDTDStream class (mentioned in code below) behaves like a stream, returning XML declaration (first line of well-formed XML file), DTD string and then - rest of XML file. It works correctly, since you can see that XMLParser returns correct errors and iterparse returns actual elements from input.xml.



from lxml import etree
if __name__ == "__main__":
      print "XMLParser"
      with open("internalSchema.dtd") as dtdFile:
          dtd = dtdFile.read()
          stream = XmlWithDTDStream(dtd, 'input.xml')
          parser = etree.XMLParser(dtd_validation=True, load_dtd=True)
          try:
              root = etree.XML(stream.read(32768), parser)
          except Exception, e:
              print "An exception:: ", e
              print "Error log:: ", parser.error_log
      print "iterparse: "
      with open("internalSchema.dtd") as dtdFile:
          dtd = dtdFile.read()
          stream = XmlWithDTDStream(dtd, 'input.xml')
          for aTuple in etree.iterparse(stream, dtd_validation=True,
load_dtd=True):
              print aTuple

The result is then as follows:

XMLParser
An exception::  No declaration for attribute badAttribute of element
pName, line 23, column 26
Error log:::23:26:ERROR:VALID:DTD_UNKNOWN_ATTRIBUTE: No
declaration for attribute badAttribute of element pName
:25:7:ERROR:VALID:DTD_UNKNOWN_ELEM: No declaration for element
property--badName
:26:5:ERROR:VALID:DTD_CONTENT_MODEL: Element propertyGroup
content does not follow the DTD, expecting (property)+, got
(property--badName )
:29:29:ERROR:VALID:DTD_UNKNOWN_ELEM: No declaration for element name

iterparse:
(u'end',)
(u'end',)
(u'end',)
(u'end',)
(u'end',)
(u'end',)
(u'end',)
(u'end',)



from lxml import etree
if __name__ == "__main__":
     print "XMLParser"
     with open("internalSchema.dtd") as dtdFile:
         dtd = dtdFile.read()
         stream = XmlWithDTDStream(dtd, 'input.xml')
         parser = etree.XMLParser(dtd_validation=True, load_dtd=True)
         try:
             root = etree.XML(stream.read(32768), parser)
         except Exception, e:
             print "An exception:: ", e
             print "Error log:: ", parser.error_log
     print "iterparse: "
     with open("internalSchema.dtd") as dtdFile:
         dtd = dtdFile.read()
         stream = XmlWithDTDStream(dtd, 'input.xml')
         for aTuple in etree.iterparse(stream, dtd_validation=True, 
load_dtd=True):
             print aTuple

The result is as follows:

XMLParser
An exception::  No declaration for attribute badAttribute of element 
pName, line 23, column 26
Error log:: :23:26:ERROR:VALID:DTD_UNKNOWN_ATTRIBUTE: No 
declaration for attribute badAttribute of element pName
:25:7:ERROR:VALID:DTD_UNKNOWN_ELEM: No declaration for element 
property--badName
:26:5:ERROR:VALID:DTD_CONTENT_MODEL: Element propertyGroup 
content does not follow the DTD, expecting (property)+, got 
(property--badName )
:29:29:ERROR:VALID:DTD_UNKNOWN_ELEM: No declaration for element name

iterparse:
(u'end', )
(u'end', )
(u'end', )
(u'end', )
(u'end', )
(u'end', )
(u'end', )
(u'end', )

Hope someone could assist me, or state that this is a bug in lxml. My issues comes from the fact that I need to validate files that are too large to be read into memory using etree.XML(). If there is a way I could do it without iterparse - I'd gladly learn it. Best regards, Adam. -------------- next part -------------- A non-text attachment was scrubbed... Name: input.xml Type: text/xml Size: 418 bytes Desc: not available Url : http://codespeak.net/pipermail/lxml-dev/attachments/20100702/67fd6cfb/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: internalSchema.dtd Type: application/xml-dtd Size: 602 bytes Desc: not available Url : http://codespeak.net/pipermail/lxml-dev/attachments/20100702/67fd6cfb/attachment-0001.bin From stefan_ml at behnel.de Fri Jul 2 17:57:54 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 17:57:54 +0200 Subject: [lxml-dev] Fwd: iterparse() doesn't do dtd_validation In-Reply-To: <4C2E0A0A.40009@rdprojekt.pl> References: <4C2E0A0A.40009@rdprojekt.pl> Message-ID: <4C2E0C82.7020808@behnel.de> Adam Biela?ski, 02.07.2010 17:47: > Can anyone help me with validating XML file against internal DTD with > iterparse()? I just can't make iterparse() use dtd_validation flag. > [...] > > > ... To reference a DTD, your XML document needs a DOCTYPE declaration. http://xmlsoft.org/xmldtd.html Once that's in the document, the "dtd_validation" flag should work. Stefan From ab at rdprojekt.pl Fri Jul 2 19:26:26 2010 From: ab at rdprojekt.pl (=?UTF-8?Q?Adam_Biela=C5=84ski?=) Date: Fri, 02 Jul 2010 10:26:26 -0700 Subject: [lxml-dev] =?utf-8?q?Fwd=3A_iterparse=28=29_doesn=27t_do_dtd=5Fva?= =?utf-8?q?lidation?= In-Reply-To: <4C2E0C82.7020808@behnel.de> References: <4C2E0A0A.40009@rdprojekt.pl> <4C2E0C82.7020808@behnel.de> Message-ID: On Fri, 02 Jul 2010 17:57:54 +0200, Stefan Behnel wrote: > Adam Biela?ski, 02.07.2010 17:47: >> Can anyone help me with validating XML file against internal DTD with >> iterparse()? I just can't make iterparse() use dtd_validation flag. > > [...] > > > > > > ... > > To reference a DTD, your XML document needs a DOCTYPE declaration. > > http://xmlsoft.org/xmldtd.html > > Once that's in the document, the "dtd_validation" flag should work. > > Stefan Well, my problem is that I need to validate XML with *internal* DTD (one that is embedded in file). It's because XML files come from untrusted parties and I can't rely on them providing correct DTD declaration, so I'm putting my DTD into XML file with this custom stream class mentioned in my previous post. It works when I use that stream output and do etree.XML(stream.read(), etree.XMLParser(dtd_validation=True)), that's why I still believe that there is a way to make it work also with iterparse(). In other words: I need to make a program that takes a potentially large XML file from user, embeds known and constant (not provided by user) DTD into it and feeds the result (XML with embedded DTD) to lxml validator without putting whole XML tree into memory. The part that combines DTD with XML already exists as a stream and works fine with etree.XML(). Now I want to have incremental validation (to save memory), so I wanted to use etree.iterparse(). Does lxml design allows it, is there some special 'catch' (like - DTD has to be surrounded with something else than just ) to make it work, or wouldn't it work at all this way? Regards, Adam. From stefan_ml at behnel.de Fri Jul 2 20:37:00 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 20:37:00 +0200 Subject: [lxml-dev] Fwd: iterparse() doesn't do dtd_validation In-Reply-To: References: <4C2E0A0A.40009@rdprojekt.pl> <4C2E0C82.7020808@behnel.de> Message-ID: <4C2E31CC.1070205@behnel.de> Adam Biela?ski, 02.07.2010 19:26: > Well, my problem is that I need to validate XML with *internal* DTD (one > that is embedded in file). I guess you mean "external" here, i.e. not the internal subset that is included in the XML document but a DTD in an external file. In this case, even one that is not referenced by the document itself. > I need to make a program that takes a potentially large XML file from > user, embeds known and constant (not provided by user) DTD into it and > feeds the result (XML with embedded DTD) to lxml validator without putting > whole XML tree into memory. The part that combines DTD with XML already > exists as a stream and works fine with etree.XML(). Now I want to have > incremental validation (to save memory), so I wanted to use > etree.iterparse(). Does lxml design allows it, is there some special > 'catch' (like - DTD has to be surrounded with something else than just > ) to make it work, or wouldn't it work at all this way? There isn't currently a way to provide an unrelated DTD to the parser, but you can push your DTD through "trang" to generate an XML Schema from it, which you can then pass into the parser. http://www.thaiopensource.com/relaxng/trang.html Stefan From piet at cs.uu.nl Sat Jul 3 05:05:33 2010 From: piet at cs.uu.nl (Piet van Oostrum) Date: Fri, 2 Jul 2010 23:05:33 -0400 Subject: [lxml-dev] html.fromstring returning encoded string from a non unicoded string source In-Reply-To: References: Message-ID: <19502.43261.171205.402707@cochabamba.vanoostrum.org> >>>>> matt barto (mb) wrote: > Hello, > I am trying to obtain a title from a website which has a Unicode tm and > register mark, but the Unicode behavior is not what I expect. > tree = html.fromstring("

Apple® - iPad™ with Wi-Fi - > 16GB

" > print tree.text_content() --------> print(tree.text_content()) > Apple? - iPad? with Wi-Fi - 16GB > I would expect the output from the print is "Apple® - iPad™ with > Wi-Fi - 16GB", but it seems some encoding occurred during the tree creation.? > The browser is able to handle these html characters correctly using > "iso-8856-1" char set (even though the document says "utf-8"). > Can you provide some insight how these two html tags are handled and what I > can do to have the expected behavior? My lxml version is 2.2.2. ® is the same as ?. 174 is the Unicode code point for ? and this has no relation to the character set used (it is a common misunderstanding that it is the code point in the current encoding but it is not). ™ isn't a valid Unicode code point, but apparently it is accepted because this is the Windows-1252 code and these are often used by mistake as if they are Unicode code points. In that case it is the code for ?. But the official code for trademark is ™ In HTML there is no difference between the numeric code and the corresponding character, therefore the parser parses the code to get the Unicode character. On output you can get the entity code back by using tostring with the default encoding (=ASCII). Note that this will also output numeric entity codes for non-ASCII characters that were in the string as characters rather than as numeric entity codes. >>> html.tostring(tree[0][0]) '

Apple® - iPad™ with Wi-Fi - 16GB

' or if you want only the text, use the string encode method: >>> tree[0][0].text.encode('ascii', 'xmlcharrefreplace') 'Apple® - iPad™ with Wi-Fi - 16GB' >>> print(tree.text_content().encode('ascii', 'xmlcharrefreplace')) Apple® - iPad™ with Wi-Fi - 16GB -- Piet van Oostrum WWW: http://pietvanoostrum.com/ PGP key: [8DAE142BE17999C4] Nu Fair Trade woonwaar op http://www.zylja.com From p.oberndoerfer at urheberrecht.org Sat Jul 3 17:59:59 2010 From: p.oberndoerfer at urheberrecht.org (=?iso-8859-1?Q?=22Pascal_Obernd=F6rfer=22?=) Date: Sat, 3 Jul 2010 17:59:59 +0200 Subject: [lxml-dev] lxml eggs for Mac OS X 10.4 (maybe 10.5?) Message-ID: All, Don't know if these eggs are of help to anybody else on "older" Macs (i.e. 10.4 on ppc and i386): I needed lxml on such machines, so here they are at your own risk! The 2.2.6 eggs should statically link to: -libxml2-2.7.7 -libxslt-1.1.26 -libiconv-1.13.1 -zlib-1.2.5 Only tested with 'selftest.py' and 'selftest2.py'. All tests pass. But further testing is certainly needed! Anybody with PyPI rights, feel free to upload them, as I cannot guarantee the above link will remain forever ;-) Thanks. Pascal From egnor at ofb.net Mon Jul 5 02:20:01 2010 From: egnor at ofb.net (Dan Egnor) Date: Sun, 4 Jul 2010 17:20:01 -0700 Subject: [lxml-dev] Handling charset properly in lxml.html.parse() and friends In-Reply-To: References: Message-ID: I'm not sure if this is a bug, a missing feature, maybe something that should be discussed in the documentation/FAQ more clearly, or maybe I'm just a moron and missed something obvious, but: Handling charsets in HTML is always challenging. Charset information may come from an HTTP Content-Type header (if the HTML comes via HTTP, which often it does), from an http-equiv meta tag in the file itself, from some other external assertion, or it may need to be guessed from the content. The lxml documentation gives this advice at one point: Similarly, you will get errors when you try the same with HTML data in a unicode string that specifies a charset in a meta tag of the header. You should generally avoid converting XML/HTML data to unicode before passing it into the parsers. It is both slower and error prone. However, if you take that advice, then I don't know any way to supply external charset information to lxml.html.parse() (or lxml.html.document_fromstring() and so on). These functions also don't seem to do much in the way of charset guessing. The *html5lib* parser has a "guess_charset" flag, but the ordinary HTML parser apparently does not. I have resorted to calling chardet myself and converting the input bytes to Unicode using whatever charset I get from chardet, and passing Unicode to lxml.html.document_fromstring(), in explicit contravention of the advice quoted above. This ignores HTTP headers (which I do have access to in my application, which is a content transforming proxy) as well as meta tags in the document, which is a bummer. Ideally, the HTML parsing functions would accept a byte string along with any charset assertions you might have access to, and then do the right thing (using your assertions, looking for meta tags, and applying chardet, in some order). _Really_ ideally it would even parse Content-type header values (since it will need to do so to understand the meta tag anyway). Maybe there's a way to get it to do something like this, but if so I couldn't figure it out. Are there currently known best practices? Searching around the forums, I find other puzzled users, but no crisp answers. I'm happy to help however I can. -- egnor -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/lxml-dev/attachments/20100704/c6875da5/attachment.htm From stefan_ml at behnel.de Mon Jul 5 07:35:54 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 07:35:54 +0200 Subject: [lxml-dev] lxml eggs for Mac OS X 10.4 (maybe 10.5?) In-Reply-To: References: Message-ID: <4C316F3A.1050602@behnel.de> "Pascal Obernd?rfer", 03.07.2010 17:59: > Don't know if these eggs are of help to anybody > else on "older" Macs (i.e. 10.4 on ppc and i386): > > > > I needed lxml on such machines, so here they are at your own risk! > > The 2.2.6 eggs should statically link to: > > -libxml2-2.7.7 > -libxslt-1.1.26 > -libiconv-1.13.1 > -zlib-1.2.5 > > Only tested with 'selftest.py' and 'selftest2.py'. All tests pass. But > further testing is certainly needed! > > Anybody with PyPI rights, feel free to upload them, as I cannot guarantee > the above link will remain forever ;-) I uploaded the fat egg. Thanks a lot! Stefan From stefan_ml at behnel.de Mon Jul 5 08:04:22 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 08:04:22 +0200 Subject: [lxml-dev] Handling charset properly in lxml.html.parse() and friends In-Reply-To: References:

SchemaVersion 21

Attribute name Regards, Adam. P.S. Seems like my previous post was incorrectly interpreted as an answer to my post so I'm resending it. Sorry for the mess. W dniu 2010-07-02 20:37, Stefan Behnel pisze: > Adam Biela?ski, 02.07.2010 19:26: >> Well, my problem is that I need to validate XML with *internal* DTD (one >> that is embedded in file). > > I guess you mean "external" here, i.e. not the internal subset that is > included in the XML document but a DTD in an external file. In this > case, even one that is not referenced by the document itself. > > >> I need to make a program that takes a potentially large XML file from >> user, embeds known and constant (not provided by user) DTD into it and >> feeds the result (XML with embedded DTD) to lxml validator without >> putting >> whole XML tree into memory. The part that combines DTD with XML already >> exists as a stream and works fine with etree.XML(). Now I want to have >> incremental validation (to save memory), so I wanted to use >> etree.iterparse(). Does lxml design allows it, is there some special >> 'catch' (like - DTD has to be surrounded with something else than just >> ) to make it work, or wouldn't it work at all this >> way? > > There isn't currently a way to provide an unrelated DTD to the parser, > but you can push your DTD through "trang" to generate an XML Schema > from it, which you can then pass into the parser. > > http://www.thaiopensource.com/relaxng/trang.html > > Stefan > From burak.arslan at arskom.com.tr Tue Jul 6 13:22:39 2010 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Tue, 06 Jul 2010 14:22:39 +0300 Subject: [lxml-dev] validating one document with multiple xml schemas In-Reply-To: <4C32F9BE.7080801@behnel.de> References: <4C31FBA1.4040906@arskom.com.tr> <20100706065051.302120@gmx.net> <4C32E22A.4080403@arskom.com.tr> <4C32E2E8.2030206@behnel.de> <4C32F61F.7000100@arskom.com.tr> <4C32F7F1.2040301@behnel.de> <4C32F9BE.7080801@behnel.de> Message-ID: <4C3311FF.5020309@arskom.com.tr> On 07/06/10 12:39, Stefan Behnel wrote: > Stefan Behnel, 06.07.2010 11:31: >> Burak Arslan, 06.07.2010 11:23: >>> On 07/06/10 11:01, Stefan Behnel wrote: >>>> Burak Arslan, 06.07.2010 09:58: >>>>> On 07/06/10 09:50, jholg at gmx.de wrote: >>>>>> Note that you need to give a hint to the imported schema's location >>>>>> with the schemaLocation attribute; if you leave this out the >>>>>> XMLSchema >>>>>> instantiation will fail: >>>>> >>>>> schemaLocation attribute is not specified for schemas that are >>>>> part of >>>>> the same wsdl file. >>>>> >>>>> See an example here: >>>>> >>>>> http://mssoapinterop.org/asmx/simple.asmx?WSDL >>>>> >>>>> I'm looking for a way to manually suppy missing schema tags to >>>>> XMLSchema >>>>> object before it throws an exception. >>>> >>>> What about a >>>> >>>> - parse schema document >>>> - insert schemaLocation >>>> - hand over to XMLSchema() >>>> >>>> approach? >>> >>> and write schema nodes to separate temporary files while doing that? >>> that'd be a horrible hack, but i guess you're suggesting this because >>> there's no easy way to patch lxml (or even libxml2) to support manual >>> schema node feeding, right? >> >> Erm, no, I was just suggesting that because you mentioned that your >> schema >> documents were missing the schemaLocation attribute, so adding it sounds >> like a viable solution to me. >> >> However, now that you mention it, I never tried if the custom document >> loaders are used by the XML Schema parser. Would be worth testing here. > > At least according to the sources, it should just work: > > http://codespeak.net/lxml/resolvers.html > > Stefan hi stefan, thanks for the pointer. there's one issue with it though: i'm generating the xml schema, not parsing it from some input. is there a way to have the resolvers work for already-parsed input? thanks, burak From stefan_ml at behnel.de Tue Jul 6 13:30:23 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 06 Jul 2010 13:30:23 +0200 Subject: [lxml-dev] validating one document with multiple xml schemas In-Reply-To: <4C3311FF.5020309@arskom.com.tr> References: <4C31FBA1.4040906@arskom.com.tr> <20100706065051.302120@gmx.net> <4C32E22A.4080403@arskom.com.tr> <4C32E2E8.2030206@behnel.de> <4C32F61F.7000100@arskom.com.tr> <4C32F7F1.2040301@behnel.de> <4C32F9BE.7080801@behnel.de> <4C3311FF.5020309@arskom.com.tr> Message-ID: <4C3313CF.9050004@behnel.de> Burak Arslan, 06.07.2010 13:22: > On 07/06/10 12:39, Stefan Behnel wrote: >> http://codespeak.net/lxml/resolvers.html > > thanks for the pointer. there's one issue with it though: i'm generating > the xml schema, not parsing it from some input. is there a way to have > the resolvers work for already-parsed input? How do you generate it? Do you build it manually through the ET API? Even then, it would be enough to parse the root element instead of calling Element(), and then appending to that. Or parse it once and then use deepcopy. Or ... Stefan From mike_mp at zzzcomputing.com Tue Jul 6 19:33:02 2010 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Tue, 6 Jul 2010 13:33:02 -0400 Subject: [lxml-dev] schema validation issue Message-ID: <9098E2B4-D8F6-4729-B709-1F8B9AC21D62@zzzcomputing.com> I was playing with the libxml versions that underlie my lxml installation in order to help someone with the "static deps" flag. I noticed that when I build against the latest libxml (2.7.7), as opposed to the older one I have lying around on my mac (2.2.6) , some validations change behavior. The script below illustrates three XML docs that validate completely with 2.2.6 but the second two fail on 2.7.7. My question is, does elem1, elem2, .. mean that *any* of elem1, elem2, etc. can be present , or that every subelement, or none, must be present ? In 2.7.7 it seems to be interpreting it as, "all of elem1, elem2, elem3, ... must be present, or none". This doesn't match what I can find in the xml-schema spec (http://www.w3.org/TR/xmlschema-0/#ref18, "All the elements in the group may appear once or not at all, and they may appear in any order.") and various online tutorials (http://www.w3schools.com/Schema/el_all.asp, "The example above indicates that the "firstname" and the "lastname" elements can appear in any order and each element CAN appear zero or one time!"). So is libxml wrong or am I misinterpreting ? How would I lay out a tag like with each child tag being independently optional ? The other one is how to make an "xs:int" that is optional, but that one I haven't researched yet. Previously, I could say without issue, now it says its an invalid integer. from lxml import etree from StringIO import StringIO schema = """ """ xmlschema = etree.XMLSchema(etree.parse(StringIO(schema))) # passes doc1 = """

some value

""" # fails. it wants both "int-attr" and "str-attr" to be present. # didn't think this was how "xs:all" worked ? doc2 = """

""" # fails. doesn't allow blank for "type='xs:int'". doc3 = """

some value

Message-ID: <20100729182408.91116and866y2r48@boilermail.purdue.edu> If you are referring to the attributes of an Element in the XML - I think you should be using the get() method of the Element class. More details on the API of the Element class can be found here http://codespeak.net/lxml/api/lxml.etree._Element-class.html ~Shankar. Quoting Dirk Rothe : > On Thu, 29 Jul 2010 18:58:25 +0200, sakshi chawla > wrote: > >> hi >> how can I read the attributes from xml file? > > hi, > > better explain, what "the attributes" are. > _______________________________________________ > lxml-dev mailing list > lxml-dev at codespeak.net > http://codespeak.net/mailman/listinfo/lxml-dev >