Dear Stefan,<br><br>I've tried the method you've suggested below, but it isn't quite working for me. It may be that I've misunderstood your suggestion. I'll explain what I've tried. Here is my python program, extract_links_dmoz.py:<br>
<br>from lxml import etree<br>infile = open("content.example.xml", "r")<br>infile.seek(0)<br>outfile = open("output_test001.txt", "w")<br>class EchoTarget():<br> def start(self, tag, attrib):<br>
if tag.endswith("xternalPage"):<br> line = attrib["about"]<br> if line != "":<br> outfile.write(line+"\n")<br> print line<br>
def close(self):<br> return "closed!"<br>parser = etree.XMLParser(target = EchoTarget())<br>result = etree.XML(infile.read(), parser)<br><br>This uses the short, example RDF file at <a href="http://rdf.dmoz.org/rdf/content.example.txt">http://rdf.dmoz.org/rdf/content.example.txt</a> (which I have renamed content.example.xml), and works fine. When I view the output_test001.txt file, it contains one URL per line, which is exactly what I want for now.<br>
<br>However, if I change the program to read content.rdf.u8.xml (i.e. the full-length DMOZ links file from <a href="http://rdf.dmoz.org/rdf/content.rdf.u8.gz">http://rdf.dmoz.org/rdf/content.rdf.u8.gz</a>) instead of content.example.xml , then when I run the program I get the following error:<br>
<br>Traceback (most recent call last):<br> File "extract_links_dmoz.py", line 26, in <module><br> result = etree.XML(infile.read(), parser)<br>MemoryError<br><br>Any help you (or others) can offer would be greatly appreciated.<br>
<br>Many thanks,<br><br>Sam<br><br><div class="gmail_quote">2008/5/22 Stefan Behnel <<a href="mailto:stefan_ml@behnel.de">stefan_ml@behnel.de</a>>:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi,<br>
<div><div></div><div class="Wj3C7c"><br>
Sam Kuper wrote:<br>
> Gosh, this is turning into a really fragmented post; apologies. I meant to<br>
> add to the first post that once parsed, my intention was to run a fairly<br>
> simple XSL transform on the document, to extract a copy of each of the URLs<br>
> it contains. Probably something like this:<br>
> <?xml version="1.0" encoding="UTF-8"?><br>
> <xsl:stylesheet version="1.0" xmlns:xsl="<br>
> <a href="http://www.w3.org/1999/XSL/Transform" target="_blank">http://www.w3.org/1999/XSL/Transform</a>"><br>
> <xsl:template match="/"><br>
> <html><br>
> <body><br>
> <h2>ODP URLs</h2><br>
> <xsl:for-each select="Topic/link"><br>
> <p><xsl:value-of select="@r:resource"/></p><br>
> </xsl:for-each><br>
> </body><br>
> </html><br>
> </xsl:template><br>
> </xsl:stylesheet><br>
<br>
</div></div>That is a problem that can be solved with extremely little memory. Take a look<br>
at the (SAX-like) target parser interface, which will not build a tree and<br>
instead just receive callbacks while parsing:<br>
<br>
<a href="http://codespeak.net/lxml/dev/parsing.html#the-target-parser-interface" target="_blank">http://codespeak.net/lxml/dev/parsing.html#the-target-parser-interface</a><br>
<br>
Write a parser target class that keeps track of being inside or outside the<br>
"Topic" tag (start/end), and whenever you find a "link" tag while inside a<br>
"Topic" tag, look for a "{whatever-namespace}resource" attribute in the attrib<br>
dictionary and and write it into a hand-generated HTML stream like the one you<br>
used above.<br>
<font color="#888888"><br>
Stefan<br>
</font></blockquote></div><br><br>