<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Ahh, randomly enough, the thread you link to is the one I started. After browsing through the lxml code, it turned out that there was no need to pass an open_http parameter, as the default method did almost exactly the same thing as the code sample given and so monkey patching the library (the standard way to add cookie support) already works. Unfortunately, I found out that passing a URL directly to lxml causes it to use libxml's native downloading support, which has no support for cookies. As such, you have to handle all of the downloading of content yourself (except when taking advantage of lxml forms).<div><br></div><div>As to waiting 2-3 seconds before requests, you can just put sleeps into your code, or find some sort of bandwidth throttling package...</div><div><br></div><div>Douglas Mayle</div><div><br><div><div>On Apr 23, 2009, at 2:41 AM, Akafubu Kibombo wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">I am trying to write a script which fetches a url, logs into the site, then fetches particular items from the page, and goes to the next page, fetching the same type of files on the new page until there are no new pages to fetch from. So I need form and cooke handling, as well as manipulating the headers. What do I need to use? I found this thread, but I don't understand it: <a href="http://codespeak.net/pipermail/lxml-dev/2008-December/004272.html">http://codespeak.net/pipermail/lxml-dev/2008-December/004272.html</a>.<br> <br>Also, I don't want to wipe out the server with so many requests, is there a "wait 2 - 3 seconds before fetching the next element" type function?..<br><br>Thank you so, so much.<br><br>-A.F.<br> _______________________________________________<br>lxml-dev mailing list<br><a href="mailto:lxml-dev@codespeak.net">lxml-dev@codespeak.net</a><br><a href="http://codespeak.net/mailman/listinfo/lxml-dev">http://codespeak.net/mailman/listinfo/lxml-dev</a><br></blockquote></div><br></div></body></html>