<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I wrote a tool to sync safari books downloads that does similar things to what you're talking about. I found the various issues you run into with form and cookie handling when using lxml (and wrote an article about it here: <a href="http://douglas.mayle.org/2009/03/05/syncing-safari-downloads-intro-screen-scraping/">http://douglas.mayle.org/2009/03/05/syncing-safari-downloads-intro-screen-scraping/</a> ). I spent some time making sure the code was clean and very well documented, so it should help you to get started. The example is here:<div><a href="http://projects.mayle.org/hg/safarisync/file/23cfad04ce3a/safarisync/safarisync/safarisync.py">http://projects.mayle.org/hg/safarisync/file/23cfad04ce3a/safarisync/safarisync/safarisync.py</a></div><div><br></div><div>Douglas Mayle</div><div><br><div><div><div>On Apr 23, 2009, at 2:41 AM, Akafubu Kibombo wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">I am trying to write a script which fetches a url, logs into the site, then fetches particular items from the page, and goes to the next page, fetching the same type of files on the new page until there are no new pages to fetch from. So I need form and cooke handling, as well as manipulating the headers. What do I need to use? I found this thread, but I don't understand it: <a href="http://codespeak.net/pipermail/lxml-dev/2008-December/004272.html">http://codespeak.net/pipermail/lxml-dev/2008-December/004272.html</a>.<br> <br>Also, I don't want to wipe out the server with so many requests, is there a "wait 2 - 3 seconds before fetching the next element" type function?..<br><br>Thank you so, so much.<br><br>-A.F.<br> _______________________________________________<br>lxml-dev mailing list<br><a href="mailto:lxml-dev@codespeak.net">lxml-dev@codespeak.net</a><br><a href="http://codespeak.net/mailman/listinfo/lxml-dev">http://codespeak.net/mailman/listinfo/lxml-dev</a><br></blockquote></div><br></div></div></body></html>