I am trying to write a script which fetches a url, logs into the site, then fetches particular items from the page, and goes to the next page, fetching the same type of files on the new page until there are no new pages to fetch from. So I need form and cooke handling, as well as manipulating the headers. What do I need to use? I found this thread, but I don't understand it: <a href="http://codespeak.net/pipermail/lxml-dev/2008-December/004272.html">http://codespeak.net/pipermail/lxml-dev/2008-December/004272.html</a>.<br>
<br>Also, I don't want to wipe out the server with so many requests, is there a "wait 2 - 3 seconds before fetching the next element" type function?..<br><br>Thank you so, so much.<br><br>-A.F.<br>