[wwwsearch-commits] r26900 - wwwsearch/mechanize/trunk
jjlee at codespeak.net
jjlee at codespeak.net
Sat May 6 22:42:10 CEST 2006
Author: jjlee
Date: Sat May 6 22:42:09 2006
New Revision: 26900
Removed:
wwwsearch/mechanize/trunk/cc-README.html.in
wwwsearch/mechanize/trunk/pp-README.html.in
Modified:
wwwsearch/mechanize/trunk/MANIFEST.in
wwwsearch/mechanize/trunk/README.html.in
wwwsearch/mechanize/trunk/doc.html.in
Log:
Minimal documentation integration
Modified: wwwsearch/mechanize/trunk/MANIFEST.in
==============================================================================
--- wwwsearch/mechanize/trunk/MANIFEST.in (original)
+++ wwwsearch/mechanize/trunk/MANIFEST.in Sat May 6 22:42:09 2006
@@ -8,5 +8,6 @@
include doc.html.in
include doc.html
include ChangeLog.txt
+include 0.1.0-changes.txt
include *.py
recursive-include examples *.py
Modified: wwwsearch/mechanize/trunk/README.html.in
==============================================================================
--- wwwsearch/mechanize/trunk/README.html.in (original)
+++ wwwsearch/mechanize/trunk/README.html.in Sat May 6 22:42:09 2006
@@ -2,10 +2,16 @@
"http://www.w3.org/TR/html4/strict.dtd">
@# This file is processed by EmPy: do not edit
@# http://wwwsearch.sf.net/bits/colorize.py
-@{from colorize import colorize}
-@{import time}
-@{import release}
-@{last_modified = release.svn_id_to_time("$Id$")}
+@{
+from colorize import colorize
+import time
+import release
+last_modified = release.svn_id_to_time("$Id$")
+try:
+ base
+except NameError:
+ base = False
+}
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
@@ -15,7 +21,7 @@
<meta name="keywords" content="cookie,HTTP,Python,web,client,client-side,HTML,META,HTTP-EQUIV,Refresh">
<title>mechanize</title>
<style type="text/css" media="screen">@@import "../styles/style.css";</style>
- <base href="http://wwwsearch.sourceforge.net/mechanize/">
+ @[if base]<base href="http://wwwsearch.sourceforge.net/mechanize/">@[end if]
</head>
<body>
@@ -34,10 +40,11 @@
</a>.
<ul>
+
<li><code>mechanize.Browser</code> is a subclass of
<code>mechanize.UserAgent</code>, which is, in turn, a subclass of
- <code>urllib2.OpenerDirector</code> (or of
- <code>ClientCookie.OpenerDirector</code> for pre-2.4 versions of Python), so:
+ <code>urllib2.OpenerDirector</code> (in fact, of
+ <code>mechanize.OpenerDirector</code>), so:
<ul>
<li>any URL can be opened, not just <code>http:</code>
<li><code>mechanize.UserAgent</code> offers easy dynamic configuration of
@@ -55,14 +62,16 @@
<li>Automatic observance of <a
href="http://www.robotstxt.org/wc/norobots.html">
<code>robots.txt</code></a>.
- <li>Automatic handling of HTTP-Equiv and Refresh, using <a
- href="../ClientCookie/">ClientCookie</a>.
+ <li>Automatic handling of HTTP-Equiv and Refresh.
</ul>
<a name="examples"></a>
<h2>Examples</h2>
+<p class="docwarning">This documentation is in need of reorganisation and
+extension!</p>
+
<p>The two below are just to give the gist. There are also some <a
href="./#tests">actual working examples</a>.
@@ -115,7 +124,7 @@
br.set_handle_robots(False)
# Don't handle cookies
br.set_cookiejar()
-# Supply your own ClientCookie.CookieJar (NOTE: cookie handling is ON by
+# Supply your own mechanize.CookieJar (NOTE: cookie handling is ON by
# default: no need to do this unless you have some reason to use a
# particular cookiejar)
br.set_cookiejar(cj)
@@ -127,30 +136,78 @@
br.set_debug_http(True)
# To make sure you're seeing all debug output:
-for logger in [
- logging.getLogger("ClientCookie"),
- logging.getLogger("cookielib"),
- ]:
- logger.addHandler(logging.StreamHandler(sys.stdout))
- logger.setLevel(logging.INFO)
+logger = logging.getLogger("mechanize")
+logger.addHandler(logging.StreamHandler(sys.stdout))
+logger.setLevel(logging.INFO)
""")}
+<a name="compatnotes"></a>
+<h2>Compatibility</h2>
+
+<p>This note explains the relationship between mechanize, ClientCookie,
+<code>cookielib</code> and <code>urllib2</code>, and which to use when.
+
+<ol>
+
+ <li>ClientCookie is no longer maintained separately, and its interface is now
+ exported through module mechanize (since mechanize 0.1.0). Old code may
+ simply be changed to <code>import mechanize as ClientCookie</code> and
+ should continue to work.
+
+ <li>The cookie handling parts of mechanize are in Python 2.4 standard library
+ as module <code>cookielib</code> and extensions to module
+ <code>urllib2</code>.
+
+ <li>mechanize works with Python 2.3, Python 2.4 and Python 2.5.
+
+ <li>Handler classes that are missing from 2.4's <code>urllib2</code>
+ (eg. <code>HTTPRefreshProcessor</code>) may be used with 2.4's
+ <code>urllib2</code> (however, note the paragraph below). With any
+ version of Python, <code>urllib2</code> handlers that are missing from
+ mechanize (eg. <code>ProxyHandler</code>) may be used with mechanize, and
+ <code>urllib2.Request</code> objects may be used with mechanize.
+ <strong>IMPORTANT:</strong> For all other code, use mechanize
+ <em><strong>exclusively</strong></em>: do NOT mix use of mechanize and
+ <code>urllib2</code>!
+
+ <li>If you want to use <code>mechanize.RefreshProcessor</code> with Python
+ >= 2.4's <code>urllib2</code>, you must also use
+ <code>mechanize.HTTPRedirectHandler</code>.
+
+ <li><code>mechanize.HTTPRefererProcessor</code> requires special support from
+ <code>mechanize.Browser</code>, so cannot be used with vanilla
+ <code>urllib2</code>.
+
+ <li><code>mechanize.HTTPRequestUpgradeProcessor</code> and
+ <code>mechanize.ResponseUpgradeProcessor</code> are not useful outside of
+ mechanize.
+
+</ol>
+
+
<a name="docs"></a>
<h2>Documentation</h2>
<p>Full documentation is in the docstrings.
+<p>The documentation in the web pages is in need of reorganisation at the
+moment, after the merge of ClientCookie into mechanize.
+
<a name="credits"></a>
<h2>Credits</h2>
-<p>Thanks to Ian Bicking, for persuading me that a <code>UserAgent</code> class
-would be useful, and to everyone who has reported bugs.
-
-<p>And of course thanks to Andy Lester for the original, <a
-href="http://search.cpan.org/dist/WWW-Mechanize/"><code>WWW::Mechanize</code>
-</a>.
+<p>Thanks to all the too-numerous-to-list people who reported bugs and provided
+patches. Also thanks to Ian Bicking, for persuading me that a
+<code>UserAgent</code> class would be useful, and to Ronald Tschalar for advice
+on Netscape cookies.
+
+<p>A lot of credit must go to Gisle Aas, who wrote libwww-perl, from which
+large parts of mechanize originally derived, and Andy Lester for the original,
+<a href="http://search.cpan.org/dist/WWW-Mechanize/"><code>WWW::Mechanize</code>
+</a>. Finally, thanks to the (coincidentally-named) Johnny Lee for the MSIE
+CookieJar Perl code from which mechanize's support for that is derived.
<a name="todo"></a>
@@ -191,10 +248,10 @@
<li>Implement RFC 3986 URL absolutization.
<li>Figure out the Right Thing (if such a thing exists) for %-encoding.
<li>How do IRIs fit into the world?
- <li>IDNA (ClientCookie) -- must read about security stuff first.
+ <li>IDNA -- must read about security stuff first.
<li>Unicode support in general (not sure yet how/when/whether this will
happen).
- <li>Provide per-connection access to timeouts (ClientCookie).
+ <li>Provide per-connection access to timeouts.
<li>Keep-alive / connection caching.
<li>Pipelining??
<li>Content negotiation.
@@ -222,9 +279,7 @@
<h2>EasyInstall / setuptools</h2>
<p>The benefit of EasyInstall and the new <code>setuptools</code>-supporting
-<code>setup.py</code> is that they grab all dependencies for you (viz,
-ClientForm, ClientCookie, and either pullparser or
-<a href="http://www.crummy.com/software/BeautifulSoup/">Beautiful Soup</a>).
+<code>setup.py</code> is that they grab all dependencies for you.
<p><strong>You need EasyInstall version 0.6a8 or newer.</strong>
@@ -259,7 +314,7 @@
<pre>easy_install "mechanize==dev"</pre>
<p>Note that that will not necessarily grab the SVN versions of dependencies,
-such as ClientCookie: It will use SVN to fetch dependencies if and only if the
+such as ClientForm: It will use SVN to fetch dependencies if and only if the
SVN HEAD version of mechanize declares itself to depend on the SVN versions of
those dependencies; even then, those declared dependencies won't necessarily be
on SVN HEAD, but rather a particular revision. If you want SVN HEAD for a
@@ -359,10 +414,18 @@
href="../ClientForm/">ClientForm web page</a> are useful for mechanize users,
and are now real run-able scripts rather than just documentation.
+<h3>Functional tests</h3>
+
+<p>To run the functional tests (which <strong>do</strong> access the network),
+run the following
+
+command:
+<pre>python functional_tests.py</pre>
+
<h3>Unit tests</h3>
-<p>Note that the dependencies of mechanize (ClientCookie, ClientForm, and
-pullparser) have their own unit tests, which must be run separately.
+<p>Note that ClientForm (a dependency of mechanize) has its own unit tests,
+which must be run separately.
<p>To run the unit tests (none of which access the network), run the following
command:
@@ -411,9 +474,7 @@
<li>Which version of Python do I need?
<p>2.3 or above.
<li>What else do I need?
- <p><a href="../ClientCookie/">ClientCookie</a>,
- <a href="../ClientForm/">ClientForm</a> and
- <a href="../pullparser/">pullparser</a>.
+ <p>mechanize depends on <a href="../ClientForm/">ClientForm</a>.
<p>The versions of those required modules are listed in the
<code>setup.py</code> for mechanize (included with the download). The
dependencies are automatically fetched by <a
@@ -429,6 +490,7 @@
included in the distribution).
</ul>
+
<p>I prefer questions and comments to be sent to the <a
href="http://lists.sourceforge.net/lists/listinfo/wwwsearch-general">
mailing list</a> rather than direct to me.
@@ -447,6 +509,7 @@
<br>
<a href="./#examples">Examples</a><br>
+<a href="./#compatnotes">Compatibility</a><br>
<a href="./#docs">Documentation</a><br>
<a href="./#todo">To-do</a><br>
<a href="./#download">Download</a><br>
Deleted: /wwwsearch/mechanize/trunk/cc-README.html.in
==============================================================================
--- /wwwsearch/mechanize/trunk/cc-README.html.in Sat May 6 22:42:09 2006
+++ (empty file)
@@ -1,236 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
- "http://www.w3.org/TR/html4/strict.dtd">
-@# This file is processed by EmPy to colorize Python source code
-@# http://wwwsearch.sf.net/bits/colorize.py
-@{from colorize import colorize}
-@{import time}
-@{import release}
-@{last_modified = release.svn_id_to_time("$Id$")}
-<html>
-<head>
- <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
- <meta name="author" content="John J. Lee <jjl@@pobox.com>">
- <meta name="date" content="@(time.strftime("%Y-%m-%d", last_modified))">
- <meta name="keywords" content="cookie,HTTP,Python,web,client,client-side,HTML,META,HTTP-EQUIV,Refresh">
- <title>ClientCookie</title>
- <style type="text/css" media="screen">@@import "../styles/style.css";</style>
- <style type="text/css" media="screen">@@import "../styles/cookie_style.css";</style>
- <base href="http://wwwsearch.sourceforge.net/ClientCookie/">
-</head>
-<body>
-
-<div id="sf"><a href="http://sourceforge.net">
-<img src="http://sourceforge.net/sflogo.php?group_id=48205&type=2"
- width="125" height="37" alt="SourceForge.net Logo"></a></div>
-<!--<img src="../images/sflogo.png"-->
-
-<h1>ClientCookie</h1>
-
-<div id="Content">
-
-<p>Please read <a href="./#compatnotes">this note</a> explaining the
-relationship between ClientCookie, <code>cookielib</code> and
-<code>urllib2</code>, and which to use when.
-
-<p>ClientCookie is a <a href="http://www.python.org/">Python</a> module for
-handling HTTP cookies on the client side, useful for accessing web sites that
-require cookies to be set and then returned later. It also provides some other
-(optional) useful stuff: <code>HTTP-EQUIV</code> and <code>Refresh</code>
-handling, automatic adding of the <code>Referer</code> [<em><a
-href="http://www.ietf.org/rfc/rfc2616.txt">sic</a></em>] header, automatic
-observance of <code>robots.txt</code> and lazily-<code>seek()</code>able
-responses. These extras are implemented using an extension that makes it
-easier to add new functionality to <code>urllib2</code>. It has developed from
-a port of Gisle Aas' Perl module <code>HTTP::Cookies</code>, from the <a
-href="http://www.linpro.no/lwp/">libwww-perl</a> library.
-
-@{colorize(r"""
- import ClientCookie
- response = ClientCookie.urlopen("http://foo.bar.com/")
-""")}
-
-<p>This function behaves identically to <code>urllib2.urlopen</code>, except
-that it deals with cookies automatically. That's probably all you need to
-know.
-
-<p>Python 2.0 or above is required, and <code>urllib2</code> is recommended.
-If you have 2.1 or above, you've already got a recent enough version of
-<code>urllib2</code>. For Python 2.0, you need the newer versions from Python
-2.1 (available from the source distribution or Python CVS: <a
-href="http://cvs.sourceforge.net/viewcvs.py/*checkout*/python/python/dist/src/Lib/urllib2.py?rev=1.13.2.2">urllib2.py</a>).
-Note that you don't need to replace the original <code>urllib2</code> /
-<code>urllib</code> - you can just make sure they're in <code>sys.path</code>
-ahead of the copies from 2.0's standard library.
-
-<p>For full documentation, see <a href="./doc.html">here</a> and the docstrings
-in the module source code.
-
-<p>Other than Gisle, particular thanks to Johnny Lee (MSIE Perl code) and
-Ronald Tschalar (advice on Netscape cookies).
-
-
-<a name="compatnotes"></a>
-<h2>Notes about ClientCookie, <code>urllib2</code> and <code>cookielib</code></h2>
-
-<p>Even if you're not using Python 2.4, please note the last of these points.
-
-<ol>
-
- <li>The cookie handling parts of ClientCookie are in Python 2.4 standard
- library as module <code>cookielib</code> and extensions to module
- <code>urllib2</code>.
-
- <li>ClientCookie works with Python 2.4.
-
- <li>For new code to run on Python 2.4, I recommend use of standard
- library modules <code>urllib2</code> and <code>cookielib</code>
- instead of ClientCookie. I recommend
- <a href="http://docs.python.org/lib/cookielib-examples.html">turning on
- RFC 2965 support</a> to work around a bug in <code>cookielib</code> in
- Python 2.4.0.
-
- <li>Handler classes thst are missing from 2.4's <code>urllib2</code>
- (eg. <code>HTTPRefreshProcessor</code>) may be used with 2.4's
- <code>urllib2</code> (however, note the paragraph below). With any
- version of Python, parts of <code>urllib2</code> that are missing from
- ClientCookie (eg. <code>ProxyHandler</code>) may be used with
- ClientCookie, and <code>urllib2.Request</code> objects may be used with
- ClientCookie. <strong>IMPORTANT:</strong> For all other code, use
- ClientCookie <em><strong>exclusively</strong></em>: do NOT mix use of
- ClientCookie and <code>urllib2</code>!
-
-</ol>
-
-<p><strong>Finally, note</strong> that, if you want to use
-<code>ClientCookie.RefreshProcessor</code> with Python 2.4's
-<code>urllib2</code>, you must also use
-<code>ClientCookie.HTTPRedirectHandler</code>.
-
-<a name="download"></a>
-<h2>Download</h2>
-
-<p>All documentation (including these web pages) is included in the
-distribution.
-
-<p>To port your code from 0.4.x to 1.0.x, see <a
-href="./porting-0.4-1.0.txt">here</a>.
-
-<p><em>Stable release.</em>
-
-<ul>
-@{version = "1.3.0"}
-<li><a href="./src/ClientCookie-@(version).tar.gz">ClientCookie-@(version).tar.gz</a>
-<li><a href="./src/ClientCookie-@(version).zip">ClientCookie-@(version).zip</a>
-<li><a href="./src/ChangeLog.txt">Change Log</a> (included in distribution)
-<li><a href="./src/">Older versions.</a>
-</ul>
-
-<br>
-
-<p><em>Old release.</em>
-
-<ul>
-@{old_version = "0.4.19"}
-@{old_win_version = release.win_version(old_version)}
-<li><a href="./src/ClientCookie-@(old_version).tar.gz">ClientCookie-@(old_version).tar.gz</a>
-<li><a href="./src/ClientCookie-@(old_win_version).zip">ClientCookie-@(old_win_version).zip</a>
-<li><a href="./src/ChangeLog.txt">Change Log</a> (included in distribution)
-<li><a href="./src/">Older versions.</a>
-</ul>
-
-<p>For installation instructions, see the INSTALL file included in the
-distribution.
-
-
-<h2>Subversion</h2>
-
-<p>The <a href="http://subversion.tigris.org/">Subversion (SVN)</a> trunk is <a href="http://codespeak.net/svn/wwwsearch/ClientCookie/trunk#egg=ClientCookie-dev">http://codespeak.net/svn/wwwsearch/ClientCookie/trunk</a>, so to check out the source:
-
-<pre>
-svn co http://codespeak.net/svn/wwwsearch/ClientCookie/trunk ClientCookie
-</pre>
-
-
-<a name="faq_pre"></a>
-<h2>FAQs - pre-install</h2>
-<ul>
- <li>Doesn't the standard Python library module, <code>Cookie</code>, do
- this?
- <p>No: Cookie.py does the server end of the job. It doesn't know when to
- accept cookies from a server or when to pass them back.
- <li>Which version of Python do I need?
- <p>2.0 or above.
- <li>Is urllib2.py required?
- <p>No. You probably want it, though.
- <li>Which urllib2.py do I need?
- <p>You don't, but if you want to use the extended <code>urllib2</code>
- callables from ClientCookie, and you have Python 2.0, you need to
- upgrade to the version from Python 2.1. Otherwise, you're OK.
- <li>Which license?
- <p>ClientCookie is dual-licensed: you may pick either the
- <a href="http://www.opensource.org/licenses/bsd-license.php">BSD license</a>,
- or the <a href="http://www.zope.org/Resources/ZPL">ZPL 2.1</a> (both are
- included in the distribution).
- <li>Where can I find out more about the HTTP cookie protocol?
- <p>There is more than one protocol, in fact (see the <a href="./doc.html">docs</a>
- for a brief explanation of the history):
- <ul>
- <li>The original <a href="http://www.netscape.com/newsref/std/cookie_spec.html">
- Netscape cookie protocol</a> - the standard still in use today, in
- theory (in reality, the protocol implemented by all the major browsers
- only bears a passing resemblance to the protocol sketched out in this
- document).
- <li><a href="http://www.ietf.org/rfcs/rfc2109.txt">RFC 2109</a> - obsoleted
- by RFC 2965.
- <li><a href="http://www.ietf.org/rfcs/rfc2965.txt">RFC 2965</a> - the
- Netscape protocol with the bugs fixed (not widely used - the Netscape
- protocol still dominates, and seems likely to remain dominant
- indefinitely, at least on the Internet).
- <a href="http://www.ietf.org/rfcs/rfc2964.txt">RFC 2964</a> discusses use
- of the protocol.
- <a href="http://kristol.org/cookie/errata.html">Errata</a> to RFC 2965
- are currently being discussed on the
- <a href="http://lists.bell-labs.com/mailman/listinfo/http-state">
- http-state mailing list</a> (update: list traffic died months ago and
- hasn't revived).
- <li>A <a href="http://doi.acm.org/10.1145/502152.502153">paper</a> by David
- Kristol setting out the history of the cookie standards in exhausting
- detail.
- <li>HTTP cookies <a href="http://www.cookiecentral.com/">FAQ</a>.
- </ul>
- <li>Which protocols does ClientCookie support?
- <p>Netscape and RFC 2965. RFC 2965 handling is switched off by default.
- <li>What about RFC 2109?
- <p>RFC 2109 cookies are currently parsed as Netscape cookies, and treated
- by default as RFC 2965 cookies thereafter if RFC 2965 handling is enabled,
- or as Netscape cookies otherwise. RFC 2109 is officially obsoleted by RFC
- 2965. Browsers do use a few RFC 2109 features in their Netscape cookie
- implementations (<code>port</code> and <code>max-age</code>), and
- ClientCookie knows about that, too.
-</ul>
-
-<p>I prefer questions and comments to be sent to the <a
-href="http://lists.sourceforge.net/lists/listinfo/wwwsearch-general">
-mailing list</a> rather than direct to me.
-
-<p><a href="mailto:jjl@@pobox.com">John J. Lee</a>,
-@(time.strftime("%B %Y", last_modified)).
-
-<hr>
-
-</div>
-
-<div id="Menu">
-
-@(release.navbar('ClientCookie'))
-
-<br>
-
-<a href="./#download">Download</a><br>
-<a href="./#faq_pre">FAQs - pre-install</a><br>
-
-</div>
-
-
-</body>
-</html>
Modified: wwwsearch/mechanize/trunk/doc.html.in
==============================================================================
--- wwwsearch/mechanize/trunk/doc.html.in (original)
+++ wwwsearch/mechanize/trunk/doc.html.in Sat May 6 22:42:09 2006
@@ -2,20 +2,24 @@
"http://www.w3.org/TR/html4/strict.dtd">
@# This file is processed by EmPy to colorize Python source code
@# http://wwwsearch.sf.net/bits/colorize.py
-@{from colorize import colorize}
-@{import time}
-@{import release}
-@{last_modified = release.svn_id_to_time("$Id$")}
+@{
+from colorize import colorize
+import time
+import release
+last_modified = release.svn_id_to_time("$Id$")
+try:
+ base
+except NameError:
+ base = False
+}
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<meta name="author" content="John J. Lee <jjl@@pobox.com>">
<meta name="date" content="@(time.strftime("%Y-%m-%d", last_modified))">
- <meta name="keywords" content="cookie,HTTP,Python,web,client,client-side,HTML,META,HTTP-EQUIV,Refresh">
- <title>ClientCookie documentation</title>
+ <title>mechanize documentation</title>
<style type="text/css" media="screen">@@import "../styles/style.css";</style>
- <style type="text/css" media="screen">@@import "../styles/cookie_style.css";</style>
- <base href="http://wwwsearch.sourceforge.net/ClientCookie/">
+ @[if base]<base href="http://wwwsearch.sourceforge.net/mechanize/">@[end if]
</head>
<body>
@@ -23,35 +27,41 @@
<img src="http://sourceforge.net/sflogo.php?group_id=48205&type=2"
width="125" height="37" alt="SourceForge.net Logo"></a></div>
-<h1>ClientCookie</h1>
+<h1>mechanize handlers</h1>
<div id="Content">
+<p class="docwarning">This documentation is in need of reorganisation!</p>
+
+<p>This page is the old ClientCookie documentation. It deals with operation on
+the level of urllib2 Handler objects, and also with adding headers, debugging,
+and cookie handling. Documentation for the higher-level browser-style
+interface is <a href="./mechanize">elsewhere</a>.
+
+
<a name="examples"></a>
<h2>Examples</h2>
@{colorize(r"""
-import ClientCookie
-response = ClientCookie.urlopen("http://foo.bar.com/")
+import mechanize
+response = mechanize.urlopen("http://foo.bar.com/")
""")}
<p>This function behaves identically to <code>urllib2.urlopen()</code>, except
-that it deals with cookies automatically. That's probably all you need to know
-(except that ClientCookie provides <a href="./doc.html#extras">features other
-than cookie handling</a> that you can turn on).
+that it deals with cookies automatically.
<p>Here is a more complicated example, involving <code>Request</code> objects
(useful if you want to pass <code>Request</code>s around, add headers to them,
etc.):
@{colorize(r"""
-import ClientCookie
-request = ClientCookie.Request("http://www.acme.com/")
-# note we're using the urlopen from ClientCookie, not urllib2
-response = ClientCookie.urlopen(request)
+import mechanize
+request = mechanize.Request("http://www.acme.com/")
+# note we're using the urlopen from mechanize, not urllib2
+response = mechanize.urlopen(request)
# let's say this next request requires a cookie that was set in response
-request2 = ClientCookie.Request("http://www.acme.com/flying_machines.html")
-response2 = ClientCookie.urlopen(request2)
+request2 = mechanize.Request("http://www.acme.com/flying_machines.html")
+response2 = mechanize.urlopen(request2)
print response2.geturl()
print response2.info() # headers
@@ -59,12 +69,12 @@
""")}
<p>(The above example would also work with <code>urllib2.Request</code> objects
-too, since <code>ClientCookie.HTTPRequestUpgradeProcessor</code> knows about
+too, since <code>mechanize.HTTPRequestUpgradeProcessor</code> knows about
that class, but don't if you can avoid it, because this is an obscure hack for
compatibility purposes only).
<p>In these examples, the workings are hidden inside the
-<code>ClientCookie.urlopen()</code> function, which is an extension of
+<code>mechanize.urlopen()</code> function, which is an extension of
<code>urllib2.urlopen()</code>. Redirects, proxies and cookies are handled
automatically by this function (note that you may need a bit of configuration
to get your proxies correctly set up: see <code>urllib2</code> documentation).
@@ -75,8 +85,8 @@
are used like any other handler. There is quite a bit of other
<code>urllib2</code>-workalike code, too. Note: This duplication has gone away
in Python 2.4, since 2.4's <code>urllib2</code> contains the processor
-extensions from ClientCookie, so you can simply use ClientCookie's processor
-classes direct with 2.4's <code>urllib2</code>; also, ClientCookie's cookie
+extensions from mechanize, so you can simply use mechanize's processor
+classes direct with 2.4's <code>urllib2</code>; also, mechanize's cookie
functionality is included in Python 2.4 as module <code>cookielib</code> and
<code>urllib2.HTTPCookieProcessor</code>.
@@ -89,23 +99,23 @@
@{colorize(r"""
# Don't copy this blindly! You probably want to follow the examples
# above, not this one.
-import ClientCookie
+import mechanize
# Build an opener that *doesn't* automatically call .add_cookie_header()
# and .extract_cookies(), so we can do it manually without interference.
-class NullCookieProcessor(ClientCookie.HTTPCookieProcessor):
+class NullCookieProcessor(mechanize.HTTPCookieProcessor):
def http_request(self, request): return request
def http_response(self, request, response): return response
-opener = ClientCookie.build_opener(NullCookieProcessor)
+opener = mechanize.build_opener(NullCookieProcessor)
-request = ClientCookie.Request("http://www.acme.com/")
-response = ClientCookie.urlopen(request)
-cj = ClientCookie.CookieJar()
+request = mechanize.Request("http://www.acme.com/")
+response = mechanize.urlopen(request)
+cj = mechanize.CookieJar()
cj.extract_cookies(response, request)
# let's say this next request requires a cookie that was set in response
-request2 = ClientCookie.Request("http://www.acme.com/flying_machines.html")
+request2 = mechanize.Request("http://www.acme.com/flying_machines.html")
cj.add_cookie_header(request2)
-response2 = ClientCookie.urlopen(request2)
+response2 = mechanize.urlopen(request2)
""")}
<p>The <code>CookieJar</code> class does all the work. There are essentially
@@ -120,8 +130,8 @@
checked for acceptability based on the host name, etc. Cookies are only set on
outgoing requests if they match the request's host name, path, etc.
-<p><strong>Note that if you're using <code>ClientCookie.urlopen()</code> (or if
-you're using <code>ClientCookie.HTTPCookieProcessor</code> by some other
+<p><strong>Note that if you're using <code>mechanize.urlopen()</code> (or if
+you're using <code>mechanize.HTTPCookieProcessor</code> by some other
means), you don't need to call <code>.extract_cookies()</code> or
<code>.add_cookie_header()</code> yourself</strong>. If, on the other hand,
you don't want to use <code>urllib2</code>, you will need to use this pair of
@@ -137,10 +147,10 @@
a human-readable file:
@{colorize(r"""
-import ClientCookie
-cj = ClientCookie.LWPCookieJar()
+import mechanize
+cj = mechanize.LWPCookieJar()
cj.revert("cookie3.txt")
-opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cj))
+opener = mechanize.build_opener(mechanize.HTTPCookieProcessor(cj))
r = opener.open("http://foobar.com/")
cj.save("cookie3.txt")
""")}
@@ -164,7 +174,7 @@
<h2>Important note</h2>
-<p>Only use names you can import directly from the <code>ClientCookie</code>
+<p>Only use names you can import directly from the <code>mechanize</code>
package, and that don't start with a single underscore. Everything else is
subject to change or disappearance without notice.
@@ -185,8 +195,8 @@
works, but there have been bugs in the past!
@{colorize(r"""
-import os, ClientCookie
-cookies = ClientCookie.MozillaCookieJar()
+import os, mechanize
+cookies = mechanize.MozillaCookieJar()
cookies.load(os.path.join(os.environ["HOME"], "/.netscape/cookies.txt"))
# see also the save and revert methods
""")}
@@ -203,8 +213,8 @@
storing copies of cookies as <code>MSIECookieJar</code> does).
@{colorize(r"""
-import ClientCookie
-cj = ClientCookie.MSIECookieJar(delayload=True)
+import mechanize
+cj = mechanize.MSIECookieJar(delayload=True)
cj.load_from_registry() # finds cookie index file from registry
""")}
@@ -229,22 +239,6 @@
(`Set-Cookie3'). Unlike <code>MozilliaCookieJar</code>, this file format
doesn't lose information.
-<a name="database"></a>
-<h2>Saving cookies in a database</h2>
-
-<p><code>BSDDBCookieJar</code> (NOT FULLY TESTED!) saves to a BSDDB database
-using the standard library's <code>bsddb</code> module. Rather than using the
-constructor directly, you probably want to use
-<code>CreateBSDDBCookieJar<code>, which will create the named database file if
-it doesn't exist; otherwise, it will be opened.
-
-@{colorize(r"""
-import ClientCookie
-cj = ClientCookie.CreateBSDDBCookieJar("cookies.db")
-""")}
-
-<p>See also <code>MSIEDBCookieJar</code> <a href="browsers">above</a>.
-
<a name="cookiejar"></a>
<h2>Using your own CookieJar instance</h2>
@@ -258,12 +252,12 @@
<code>CookieJar</code> it should use:
@{colorize(r"""
-import ClientCookie
-cookies = ClientCookie.CookieJar()
+import mechanize
+cookies = mechanize.CookieJar()
# build_opener() adds standard handlers (such as HTTPHandler and
# HTTPCookieProcessor) by default. The cookie processor we supply
# will replace the default one.
-opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cookies))
+opener = mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies))
r = opener.open("http://acme.com/") # GET
r = opener.open("http://acme.com/", data) # POST
@@ -273,11 +267,11 @@
<code>OpenerDirector</code> instance to do its work, so if you want to use
<code>urlopen()</code> with your own <code>CookieJar</code>, install the
<code>OpenerDirector</code> you built with <code>build_opener()</code> using
-the <code>ClientCookie.install_opener()</code> function, then proceed as usual:
+the <code>mechanize.install_opener()</code> function, then proceed as usual:
@{colorize(r"""
-ClientCookie.install_opener(opener)
-r = ClientCookie.urlopen("http://www.acme.com/")
+mechanize.install_opener(opener)
+r = mechanize.urlopen("http://www.acme.com/")
""")}
<p>Of course, everyone using <code>urlopen</code> is using the same global
@@ -286,13 +280,13 @@
<a name="policy"></a>
<p>You can set a policy object (must satisfy the interface defined by
-<code>ClientCookie.CookiePolicy</code>), which determines which cookies are
+<code>mechanize.CookiePolicy</code>), which determines which cookies are
allowed to be set and returned. Use the policy argument to the
<code>CookieJar</code> constructor, or use the .set_policy() method. The
default implementation has some useful switches:
@{colorize(r"""
-from ClientCookie import CookieJar, DefaultCookiePolicy as Policy
+from mechanize import CookieJar, DefaultCookiePolicy as Policy
cookies = CookieJar()
# turn on RFC 2965 cookies, be more strict about domains when setting and
# returning Netscape cookies, and block some domains from setting cookies
@@ -322,7 +316,7 @@
robots.txt</code> file by which web site operators can request robots to keep
out of their site, or out of particular areas of it. This processor uses the
standard Python library's <code>robotparser</code> module. It raises
-<code>ClientCookie.RobotExclusionError</code> (subclass of
+<code>mechanize.RobotExclusionError</code> (subclass of
<code>urllib2.HTTPError</code>) if an attempt is made to open a URL prohibited
by <code>robots.txt</code>. XXX ATM, this makes use of code in the
<code>robotparser</code> module that uses <code>urllib</code> - this will
@@ -331,7 +325,7 @@
<dt><code>HTTPEquivProcessor</code>
<dd><p>The <code><META HTTP-EQUIV></code> tag is a way of including data
-in HTML to be treated as if it were part of the HTTP headers. ClientCookie can
+in HTML to be treated as if it were part of the HTTP headers. mechanize can
automatically read these tags and add the <code>HTTP-EQUIV</code> headers to
the response object's real HTTP headers. The HTML is left unchanged.
@@ -339,14 +333,14 @@
<dd><p>The <code>Refresh</code> HTTP header is a non-standard header which is
widely used. It requests that the user-agent follow a URL after a specified
-time delay. ClientCookie can treat these headers (which may have been set in
+time delay. mechanize can treat these headers (which may have been set in
<code><META HTTP-EQUIV></code> tags) as if they were 302 redirections.
Exactly when and how <code>Refresh</code> headers are handled is configurable
using the constructor arguments.
<dt><code>SeekableProcessor</code>
-<dd><p>This makes ClientCookie's response objects <code>seek()</code>able.
+<dd><p>This makes mechanize's response objects <code>seek()</code>able.
Seeking is done lazily (ie. the response object only reads from the socket as
necessary, rather than slurping in all the data before the response is returned
to you).
@@ -364,13 +358,13 @@
package does this properly.</p></dd>
@{colorize(r"""
-import ClientCookie
-cookies = ClientCookie.CookieJar()
+import mechanize
+cookies = mechanize.CookieJar()
-opener = ClientCookie.build_opener(ClientCookie.HTTPRefererProcessor,
- ClientCookie.HTTPEquivProcessor,
- ClientCookie.HTTPRefreshProcessor,
- ClientCookie.SeekableProcessor)
+opener = mechanize.build_opener(mechanize.HTTPRefererProcessor,
+ mechanize.HTTPEquivProcessor,
+ mechanize.HTTPRefreshProcessor,
+ mechanize.SeekableProcessor)
opener.open("http://www.rhubarb.com/")
""")}
@@ -380,10 +374,10 @@
<a name="requests"></a>
<h2>Confusing fact about headers and Requests</h2>
-<p>ClientCookie automatically upgrades <code>urllib2.Request</code> objects to
-<code>ClientCookie.Request</code>, as a backwards-compatibility hack. This
+<p>mechanize automatically upgrades <code>urllib2.Request</code> objects to
+<code>mechanize.Request</code>, as a backwards-compatibility hack. This
means that you won't see any headers that are added to Request objects by
-handlers unless you use <code>ClientCookie.Request</code> in the first place.
+handlers unless you use <code>mechanize.Request</code> in the first place.
Sorry about that.
@@ -393,16 +387,16 @@
<p>Adding headers is done like so:
@{colorize(r"""
-import ClientCookie, urllib2
+import mechanize, urllib2
req = urllib2.Request("http://foobar.com/")
-req.add_header("Referer", "http://wwwsearch.sourceforge.net/ClientCookie/")
-r = ClientCookie.urlopen(req)
+req.add_header("Referer", "http://wwwsearch.sourceforge.net/mechanize/")
+r = mechanize.urlopen(req)
""")}
<p>You can also use the headers argument to the <code>urllib2.Request</code>
constructor.
-<p><code>urllib2</code> (in fact, ClientCookie takes over this task from
+<p><code>urllib2</code> (in fact, mechanize takes over this task from
<code>urllib2</code>) adds some headers to <code>Request</code> objects
automatically - see the next section for details.
@@ -416,9 +410,9 @@
<code>OpenerDirector</code>:
@{colorize(r"""
-import ClientCookie
-cookies = ClientCookie.CookieJar()
-opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cookies))
+import mechanize
+cookies = mechanize.CookieJar()
+opener = mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies))
opener.addheaders = [("User-agent", "Mozilla/5.0 (compatible; MyProgram/0.1)"),
("From", "responsible.person at example.com")]
""")}
@@ -427,14 +421,14 @@
<code>OpenerDirector</code> globally:
@{colorize(r"""
-ClientCookie.install_opener(opener)
-r = ClientCookie.urlopen("http://acme.com/")
+mechanize.install_opener(opener)
+r = mechanize.urlopen("http://acme.com/")
""")}
<p>Also, a few standard headers (<code>Content-Length</code>,
<code>Content-Type</code> and <code>Host</code>) are added when the
<code>Request</code> is passed to <code>urlopen()</code> (or
-<code>OpenerDirector.open()</code>). ClientCookie explictly adds these (and
+<code>OpenerDirector.open()</code>). mechanize explictly adds these (and
<code>User-Agent</code>) to the <code>Request</code> object, unlike versions of
<code>urllib2</code> before Python 2.4 (but <strong>note</strong> that
Content-Length is an exception to this rule: it is sent, but not explicitly
@@ -465,7 +459,7 @@
by the user. For example, if the request is for an image in an HTML document,
this is the request-host of the request for the page containing the image.
-<p><strong>ClientCookie knows that redirected transactions are unverifiable,
+<p><strong>mechanize knows that redirected transactions are unverifiable,
and will handle that on its own (ie. you don't need to think about the origin
request-host or verifiability yourself).</strong>
@@ -485,7 +479,7 @@
""")}
<li>If you're using a <code>urllib2.Request</code> from Python 2.4 or later,
- or you're using a <code>ClientCookie.Request<code>, use the
+ or you're using a <code>mechanize.Request<code>, use the
<code>unverifiable</code> and <code>origin_req_host</code> arguments to the
constructor:
@@ -510,9 +504,9 @@
<!--XXX move as much as poss. to General page-->
<p>First, a few common problems. The most frequent mistake people seem to make
-is to use <code>ClientCookie.urlopen()</code>, <em>and</em> the
+is to use <code>mechanize.urlopen()</code>, <em>and</em> the
<code>.extract_cookies()</code> and <code>.add_cookie_header()</code> methods
-on a cookie object themselves. If you use <code>ClientCookie.urlopen()</code>
+on a cookie object themselves. If you use <code>mechanize.urlopen()</code>
(or <code>OpenerDirector.open()</code>), the module handles extraction and
adding of cookies by itself, so you should not call
<code>.extract_cookies()</code> or <code>.add_cookie_header()</code>.
@@ -544,11 +538,11 @@
cookies are going away after saving and loading.
@{colorize(r"""
-import ClientCookie
-cj = ClientCookie.LWPCookieJar()
-opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cj))
-ClientCookie.install_opener(opener)
-r = ClientCookie.urlopen("http://foobar.com/")
+import mechanize
+cj = mechanize.LWPCookieJar()
+opener = mechanize.build_opener(mechanize.HTTPCookieProcessor(cj))
+mechanize.install_opener(opener)
+r = mechanize.urlopen("http://foobar.com/")
cj.save("/some/file", ignore_discard=True, ignore_expires=True)
""")}
@@ -573,24 +567,24 @@
<p>You can turn on display of HTTP headers:
@{colorize(r"""
-import ClientCookie
-hh = ClientCookie.HTTPHandler() # you might want HTTPSHandler, too
+import mechanize
+hh = mechanize.HTTPHandler() # you might want HTTPSHandler, too
hh.set_http_debuglevel(1)
-opener = ClientCookie.build_opener(hh)
+opener = mechanize.build_opener(hh)
response = opener.open(url)
""")}
<p>Alternatively, you can examine your individual request and response objects
-to see what's going on. Note, though, that ClientCookie upgrades
-urllib2.Request objects to ClientCookie.Request, so you won't see any headers
-that are added to requests by handlers unless you use ClientCookie.Request in
-the first place. ClientCookie's responses can be made <code>.seek()</code>able
+to see what's going on. Note, though, that mechanize upgrades
+urllib2.Request objects to mechanize.Request, so you won't see any headers
+that are added to requests by handlers unless you use mechanize.Request in
+the first place. mechanize's responses can be made <code>.seek()</code>able
using <code>SeekableProcessor</code>. It's often useful to use the
<code>.seek()</code> method like this during debugging:
@{colorize(r"""
...
-response = ClientCookie.urlopen("http://spam.eggs.org/")
+response = mechanize.urlopen("http://spam.eggs.org/")
print response.read()
response.seek(0)
# rest of code continues as if you'd never .read() the response
@@ -605,16 +599,16 @@
<code>build_opener()</code>) you have to turn on logging at the
<code>INFO</code> level or lower in order to see any output.
-<p>If you would like to see what is going on in ClientCookie's tiny mind, do
+<p>If you would like to see what is going on in mechanize's tiny mind, do
this:
@{colorize(r"""
-import ClientCookie
-# ClientCookie.DEBUG covers masses of debugging information,
-# ClientCookie.INFO just shows the output from HTTPRedirectDebugProcessor,
-logger = ClientCookie.getLogger("ClientCookie")
-logger.addHandler(ClientCookie.StreamHandler())
-logger.setLevel(ClientCookie.DEBUG)
+import sys, logging
+# logging.DEBUG covers masses of debugging information,
+# logging.INFO just shows the output from HTTPRedirectDebugProcessor,
+logger = logging.getLogger("mechanize")
+logger.addHandler(logging.StreamHandler(sys.stdout))
+logger.setLevel(logging.DEBUG)
""")}
<p>(In Python 2.3 or newer, <code>logging.getLogger</code>,
@@ -627,7 +621,7 @@
<p>One final thing to note is that there are some catch-all bare
<code>except:</code> statements in the module, which are there to handle
unexpected bad input without crashing your program. If this happens, it's a
-bug in ClientCookie, so please mail me the warning text.
+bug in mechanize, so please mail me the warning text.
<a name="script"></a>
@@ -661,11 +655,11 @@
@{colorize("""\
import copy
-import ClientCookie
-class CommentCleanProcessor(ClientCookie.BaseProcessor):
+import mechanize
+class CommentCleanProcessor(mechanize.BaseProcessor):
def http_response(self, request, response):
if not hasattr(response, "seek"):
- response = ClientCookie.response_seek_wrapper(response)
+ response = mechanize.response_seek_wrapper(response)
response.seek(0)
new_response = copy.copy(response)
new_response.set_data(
@@ -692,7 +686,7 @@
protocol</a> (cookie_spec.html) is still the only standard supported by most
browsers (including Internet Explorer and Netscape). Be aware that
cookie_spec.html is not, and never was, actually followed to the letter (or
-anything close) by anyone (including Netscape, IE and ClientCookie): the
+anything close) by anyone (including Netscape, IE and mechanize): the
Netscape protocol standard is really defined by the behaviour of Netscape (and
now IE). Netscape cookies are also known as V0 cookies, to distinguish them
from RFC 2109 or RFC 2965 cookies, which have a version cookie-attribute with a
@@ -730,7 +724,7 @@
and MSIE & Mozilla options
<p>XXX Apparently MSIE implements bits of RFC 2109 - but not very compliant
-(surprise). Presumably other browsers do too, as a result. ClientCookie
+(surprise). Presumably other browsers do too, as a result. mechanize
already does allow Netscape cookies to have <code>max-age</code> and
<code>port</code> cookie-attributes, and as far as I know that's the extent of
the support present in MSIE. I haven't tested, though!
@@ -763,6 +757,55 @@
and open holes where required for Netscape protocol-compatibility. RFC
2965 cookies are <em>always</em> treated as RFC 2965 requires, of course!
+
+<a name="faq_pre"></a>
+<h2>FAQs - pre install</h2>
+<ul>
+ <li>Doesn't the standard Python library module, <code>Cookie</code>, do
+ this?
+ <p>No: Cookie.py does the server end of the job. It doesn't know when to
+ accept cookies from a server or when to pass them back.
+ <li>Is urllib2.py required?
+ <p>No. You probably want it, though.
+ <li>Where can I find out more about the HTTP cookie protocol?
+ <p>There is more than one protocol, in fact (see the <a href="./doc.html">docs</a>
+ for a brief explanation of the history):
+ <ul>
+ <li>The original <a href="http://www.netscape.com/newsref/std/cookie_spec.html">
+ Netscape cookie protocol</a> - the standard still in use today, in
+ theory (in reality, the protocol implemented by all the major browsers
+ only bears a passing resemblance to the protocol sketched out in this
+ document).
+ <li><a href="http://www.ietf.org/rfcs/rfc2109.txt">RFC 2109</a> - obsoleted
+ by RFC 2965.
+ <li><a href="http://www.ietf.org/rfcs/rfc2965.txt">RFC 2965</a> - the
+ Netscape protocol with the bugs fixed (not widely used - the Netscape
+ protocol still dominates, and seems likely to remain dominant
+ indefinitely, at least on the Internet).
+ <a href="http://www.ietf.org/rfcs/rfc2964.txt">RFC 2964</a> discusses use
+ of the protocol.
+ <a href="http://kristol.org/cookie/errata.html">Errata</a> to RFC 2965
+ are currently being discussed on the
+ <a href="http://lists.bell-labs.com/mailman/listinfo/http-state">
+ http-state mailing list</a> (update: list traffic died months ago and
+ hasn't revived).
+ <li>A <a href="http://doi.acm.org/10.1145/502152.502153">paper</a> by David
+ Kristol setting out the history of the cookie standards in exhausting
+ detail.
+ <li>HTTP cookies <a href="http://www.cookiecentral.com/">FAQ</a>.
+ </ul>
+ <li>Which protocols does ClientCookie support?
+ <p>Netscape and RFC 2965. RFC 2965 handling is switched off by default.
+ <li>What about RFC 2109?
+ <p>RFC 2109 cookies are currently parsed as Netscape cookies, and treated
+ by default as RFC 2965 cookies thereafter if RFC 2965 handling is enabled,
+ or as Netscape cookies otherwise. RFC 2109 is officially obsoleted by RFC
+ 2965. Browsers do use a few RFC 2109 features in their Netscape cookie
+ implementations (<code>port</code> and <code>max-age</code>), and
+ ClientCookie knows about that, too.
+</ul>
+
+
<a name="faq_use"></a>
<h2>FAQs - usage</h2>
<ul>
@@ -786,7 +829,7 @@
the loading fails).
<li>Is it threadsafe?
<p>No. <em>Tested</em> patches welcome. Clarification: As far as I know,
- it's perfectly possible to use ClientCookie in threaded code, but it
+ it's perfectly possible to use mechanize in threaded code, but it
provides no synchronisation: you have to provide that yourself.
<li>How do I do <X>
<p>The module docstrings are worth reading if you want to do something
@@ -799,7 +842,7 @@
in Python 2.4.
<li>How do I use it without urllib2.py?
<p>@{colorize(r"""
-from ClientCookie import CookieJar
+from mechanize import CookieJar
print CookieJar.extract_cookies.__doc__
print CookieJar.add_cookie_header.__doc__
""")}
@@ -825,7 +868,6 @@
<a href="./doc.html#examples">Examples</a><br>
<a href="./doc.html#browsers">Mozilla & MSIE</a><br>
<a href="./doc.html#file">Cookies in a file</a><br>
-<a href="./doc.html#database">Cookies in a database</a><br>
<a href="./doc.html#cookiejar">Using a <code>CookieJar</code></a><br>
<a href="./doc.html#extras">Processors</a><br>
<a href="./doc.html#requests">Request confusion</a><br>
Deleted: /wwwsearch/mechanize/trunk/pp-README.html.in
==============================================================================
--- /wwwsearch/mechanize/trunk/pp-README.html.in Sat May 6 22:42:09 2006
+++ (empty file)
@@ -1,160 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
- "http://www.w3.org/TR/html4/strict.dtd">
-@# This file is processed by EmPy to colorize Python source code
-@# http://wwwsearch.sf.net/bits/colorize.py
-@{from colorize import colorize}
-@{import time}
-@{import release}
-@{last_modified = release.svn_id_to_time("$Id$")}
-<html>
-<head>
- <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
- <meta name="author" content="John J. Lee <jjl@@pobox.com>">
- <meta name="date" content="@(time.strftime("%Y-%m-%d", last_modified))">
- <meta name="keywords" content="Python,HTML,token,parser,pull,API,web,client,client-side">
- <title>pullparser</title>
- <style type="text/css" media="screen">@@import "../styles/style.css";</style>
- <base href="http://wwwsearch.sourceforge.net/pullparser/">
-</head>
-<body>
-
-<div id="sf"><a href="http://sourceforge.net">
-<img src="http://sourceforge.net/sflogo.php?group_id=48205&type=2"
- width="125" height="37" alt="SourceForge.net Logo"></a></div>
-<!--<img src="../images/sflogo.png"-->
-
-<h1>pullparser</h1>
-
-<div id="Content">
-
-<p>A simple "pull API" for HTML parsing, after Perl's
-<code>HTML::TokeParser</code>. Many simple HTML parsing tasks are
-simpler this way than with the <code>HTMLParser</code> module.
-<code>pullparser.PullParser</code> is a subclass of
-<code>HTMLParser.HTMLParser</code>.
-
-<p>Examples:
-
-<p>This program extracts all links from a document. It will print one line for
-each link, containing the URL and the textual description between the
-<code><a>...</a></code> tags:
-
-@{colorize(r"""
-import pullparser, sys
-f = file(sys.argv[1])
-p = pullparser.PullParser(f)
-for token in p.tags("a"):
- if token.type == "endtag": continue
- url = dict(token.attrs).get("href", "-")
- text = p.get_compressed_text(endat=("endtag", "a"))
- print "%s\t%s" % (url, text)
-""")}
-
-<p>This program extracts the <code><title></code> from the document:
-
-@{colorize(r"""
-import pullparser, sys
-f = file(sys.argv[1])
-p = pullparser.PullParser(f)
-if p.get_tag("title"):
- title = p.get_compressed_text()
- print "Title: %s" % title
-""")}
-
-<p>Thanks to Gisle Aas, who wrote <code>HTML::TokeParser</code>.
-
-
-<a name="download"></a>
-<h2>Download</h2>
-<p>All documentation (including this web page) is included in the distribution.
-
-<p><em>Stable release.</em>
-
-<ul>
-@{version = "0.1.0"}
-<li><a href="./src/pullparser-@(version).tar.gz">pullparser-@(version).tar.gz</a>
-<li><a href="./src/pullparser-@(version).zip">pullparser-@(version).zip</a>
-<li><a href="./src/ChangeLog.txt">Change Log</a> (included in distribution)
-<li><a href="./src/">Older versions.</a>
-</ul>
-
-<p>For installation instructions, see the INSTALL file included in the
-distribution.
-
-
-<h2>Subversion</h2>
-
-<p>The <a href="http://subversion.tigris.org/">Subversion (SVN)</a> trunk is <a href="http://codespeak.net/svn/wwwsearch/pullparser/trunk#egg=pullparser-dev">http://codespeak.net/svn/wwwsearch/pullparser/trunk</a>, so to check out the source:
-
-<pre>
-svn co http://codespeak.net/svn/wwwsearch/pullparser/trunk pullparser
-</pre>
-
-
-<h2>See also</h2>
-
-<p><a href="http://www.crummy.com/software/BeautifulSoup/">Beautiful
-Soup</a> is widely recommended. More robust than this module.
-
-<p>I recommend <a
-href="http://www.crummy.com/software/BeautifulSoup/">Beautiful Soup</a> over
-pullparser for new web scraping code. More robust and flexible than this
-module.
-
-<a name="faq"></a>
-<h2>FAQs</h2>
-<ul>
- <li>Which version of Python do I need?
- <p>2.2.1 or above.
- <li>Which license?
- <p>pullparser is dual-licensed: you may pick either the
- <a href="http://www.opensource.org/licenses/bsd-license.php">BSD license</a>,
- or the <a href="http://www.zope.org/Resources/ZPL">ZPL 2.1</a> (both are
- included in the distribution).
- <li>Why does it fail to parse my HTML?
- <p>Because module <code>HTMLParser</code> is fussy. Try
- <code>pullparser.TolerantPullParser</code> instead, which uses module
- <code>sgmllib</code> instead. Note that self-closing tags (<foo/>)
- will show up as 'starttag' tags, not 'startendtag' tags if you use this
- class - this is a limitation of module <code>sgmllib</code>.
- <li>Why don't I see the tokens I expect?
- <p>
- <ul>
- <li>Are there missing end-tags in your HTML? (Maybe this will improve in
- future.)
- <li>Element names passed to methods such as PullParser.get_token() must be
- given in lower case - maybe you forgot that? (Element names <em>in the
- HTML</em> can be any case, of course.)
- <li><code>HTMLParser.HTMLParser</code> isn't very robust. Would be fairly
- easy to (perhaps optionally) rebase on the other standard library HTML
- parsing module, <code>sgmllib.SGMLParser</code> (which is really an
- HTML parser, not a full SGML parser, despite the name). I'm not going
- to do that, though.
- </ul>
-</ul>
-
-<p>I prefer questions and comments to be sent to the <a
-href="http://lists.sourceforge.net/lists/listinfo/wwwsearch-general">
-mailing list</a> rather than direct to me.
-
-<p><a href="mailto:jjl@@pobox.com">John J. Lee</a>,
-@(time.strftime("%B %Y", last_modified)).
-
-<hr>
-
-</div>
-
-<div id="Menu">
-
-@(release.navbar('pullparser'))
-
-<br>
-
-<a href="./#download">Download</a><br>
-<a href="./#faq">FAQs</a><br>
-
-</div>
-
-
-</body>
-</html>
More information about the wwwsearch-commits
mailing list