[wwwsearch-commits] r26861 - in wwwsearch/mechanize/trunk: .
examples mechanize test
jjlee at codespeak.net
jjlee at codespeak.net
Sat May 6 13:30:53 CEST 2006
Author: jjlee
Date: Sat May 6 13:30:47 2006
New Revision: 26861
Added:
wwwsearch/mechanize/trunk/cc-README.html.in
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/cc-README.html.in
wwwsearch/mechanize/trunk/doc.html.in
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/doc.html.in
wwwsearch/mechanize/trunk/examples/BSDDBCookieJar.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/examples/BSDDBCookieJar.py
wwwsearch/mechanize/trunk/examples/MSIEDBCookieJar.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/examples/MSIEDBCookieJar.py
wwwsearch/mechanize/trunk/examples/cookietest.cgi
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/examples/cookietest.cgi
wwwsearch/mechanize/trunk/mechanize/_ClientCookie.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_ClientCookie.py
wwwsearch/mechanize/trunk/mechanize/_ConnCache.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_ConnCache.py
wwwsearch/mechanize/trunk/mechanize/_Debug.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_Debug.py
wwwsearch/mechanize/trunk/mechanize/_HeadersUtil.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_HeadersUtil.py
wwwsearch/mechanize/trunk/mechanize/_LWPCookieJar.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_LWPCookieJar.py
wwwsearch/mechanize/trunk/mechanize/_MSIECookieJar.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_MSIECookieJar.py
wwwsearch/mechanize/trunk/mechanize/_MozillaCookieJar.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_MozillaCookieJar.py
wwwsearch/mechanize/trunk/mechanize/_Opener.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_Opener.py
wwwsearch/mechanize/trunk/mechanize/_Request.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_Request.py
wwwsearch/mechanize/trunk/mechanize/_Util.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_Util.py
wwwsearch/mechanize/trunk/mechanize/_pullparser.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_pullparser.py
wwwsearch/mechanize/trunk/mechanize/_urllib2_support.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/mechanize/_urllib2_support.py
wwwsearch/mechanize/trunk/pp-README.html.in
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/pp-README.html.in
wwwsearch/mechanize/trunk/test/
- copied from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/
wwwsearch/mechanize/trunk/test.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test.py
wwwsearch/mechanize/trunk/test/test_conncache.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/test_conncache.py
wwwsearch/mechanize/trunk/test/test_cookies.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/test_cookies.py
wwwsearch/mechanize/trunk/test/test_date.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/test_date.py
wwwsearch/mechanize/trunk/test/test_headers.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/test_headers.py
wwwsearch/mechanize/trunk/test/test_mechanize.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/test_mechanize.py
wwwsearch/mechanize/trunk/test/test_misc.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/test_misc.py
wwwsearch/mechanize/trunk/test/test_pullparser.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/test_pullparser.py
wwwsearch/mechanize/trunk/test/test_urllib2.py
- copied unchanged from r26778, wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/test/test_urllib2.py
Modified:
wwwsearch/mechanize/trunk/0.1.0-changes.txt
wwwsearch/mechanize/trunk/COPYING.txt
wwwsearch/mechanize/trunk/INSTALL.txt
wwwsearch/mechanize/trunk/MANIFEST.in
wwwsearch/mechanize/trunk/README.html.in
wwwsearch/mechanize/trunk/functional_tests.py
wwwsearch/mechanize/trunk/mechanize/__init__.py
wwwsearch/mechanize/trunk/mechanize/_html.py
wwwsearch/mechanize/trunk/mechanize/_mechanize.py
wwwsearch/mechanize/trunk/mechanize/_useragent.py
wwwsearch/mechanize/trunk/setup.py
Log:
Merge merge of ClientCookie and pullparser with mechanize (they're part of all a single package now, named 'mechanize'): -r 26768:26778 http://codespeak.net/svn/wwwsearch/mechanize/branch/mechanize-clientcookie-pullparser-merge/
Modified: wwwsearch/mechanize/trunk/0.1.0-changes.txt
==============================================================================
--- wwwsearch/mechanize/trunk/0.1.0-changes.txt (original)
+++ wwwsearch/mechanize/trunk/0.1.0-changes.txt Sat May 6 13:30:47 2006
@@ -1,7 +1,20 @@
Recent public API changes:
+- ClientCookie has been moved into the mechanize package and is no
+ longer a separate package. The ClientCookie interface is still
+ supported, but all names must be imported from module mechanize
+ instead of from module ClientCookie. Python 2.3 is now required. (I
+ have no plans to merge ClientForm with mechanize.) Note that the
+ logging work-alike facility is gone. Also, the experimental BSDDB
+ support is now only included as example code, and not installed.
+
+- pullparser has been moved into the mechanize package and is no
+ longer a separate package. Also, the interface of pullparser is no
+ longer supported. Instead, it's just a purely internal
+ implementation detail of mechanize.
+
- ATM, auth support requires Python 2.5 (I hope to fix this soon for
- older Pythons). As a result, some tests fail for 2.2, 2.3, and 2.4.
+ older Pythons). As a result, some tests fail for 2.3 and 2.4.
- Some mechanize.Browser constructor args have been moved to
mechanize.Factory (default_encoding, ...).
Modified: wwwsearch/mechanize/trunk/COPYING.txt
==============================================================================
--- wwwsearch/mechanize/trunk/COPYING.txt (original)
+++ wwwsearch/mechanize/trunk/COPYING.txt Sat May 6 13:30:47 2006
@@ -1,4 +1,6 @@
Copyright (c) 2002-2006 John J. Lee <jjl at pobox.com>
+Copyright (c) 1997-1999 Gisle Aas
+Copyright (c) 1997-1999 Johnny Lee
Copyright (c) 2003 Andy Lester
All rights reserved.
Modified: wwwsearch/mechanize/trunk/INSTALL.txt
==============================================================================
--- wwwsearch/mechanize/trunk/INSTALL.txt (original)
+++ wwwsearch/mechanize/trunk/INSTALL.txt Sat May 6 13:30:47 2006
@@ -32,8 +32,8 @@
python setup.py easy_install --no-deps .
-Alternatively, just copy the whole mechanize directory into your
-Python path (eg. unix: /usr/local/lib/python2.4/site-packages,
+Alternatively, just copy the whole mechanize directory into a directory
+on your Python path (eg. unix: /usr/local/lib/python2.4/site-packages,
Windows: C:\Python24\Lib\site-packages). Only copy the mechanize
directory that's inside the distributed tarball / zip archive, not the
entire mechanize-x.x.x directory!
@@ -44,13 +44,21 @@
python test.py
-This runs the tests against the source files extracted from the
-package. For help on command line options:
+This runs the tests against the source files extracted from the package.
+For help on command line options:
python test.py --help
+To run the functional tests (which DO access the network), run the
+following command:
-Bugs and comments to jjl at pobox.com.
+ python functional_tests.py
+
+
+Please send bugs and comments to the mailing list (or failing that, to
+jjl at pobox.com):
+
+https://lists.sourceforge.net/lists/listinfo/wwwsearch-general
NO WARRANTY
@@ -62,12 +70,14 @@
Copyright Notices
(C) 2002-2006 John J. Lee. All rights reserved.
- (C) 2003 Andy Lester. All rights reserved. (Perl code from which
- this module is derived)
+ (C) 1995-2001 Gisle Aas. All rights reserved. (Original LWP code)
+ (C) 2002-2003 Johhny Lee. All rights reserved. (MSIE Perl code)
+ (C) 2003 Andy Lester. All rights reserved. (Original WWW::Mechanize
+ Perl code)
This code in this package is free software; you can redistribute it
-and/or modify it under the terms of the BSD or ZPL 2.1 licenses (see
-the file COPYING.txt).
+and/or modify it under the terms of the BSD or ZPL 2.1 licenses (see the
+file COPYING.txt).
John J. Lee <jjl at pobox.com>
May 2006
Modified: wwwsearch/mechanize/trunk/MANIFEST.in
==============================================================================
--- wwwsearch/mechanize/trunk/MANIFEST.in (original)
+++ wwwsearch/mechanize/trunk/MANIFEST.in Sat May 6 13:30:47 2006
@@ -5,6 +5,8 @@
include README.html.in
include README.html
include README.txt
+include doc.html.in
+include doc.html
include ChangeLog.txt
include *.py
recursive-include examples *.py
Modified: wwwsearch/mechanize/trunk/README.html.in
==============================================================================
--- wwwsearch/mechanize/trunk/README.html.in (original)
+++ wwwsearch/mechanize/trunk/README.html.in Sat May 6 13:30:47 2006
@@ -11,7 +11,8 @@
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<meta name="author" content="John J. Lee <jjl@@pobox.com>">
<meta name="date" content="@(time.strftime("%Y-%m-%d", last_modified))">
- <meta name="keywords" content="Python,HTML,browser,stateful,web,client,client-side,mechanize,form,ClientForm,ClientCookie,pullparser,WWW::Mechanize">
+ <meta name="keywords" content="Python,HTML,HTTP,browser,stateful,web,client,client-side,mechanize,cookie,form,META,HTTP-EQUIV,Refresh,ClientForm,ClientCookie,pullparser,WWW::Mechanize">
+ <meta name="keywords" content="cookie,HTTP,Python,web,client,client-side,HTML,META,HTTP-EQUIV,Refresh">
<title>mechanize</title>
<style type="text/css" media="screen">@@import "../styles/style.css";</style>
<base href="http://wwwsearch.sourceforge.net/mechanize/">
@@ -408,7 +409,7 @@
<h2>FAQs</h2>
<ul>
<li>Which version of Python do I need?
- <p>2.2.1 or above.
+ <p>2.3 or above.
<li>What else do I need?
<p><a href="../ClientCookie/">ClientCookie</a>,
<a href="../ClientForm/">ClientForm</a> and
Modified: wwwsearch/mechanize/trunk/functional_tests.py
==============================================================================
--- wwwsearch/mechanize/trunk/functional_tests.py (original)
+++ wwwsearch/mechanize/trunk/functional_tests.py Sat May 6 13:30:47 2006
@@ -1,9 +1,27 @@
#!/usr/bin/env python
+# These tests access the network.
+
import os
from unittest import TestCase
import mechanize
+from mechanize import build_opener, install_opener, urlopen, urlretrieve
+from mechanize import CookieJar, HTTPCookieProcessor, \
+ HTTPHandler, HTTPRefreshProcessor, \
+ HTTPEquivProcessor, HTTPRedirectHandler, \
+ HTTPRedirectDebugProcessor, HTTPResponseDebugProcessor
+
+#from cookielib import CookieJar
+#from urllib2 import build_opener, install_opener, urlopen
+#from urllib2 import HTTPCookieProcessor, HTTPHandler
+
+#from ClientCookie import CreateBSDDBCookieJar
+
+## logger = ClientCookie.getLogger("ClientCookie")
+## logger.addHandler(ClientCookie.StreamHandler())
+## logger.setLevel(ClientCookie.DEBUG)
+
def sanepathname2url(path):
import urllib
@@ -60,6 +78,16 @@
r.seek(0)
self.assertEqual(r.read(), html)
+ def test_response_close_and_read(self):
+ opener = ClientCookie.build_opener(ClientCookie.SeekableProcessor)
+ r = opener.open("http://wwwsearch.sf.net/bits/cctest2.txt")
+ # closing response shouldn't stop methods working if we're using
+ # SeekableProcessor (ie. _Util.response_seek_wrapper)
+ r.read()
+ r.close()
+ r.seek(0)
+ self.assertEqual(r.read(), "Hello ClientCookie functional test suite.\n")
+
def test_set_response(self):
from mechanize import Browser
br = Browser()
@@ -104,6 +132,83 @@
"Hello ClientCookie functional test suite.\n")
+class FunctionalTests(TestCase):
+ def test_clientcookie(self):
+ import urllib2
+ # this test page depends on cookies, and an http-equiv refresh
+ #cj = CreateBSDDBCookieJar("/home/john/db.db")
+ cj = CookieJar()
+ handlers = [
+ HTTPCookieProcessor(cj),
+ HTTPRefreshProcessor(max_time=None, honor_time=False),
+ HTTPEquivProcessor(),
+
+ HTTPRedirectHandler(), # needed for Refresh handling in 2.4.0
+# HTTPHandler(True),
+# HTTPRedirectDebugProcessor(),
+# HTTPResponseDebugProcessor(),
+ ]
+
+ o = apply(build_opener, handlers)
+ try:
+ install_opener(o)
+ try:
+ r = urlopen("http://wwwsearch.sf.net/cgi-bin/cookietest.cgi")
+ except urllib2.URLError, e:
+ #print e.read()
+ raise
+ data = r.read()
+ #print data
+ self.assert_(
+ data.find("Your browser supports cookies!") >= 0)
+ self.assert_(len(cj) == 1)
+
+ # test response.seek() (added by HTTPEquivProcessor)
+ r.seek(0)
+ samedata = r.read()
+ r.close()
+ self.assert_(samedata == data)
+ finally:
+ o.close()
+ # uninstall opener (don't try this at home)
+ ClientCookie._urllib2_support._opener = None
+
+ def test_urlretrieve(self):
+ url = "http://www.python.org/"
+ verif = CallbackVerifier(self)
+ fn, hdrs = urlretrieve(url, "python.html", verif.callback)
+ try:
+ f = open(fn)
+ data = f.read()
+ f.close()
+ finally:
+ os.remove(fn)
+ r = urlopen(url)
+ self.assert_(data == r.read())
+ r.close()
+
+## def test_cacheftp(self):
+## from urllib2 import CacheFTPHandler, build_opener
+## o = build_opener(CacheFTPHandler())
+## r = o.open("ftp://ftp.python.org/pub/www.python.org/robots.txt")
+## data1 = r.read()
+## r.close()
+## r = o.open("ftp://ftp.python.org/pub/www.python.org/2.3.2/announce.txt")
+## data2 = r.read()
+## r.close()
+## self.assert_(data1 != data2)
+
+class CallbackVerifier:
+ # for .test_urlretrieve()
+ def __init__(self, testcase):
+ self._count = 0
+ self._testcase = testcase
+ def callback(self, block_nr, block_size, total_size):
+ if block_nr != self._count:
+ self._testcase.fail()
+ self._count = self._count + 1
+
+
if __name__ == "__main__":
import unittest
unittest.main()
Modified: wwwsearch/mechanize/trunk/mechanize/__init__.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/__init__.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/__init__.py Sat May 6 13:30:47 2006
@@ -1,9 +1,45 @@
-from _useragent import UserAgent, HTTPProxyPasswordMgr
from _mechanize import Browser, \
BrowserStateError, LinkNotFoundError, FormNotFoundError, \
__version__
+from _useragent import UserAgent, HTTPProxyPasswordMgr
+
from _html import Link, \
Factory, DefaultFactory, RobustFactory, \
FormsFactory, LinksFactory, TitleFactory, \
RobustFormsFactory, RobustLinksFactory, RobustTitleFactory
+
+# If you hate the idea of turning bugs into warnings, do:
+# import mechanize; mechanize.USE_BARE_EXCEPT = False
+USE_BARE_EXCEPT = True
+
+from _ClientCookie import Cookie, CookiePolicy, DefaultCookiePolicy, \
+ CookieJar, FileCookieJar, LoadError, request_host
+from _LWPCookieJar import LWPCookieJar, lwp_cookie_str
+from _MozillaCookieJar import MozillaCookieJar
+from _MSIECookieJar import MSIECookieJar
+from _urllib2_support import \
+ Request, \
+ OpenerDirector, build_opener, install_opener, urlopen, \
+ OpenerFactory, urlretrieve, BaseHandler, HeadParser, \
+ RobotExclusionError
+try:
+ from _urllib2_support import XHTMLCompatibleHeadParser
+except ImportError:
+ pass
+from ClientCookie._urllib2_support import \
+ HTTPHandler, HTTPRedirectHandler, \
+ HTTPRequestUpgradeProcessor, \
+ HTTPEquivProcessor, SeekableProcessor, HTTPCookieProcessor, \
+ HTTPRefererProcessor, \
+ HTTPRefreshProcessor, HTTPErrorProcessor, \
+ HTTPResponseDebugProcessor, HTTPRedirectDebugProcessor, \
+ HTTPRobotRulesProcessor
+
+import httplib
+if hasattr(httplib, 'HTTPS'):
+ from ClientCookie._urllib2_support import HTTPSHandler
+del httplib
+
+from _Util import http2time as str2time
+from _Util import response_seek_wrapper
Modified: wwwsearch/mechanize/trunk/mechanize/_html.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/_html.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/_html.py Sat May 6 13:30:47 2006
@@ -8,13 +8,11 @@
"""
-from __future__ import generators
-
import re, copy, urllib, htmlentitydefs
from urlparse import urljoin
-import ClientCookie
-from ClientCookie._HeadersUtil import split_header_words, is_html as _is_html
+import _Request
+from _HeadersUtil import split_header_words, is_html as _is_html
## # XXXX miserable hack
## def urljoin(base, url):
@@ -120,9 +118,9 @@
link_class=Link,
urltags=None,
):
- import pullparser
+ import _pullparser
if link_parser_class is None:
- link_parser_class = pullparser.TolerantPullParser
+ link_parser_class = _pullparser.TolerantPullParser
self.link_parser_class = link_parser_class
self.link_class = link_class
if urltags is None:
@@ -143,7 +141,6 @@
def links(self):
"""Return an iterator that provides links of the document."""
- import pullparser
response = self._response
encoding = self._encoding
base_url = self._base_url
@@ -200,7 +197,7 @@
form_parser_class = ClientForm.FormParser
self.form_parser_class = form_parser_class
if request_class is None:
- request_class = ClientCookie.Request
+ request_class = _Request.Request
self.request_class = request_class
self.backwards_compat = backwards_compat
self._response = None
@@ -231,12 +228,12 @@
self._encoding = encoding
def title(self):
- import pullparser
- p = pullparser.TolerantPullParser(
+ import _pullparser
+ p = _pullparser.TolerantPullParser(
self._response, encoding=self._encoding)
try:
p.get_tag("title")
- except pullparser.NoMoreTokensError:
+ except _pullparser.NoMoreTokensError:
return None
else:
return p.get_text()
@@ -384,7 +381,7 @@
url = clean_url(url, encoding)
text = link.firstText(lambda t: True)
if text is BeautifulSoup.Null:
- # follow pullparser's weird behaviour rigidly
+ # follow _pullparser's weird behaviour rigidly
if link.name == "a":
text = ""
else:
Modified: wwwsearch/mechanize/trunk/mechanize/_mechanize.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/_mechanize.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/_mechanize.py Sat May 6 13:30:47 2006
@@ -22,10 +22,10 @@
import urllib2, urlparse, sys, copy
-import ClientCookie
-
from _useragent import UserAgent
from _html import DefaultFactory
+from _Util import response_seek_wrapper
+import _Request
__version__ = (0, 1, 0, "a", None) # 0.1.0a
@@ -62,7 +62,7 @@
if sys.version_info[:2] >= (2, 4):
- from ClientCookie._Opener import OpenerMixin
+ from _Opener import OpenerMixin
else:
class OpenerMixin: pass
@@ -76,7 +76,7 @@
Public attributes:
- request: current request (ClientCookie.Request or urllib2.Request)
+ request: current request (mechanize.Request or urllib2.Request)
form: currently selected form (see .select_form())
"""
@@ -93,7 +93,7 @@
factory: object implementing the mechanize.Factory interface.
history: object implementing the mechanize.History interface. Note this
interface is still experimental and may change in future.
- request_class: Request class to use. Defaults to ClientCookie.Request
+ request_class: Request class to use. Defaults to mechanize.Request
by default for Pythons older than 2.4, urllib2.Request otherwise.
The Factory and History objects passed in are 'owned' by the Browser,
@@ -113,7 +113,7 @@
if request_class is None:
if not hasattr(urllib2.Request, "add_unredirected_header"):
- request_class = ClientCookie.Request
+ request_class = _Request.Request
else:
request_class = urllib2.Request # Python >= 2.4
@@ -168,9 +168,8 @@
response = error
## except (IOError, socket.error, OSError), error:
## # Yes, urllib2 really does raise all these :-((
-## # See test_urllib2.py in stdlib and in ClientCookie for examples
-## # of socket.gaierror and OSError, plus note that FTPHandler raises
-## # IOError.
+## # See test_urllib2.py for examples of socket.gaierror and OSError,
+## # plus note that FTPHandler raises IOError.
## # XXX I don't seem to have an example of exactly socket.error being
## # raised, only socket.gaierror...
## # I don't want to start fixing these here, though, since this is a
@@ -194,7 +193,7 @@
def set_response(self, response):
"""Replace current response with (a copy of) response."""
- from ClientCookie._Util import closeable_response
+ from _Util import closeable_response
# sanity check, necessary but far from sufficient
if not (hasattr(response, "info") and hasattr(response, "geturl") and
hasattr(response, "read")):
@@ -207,8 +206,8 @@
if not hasattr(response, 'closeable_response'):
# we expect to get here if a urllib2 handler constructed the
# response, i.e. the response is an urllib.addinfourl, instead of a
- # ClientCookie._Util.closeable_response as returned by
- # e.g. ClientCookie.HTTPHandler
+ # _Util.closeable_response as returned by
+ # e.g. mechanize.HTTPHandler
try:
code = response.code
except AttributeError:
@@ -222,7 +221,7 @@
response = closeable_response(
response.fp, response.info(), response.geturl(), code, msg)
if not hasattr(response, "seek"):
- response = ClientCookie.response_seek_wrapper(response)
+ response = response_seek_wrapper(response)
# 0) don't want to copy here, but
# 1) don't want to copy some of the time and not other times
# 2) need response to be .close()able and .seek()able
Modified: wwwsearch/mechanize/trunk/mechanize/_useragent.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/_useragent.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/_useragent.py Sat May 6 13:30:47 2006
@@ -13,11 +13,14 @@
import sys
import urllib2
-import ClientCookie
+
if sys.version_info[:2] >= (2, 4):
from urllib2 import OpenerDirector, BaseHandler, HTTPErrorProcessor
else:
- from ClientCookie import OpenerDirector, BaseHandler, HTTPErrorProcessor
+ from _Opener import OpenerDirector
+ from _urllib2_support import BaseHandler, HTTPErrorProcessor
+
+import _urllib2_support
class HTTPRefererProcessor(BaseHandler):
def http_request(self, request):
@@ -99,7 +102,7 @@
handler_classes = {
# scheme handlers
- "http": ClientCookie.HTTPHandler,
+ "http": _urllib2_support.HTTPHandler,
"ftp": urllib2.FTPHandler, # CacheFTPHandler is buggy in 2.3
"file": urllib2.FileHandler,
"gopher": urllib2.GopherHandler,
@@ -108,25 +111,26 @@
"_unknown": urllib2.UnknownHandler,
# HTTP{S,}Handler depend on HTTPErrorProcessor too
"_http_error": HTTPErrorProcessor,
- "_http_request_upgrade": ClientCookie.HTTPRequestUpgradeProcessor,
+ "_http_request_upgrade": _urllib2_support.HTTPRequestUpgradeProcessor,
"_http_default_error": urllib2.HTTPDefaultErrorHandler,
# feature handlers
"_basicauth": urllib2.HTTPBasicAuthHandler,
"_digestauth": urllib2.HTTPBasicAuthHandler,
- "_redirect": ClientCookie.HTTPRedirectHandler,
- "_cookies": ClientCookie.HTTPCookieProcessor,
- "_refresh": ClientCookie.HTTPRefreshProcessor,
+ "_redirect": _urllib2_support.HTTPRedirectHandler,
+ "_cookies": _urllib2_support.HTTPCookieProcessor,
+ "_refresh": _urllib2_support.HTTPRefreshProcessor,
"_referer": HTTPRefererProcessor, # from this module, note
- "_equiv": ClientCookie.HTTPEquivProcessor,
- "_seek": ClientCookie.SeekableProcessor,
+ "_equiv": _urllib2_support.HTTPEquivProcessor,
+ "_seek": _urllib2_support.SeekableProcessor,
"_proxy": urllib2.ProxyHandler,
"_proxy_basicauth": urllib2.ProxyBasicAuthHandler,
"_proxy_digestauth": urllib2.ProxyDigestAuthHandler,
+ "_robots": _urllib2_support.HTTPRobotRulesProcessor,
# debug handlers
- "_debug_redirect": ClientCookie.HTTPRedirectDebugProcessor,
- "_debug_response_body": ClientCookie.HTTPResponseDebugProcessor,
+ "_debug_redirect": _urllib2_support.HTTPRedirectDebugProcessor,
+ "_debug_response_body": _urllib2_support.HTTPResponseDebugProcessor,
}
default_schemes = ["http", "ftp", "file", "gopher"]
@@ -137,14 +141,11 @@
"_refresh", "_equiv",
"_basicauth", "_digestauth",
"_proxy", "_proxy_basicauth", "_proxy_digestauth",
- "_seek",
+ "_seek", "_robots",
]
- if hasattr(ClientCookie, 'HTTPSHandler'):
- handler_classes["https"] = ClientCookie.HTTPSHandler
+ if hasattr(_urllib2_support, 'HTTPSHandler'):
+ handler_classes["https"] = _urllib2_support.HTTPSHandler
default_schemes.append("https")
- if hasattr(ClientCookie, "HTTPRobotRulesProcessor"):
- handler_classes["_robots"] = ClientCookie.HTTPRobotRulesProcessor
- default_features.append("_robots")
def __init__(self):
OpenerDirector.__init__(self)
@@ -224,7 +225,7 @@
"this class can't do HTTP Referer: use mechanize.Browser instead")
def set_cookiejar(self, cookiejar):
- """Set a ClientCookie.CookieJar, or None."""
+ """Set a mechanize.CookieJar, or None."""
self._set_handler("_cookies", obj=cookiejar)
# XXX could use Greg Stein's httpx for some of this instead?
@@ -289,36 +290,33 @@
"""
self._set_handler("_referer", handle)
self._handle_referer = bool(handle)
+ # XXXXX isn't this always true?
def set_seekable_responses(self, handle):
"""Make response objects .seek()able."""
self._set_handler("_seek", handle)
def set_debug_redirects(self, handle):
- """Log information about HTTP redirects.
-
- This includes refreshes, which show up as faked 302 redirections at the
- moment.
+ """Log information about HTTP redirects (including refreshes).
Logging is performed using module logging. The logger name is
- "ClientCookie.http_redirects". To actually print some debug output,
+ "mechanize.http_redirects". To actually print some debug output,
eg:
- logger = logging.getLogger("ClientCookie.http_redirects")
- logger.addHandler(logging.StreamHandler())
+ import sys, logging
+ logger = logging.getLogger("mechanize.http_redirects")
+ logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.INFO)
Other logger names relevant to this module:
- "ClientCookie.http_responses"
- "ClientCookie.cookies" (or "cookielib" if running Python 2.4)
+ "mechanize.http_responses"
+ "mechanize.cookies" (or "cookielib" if running Python 2.4)
To turn on everything:
- for logger in [
- logging.getLogger("ClientCookie"),
- logging.getLogger("cookielib"),
- ]:
- logger.addHandler(logging.StreamHandler())
- logger.setLevel(logging.INFO)
+ import sys, logging
+ logger = logging.getLogger("mechanize")
+ logger.addHandler(logging.StreamHandler(sys.stdout))
+ logger.setLevel(logging.INFO)
"""
self._set_handler("_debug_redirect", handle)
@@ -371,6 +369,7 @@
self.add_handler(newhandler)
self._ua_handlers[name] = newhandler
+# XXXXX cruft
def remove(sequence, obj):
# for use when can't use .remove() because of obj.__cmp__ :-(
# (ClientCookie only requires Python 2.0, which doesn't have __lt__)
Modified: wwwsearch/mechanize/trunk/setup.py
==============================================================================
--- wwwsearch/mechanize/trunk/setup.py (original)
+++ wwwsearch/mechanize/trunk/setup.py Sat May 6 13:30:47 2006
@@ -3,6 +3,20 @@
Stateful programmatic web browsing, after Andy Lester's Perl module
WWW::Mechanize.
+
+The library is layered: mechanize.Browser (stateful web browser),
+mechanize.UserAgent (configurable URL opener), plus urllib2 handlers.
+
+Features include: ftp:, http: and file: URL schemes, browser history,
+high-level hyperlink and HTML form support, HTTP cookies, HTTP-EQUIV and
+Refresh, Referer [sic] header, robots.txt, redirections, proxies, and
+Basic and Digest HTTP authentication. mechanize's response objects are
+(lazily-) .seek()able and still work after .close().
+
+Much of the code originally derived from Perl code by Gisle Aas
+(libwww-perl), Johnny Lee (MSIE Cookie support) and last but not least
+Andy Lester (WWW::Mechanize). urllib2 was written by Jeremy Hylton.
+
"""
def unparse_version(tup):
@@ -39,10 +53,7 @@
## open("mechanize/_mechanize.py").read())
## VERSION = unparse_version(str_to_tuple(VERSION_MATCH.group(1)))
VERSION = "0.1.0a"
-INSTALL_REQUIRES = [
- "ClientForm>=0.2.2, ==dev",
- "ClientCookie>=1.3.0, ==dev",
- "pullparser>=0.1.0, ==dev"]
+INSTALL_REQUIRES = ["ClientForm>=0.2.2, ==dev"]
NAME = "mechanize"
PACKAGE = True
LICENSE = "BSD" # or ZPL 2.1
@@ -71,6 +82,7 @@
Topic :: System :: Archiving :: Mirroring
Topic :: System :: Networking :: Monitoring
Topic :: System :: Systems Administration
+Topic :: Text Processing
Topic :: Text Processing :: Markup
Topic :: Text Processing :: Markup :: HTML
Topic :: Text Processing :: Markup :: XML
More information about the wwwsearch-commits
mailing list