Alternatively, you can examine your individual request and response
objects to see what's going on. Note, though, that mechanize upgrades
-urllib2.Request objects to mechanize.Request, so you won't see any
-headers that are added to requests by handlers unless you use
-mechanize.Request in the first place. In addition, requests may
-involve "sub-requests" in cases such as redirection, in which case you
-will also not see everything that's going on just by examining the
-original request and final response. mechanize's responses can be
-made to have .seek() and .get_data()
-methods. It's often useful to use the .get_data() method
-during debugging (see above).
+urllib2.Request objects to mechanize.Request, so you
+won't see any headers that are added to requests by handlers unless you use
+mechanize.Request in the first place. In addition, requests may
+involve "sub-requests" in cases such as redirection, in which case you will
+also not see everything that's going on just by examining the original request
+and final response. mechanize's responses can be made to
+have .seek() and .get_data() methods. It's often
+useful to use the .get_data() method during debugging
+(see above).
What about RFC 2109?
RFC 2109 cookies are currently parsed as Netscape cookies, and treated
@@ -862,7 +862,7 @@
or as Netscape cookies otherwise. RFC 2109 is officially obsoleted by RFC
2965. Browsers do use a few RFC 2109 features in their Netscape cookie
implementations (port and max-age), and
- ClientCookie knows about that, too.
+ mechanize knows about that, too.
From jjlee at codespeak.net Sat Feb 23 20:11:32 2008
From: jjlee at codespeak.net (jjlee at codespeak.net)
Date: Sat, 23 Feb 2008 20:11:32 +0100 (CET)
Subject: [wwwsearch-commits] r51827 - in wwwsearch/mechanize/trunk: . test
Message-ID: <20080223191132.E7C071684C4@codespeak.net>
Author: jjlee
Date: Sat Feb 23 20:11:32 2008
New Revision: 51827
Added:
wwwsearch/mechanize/trunk/test/test_password_manager.special_doctest
- copied unchanged from r45426, wwwsearch/mechanize/trunk/test/test_password_manager.doctest
wwwsearch/mechanize/trunk/test/test_robotfileparser.special_doctest
- copied unchanged from r48480, wwwsearch/mechanize/trunk/test/test_robotfileparser.doctest
Removed:
wwwsearch/mechanize/trunk/test/test_password_manager.doctest
wwwsearch/mechanize/trunk/test/test_robotfileparser.doctest
Modified:
wwwsearch/mechanize/trunk/test.py
Log:
Avoid running doctest files that need special context by naming convention rather than special-purpose code
Modified: wwwsearch/mechanize/trunk/test.py
==============================================================================
--- wwwsearch/mechanize/trunk/test.py (original)
+++ wwwsearch/mechanize/trunk/test.py Sat Feb 23 20:11:32 2008
@@ -76,36 +76,23 @@
# run .doctest files needing special support
common_globs = {"mechanize": mechanize}
pm_doctest_filename = os.path.join(
- "test", "test_password_manager.doctest")
+ "test", "test_password_manager.special_doctest")
for globs in [
{"mgr_class": mechanize.HTTPPasswordMgr},
{"mgr_class": mechanize.HTTPProxyPasswordMgr},
]:
globs.update(common_globs)
- doctest.testfile(
- pm_doctest_filename,
- #os.path.join("test", "test_scratch.doctest"),
- globs=globs,
- )
+ doctest.testfile(pm_doctest_filename, globs=globs)
try:
import robotparser
except ImportError:
pass
else:
- doctest.testfile(os.path.join("test",
- "test_robotfileparser.doctest"))
+ doctest.testfile(os.path.join(
+ "test", "test_robotfileparser.special_doctest"))
# run .doctest files
- special_doctests = [pm_doctest_filename,
- os.path.join("test", "test_scratch.doctest"),
- os.path.join("test",
- "test_robotfileparser.doctest"),
- ]
doctest_files = glob.glob(os.path.join("test", "*.doctest"))
-
- for dt in special_doctests:
- if dt in doctest_files:
- doctest_files.remove(dt)
for df in doctest_files:
doctest.testfile(df)
Deleted: /wwwsearch/mechanize/trunk/test/test_password_manager.doctest
==============================================================================
--- /wwwsearch/mechanize/trunk/test/test_password_manager.doctest Sat Feb 23 20:11:32 2008
+++ (empty file)
@@ -1,148 +0,0 @@
-Features common to HTTPPasswordMgr and HTTPProxyPasswordMgr
-===========================================================
-
-(mgr_class gets here through globs argument)
-
->>> mgr = mgr_class()
->>> add = mgr.add_password
-
->>> add("Some Realm", "http://example.com/", "joe", "password")
->>> add("Some Realm", "http://example.com/ni", "ni", "ni")
->>> add("c", "http://example.com/foo", "foo", "ni")
->>> add("c", "http://example.com/bar", "bar", "nini")
->>> add("b", "http://example.com/", "first", "blah")
->>> add("b", "http://example.com/", "second", "spam")
->>> add("a", "http://example.com", "1", "a")
->>> add("Some Realm", "http://c.example.com:3128", "3", "c")
->>> add("Some Realm", "d.example.com", "4", "d")
->>> add("Some Realm", "e.example.com:3128", "5", "e")
-
->>> mgr.find_user_password("Some Realm", "example.com")
-('joe', 'password')
->>> mgr.find_user_password("Some Realm", "http://example.com")
-('joe', 'password')
->>> mgr.find_user_password("Some Realm", "http://example.com/")
-('joe', 'password')
->>> mgr.find_user_password("Some Realm", "http://example.com/spam")
-('joe', 'password')
->>> mgr.find_user_password("Some Realm", "http://example.com/spam/spam")
-('joe', 'password')
->>> mgr.find_user_password("c", "http://example.com/foo")
-('foo', 'ni')
->>> mgr.find_user_password("c", "http://example.com/bar")
-('bar', 'nini')
-
-Actually, this is really undefined ATM
-#Currently, we use the highest-level path where more than one match:
-#
-#>>> mgr.find_user_password("Some Realm", "http://example.com/ni")
-#('joe', 'password')
-
-Use latest add_password() in case of conflict:
-
->>> mgr.find_user_password("b", "http://example.com/")
-('second', 'spam')
-
-No special relationship between a.example.com and example.com:
-
->>> mgr.find_user_password("a", "http://example.com/")
-('1', 'a')
->>> mgr.find_user_password("a", "http://a.example.com/")
-(None, None)
-
-Ports:
-
->>> mgr.find_user_password("Some Realm", "c.example.com")
-(None, None)
->>> mgr.find_user_password("Some Realm", "c.example.com:3128")
-('3', 'c')
->>> mgr.find_user_password("Some Realm", "http://c.example.com:3128")
-('3', 'c')
->>> mgr.find_user_password("Some Realm", "d.example.com")
-('4', 'd')
->>> mgr.find_user_password("Some Realm", "e.example.com:3128")
-('5', 'e')
-
-
-Default port tests
-------------------
-
->>> mgr = mgr_class()
->>> add = mgr.add_password
-
-The point to note here is that we can't guess the default port if there's
-no scheme. This applies to both add_password and find_user_password.
-
->>> add("f", "http://g.example.com:80", "10", "j")
->>> add("g", "http://h.example.com", "11", "k")
->>> add("h", "i.example.com:80", "12", "l")
->>> add("i", "j.example.com", "13", "m")
->>> mgr.find_user_password("f", "g.example.com:100")
-(None, None)
->>> mgr.find_user_password("f", "g.example.com:80")
-('10', 'j')
->>> mgr.find_user_password("f", "g.example.com")
-(None, None)
->>> mgr.find_user_password("f", "http://g.example.com:100")
-(None, None)
->>> mgr.find_user_password("f", "http://g.example.com:80")
-('10', 'j')
->>> mgr.find_user_password("f", "http://g.example.com")
-('10', 'j')
->>> mgr.find_user_password("g", "h.example.com")
-('11', 'k')
->>> mgr.find_user_password("g", "h.example.com:80")
-('11', 'k')
->>> mgr.find_user_password("g", "http://h.example.com:80")
-('11', 'k')
->>> mgr.find_user_password("h", "i.example.com")
-(None, None)
->>> mgr.find_user_password("h", "i.example.com:80")
-('12', 'l')
->>> mgr.find_user_password("h", "http://i.example.com:80")
-('12', 'l')
->>> mgr.find_user_password("i", "j.example.com")
-('13', 'm')
->>> mgr.find_user_password("i", "j.example.com:80")
-(None, None)
->>> mgr.find_user_password("i", "http://j.example.com")
-('13', 'm')
->>> mgr.find_user_password("i", "http://j.example.com:80")
-(None, None)
-
-
-Features specific to HTTPProxyPasswordMgr
-=========================================
-
-Default realm:
-
->>> mgr = mechanize.HTTPProxyPasswordMgr()
->>> add = mgr.add_password
-
->>> mgr.find_user_password("d", "f.example.com")
-(None, None)
->>> add(None, "f.example.com", "6", "f")
->>> mgr.find_user_password("d", "f.example.com")
-('6', 'f')
-
-Default host/port:
-
->>> mgr.find_user_password("e", "g.example.com")
-(None, None)
->>> add("e", None, "7", "g")
->>> mgr.find_user_password("e", "g.example.com")
-('7', 'g')
-
-Default realm and host/port:
-
->>> mgr.find_user_password("f", "h.example.com")
-(None, None)
->>> add(None, None, "8", "h")
->>> mgr.find_user_password("f", "h.example.com")
-('8', 'h')
-
-Default realm beats default host/port:
-
->>> add("d", None, "9", "i")
->>> mgr.find_user_password("d", "f.example.com")
-('6', 'f')
Deleted: /wwwsearch/mechanize/trunk/test/test_robotfileparser.doctest
==============================================================================
--- /wwwsearch/mechanize/trunk/test/test_robotfileparser.doctest Sat Feb 23 20:11:32 2008
+++ (empty file)
@@ -1,8 +0,0 @@
->>> from mechanize._http import MechanizeRobotFileParser
-
-Calling .set_opener() without args sets a default opener.
-
->>> rfp = MechanizeRobotFileParser()
->>> rfp.set_opener()
->>> rfp._opener # doctest: +ELLIPSIS
-
From jjlee at codespeak.net Sat Feb 23 20:49:29 2008
From: jjlee at codespeak.net (jjlee at codespeak.net)
Date: Sat, 23 Feb 2008 20:49:29 +0100 (CET)
Subject: [wwwsearch-commits] r51828 - in wwwsearch/mechanize/trunk:
mechanize test
Message-ID: <20080223194929.D05441684C9@codespeak.net>
Author: jjlee
Date: Sat Feb 23 20:49:27 2008
New Revision: 51828
Modified:
wwwsearch/mechanize/trunk/mechanize/_mozillacookiejar.py
wwwsearch/mechanize/trunk/test/test_cookies.py
Log:
* Handle cookies containing embedded tabs in mozilla format files
* Remove an assertion about mozilla format cookies file contents (raise LoadError instead)
Modified: wwwsearch/mechanize/trunk/mechanize/_mozillacookiejar.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/_mozillacookiejar.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/_mozillacookiejar.py Sat Feb 23 20:49:27 2008
@@ -80,7 +80,7 @@
continue
domain, domain_specified, path, secure, expires, name, value = \
- line.split("\t")
+ line.split("\t", 6)
secure = (secure == "TRUE")
domain_specified = (domain_specified == "TRUE")
if name == "":
@@ -88,7 +88,9 @@
value = None
initial_dot = domain.startswith(".")
- assert domain_specified == initial_dot
+ if domain_specified != initial_dot:
+ raise LoadError("domain and domain specified flag don't "
+ "match in %s: %s" % (filename, line))
discard = False
if expires == "":
@@ -113,9 +115,9 @@
self.set_cookie(c)
except:
- reraise_unmasked_exceptions((IOError,))
+ reraise_unmasked_exceptions((IOError, LoadError))
raise LoadError("invalid Netscape format file %s: %s" %
- (filename, line))
+ (filename, line))
def save(self, filename=None, ignore_discard=False, ignore_expires=False):
if filename is None:
Modified: wwwsearch/mechanize/trunk/test/test_cookies.py
==============================================================================
--- wwwsearch/mechanize/trunk/test/test_cookies.py (original)
+++ wwwsearch/mechanize/trunk/test/test_cookies.py Sat Feb 23 20:49:27 2008
@@ -1,6 +1,6 @@
"""Tests for _ClientCookie."""
-import urllib2, re, os, StringIO, mimetools, time
+import urllib2, re, os, StringIO, mimetools, time, tempfile, errno
from time import localtime
from unittest import TestCase
@@ -1344,6 +1344,46 @@
assert len(new_c) == 4 # 2 of them discarded on save
assert repr(new_c).find("name='foo1', value='bar'") != -1
+ def test_mozilla_cookiejar_embedded_tab(self):
+ from mechanize import MozillaCookieJar
+ filename = tempfile.mktemp()
+ fh = open(filename, "w")
+ try:
+ fh.write(
+ MozillaCookieJar.header + "\n" +
+ "a.com\tFALSE\t/\tFALSE\t\tname\tval\tstillthevalue\n"
+ "a.com\tFALSE\t/\tFALSE\t\tname2\tvalue\n")
+ fh.close()
+ cj = MozillaCookieJar(filename)
+ cj.revert(ignore_discard=True)
+ cookies = cj._cookies["a.com"]["/"]
+ self.assertEquals(cookies["name"].value, "val\tstillthevalue")
+ self.assertEquals(cookies["name2"].value, "value")
+ finally:
+ try:
+ os.remove(filename)
+ except OSError, exc:
+ if exc.errno != errno.EEXIST:
+ raise
+
+ def test_mozilla_cookiejar_initial_dot_violation(self):
+ from mechanize import MozillaCookieJar, LoadError
+ filename = tempfile.mktemp()
+ fh = open(filename, "w")
+ try:
+ fh.write(
+ MozillaCookieJar.header + "\n" +
+ ".a.com\tFALSE\t/\tFALSE\t\tname\tvalue\n")
+ fh.close()
+ cj = MozillaCookieJar(filename)
+ self.assertRaises(LoadError, cj.revert, ignore_discard=True)
+ finally:
+ try:
+ os.remove(filename)
+ except OSError, exc:
+ if exc.errno != errno.EEXIST:
+ raise
+
def test_netscape_misc(self):
# Some additional Netscape cookies tests.
from mechanize import CookieJar, Request
From jjlee at codespeak.net Sat Feb 23 21:56:33 2008
From: jjlee at codespeak.net (jjlee at codespeak.net)
Date: Sat, 23 Feb 2008 21:56:33 +0100 (CET)
Subject: [wwwsearch-commits] r51829 - wwwsearch/mechanize/trunk/test
Message-ID: <20080223205633.53AC91684C8@codespeak.net>
Author: jjlee
Date: Sat Feb 23 21:56:31 2008
New Revision: 51829
Modified:
wwwsearch/mechanize/trunk/test/test_cookies.py
Log:
Stop tests from clobbering files that happen to be lying around in cwd (!)
Modified: wwwsearch/mechanize/trunk/test/test_cookies.py
==============================================================================
--- wwwsearch/mechanize/trunk/test/test_cookies.py (original)
+++ wwwsearch/mechanize/trunk/test/test_cookies.py Sat Feb 23 21:56:31 2008
@@ -118,7 +118,7 @@
# missing = sign in Cookie: header is regarded by Mozilla as a missing
# NAME. WE regard it as a missing VALUE.
- filename = os.path.abspath("cookies2.txt")
+ filename = tempfile.mktemp()
c = MozillaCookieJar(filename)
interact_netscape(c, "http://www.acme.com/", 'eggs')
interact_netscape(c, "http://www.acme.com/", '"spam"; path=/foo/')
@@ -1262,7 +1262,7 @@
assert len(c) == 6
# save and restore
- filename = "lwp-cookies.txt"
+ filename = tempfile.mktemp()
try:
c.save(filename, ignore_discard=True)
@@ -1306,7 +1306,7 @@
year_plus_one = localtime(time.time())[0] + 1
- filename = "cookies.txt"
+ filename = tempfile.mktemp()
c = MozillaCookieJar(filename,
policy=DefaultCookiePolicy(rfc2965=True))
From wwwsearch-commits at codespeak.net Mon Feb 25 20:51:30 2008
From: wwwsearch-commits at codespeak.net (zkzo ® Official Site)
Date: Mon, 25 Feb 2008 20:51:30 +0100 (CET)
Subject: [wwwsearch-commits] February %71 OFF
Message-ID: <20080225-15058.3914.qmail@tdev161-240.codetel.net.do>
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/wwwsearch-commits/attachments/20080225/71e8ae6b/attachment.htm
From wwwsearch-commits at codespeak.net Wed Feb 27 23:55:43 2008
From: wwwsearch-commits at codespeak.net (wwwsearch-commits at codespeak.net)
Date: Wed, 27 Feb 2008 23:55:43 +0100 (CET)
Subject: [wwwsearch-commits] February 74% OFF
Message-ID: <20080227125510.35443.qmail@catv-5985e861.catv.broadband.hu>
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/wwwsearch-commits/attachments/20080227/b70e6ea3/attachment-0001.htm
From jjlee at codespeak.net Thu Feb 28 00:17:16 2008
From: jjlee at codespeak.net (jjlee at codespeak.net)
Date: Thu, 28 Feb 2008 00:17:16 +0100 (CET)
Subject: [wwwsearch-commits] r51910 - wwwsearch/mechanize/trunk/mechanize
Message-ID: <20080227231716.C17961683E3@codespeak.net>
Author: jjlee
Date: Thu Feb 28 00:17:16 2008
New Revision: 51910
Modified:
wwwsearch/mechanize/trunk/mechanize/_clientcookie.py
Log:
Handle missing cookie max-age value. Previously, a warning was emitted in this case.
Modified: wwwsearch/mechanize/trunk/mechanize/_clientcookie.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/_clientcookie.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/_clientcookie.py Thu Feb 28 00:17:16 2008
@@ -1212,6 +1212,10 @@
continue
if k == "max-age":
max_age_set = True
+ if v is None:
+ debug(" missing value for max-age attribute")
+ bad_cookie = True
+ break
try:
v = int(v)
except ValueError:
From jjlee at codespeak.net Thu Feb 28 00:22:23 2008
From: jjlee at codespeak.net (jjlee at codespeak.net)
Date: Thu, 28 Feb 2008 00:22:23 +0100 (CET)
Subject: [wwwsearch-commits] r51911 - in wwwsearch/mechanize/trunk: .
mechanize test test-tools test-tools/support
Message-ID: <20080227232223.7CAD41683D9@codespeak.net>
Author: jjlee
Date: Thu Feb 28 00:22:22 2008
New Revision: 51911
Added:
wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py
wwwsearch/mechanize/trunk/test-tools/support/
Modified:
wwwsearch/mechanize/trunk/functional_tests.py
wwwsearch/mechanize/trunk/mechanize/__init__.py
wwwsearch/mechanize/trunk/test-tools/cookietest.cgi
wwwsearch/mechanize/trunk/test/test_cookies.py
Log:
* Add support for Firefox 3 cookie jars ("cookies.sqlite")
* Fix a couple of test typos re tempfile cleanup
* Add a functional test for seek-wrapped HTTPError repr
Modified: wwwsearch/mechanize/trunk/functional_tests.py
==============================================================================
--- wwwsearch/mechanize/trunk/functional_tests.py (original)
+++ wwwsearch/mechanize/trunk/functional_tests.py Thu Feb 28 00:22:22 2008
@@ -4,7 +4,7 @@
# thanks Moof (aka Giles Antonio Radford) for some of these
-import os, sys, urllib
+import os, sys, urllib, tempfile, errno
from unittest import TestCase
import mechanize
@@ -159,6 +159,14 @@
r.get_data(),
"Hello ClientCookie functional test suite.\n")
+ def test_seek_wrapper_class_name(self):
+ opener = mechanize.UserAgent()
+ opener.set_seekable_responses(True)
+ try:
+ opener.open(urljoin(self.uri, "nonexistent"))
+ except mechanize.HTTPError, exc:
+ self.assert_("HTTPError instance" in repr(exc))
+
def test_no_seek(self):
# should be possible to turn off UserAgent's .seek() functionality
def check_no_seek(opener):
@@ -395,6 +403,65 @@
## r.close()
## self.assert_(data1 != data2)
+
+class CookieJarTests(TestCase):
+
+ def test_mozilla_cookiejar(self):
+ filename = tempfile.mktemp()
+ try:
+ def get_cookiejar():
+ cj = mechanize.MozillaCookieJar(filename=filename)
+ try:
+ cj.revert()
+ except IOError, exc:
+ if exc.errno != errno.ENOENT:
+ raise
+ return cj
+ def commit(cj):
+ cj.save()
+ self._test_cookiejar(get_cookiejar, commit)
+ finally:
+ try:
+ os.remove(filename)
+ except OSError, exc:
+ if exc.errno != errno.ENOENT:
+ raise
+
+ def test_firefox3_cookiejar(self):
+ filename = tempfile.mktemp()
+ try:
+ def get_cookiejar():
+ cj = mechanize.Firefox3CookieJar(filename=filename)
+ cj.connect()
+ return cj
+ def commit(cj):
+ pass
+ self._test_cookiejar(get_cookiejar, commit)
+ finally:
+ os.remove(filename)
+
+ def _test_cookiejar(self, get_cookiejar, commit):
+ cookiejar = get_cookiejar()
+ br = mechanize.Browser()
+ br.set_cookiejar(cookiejar)
+ br.set_handle_refresh(False)
+ url = urljoin(self.uri, "/cgi-bin/cookietest.cgi")
+ # no cookie was set on the first request
+ html = br.open(url).read()
+ self.assertEquals(html.find("Your browser supports cookies!"), -1)
+ self.assertEquals(len(cookiejar), 1)
+ # ... but now we have the cookie
+ html = br.open(url).read()
+ self.assert_("Your browser supports cookies!" in html)
+ commit(cookiejar)
+
+ # should still have the cookie when we load afresh
+ cookiejar = get_cookiejar()
+ br.set_cookiejar(cookiejar)
+ html = br.open(url).read()
+ self.assert_("Your browser supports cookies!" in html)
+
+
class CallbackVerifier:
# for .test_urlretrieve()
def __init__(self, testcase):
Modified: wwwsearch/mechanize/trunk/mechanize/__init__.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/__init__.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/__init__.py Thu Feb 28 00:22:22 2008
@@ -115,6 +115,7 @@
from _clientcookie import Cookie, CookiePolicy, DefaultCookiePolicy, \
CookieJar, FileCookieJar, LoadError, request_host
from _lwpcookiejar import LWPCookieJar, lwp_cookie_str
+from _firefox3cookiejar import Firefox3CookieJar
from _mozillacookiejar import MozillaCookieJar
from _msiecookiejar import MSIECookieJar
Added: wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py
==============================================================================
--- (empty file)
+++ wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py Thu Feb 28 00:22:22 2008
@@ -0,0 +1,245 @@
+"""Firefox 3 "cookies.sqlite" cookie persistence.
+
+Copyright 2008 John J Lee
+
+This code is free software; you can redistribute it and/or modify it
+under the terms of the BSD or ZPL 2.1 licenses (see the file
+COPYING.txt included with the distribution).
+
+"""
+
+import logging
+import time
+
+from _clientcookie import CookieJar, Cookie, MappingIterator
+from _util import isstringlike
+debug = logging.getLogger("mechanize.cookies").debug
+
+
+try:
+ import sqlite3
+except ImportError:
+ pass
+else:
+ class Firefox3CookieJar(CookieJar):
+
+ """Firefox 3 cookie jar.
+
+ The cookies are stored in Firefox 3's "cookies.sqlite" format.
+
+ Constructor arguments:
+
+ filename: filename of cookies.sqlite (typically found at the top level
+ of a firefox profile directory)
+ autoconnect: as a convenience, connect to the SQLite cookies database at
+ Firefox3CookieJar construction time (default True)
+ policy: an object satisfying the mechanize.CookiePolicy interface
+
+ Note that this is NOT a FileCookieJar, and there are no .load(),
+ .save() or .restore() methods. The database is in sync with the
+ cookiejar object's state after each public method call.
+
+ Following Firefox's own behaviour, session cookies are never saved to
+ the database.
+
+ The file is created, and an sqlite database written to it, if it does
+ not already exist. The moz_cookies database table is created if it does
+ not already exist.
+ """
+
+ def __init__(self, filename, autoconnect=True, policy=None):
+ CookieJar.__init__(self, policy)
+ if filename is not None and not isstringlike(filename):
+ raise ValueError("filename must be string-like")
+ self.filename = filename
+ self._conn = None
+ if autoconnect:
+ self.connect()
+
+ def connect(self):
+ self._conn = sqlite3.connect(self.filename)
+ self._conn.isolation_level = "DEFERRED"
+ self._create_table_if_necessary()
+
+ def close(self):
+ self._conn.close()
+
+ def _transaction(self, func):
+ try:
+ cur = self._conn.cursor()
+ try:
+ result = func(cur)
+ finally:
+ cur.close()
+ except:
+ self._conn.rollback()
+ raise
+ else:
+ self._conn.commit()
+ return result
+
+ def _execute(self, query, params=()):
+ return self._transaction(lambda cur: cur.execute(query, params))
+
+ def _query(self, query, params=()):
+ # XXX should we bother with a transaction?
+ cur = self._conn.cursor()
+ try:
+ cur.execute(query, params)
+ for row in cur.fetchall():
+ yield row
+ finally:
+ cur.close()
+
+ def _create_table_if_necessary(self):
+ self._execute("""\
+CREATE TABLE IF NOT EXISTS moz_cookies (id INTEGER PRIMARY KEY, name TEXT,
+ value TEXT, host TEXT, path TEXT,expiry INTEGER,
+ lastAccessed INTEGER, isSecure INTEGER, isHttpOnly INTEGER)""")
+
+ def _cookie_from_row(self, row):
+ (pk, name, value, domain, path, expires,
+ last_accessed, secure, http_only) = row
+
+ version = 0
+ domain = domain.encode("ascii", "ignore")
+ path = path.encode("ascii", "ignore")
+ name = name.encode("ascii", "ignore")
+ value = value.encode("ascii", "ignore")
+ secure = bool(secure)
+
+ # last_accessed isn't a cookie attribute, so isn't added to rest
+ rest = {}
+ if http_only:
+ rest["HttpOnly"] = None
+
+ if name == "":
+ name = value
+ value = None
+
+ initial_dot = domain.startswith(".")
+ domain_specified = initial_dot
+
+ discard = False
+ if expires == "":
+ expires = None
+ discard = True
+
+ return Cookie(version, name, value,
+ None, False,
+ domain, domain_specified, initial_dot,
+ path, False,
+ secure,
+ expires,
+ discard,
+ None,
+ None,
+ rest)
+
+ def clear(self, domain=None, path=None, name=None):
+ CookieJar.clear(self, domain, path, name)
+ where_parts = []
+ sql_params = []
+ if domain is not None:
+ where_parts.append("host = ?")
+ sql_params.append(domain)
+ if path is not None:
+ where_parts.append("path = ?")
+ sql_params.append(path)
+ if name is not None:
+ where_parts.append("name = ?")
+ sql_params.append(name)
+ where = " AND ".join(where_parts)
+ if where:
+ where = " WHERE " + where
+ def clear(cur):
+ cur.execute("DELETE FROM moz_cookies%s" % where,
+ tuple(sql_params))
+ self._transaction(clear)
+
+ def _row_from_cookie(self, cookie, cur):
+ expires = cookie.expires
+ if cookie.discard:
+ expires = ""
+
+ domain = unicode(cookie.domain)
+ path = unicode(cookie.path)
+ name = unicode(cookie.name)
+ value = unicode(cookie.value)
+ secure = bool(int(cookie.secure))
+
+ if value is None:
+ value = name
+ name = ""
+
+ last_accessed = int(time.time())
+ http_only = cookie.has_nonstandard_attr("HttpOnly")
+
+ query = cur.execute("""SELECT MAX(id) + 1 from moz_cookies""")
+ pk = query.fetchone()[0]
+ if pk is None:
+ pk = 1
+
+ return (pk, name, value, domain, path, expires,
+ last_accessed, secure, http_only)
+
+ def set_cookie(self, cookie):
+ if cookie.discard:
+ CookieJar.set_cookie(self, cookie)
+ return
+
+ def set_cookie(cur):
+ row = self._row_from_cookie(cookie, cur)
+ name, unused, domain, path = row[1:5]
+ cur.execute("""\
+DELETE FROM moz_cookies WHERE host = ? AND path = ? AND name = ?""",
+ (domain, path, name))
+ cur.execute("""\
+INSERT INTO moz_cookies VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
+""", row)
+ self._transaction(set_cookie)
+
+ def __iter__(self):
+ # session (non-persistent) cookies
+ for cookie in MappingIterator(self._cookies):
+ yield cookie
+ # persistent cookies
+ for row in self._query("""\
+SELECT * FROM moz_cookies ORDER BY name, path, host"""):
+ yield self._cookie_from_row(row)
+
+ def _cookies_for_request(self, request):
+ session_cookies = CookieJar._cookies_for_request(self, request)
+ def get_cookies(cur):
+ query = cur.execute("SELECT host from moz_cookies")
+ domains = [row[0] for row in query.fetchmany()]
+ cookies = []
+ for domain in domains:
+ cookies += self._persistent_cookies_for_domain(domain,
+ request, cur)
+ return cookies
+ persistent_coookies = self._transaction(get_cookies)
+ return session_cookies + persistent_coookies
+
+ def _persistent_cookies_for_domain(self, domain, request, cur):
+ cookies = []
+ if not self._policy.domain_return_ok(domain, request):
+ return []
+ debug("Checking %s for cookies to return", domain)
+ query = cur.execute("""\
+SELECT * from moz_cookies WHERE host = ? ORDER BY path""",
+ (domain,))
+ cookies = [self._cookie_from_row(row) for row in query.fetchmany()]
+ last_path = None
+ r = []
+ for cookie in cookies:
+ if (cookie.path != last_path and
+ not self._policy.path_return_ok(cookie.path, request)):
+ last_path = cookie.path
+ continue
+ if not self._policy.return_ok(cookie, request):
+ debug(" not returning cookie")
+ continue
+ debug(" it's a match")
+ r.append(cookie)
+ return r
Modified: wwwsearch/mechanize/trunk/test-tools/cookietest.cgi
==============================================================================
--- wwwsearch/mechanize/trunk/test-tools/cookietest.cgi (original)
+++ wwwsearch/mechanize/trunk/test-tools/cookietest.cgi Thu Feb 28 00:22:22 2008
@@ -5,8 +5,12 @@
#import cgitb; cgitb.enable()
+import time
+
print "Content-Type: text/html"
-print "Set-Cookie: foo=bar\n"
+year_plus_one = time.localtime(time.time())[0] + 1
+expires = "expires=09-Nov-%d 23:12:40 GMT" % (year_plus_one,)
+print "Set-Cookie: foo=bar; %s\n" % expires
import sys, os, string, cgi, Cookie, urllib
from xml.sax import saxutils
Modified: wwwsearch/mechanize/trunk/test/test_cookies.py
==============================================================================
--- wwwsearch/mechanize/trunk/test/test_cookies.py (original)
+++ wwwsearch/mechanize/trunk/test/test_cookies.py Thu Feb 28 00:22:22 2008
@@ -35,6 +35,25 @@
return cookie_hdr
+class TempfileTestMixin():
+
+ def setUp(self):
+ self._tempfiles = []
+
+ def tearDown(self):
+ for fn in self._tempfiles:
+ try:
+ os.remove(fn)
+ except IOError, exc:
+ if exc.errno != errno.ENOENT:
+ raise
+
+ def mktemp(self):
+ fn = tempfile.mktemp()
+ self._tempfiles.append(fn)
+ return fn
+
+
class CookieTests(TestCase):
# XXX
# Get rid of string comparisons where not actually testing str / repr.
@@ -864,7 +883,196 @@
assert cookie.expires is None
-class LWPCookieTests(TestCase):
+class CookieJarPersistenceTests(TempfileTestMixin, TestCase):
+
+ def _interact(self, cj):
+ year_plus_one = localtime(time.time())[0] + 1
+ interact_2965(cj, "http://www.acme.com/",
+ "foo1=bar; max-age=100; Version=1")
+ interact_2965(cj, "http://www.acme.com/",
+ 'foo2=bar; port="80"; max-age=100; Discard; Version=1')
+ interact_2965(cj, "http://www.acme.com/", "foo3=bar; secure; Version=1")
+
+ expires = "expires=09-Nov-%d 23:12:40 GMT" % (year_plus_one,)
+ interact_netscape(cj, "http://www.foo.com/",
+ "fooa=bar; %s" % expires)
+ interact_netscape(cj, "http://www.foo.com/",
+ "foob=bar; Domain=.foo.com; %s" % expires)
+ interact_netscape(cj, "http://www.foo.com/",
+ "fooc=bar; Domain=www.foo.com; %s" % expires)
+
+ def test_firefox3_cookiejar_restore(self):
+ try:
+ from mechanize import Firefox3CookieJar
+ except ImportError:
+ pass
+ else:
+ from mechanize import DefaultCookiePolicy
+ filename = self.mktemp()
+ def create_cookiejar():
+ cj = Firefox3CookieJar(filename,
+ policy=DefaultCookiePolicy(rfc2965=True))
+ cj.connect()
+ return cj
+ cj = create_cookiejar()
+ self._interact(cj)
+ self.assertEquals(len(cj), 6)
+ cj.close()
+ cj = create_cookiejar()
+ self.assert_("name='foo1', value='bar'" in repr(cj))
+ self.assertEquals(len(cj), 4)
+
+ def test_firefox3_cookiejar_iteration(self):
+ try:
+ from mechanize import Firefox3CookieJar
+ except ImportError:
+ pass
+ else:
+ from mechanize import DefaultCookiePolicy, Cookie
+ filename = self.mktemp()
+ cj = Firefox3CookieJar(filename,
+ policy=DefaultCookiePolicy(rfc2965=True))
+ cj.connect()
+ self._interact(cj)
+ summary = "\n".join([str(cookie) for cookie in cj])
+ self.assertEquals(summary,
+ """\
+
+
+
+
+
+""")
+
+ def test_firefox3_cookiejar_clear(self):
+ try:
+ from mechanize import Firefox3CookieJar
+ except ImportError:
+ pass
+ else:
+ from mechanize import DefaultCookiePolicy, Cookie
+ filename = self.mktemp()
+ cj = Firefox3CookieJar(filename,
+ policy=DefaultCookiePolicy(rfc2965=True))
+ cj.connect()
+ self._interact(cj)
+ cj.clear("www.acme.com", "/", "foo2")
+ def summary(): return "\n".join([str(cookie) for cookie in cj])
+ self.assertEquals(summary(),
+ """\
+
+
+
+
+""")
+ cj.clear("www.acme.com")
+ self.assertEquals(summary(),
+ """\
+
+
+""")
+ # if name is given, so must path and domain
+ self.assertRaises(ValueError, cj.clear, domain=".foo.com",
+ name="foob")
+ # nonexistent domain
+ self.assertRaises(KeyError, cj.clear, domain=".spam.com")
+
+ def test_firefox3_cookiejar_add_cookie_header(self):
+ try:
+ from mechanize import Firefox3CookieJar
+ except ImportError:
+ pass
+ else:
+ from mechanize import DefaultCookiePolicy, Request
+ filename = self.mktemp()
+ cj = Firefox3CookieJar(filename)
+ cj.connect()
+ # Session cookies (true .discard) and persistent cookies (false
+ # .discard) are stored differently. Check they both get sent.
+ year_plus_one = localtime(time.time())[0] + 1
+ expires = "expires=09-Nov-%d 23:12:40 GMT" % (year_plus_one,)
+ interact_netscape(cj, "http://www.foo.com/", "fooa=bar")
+ interact_netscape(cj, "http://www.foo.com/",
+ "foob=bar; %s" % expires)
+ ca, cb = cj
+ self.assert_(ca.discard)
+ self.assertFalse(cb.discard)
+ request = Request("http://www.foo.com/")
+ cj.add_cookie_header(request)
+ self.assertEquals(request.get_header("Cookie"),
+ "fooa=bar; foob=bar")
+
+ def test_mozilla_cookiejar(self):
+ # Save / load Mozilla/Netscape cookie file format.
+ from mechanize import MozillaCookieJar, DefaultCookiePolicy
+ filename = tempfile.mktemp()
+ c = MozillaCookieJar(filename,
+ policy=DefaultCookiePolicy(rfc2965=True))
+ self._interact(c)
+
+ def save_and_restore(cj, ignore_discard, filename=filename):
+ from mechanize import MozillaCookieJar, DefaultCookiePolicy
+ try:
+ cj.save(ignore_discard=ignore_discard)
+ new_c = MozillaCookieJar(filename,
+ DefaultCookiePolicy(rfc2965=True))
+ new_c.load(ignore_discard=ignore_discard)
+ finally:
+ try: os.unlink(filename)
+ except OSError: pass
+ return new_c
+
+ new_c = save_and_restore(c, True)
+ assert len(new_c) == 6 # none discarded
+ assert repr(new_c).find("name='foo1', value='bar'") != -1
+
+ new_c = save_and_restore(c, False)
+ assert len(new_c) == 4 # 2 of them discarded on save
+ assert repr(new_c).find("name='foo1', value='bar'") != -1
+
+ def test_mozilla_cookiejar_embedded_tab(self):
+ from mechanize import MozillaCookieJar
+ filename = tempfile.mktemp()
+ fh = open(filename, "w")
+ try:
+ fh.write(
+ MozillaCookieJar.header + "\n" +
+ "a.com\tFALSE\t/\tFALSE\t\tname\tval\tstillthevalue\n"
+ "a.com\tFALSE\t/\tFALSE\t\tname2\tvalue\n")
+ fh.close()
+ cj = MozillaCookieJar(filename)
+ cj.revert(ignore_discard=True)
+ cookies = cj._cookies["a.com"]["/"]
+ self.assertEquals(cookies["name"].value, "val\tstillthevalue")
+ self.assertEquals(cookies["name2"].value, "value")
+ finally:
+ try:
+ os.remove(filename)
+ except IOError, exc:
+ if exc.errno != errno.ENOENT:
+ raise
+
+ def test_mozilla_cookiejar_initial_dot_violation(self):
+ from mechanize import MozillaCookieJar, LoadError
+ filename = tempfile.mktemp()
+ fh = open(filename, "w")
+ try:
+ fh.write(
+ MozillaCookieJar.header + "\n" +
+ ".a.com\tFALSE\t/\tFALSE\t\tname\tvalue\n")
+ fh.close()
+ cj = MozillaCookieJar(filename)
+ self.assertRaises(LoadError, cj.revert, ignore_discard=True)
+ finally:
+ try:
+ os.remove(filename)
+ except IOError, exc:
+ if exc.errno != errno.ENOENT:
+ raise
+
+
+
+class LWPCookieTests(TestCase, TempfileTestMixin):
# Tests taken from libwww-perl, with a few modifications.
def test_netscape_example_1(self):
@@ -1300,90 +1508,6 @@
# unicode URL doesn't raise exception, as it used to!
cookie = interact_2965(c, u"http://www.acme.com/\xfc")
- def test_mozilla(self):
- # Save / load Mozilla/Netscape cookie file format.
- from mechanize import MozillaCookieJar, DefaultCookiePolicy
-
- year_plus_one = localtime(time.time())[0] + 1
-
- filename = tempfile.mktemp()
-
- c = MozillaCookieJar(filename,
- policy=DefaultCookiePolicy(rfc2965=True))
- interact_2965(c, "http://www.acme.com/",
- "foo1=bar; max-age=100; Version=1")
- interact_2965(c, "http://www.acme.com/",
- 'foo2=bar; port="80"; max-age=100; Discard; Version=1')
- interact_2965(c, "http://www.acme.com/", "foo3=bar; secure; Version=1")
-
- expires = "expires=09-Nov-%d 23:12:40 GMT" % (year_plus_one,)
- interact_netscape(c, "http://www.foo.com/",
- "fooa=bar; %s" % expires)
- interact_netscape(c, "http://www.foo.com/",
- "foob=bar; Domain=.foo.com; %s" % expires)
- interact_netscape(c, "http://www.foo.com/",
- "fooc=bar; Domain=www.foo.com; %s" % expires)
-
- def save_and_restore(cj, ignore_discard, filename=filename):
- from mechanize import MozillaCookieJar, DefaultCookiePolicy
- try:
- cj.save(ignore_discard=ignore_discard)
- new_c = MozillaCookieJar(filename,
- DefaultCookiePolicy(rfc2965=True))
- new_c.load(ignore_discard=ignore_discard)
- finally:
- try: os.unlink(filename)
- except OSError: pass
- return new_c
-
- new_c = save_and_restore(c, True)
- assert len(new_c) == 6 # none discarded
- assert repr(new_c).find("name='foo1', value='bar'") != -1
-
- new_c = save_and_restore(c, False)
- assert len(new_c) == 4 # 2 of them discarded on save
- assert repr(new_c).find("name='foo1', value='bar'") != -1
-
- def test_mozilla_cookiejar_embedded_tab(self):
- from mechanize import MozillaCookieJar
- filename = tempfile.mktemp()
- fh = open(filename, "w")
- try:
- fh.write(
- MozillaCookieJar.header + "\n" +
- "a.com\tFALSE\t/\tFALSE\t\tname\tval\tstillthevalue\n"
- "a.com\tFALSE\t/\tFALSE\t\tname2\tvalue\n")
- fh.close()
- cj = MozillaCookieJar(filename)
- cj.revert(ignore_discard=True)
- cookies = cj._cookies["a.com"]["/"]
- self.assertEquals(cookies["name"].value, "val\tstillthevalue")
- self.assertEquals(cookies["name2"].value, "value")
- finally:
- try:
- os.remove(filename)
- except OSError, exc:
- if exc.errno != errno.EEXIST:
- raise
-
- def test_mozilla_cookiejar_initial_dot_violation(self):
- from mechanize import MozillaCookieJar, LoadError
- filename = tempfile.mktemp()
- fh = open(filename, "w")
- try:
- fh.write(
- MozillaCookieJar.header + "\n" +
- ".a.com\tFALSE\t/\tFALSE\t\tname\tvalue\n")
- fh.close()
- cj = MozillaCookieJar(filename)
- self.assertRaises(LoadError, cj.revert, ignore_discard=True)
- finally:
- try:
- os.remove(filename)
- except OSError, exc:
- if exc.errno != errno.EEXIST:
- raise
-
def test_netscape_misc(self):
# Some additional Netscape cookies tests.
from mechanize import CookieJar, Request
From jjlee at codespeak.net Thu Feb 28 01:31:51 2008
From: jjlee at codespeak.net (jjlee at codespeak.net)
Date: Thu, 28 Feb 2008 01:31:51 +0100 (CET)
Subject: [wwwsearch-commits] r51912 - wwwsearch/mechanize/trunk/mechanize
Message-ID: <20080228003151.B0D671683BC@codespeak.net>
Author: jjlee
Date: Thu Feb 28 01:31:47 2008
New Revision: 51912
Modified:
wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py
Log:
Note a couple of issues with Firefox 3 cookie jar implementation
Modified: wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py Thu Feb 28 01:31:47 2008
@@ -47,6 +47,8 @@
not already exist.
"""
+ # XXX handle DatabaseError exceptions
+
def __init__(self, filename, autoconnect=True, policy=None):
CookieJar.__init__(self, policy)
if filename is not None and not isstringlike(filename):
@@ -189,6 +191,9 @@
return
def set_cookie(cur):
+ # XXX
+ # is this RFC 2965-correct?
+ # could this do an UPDATE instead?
row = self._row_from_cookie(cookie, cur)
name, unused, domain, path = row[1:5]
cur.execute("""\
From jjlee at codespeak.net Fri Feb 29 21:58:00 2008
From: jjlee at codespeak.net (jjlee at codespeak.net)
Date: Fri, 29 Feb 2008 21:58:00 +0100 (CET)
Subject: [wwwsearch-commits] r51967 - wwwsearch/mechanize/trunk/mechanize
Message-ID: <20080229205800.1E178168539@codespeak.net>
Author: jjlee
Date: Fri Feb 29 21:57:59 2008
New Revision: 51967
Modified:
wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py
Log:
Add note re work to do on Firefox 3 cookies support
Modified: wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py (original)
+++ wwwsearch/mechanize/trunk/mechanize/_firefox3cookiejar.py Fri Feb 29 21:57:59 2008
@@ -47,7 +47,9 @@
not already exist.
"""
- # XXX handle DatabaseError exceptions
+ # XXX
+ # handle DatabaseError exceptions
+ # add a FileCookieJar (explicit .save() / .revert() / .load() methods)
def __init__(self, filename, autoconnect=True, policy=None):
CookieJar.__init__(self, policy)
From wwwsearch-commits at codespeak.net Tue Mar 4 23:45:03 2008
From: wwwsearch-commits at codespeak.net (wwwsearch-commits at codespeak.net)
Date: Tue, 4 Mar 2008 23:45:03 +0100 (CET)
Subject: [wwwsearch-commits] Pharmacy Online March 70% OFF
Message-ID: <20080304004429.4450.qmail@adsl190-28-38-111.epm.net.co>
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/wwwsearch-commits/attachments/20080304/7eec8f28/attachment.htm
From wwwsearch-commits at codespeak.net Sun Mar 16 12:29:33 2008
From: wwwsearch-commits at codespeak.net (wwwsearch-commits at codespeak.net)
Date: Sun, 16 Mar 2008 12:29:33 +0100 (CET)
Subject: [wwwsearch-commits] Discount. Coupon #pprwf
Message-ID: <20080316132850.5034.qmail@fej49.internetdsl.tpnet.pl>
Dear wwwsearch-commits at codespeak.net, be clever, purchase your pharmaceuticals from the best shop since 1992.
http://www.google.com/pagead/iclk?sa=l&ai=aqbfk&num=7163396&adurl=http://zjm.darksidehome.com
From wwwsearch-commits at codespeak.net Wed Mar 19 08:47:46 2008
From: wwwsearch-commits at codespeak.net (wwwsearch-commits at codespeak.net)
Date: Wed, 19 Mar 2008 08:47:46 +0100 (CET)
Subject: [wwwsearch-commits] Men's Health id 4943907
Message-ID: <20080319094655.10122.qmail@a227.net156.okay.pl>
Canadian Doctor Amado Link Best Price On Net March 84% OFF!
http://www.google.in/pagead/iclk?sa=l&ai=pgohn&num=13281&adurl=http://www.samegentle.com
From wwwsearch-commits at codespeak.net Thu Mar 27 10:13:24 2008
From: wwwsearch-commits at codespeak.net (MasterDeals Admin 24153426357 ® Official Site)
Date: Thu, 27 Mar 2008 10:13:24 +0100 (CET)
Subject: [wwwsearch-commits] March %76 OFF
Message-ID: <20080327111441.14059.qmail@static-46-135-155-203.ksc.net.th>
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/wwwsearch-commits/attachments/20080327/d1be4692/attachment.htm
From wwwsearch-commits at codespeak.net Mon Mar 31 12:30:29 2008
From: wwwsearch-commits at codespeak.net (wwwsearch-commits at codespeak.net)
Date: Mon, 31 Mar 2008 12:30:29 +0200 (CEST)
Subject: [wwwsearch-commits] MedHelp 41683
Message-ID: <20080331132928.5816.qmail@dslb-088-073-120-051.pools.arcor-ip.net>
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/wwwsearch-commits/attachments/20080331/429e7eb3/attachment.htm