From jjlee at codespeak.net  Wed Apr  5 20:40:23 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Wed Apr  5 20:40:25 2006
Subject: [wwwsearch-commits] r25416 -
	wwwsearch/ClientCookie/trunk/ClientCookie
Message-ID: <20060405184023.AF90A10235@code0.codespeak.net>

Author: jjlee
Date: Wed Apr  5 20:40:22 2006
New Revision: 25416

Modified:
   wwwsearch/ClientCookie/trunk/ClientCookie/_urllib2_support.py
Log:
Fix unwarranted case assumption in Refresh header value

Modified: wwwsearch/ClientCookie/trunk/ClientCookie/_urllib2_support.py
==============================================================================
--- wwwsearch/ClientCookie/trunk/ClientCookie/_urllib2_support.py	(original)
+++ wwwsearch/ClientCookie/trunk/ClientCookie/_urllib2_support.py	Wed Apr  5 20:40:22 2006
@@ -584,7 +584,7 @@
                     jj = string.find(newurl_spec, "=")
                     if jj != -1:
                         key, newurl = newurl_spec[:jj], newurl_spec[jj+1:]
-                    if key.strip() != "url":
+                    if key.strip().lower() != "url":
                         debug("bad Refresh header: %r" % refresh)
                         return response
                 else:
From jjlee at codespeak.net  Thu Apr  6 00:30:52 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Thu Apr  6 00:30:53 2006
Subject: [wwwsearch-commits] r25427 - wwwsearch/ClientCookie/trunk
Message-ID: <20060405223052.1833D1022D@code0.codespeak.net>

Author: jjlee
Date: Thu Apr  6 00:30:51 2006
New Revision: 25427

Modified:
   wwwsearch/ClientCookie/trunk/   (props changed)
Log:
Update to latest setuptools
From jjlee at codespeak.net  Thu Apr  6 00:34:24 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Thu Apr  6 00:34:24 2006
Subject: [wwwsearch-commits] r25428 -
	wwwsearch/ClientCookie/tag/release/1.2.1-2006-04-05T22:03:57
Message-ID: <20060405223424.35A031022D@code0.codespeak.net>

Author: jjlee
Date: Thu Apr  6 00:34:23 2006
New Revision: 25428

Added:
   wwwsearch/ClientCookie/tag/release/1.2.1-2006-04-05T22:03:57/
      - copied from r25427, wwwsearch/ClientCookie/trunk/
Modified:
   wwwsearch/ClientCookie/tag/release/1.2.1-2006-04-05T22:03:57/setup.cfg
Log:
Tagged ClientCookie (trunk -r HEAD, from working copy) release 1.2.1-2006-04-05T22:03:57

Modified: wwwsearch/ClientCookie/tag/release/1.2.1-2006-04-05T22:03:57/setup.cfg
==============================================================================
--- wwwsearch/ClientCookie/trunk/setup.cfg	(original)
+++ wwwsearch/ClientCookie/tag/release/1.2.1-2006-04-05T22:03:57/setup.cfg	Thu Apr  6 00:34:23 2006
@@ -1,3 +0,0 @@
-[egg_info]
-tag_build = .dev
-tag_svn_revision = 1
From jjlee at codespeak.net  Thu Apr  6 00:43:56 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Thu Apr  6 00:43:56 2006
Subject: [wwwsearch-commits] r25429 - wwwsearch/ClientCookie/common
Message-ID: <20060405224356.3415B1022D@code0.codespeak.net>

Author: jjlee
Date: Thu Apr  6 00:43:55 2006
New Revision: 25429

Modified:
   wwwsearch/ClientCookie/common/ChangeLog
Log:
Update ChangeLog

Modified: wwwsearch/ClientCookie/common/ChangeLog
==============================================================================
--- wwwsearch/ClientCookie/common/ChangeLog	(original)
+++ wwwsearch/ClientCookie/common/ChangeLog	Thu Apr  6 00:43:55 2006
@@ -1,6 +1,16 @@
 This isn't really in proper GNU ChangeLog format, it just happens to
 look that way.
 
+2006-04-23 John J Lee <jjl@pobox.com>
+	* 1.2.1 release:
+	* Fix Refresh in case where the string "url" in the header value
+	  is in upper-case.
+	* Don't do broken XHTML handling by default (need to review code
+	  before switching this back on, e.g. should use a real XML parser
+	  for first-try at parsing).  HTTPEquivProcessor's
+	  i_want_broken_xhtml_support constructor argument gives the old
+	  behaviour.
+
 2006-03-23 John J Lee <jjl@pobox.com>
 	* 1.2.0 release:
 	* Stop trying to record precise dates in changelog, since that's
From jjlee at codespeak.net  Thu Apr  6 01:12:34 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Thu Apr  6 01:12:39 2006
Subject: [wwwsearch-commits] r25430 - wwwsearch/ClientCookie/trunk/test
Message-ID: <20060405231234.03BB810232@code0.codespeak.net>

Author: jjlee
Date: Thu Apr  6 01:12:32 2006
New Revision: 25430

Modified:
   wwwsearch/ClientCookie/trunk/test/test_headers.py
   wwwsearch/ClientCookie/trunk/test/test_urllib2.py
Log:
Silence one gaierror that has showed up in tests for ages on some machines (another is fixed in stdlib for 2.5); Fix old-Python backwards-compatibility test failure

Modified: wwwsearch/ClientCookie/trunk/test/test_headers.py
==============================================================================
--- wwwsearch/ClientCookie/trunk/test/test_headers.py	(original)
+++ wwwsearch/ClientCookie/trunk/test/test_headers.py	Thu Apr  6 01:12:32 2006
@@ -2,6 +2,11 @@
 
 from unittest import TestCase
 
+try: True
+except NameError:
+    True = 1
+    False = 0
+
 class IsHtmlTests(TestCase):
     def test_is_html(self):
         from ClientCookie._HeadersUtil import is_html

Modified: wwwsearch/ClientCookie/trunk/test/test_urllib2.py
==============================================================================
--- wwwsearch/ClientCookie/trunk/test/test_urllib2.py	(original)
+++ wwwsearch/ClientCookie/trunk/test/test_urllib2.py	Thu Apr  6 01:12:32 2006
@@ -370,12 +370,15 @@
             TESTFN = "test.txt"
             urlpath = sanepathname2url(os.path.abspath(TESTFN))
             towrite = "hello, world\n"
+            try:
+                fqdn = socket.gethostbyname(socket.gethostname())
+            except socket.gaierror:
+                fqdn = "localhost"
             for url in [
                 "file://localhost%s" % urlpath,
                 "file://%s" % urlpath,
                 "file://%s%s" % (socket.gethostbyname('localhost'), urlpath),
-                "file://%s%s" % (socket.gethostbyname(socket.gethostname()),
-                                 urlpath),
+                "file://%s%s" % (fqdn, urlpath)
                 ]:
                 f = open(TESTFN, "wb")
                 try:
From jjlee at codespeak.net  Thu Apr  6 01:13:44 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Thu Apr  6 01:13:45 2006
Subject: [wwwsearch-commits] r25431 -
	wwwsearch/ClientCookie/tag/release/1.2.1-2006-04-05T22:03:57
Message-ID: <20060405231344.9A33E10232@code0.codespeak.net>

Author: jjlee
Date: Thu Apr  6 01:13:43 2006
New Revision: 25431

Removed:
   wwwsearch/ClientCookie/tag/release/1.2.1-2006-04-05T22:03:57/
Log:
Remove release tag (tests needed fixing)
From jjlee at codespeak.net  Sat Apr  8 15:28:44 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 15:28:45 2006
Subject: [wwwsearch-commits] r25545 - wwwsearch/pullparser/trunk
Message-ID: <20060408132844.04CC61026E@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 15:28:44 2006
New Revision: 25545

Modified:
   wwwsearch/pullparser/trunk/pullparser.py
Log:
Fix entitydefs documentation

Modified: wwwsearch/pullparser/trunk/pullparser.py
==============================================================================
--- wwwsearch/pullparser/trunk/pullparser.py	(original)
+++ wwwsearch/pullparser/trunk/pullparser.py	Sat Apr  8 15:28:44 2006
@@ -283,10 +283,10 @@
         .get_tag() first (unless you want an empty string returned when you
         next call .get_text()).
 
-        Entity references are translated using the entitydefs attribute (a
-        mapping from names to characters like that provided by the standard
-        module htmlentitydefs).  Named entity references that are not in this
-        mapping are left unchanged.
+        Entity references are translated using the value of the entitydefs
+        constructor argument (a mapping from names to characters like that
+        provided by the standard module htmlentitydefs).  Named entity
+        references that are not in this mapping are left unchanged.
 
         The textify attribute is used to translate opening tags into text: see
         the class docstring.
From jjlee at codespeak.net  Sat Apr  8 17:16:53 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 17:16:55 2006
Subject: [wwwsearch-commits] r25551 - wwwsearch/pullparser/trunk
Message-ID: <20060408151653.9D4BA1025C@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 17:16:51 2006
New Revision: 25551

Modified:
   wwwsearch/pullparser/trunk/README.html.in
   wwwsearch/pullparser/trunk/pullparser.py
   wwwsearch/pullparser/trunk/setup.py
Log:
Update version to reflect backwards-incompatible .entitydefs change (change was made some time ago, after 0.0.7 released)

Modified: wwwsearch/pullparser/trunk/README.html.in
==============================================================================
--- wwwsearch/pullparser/trunk/README.html.in	(original)
+++ wwwsearch/pullparser/trunk/README.html.in	Sat Apr  8 17:16:51 2006
@@ -71,7 +71,7 @@
 <p><em>Stable release.</em>
 
 <ul>
-@{version = "0.0.8"}
+@{version = "0.1.0"}
 @{win_version = release.win_version(version)}
 <li><a href="./src/pullparser-@(version).tar.gz">pullparser-@(version).tar.gz</a>
 <li><a href="./src/pullparser-@(win_version).zip">pullparser-@(win_version).zip</a>

Modified: wwwsearch/pullparser/trunk/pullparser.py
==============================================================================
--- wwwsearch/pullparser/trunk/pullparser.py	(original)
+++ wwwsearch/pullparser/trunk/pullparser.py	Sat Apr  8 17:16:51 2006
@@ -38,7 +38,7 @@
 import re, htmlentitydefs
 import HTMLParser
 
-__version__ = (0, 0, 8, None, None)  # 0.0.8
+__version__ = (0, 1, 0, None, None)  # 0.1.0
 
 
 class NoMoreTokensError(Exception): pass

Modified: wwwsearch/pullparser/trunk/setup.py
==============================================================================
--- wwwsearch/pullparser/trunk/setup.py	(original)
+++ wwwsearch/pullparser/trunk/setup.py	Sat Apr  8 17:16:51 2006
@@ -41,7 +41,7 @@
 ## VERSION_MATCH = re.search(r'__version__ = \((.*)\)',
 ##                           open("pullparser.py").read())
 ## VERSION = unparse_version(str_to_tuple(VERSION_MATCH.group(1)))
-VERSION = '0.0.8'
+VERSION = '0.1.0'
 INSTALL_REQUIRES = []
 NAME = "pullparser"
 PACKAGE = False
From jjlee at codespeak.net  Sat Apr  8 17:17:30 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 17:17:31 2006
Subject: [wwwsearch-commits] r25552 - wwwsearch/pullparser/trunk
Message-ID: <20060408151730.66EAE1025C@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 17:17:29 2006
New Revision: 25552

Modified:
   wwwsearch/pullparser/trunk/README.html.in
Log:
Recommend BeautifulSoup

Modified: wwwsearch/pullparser/trunk/README.html.in
==============================================================================
--- wwwsearch/pullparser/trunk/README.html.in	(original)
+++ wwwsearch/pullparser/trunk/README.html.in	Sat Apr  8 17:17:29 2006
@@ -97,6 +97,11 @@
 <p><a href="http://www.crummy.com/software/BeautifulSoup/">Beautiful
 Soup</a> is widely recommended.  More robust than this module.
 
+<p>I recommend <a
+href="http://www.crummy.com/software/BeautifulSoup/">Beautiful Soup</a> over
+pullparser for new web scraping code.  More robust and flexible than this
+module.
+
 <a name="faq"></a>
 <h2>FAQs</h2>
 <ul>
From jjlee at codespeak.net  Sat Apr  8 17:21:16 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 17:21:17 2006
Subject: [wwwsearch-commits] r25553 - wwwsearch/pullparser/trunk
Message-ID: <20060408152116.B01601025C@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 17:21:15 2006
New Revision: 25553

Modified:
   wwwsearch/pullparser/trunk/   (props changed)
Log:
Use latest ez_setup SVN
From jjlee at codespeak.net  Sat Apr  8 17:22:55 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 17:22:56 2006
Subject: [wwwsearch-commits] r25554 -
	wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T14:52:18
Message-ID: <20060408152255.4EA861025C@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 17:22:54 2006
New Revision: 25554

Added:
   wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T14:52:18/
      - copied from r25553, wwwsearch/pullparser/trunk/
Modified:
   wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T14:52:18/setup.cfg
Log:
Tagged pullparser (trunk -r HEAD, from working copy) release 0.1.0-2006-04-08T14:52:18

Modified: wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T14:52:18/setup.cfg
==============================================================================
--- wwwsearch/pullparser/trunk/setup.cfg	(original)
+++ wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T14:52:18/setup.cfg	Sat Apr  8 17:22:54 2006
@@ -1,3 +0,0 @@
-[egg_info]
-tag_build = .dev
-tag_svn_revision = 1
From jjlee at codespeak.net  Sat Apr  8 19:00:35 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 19:00:36 2006
Subject: [wwwsearch-commits] r25570 -
	wwwsearch/pullparser/tag/release/0.1.0-2005-11-19T00:10:00
Message-ID: <20060408170035.6E3F5102AD@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 19:00:34 2006
New Revision: 25570

Added:
   wwwsearch/pullparser/tag/release/0.1.0-2005-11-19T00:10:00/
      - copied from r20050, wwwsearch/pullparser/trunk/
Log:
Tag 0.0.7 pullparser release (forgot to make it when the release went out)
From jjlee at codespeak.net  Sat Apr  8 19:01:38 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 19:01:39 2006
Subject: [wwwsearch-commits] r25571 -
	wwwsearch/pullparser/tag/release/0.1.0-2005-11-19T00:10:00
Message-ID: <20060408170138.D73321025F@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 19:01:38 2006
New Revision: 25571

Removed:
   wwwsearch/pullparser/tag/release/0.1.0-2005-11-19T00:10:00/
Log:
Oops, tagged with wrong version number: remove it again
From jjlee at codespeak.net  Sat Apr  8 19:01:46 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 19:01:46 2006
Subject: [wwwsearch-commits] r25572 -
	wwwsearch/pullparser/tag/release/0.0.7-2005-11-19T00:10:00
Message-ID: <20060408170146.42C431025F@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 19:01:45 2006
New Revision: 25572

Added:
   wwwsearch/pullparser/tag/release/0.0.7-2005-11-19T00:10:00/
      - copied from r20050, wwwsearch/pullparser/trunk/
Log:
Tag 0.0.7 pullparser release (forgot to make it when the release went out)
From jjlee at codespeak.net  Sat Apr  8 19:26:07 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 19:26:08 2006
Subject: [wwwsearch-commits] r25574 - wwwsearch/pullparser/common
Message-ID: <20060408172607.0CE55102A9@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 19:26:06 2006
New Revision: 25574

Modified:
   wwwsearch/pullparser/common/ChangeLog
Log:
Update ChangeLog

Modified: wwwsearch/pullparser/common/ChangeLog
==============================================================================
--- wwwsearch/pullparser/common/ChangeLog	(original)
+++ wwwsearch/pullparser/common/ChangeLog	Sat Apr  8 19:26:06 2006
@@ -1,6 +1,29 @@
 This isn't really in proper GNU ChangeLog format, it just happens to
 look that way.
 
+2006-04-08 John J Lee <jjl@pobox.com>
+	* 0.1.0 release:
+	* Backwards-incompatible change: .entitydefs attribute is no
+	  longer supported: use entitydefs constructor argument instead.
+	* Fix entity reference bugs, including:
+	* Don't unescape entity refs where encoding does not allow it
+	  (previously, UnicodeError was raised).
+	* Unescape numeric entity refs in attribute values and text.
+	* setup.py changes to support Pythons older than 2.3 again, using
+	  plain-distutils install.
+	* Add setup.cfg for setuptools.
+	* Pin ez_setup svn:externals property to specific revision.
+	* TolerantPullParser: monkeypatch sgmllib to fix
+	  http://www.python.org/sf/803422 :-(
+	* Minor documentation updates.
+
+2005-11-19 John J Lee <jjl@pobox.com>
+	* Stop trying to record precise dates in changelog, since that's
+	  silly ;-)
+	* Correct setup.py Development Status classifier.
+	* setuptools changes.
+	* Released 0.0.7.
+
 2005-10-31 John J Lee <jjl@pobox.com>
 	* Add setuptools support.
 
From jjlee at codespeak.net  Sat Apr  8 20:05:32 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 20:05:33 2006
Subject: [wwwsearch-commits] r25578 - wwwsearch/pullparser/trunk
Message-ID: <20060408180532.64971102AC@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 20:05:30 2006
New Revision: 25578

Added:
   wwwsearch/pullparser/trunk/COPYING.txt
      - copied, changed from r25552, wwwsearch/pullparser/trunk/COPYING
Removed:
   wwwsearch/pullparser/trunk/COPYING
Modified:
   wwwsearch/pullparser/trunk/INSTALL
   wwwsearch/pullparser/trunk/MANIFEST.in
   wwwsearch/pullparser/trunk/README.html.in
   wwwsearch/pullparser/trunk/pullparser.py
   wwwsearch/pullparser/trunk/setup.py
Log:
Explicitly allow ZPL 2.1; Update some copyright dates &c

Deleted: /wwwsearch/pullparser/trunk/COPYING
==============================================================================
--- /wwwsearch/pullparser/trunk/COPYING	Sat Apr  8 20:05:30 2006
+++ (empty file)
@@ -1,31 +0,0 @@
-Copyright (c) 2002-2005 John J. Lee <jjl@pobox.com>
-Copyright (c) 1998-2001 Gisle Aas (original libwww-perl code)
-
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are
-met:
-
-Redistributions of source code must retain the above copyright notice,
-this list of conditions and the following disclaimer.
-
-Redistributions in binary form must reproduce the above copyright
-notice, this list of conditions and the following disclaimer in the
-documentation and/or other materials provided with the distribution.
-
-Neither the name of the contributors nor the names of their employers
-may be used to endorse or promote products derived from this software
-without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Copied: wwwsearch/pullparser/trunk/COPYING.txt (from r25552, wwwsearch/pullparser/trunk/COPYING)
==============================================================================
--- wwwsearch/pullparser/trunk/COPYING	(original)
+++ wwwsearch/pullparser/trunk/COPYING.txt	Sat Apr  8 20:05:30 2006
@@ -1,6 +1,65 @@
-Copyright (c) 2002-2005 John J. Lee <jjl@pobox.com>
+Copyright (c) 2002-2006 John J. Lee <jjl@pobox.com>
 Copyright (c) 1998-2001 Gisle Aas (original libwww-perl code)
 
+This software is dual-licensed: you may pick either of the licenses
+below.
+
+
+BSD-style license
+==================
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+Redistributions of source code must retain the above copyright notice,
+this list of conditions and the following disclaimer.
+
+Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+
+Neither the name of the contributors nor the names of their employers
+may be used to endorse or promote products derived from this software
+without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+
+
+ZPL 2.1
+==================
+
+Zope Public License (ZPL) Version 2.1
+
+A copyright notice accompanies this license document that identifies the copyright holders.
+
+This license has been certified as open source. It has also been designated as GPL compatible by the Free Software Foundation (FSF).
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+   1. Redistributions in source code must retain the accompanying copyright notice, this list of conditions, and the following disclaimer.
+   2. Redistributions in binary form must reproduce the accompanying copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution.
+   3. Names of the copyright holders must not be used to endorse or promote products derived from this software without prior written permission from the copyright holders.
+   4. The right to distribute this software or to use it for any purpose does not give you the right to use Servicemarks (sm) or Trademarks (tm) of the copyright holders. Use of them is covered by separate agreement with the copyright holders.
+   5. If any files are modified, you must cause the modified files to carry prominent notices stating that you changed the files and the date of any change.
+
+Disclaimer
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

Modified: wwwsearch/pullparser/trunk/INSTALL
==============================================================================
--- wwwsearch/pullparser/trunk/INSTALL	(original)
+++ wwwsearch/pullparser/trunk/INSTALL	Sat Apr  8 20:05:30 2006
@@ -21,9 +21,8 @@
 
 
 Alternatively, just copy pullparser.py into your Python path
-(eg. unix: /usr/local/lib/python2.2/site-packages, Windows:
-C:\Python21, or C:\Python22\Lib\site-packages).  That's all that
-setup.py does.
+(eg. unix: /usr/local/lib/python2.4/site-packages, Windows:
+C:\Python24\Lib\site-packages).
 
 
 To run the tests (none of which access the network), run the following
@@ -48,12 +47,12 @@
 
 Copyright Notices
 
-  (C) 2002-2003 John J. Lee.  All rights reserved.
+  (C) 2002-2006 John J. Lee.  All rights reserved.
   (C) 1998-2001 Gisle Aas.  All rights reserved.   (Original LWP code)
 
 This code in this package is free software; you can redistribute it
-and/or modify it under the terms of the BSD license (see the file
-COPYING).
+and/or modify it under the terms of the BSD or ZPL 2.1 licenses (see
+the file COPYING.txt).
 
 John J. Lee <jjl@pobox.com>
-December 2003
+April 2006

Modified: wwwsearch/pullparser/trunk/MANIFEST.in
==============================================================================
--- wwwsearch/pullparser/trunk/MANIFEST.in	(original)
+++ wwwsearch/pullparser/trunk/MANIFEST.in	Sat Apr  8 20:05:30 2006
@@ -5,5 +5,5 @@
 include README.html.in
 include README.html
 include README.txt
-include ChangeLog
+include ChangeLog.txt
 include *.py

Modified: wwwsearch/pullparser/trunk/README.html.in
==============================================================================
--- wwwsearch/pullparser/trunk/README.html.in	(original)
+++ wwwsearch/pullparser/trunk/README.html.in	Sat Apr  8 20:05:30 2006
@@ -108,8 +108,10 @@
   <li>Which version of Python do I need?
   <p>2.2.1 or above.
   <li>Which license?
-  <p>The <a href="http://www.opensource.org/licenses/bsd-license.php">
-   BSD license</a> (included in distribution).
+  <p>pullparser is dual-licensed: you may pick either the
+     <a href="http://www.opensource.org/licenses/bsd-license.php">BSD license</a>,
+     or the <a href="http://www.zope.org/Resources/ZPL">ZPL 2.1</a> (both are
+     included in the distribution).
   <li>Why does it fail to parse my HTML?
     <p>Because module <code>HTMLParser</code> is fussy.  Try
     <code>pullparser.TolerantPullParser</code> instead, which uses module

Modified: wwwsearch/pullparser/trunk/pullparser.py
==============================================================================
--- wwwsearch/pullparser/trunk/pullparser.py	(original)
+++ wwwsearch/pullparser/trunk/pullparser.py	Sat Apr  8 20:05:30 2006
@@ -29,7 +29,7 @@
 Copyright 1998-2001 Gisle Aas (original libwww-perl code)
 
 This code is free software; you can redistribute it and/or modify it
-under the terms of the BSD License.
+under the terms of the BSD or ZPL 2.1 licenses.
 
 """
 

Modified: wwwsearch/pullparser/trunk/setup.py
==============================================================================
--- wwwsearch/pullparser/trunk/setup.py	(original)
+++ wwwsearch/pullparser/trunk/setup.py	Sat Apr  8 20:05:30 2006
@@ -45,7 +45,7 @@
 INSTALL_REQUIRES = []
 NAME = "pullparser"
 PACKAGE = False
-LICENSE = "BSD"
+LICENSE = "BSD"  # or ZPL 2.1
 PLATFORMS = ["any"]
 ZIP_SAFE = True
 CLASSIFIERS = """\
@@ -53,6 +53,7 @@
 Intended Audience :: Developers
 Intended Audience :: System Administrators
 License :: OSI Approved :: BSD License
+License :: OSI Approved :: Zope Public License
 Natural Language :: English
 Operating System :: OS Independent
 Programming Language :: Python
From jjlee at codespeak.net  Sat Apr  8 20:05:59 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 20:06:00 2006
Subject: [wwwsearch-commits] r25579 - wwwsearch/pullparser/trunk
Message-ID: <20060408180559.55E44102B0@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 20:05:58 2006
New Revision: 25579

Added:
   wwwsearch/pullparser/trunk/INSTALL.txt
      - copied unchanged from r25578, wwwsearch/pullparser/trunk/INSTALL
Removed:
   wwwsearch/pullparser/trunk/INSTALL
Log:
Rename INSTALL to have .txt file extension

Deleted: /wwwsearch/pullparser/trunk/INSTALL
==============================================================================
--- /wwwsearch/pullparser/trunk/INSTALL	Sat Apr  8 20:05:58 2006
+++ (empty file)
@@ -1,58 +0,0 @@
-pullparser installation instructions
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-Dependencies
-~~~~~~~~~~~~
-
-Python 2.2.1 or above is required.
-
-
-Installation
-~~~~~~~~~~~~
-
-To install the package, run the following command:
-
- python setup.py build
-
-then (with appropriate permissions)
-
- python setup.py install
-
-
-Alternatively, just copy pullparser.py into your Python path
-(eg. unix: /usr/local/lib/python2.4/site-packages, Windows:
-C:\Python24\Lib\site-packages).
-
-
-To run the tests (none of which access the network), run the following
-command:
-
- python test.py
-
-This runs the tests against the source files extracted from the
-package.  For help on command line options:
-
- python test.py --help
-
-
-Bugs and comments to jjl@pobox.com.
-
-
-NO WARRANTY
-
-THIS PACKAGE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
-WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
-MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
-
-Copyright Notices
-
-  (C) 2002-2006 John J. Lee.  All rights reserved.
-  (C) 1998-2001 Gisle Aas.  All rights reserved.   (Original LWP code)
-
-This code in this package is free software; you can redistribute it
-and/or modify it under the terms of the BSD or ZPL 2.1 licenses (see
-the file COPYING.txt).
-
-John J. Lee <jjl@pobox.com>
-April 2006
From jjlee at codespeak.net  Sat Apr  8 20:06:23 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 20:06:24 2006
Subject: [wwwsearch-commits] r25580 - wwwsearch/pullparser/trunk
Message-ID: <20060408180623.D9AFC102AF@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 20:06:23 2006
New Revision: 25580

Modified:
   wwwsearch/pullparser/trunk/MANIFEST.in
Log:
Update MANIFEST.in

Modified: wwwsearch/pullparser/trunk/MANIFEST.in
==============================================================================
--- wwwsearch/pullparser/trunk/MANIFEST.in	(original)
+++ wwwsearch/pullparser/trunk/MANIFEST.in	Sat Apr  8 20:06:23 2006
@@ -1,6 +1,6 @@
 include MANIFEST.in
-include COPYING
-include INSTALL
+include COPYING.txt
+include INSTALL.txt
 include GeneralFAQ.html
 include README.html.in
 include README.html
From jjlee at codespeak.net  Sat Apr  8 20:12:52 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 20:12:53 2006
Subject: [wwwsearch-commits] r25582 -
	wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T14:52:18
Message-ID: <20060408181252.1DDC0102B0@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 20:12:51 2006
New Revision: 25582

Removed:
   wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T14:52:18/
Log:
Remove tag: not used for release
From jjlee at codespeak.net  Sat Apr  8 20:13:08 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 20:13:09 2006
Subject: [wwwsearch-commits] r25583 -
	wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:42:31
Message-ID: <20060408181308.33AEE102B0@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 20:13:07 2006
New Revision: 25583

Added:
   wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:42:31/
      - copied from r25582, wwwsearch/pullparser/trunk/
Modified:
   wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:42:31/setup.cfg
Log:
Tagged pullparser (trunk -r HEAD, from working copy) release 0.1.0-2006-04-08T17:42:31

Modified: wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:42:31/setup.cfg
==============================================================================
--- wwwsearch/pullparser/trunk/setup.cfg	(original)
+++ wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:42:31/setup.cfg	Sat Apr  8 20:13:07 2006
@@ -1,3 +0,0 @@
-[egg_info]
-tag_build = .dev
-tag_svn_revision = 1
From jjlee at codespeak.net  Sat Apr  8 20:27:22 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 20:27:23 2006
Subject: [wwwsearch-commits] r25584 - wwwsearch/pullparser/trunk
Message-ID: <20060408182722.7C4D9100F9@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 20:27:21 2006
New Revision: 25584

Modified:
   wwwsearch/pullparser/trunk/README.html.in
Log:
Fix README link to use dots instead of underscores in .zip source package

Modified: wwwsearch/pullparser/trunk/README.html.in
==============================================================================
--- wwwsearch/pullparser/trunk/README.html.in	(original)
+++ wwwsearch/pullparser/trunk/README.html.in	Sat Apr  8 20:27:21 2006
@@ -72,9 +72,8 @@
 
 <ul>
 @{version = "0.1.0"}
-@{win_version = release.win_version(version)}
 <li><a href="./src/pullparser-@(version).tar.gz">pullparser-@(version).tar.gz</a>
-<li><a href="./src/pullparser-@(win_version).zip">pullparser-@(win_version).zip</a>
+<li><a href="./src/pullparser-@(version).zip">pullparser-@(version).zip</a>
 <li><a href="./src/ChangeLog.txt">Change Log</a> (included in distribution)
 <li><a href="./src/">Older versions.</a>
 </ul>
From jjlee at codespeak.net  Sat Apr  8 20:29:15 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 20:29:16 2006
Subject: [wwwsearch-commits] r25585 -
	wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:42:31
Message-ID: <20060408182915.ABAA0100F9@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 20:29:14 2006
New Revision: 25585

Removed:
   wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:42:31/
Log:
Remove tag: not used for release
From jjlee at codespeak.net  Sat Apr  8 20:29:30 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 20:29:31 2006
Subject: [wwwsearch-commits] r25586 -
	wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:58:53
Message-ID: <20060408182930.278ED100F9@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 20:29:29 2006
New Revision: 25586

Added:
   wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:58:53/
      - copied from r25585, wwwsearch/pullparser/trunk/
Modified:
   wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:58:53/setup.cfg
Log:
Tagged pullparser (trunk -r HEAD, from working copy) release 0.1.0-2006-04-08T17:58:53

Modified: wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:58:53/setup.cfg
==============================================================================
--- wwwsearch/pullparser/trunk/setup.cfg	(original)
+++ wwwsearch/pullparser/tag/release/0.1.0-2006-04-08T17:58:53/setup.cfg	Sat Apr  8 20:29:29 2006
@@ -1,3 +0,0 @@
-[egg_info]
-tag_build = .dev
-tag_svn_revision = 1
From jjlee at codespeak.net  Sat Apr  8 21:26:11 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 21:26:13 2006
Subject: [wwwsearch-commits] r25587 - wwwsearch/ClientCookie/common
Message-ID: <20060408192611.BC23D10254@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 21:26:10 2006
New Revision: 25587

Modified:
   wwwsearch/ClientCookie/common/ChangeLog
Log:
Fix ChangeLog

Modified: wwwsearch/ClientCookie/common/ChangeLog
==============================================================================
--- wwwsearch/ClientCookie/common/ChangeLog	(original)
+++ wwwsearch/ClientCookie/common/ChangeLog	Sat Apr  8 21:26:10 2006
@@ -1,8 +1,8 @@
 This isn't really in proper GNU ChangeLog format, it just happens to
 look that way.
 
-2006-04-23 John J Lee <jjl@pobox.com>
-	* 1.2.1 release:
+2006-04-08 John J Lee <jjl@pobox.com>
+	* 1.3.0 release:
 	* Fix Refresh in case where the string "url" in the header value
 	  is in upper-case.
 	* Don't do broken XHTML handling by default (need to review code
From jjlee at codespeak.net  Sat Apr  8 21:26:55 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 21:26:57 2006
Subject: [wwwsearch-commits] r25588 - wwwsearch/ClientCookie/trunk
Message-ID: <20060408192655.D9AA410254@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 21:26:54 2006
New Revision: 25588

Modified:
   wwwsearch/ClientCookie/trunk/README.html.in
   wwwsearch/ClientCookie/trunk/setup.py
Log:
Update version

Modified: wwwsearch/ClientCookie/trunk/README.html.in
==============================================================================
--- wwwsearch/ClientCookie/trunk/README.html.in	(original)
+++ wwwsearch/ClientCookie/trunk/README.html.in	Sat Apr  8 21:26:54 2006
@@ -118,7 +118,7 @@
 <p><em>Stable release.</em>
 
 <ul>
-@{version = "1.2.1"}
+@{version = "1.3.0"}
 <li><a href="./src/ClientCookie-@(version).tar.gz">ClientCookie-@(version).tar.gz</a>
 <li><a href="./src/ClientCookie-@(version).zip">ClientCookie-@(version).zip</a>
 <li><a href="./src/ChangeLog.txt">Change Log</a> (included in distribution)

Modified: wwwsearch/ClientCookie/trunk/setup.py
==============================================================================
--- wwwsearch/ClientCookie/trunk/setup.py	(original)
+++ wwwsearch/ClientCookie/trunk/setup.py	Sat Apr  8 21:26:54 2006
@@ -19,11 +19,11 @@
 import re
 #VERSION_MATCH = re.search(r'VERSION = "(.*)"', open("ClientCookie/_ClientCookie.py").read())
 #VERSION = VERSION_MATCH.group(1)
-VERSION = '1.2.1'
+VERSION = '1.3.0'
 INSTALL_REQUIRES = []
 NAME = "ClientCookie"
 PACKAGE = True
-LICENSE = "BSD"
+LICENSE = "BSD"  # or ZPL 2.1
 PLATFORMS = ["any"]
 ZIP_SAFE = True
 CLASSIFIERS = """\
From jjlee at codespeak.net  Sat Apr  8 21:27:09 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 21:27:11 2006
Subject: [wwwsearch-commits] r25589 - wwwsearch/ClientCookie/trunk
Message-ID: <20060408192709.ADBDB10254@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 21:27:08 2006
New Revision: 25589

Modified:
   wwwsearch/ClientCookie/trunk/COPYING.txt
Log:
Update a copyright date

Modified: wwwsearch/ClientCookie/trunk/COPYING.txt
==============================================================================
--- wwwsearch/ClientCookie/trunk/COPYING.txt	(original)
+++ wwwsearch/ClientCookie/trunk/COPYING.txt	Sat Apr  8 21:27:08 2006
@@ -1,4 +1,4 @@
-Copyright (c) 2002-2004 John J. Lee <jjl@pobox.com>
+Copyright (c) 2002-2006 John J. Lee <jjl@pobox.com>
 Copyright (c) 1997-1999 Gisle Aas
 Copyright (c) 1997-1999 Johnny Lee
 
From jjlee at codespeak.net  Sat Apr  8 21:27:55 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 21:27:56 2006
Subject: [wwwsearch-commits] r25590 -
	wwwsearch/ClientCookie/tag/release/1.2.0-2006-03-23T23:59:28
Message-ID: <20060408192755.77F9D10254@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 21:27:54 2006
New Revision: 25590

Removed:
   wwwsearch/ClientCookie/tag/release/1.2.0-2006-03-23T23:59:28/
Log:
Removed unused release tag
From jjlee at codespeak.net  Sat Apr  8 21:28:09 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 21:28:10 2006
Subject: [wwwsearch-commits] r25591 -
	wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:27:31
Message-ID: <20060408192809.EFAF610254@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 21:28:09 2006
New Revision: 25591

Added:
   wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:27:31/
      - copied from r25590, wwwsearch/ClientCookie/trunk/
Modified:
   wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:27:31/setup.cfg
Log:
Tagged ClientCookie (trunk -r HEAD, from working copy) release 1.3.0-2006-04-08T19:27:31

Modified: wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:27:31/setup.cfg
==============================================================================
--- wwwsearch/ClientCookie/trunk/setup.cfg	(original)
+++ wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:27:31/setup.cfg	Sat Apr  8 21:28:09 2006
@@ -1,3 +0,0 @@
-[egg_info]
-tag_build = .dev
-tag_svn_revision = 1
From jjlee at codespeak.net  Sat Apr  8 21:29:35 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 21:29:36 2006
Subject: [wwwsearch-commits] r25592 -
	wwwsearch/ClientCookie/trunk/ClientCookie
Message-ID: <20060408192935.C620710254@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 21:29:35 2006
New Revision: 25592

Modified:
   wwwsearch/ClientCookie/trunk/ClientCookie/_ClientCookie.py
Log:
Update version

Modified: wwwsearch/ClientCookie/trunk/ClientCookie/_ClientCookie.py
==============================================================================
--- wwwsearch/ClientCookie/trunk/ClientCookie/_ClientCookie.py	(original)
+++ wwwsearch/ClientCookie/trunk/ClientCookie/_ClientCookie.py	Sat Apr  8 21:29:35 2006
@@ -32,7 +32,7 @@
 
 """
 
-VERSION = "1.2.1"
+VERSION = "1.3.0"
 
 
 # Public health warning: anyone who thought 'cookies are simple, aren't they?',
From jjlee at codespeak.net  Sat Apr  8 21:30:02 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 21:30:03 2006
Subject: [wwwsearch-commits] r25593 -
	wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:27:31
Message-ID: <20060408193002.86BAC1025D@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 21:30:01 2006
New Revision: 25593

Removed:
   wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:27:31/
Log:
Removed unused release tag
From jjlee at codespeak.net  Sat Apr  8 21:30:11 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 21:30:13 2006
Subject: [wwwsearch-commits] r25594 -
	wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:29:33
Message-ID: <20060408193011.B78BC1025D@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 21:30:10 2006
New Revision: 25594

Added:
   wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:29:33/
      - copied from r25593, wwwsearch/ClientCookie/trunk/
Modified:
   wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:29:33/setup.cfg
Log:
Tagged ClientCookie (trunk -r HEAD, from working copy) release 1.3.0-2006-04-08T19:29:33

Modified: wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:29:33/setup.cfg
==============================================================================
--- wwwsearch/ClientCookie/trunk/setup.cfg	(original)
+++ wwwsearch/ClientCookie/tag/release/1.3.0-2006-04-08T19:29:33/setup.cfg	Sat Apr  8 21:30:10 2006
@@ -1,3 +0,0 @@
-[egg_info]
-tag_build = .dev
-tag_svn_revision = 1
From jjlee at codespeak.net  Sat Apr  8 22:02:21 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 22:02:23 2006
Subject: [wwwsearch-commits] r25595 - wwwsearch/release_scripts
Message-ID: <20060408200221.999DF10225@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 22:02:20 2006
New Revision: 25595

Modified:
   wwwsearch/release_scripts/release.py
Log:
Build source distribitions with same python used to run release script

Modified: wwwsearch/release_scripts/release.py
==============================================================================
--- wwwsearch/release_scripts/release.py	(original)
+++ wwwsearch/release_scripts/release.py	Sat Apr  8 22:02:20 2006
@@ -167,7 +167,7 @@
         self._without_setup_cfg(
             self.source_dir,
             lambda :
-            system("python setup.py sdist --formats=gztar,zip",
+            system("%s setup.py sdist --formats=gztar,zip" % sys.executable,
                    self.pretend))
 
     def check_versions(self):
From jjlee at codespeak.net  Sat Apr  8 22:02:34 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sat Apr  8 22:02:35 2006
Subject: [wwwsearch-commits] r25596 - wwwsearch/release_scripts
Message-ID: <20060408200234.3877B10225@code0.codespeak.net>

Author: jjlee
Date: Sat Apr  8 22:02:33 2006
New Revision: 25596

Modified:
   wwwsearch/release_scripts/pprelease.py
Log:
Update pullparser release script to new scheme

Modified: wwwsearch/release_scripts/pprelease.py
==============================================================================
--- wwwsearch/release_scripts/pprelease.py	(original)
+++ wwwsearch/release_scripts/pprelease.py	Sat Apr  8 22:02:33 2006
@@ -1,67 +1,65 @@
 #!/usr/bin/env python
 
-"""
-pprelease.py version branch
-
-version is the version string, eg. "0.0.1a"
-branch is the svn branch, eg. "trunk", or "branch/stable"
-
-"""
-
-import sys, os
-from release import clean, check_versions, win_version, check_date, \
-     REPOSITORY_URL, maketag
+import sys, os, posixpath
+from release import REPOSITORY_URL, getargs
 
 NAME = "pullparser"
 
-def main():
-    if len(sys.argv) <= 1:
-        sys.exit("usage: %s version [branch]" % sys.argv[0])
-    version = sys.argv[1]
-    if len(sys.argv) > 2 and sys.argv[2] != "--tag":
-        branch = sys.argv[2]
-    else:
-        branch = "trunk"
-    if "--tag" in sys.argv:
-        maketag(version, REPOSITORY_URL, NAME, branch)
-        sys.exit()
-
-    os.system("svn co %s/%s/%s %s" % (REPOSITORY_URL, NAME, branch, NAME))
-
-    os.system("svn co %s/www" % REPOSITORY_URL)
-    os.system("svn co %s/%s/common" % (REPOSITORY_URL, NAME))
-    os.system("mv www/GeneralFAQ.html common/ChangeLog %s" % NAME)
-
-    release_dir = "%s-%s" % (NAME, version)
-    os.mkdir(release_dir)
-    os.chdir(NAME)
-    os.system("em.py README.html.in > README.html")
-    bad_files = check_versions(
-        ["%s.py" % NAME, "README.html"], version)
-    if bad_files:
-        sys.exit("version doesn't match in %s" % " ".join(bad_files))
-    os.system("lynx -dump README.html > README.txt")
-    os.system("cp README.html ../%s/README-%s.html" % (
-        release_dir, win_version(version)))
-    os.system("cp ChangeLog ../%s/ChangeLog.txt" % release_dir)
-    os.system("rm MANIFEST")
-    os.system("python setup.py sdist --formats=gztar,zip")
-    os.chdir("dist")
-    distutils_zip = "%s-%s.zip" % (NAME, version)
-    zip = "%s-%s.zip" % (NAME, win_version(version))
-    tarball = "%s-%s.tar.gz" % (NAME, version)
-    os.system("mv %s ../../%s/%s" % (distutils_zip, release_dir, zip))
-    os.system("mv %s ../../%s" % (tarball, release_dir))
-    os.chdir("..")
-    os.system("rm -rf dist")
-    os.system("rm -rf build")
-    os.chdir("..")
-    print "tarring:"
-    os.system("tar -cvf %s.tar %s" % (release_dir, release_dir))
-    os.system("rm -rf %s" % release_dir)
-
-    os.chdir(NAME)
-    for py in ("python2.4", "python2.3", "python2.2"):
-        os.system("%s test.py" % py)
+def main(argv):
+    import release
+    tag, rel, upload, version, branch, revision, options = getargs(argv)
+
+    release.display_log_messages()
+
+    pretend = options.pretend
+
+    if branch == "tagged":
+        last_tag = release.gettags(NAME)[-1]
+        project_svn_base = posixpath.join(REPOSITORY_URL, NAME)
+        branch = release.relative_location(project_svn_base, last_tag)
+        print "using tagged branch", branch
+
+    build_dir = os.path.abspath("build")
+    www = release.create_svn_www_wc_instance(
+        build_dir,
+        pretend=pretend)
+    common = release.create_svn_common_wc_instance(
+        build_dir, NAME,
+        pretend=pretend)
+    src = release.create_svn_proj_wc_instance(
+        build_dir, NAME, branch, revision,
+        pretend=pretend)
+
+    sdist = release.SDist(NAME, src.fs_path, version, pretend=pretend)
+
+    sdist.test_pythons = [(2,4), (2,3), (2,2)]
+    sdist.add_svn_working_copies([www, common, src])
+    def make_readmes(fn, pretend=False):
+        readme_html = release.empy(fn, pretend=pretend)
+        readme_txt = release.lynx_dump(readme_html, pretend=pretend)
+        return [readme_html, readme_txt]
+    sdist.add_file_builders([
+        lambda pretend: make_readmes(src.filename("README.html.in"),
+                                     pretend=pretend),
+        lambda pretend: [release.empy(www.filename("GeneralFAQ.html.in"),
+                                      pretend=pretend)],
+        ])
+    sdist.add_files([
+        (common.filename("ChangeLog"), src.filename("ChangeLog.txt")),
+        ])
+    sdist.add_versioned_files([
+        NAME+".py", "ChangeLog.txt", "README.html", "setup.py"])
+    sdist.add_uploadable_files([
+        (src.filename("ChangeLog.txt"), None),
+        (src.filename("README.html"), ("README-%s.html" % version)),
+        ])
+
+    if tag:
+        sdist.tag(src, clean=options.clean)
+    if rel:
+        sdist.build(update=options.update, clean=options.clean)
+    if upload:
+        sdist.upload_to_sourceforge()
 
-main()
+if __name__ == "__main__":
+    main(sys.argv)
From jjlee at codespeak.net  Thu Apr 13 23:31:01 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Thu Apr 13 23:31:03 2006
Subject: [wwwsearch-commits] r25820 - wwwsearch/mechanize/trunk
Message-ID: <20060413213101.CADA710081@code0.codespeak.net>

Author: jjlee
Date: Thu Apr 13 23:31:00 2006
New Revision: 25820

Modified:
   wwwsearch/mechanize/trunk/README.html.in
Log:
Fix docs re debug output

Modified: wwwsearch/mechanize/trunk/README.html.in
==============================================================================
--- wwwsearch/mechanize/trunk/README.html.in	(original)
+++ wwwsearch/mechanize/trunk/README.html.in	Thu Apr 13 23:31:00 2006
@@ -118,12 +118,20 @@
 # default: no need to do this unless you have some reason to use a
 # particular cookiejar)
 br.set_cookiejar(cj)
-# Print information about HTTP redirects and Refreshes.
+# Log information about HTTP redirects and Refreshes.
 br.set_debug_redirects(True)
-# Print HTTP response bodies (ie. the HTML, most of the time).
+# Log HTTP response bodies (ie. the HTML, most of the time).
 br.set_debug_responses(True)
 # Print HTTP headers.
 br.set_debug_http(True)
+
+# To make sure you're seeing all debug output:
+for logger in [
+    logging.getLogger("ClientCookie"),
+    logging.getLogger("cookielib"),
+    ]:
+    logger.addHandler(logging.StreamHandler(sys.stdout))
+    logger.setLevel(logging.INFO)
 """)}
 
 
From jjlee at codespeak.net  Fri Apr 21 00:02:26 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Fri Apr 21 00:02:28 2006
Subject: [wwwsearch-commits] r26069 - in wwwsearch/mechanize/trunk: .
	mechanize
Message-ID: <20060420220226.A0EFA1007D@code0.codespeak.net>

Author: jjlee
Date: Fri Apr 21 00:02:25 2006
New Revision: 26069

Added:
   wwwsearch/mechanize/trunk/mechanize/_html.py
Modified:
   wwwsearch/mechanize/trunk/mechanize/__init__.py
   wwwsearch/mechanize/trunk/mechanize/_mechanize.py
   wwwsearch/mechanize/trunk/test.py
Log:
Shift HTML stuff into new module

Modified: wwwsearch/mechanize/trunk/mechanize/__init__.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/__init__.py	(original)
+++ wwwsearch/mechanize/trunk/mechanize/__init__.py	Fri Apr 21 00:02:25 2006
@@ -1,7 +1,9 @@
 from _useragent import UserAgent
-from _mechanize import Browser, Link, \
-     Factory, DefaultFactory, RobustFactory, \
-     FormsFactory, LinksFactory, pp_get_title, \
-     RobustFormsFactory, RobustLinksFactory, bs_get_title, \
+from _mechanize import Browser, \
      BrowserStateError, LinkNotFoundError, FormNotFoundError, \
      __version__
+
+from _html import Link, \
+     Factory, DefaultFactory, RobustFactory, \
+     FormsFactory, LinksFactory, pp_get_title, \
+     RobustFormsFactory, RobustLinksFactory, bs_get_title

Added: wwwsearch/mechanize/trunk/mechanize/_html.py
==============================================================================
--- (empty file)
+++ wwwsearch/mechanize/trunk/mechanize/_html.py	Fri Apr 21 00:02:25 2006
@@ -0,0 +1,400 @@
+from __future__ import generators
+
+import re, urllib, htmlentitydefs
+from urlparse import urljoin
+
+import ClientCookie
+
+## # XXXX miserable hack
+## def urljoin(base, url):
+##     if url.startswith("?"):
+##         return base+url
+##     else:
+##         return urlparse.urljoin(base, url)
+
+## def chr_range(a, b):
+##     return "".join(map(chr, range(ord(a), ord(b)+1)))
+
+## RESERVED_URL_CHARS = ("ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+##                       "abcdefghijklmnopqrstuvwxyz"
+##                       "-_.~")
+## UNRESERVED_URL_CHARS = "!*'();:@&=+$,/?%#[]"
+# we want (RESERVED_URL_CHARS+UNRESERVED_URL_CHARS), minus those
+# 'safe'-by-default characters that urllib.urlquote never quotes
+URLQUOTE_SAFE_URL_CHARS = "!*'();:@&=+$,/?%#[]~"
+
+# idea for this argument-processing trick is from Peter Otten
+class Args:
+    def __init__(self, args_map):
+        self.dictionary = dict(args_map)
+    def __getattr__(self, key):
+        try:
+            return self.dictionary[key]
+        except KeyError:
+            return getattr(self.__class__, key)
+
+def form_parser_args(
+    select_default=False,
+    form_parser_class=None,
+    request_class=None,
+    backwards_compat=False,
+    encoding="latin-1",  # deprecated
+    ):
+    return Args(locals())
+
+
+class Link:
+    def __init__(self, base_url, url, text, tag, attrs):
+        assert None not in [url, tag, attrs]
+        self.base_url = base_url
+        self.absolute_url = urljoin(base_url, url)
+        self.url, self.text, self.tag, self.attrs = url, text, tag, attrs
+    def __cmp__(self, other):
+        try:
+            for name in "url", "text", "tag", "attrs":
+                if getattr(self, name) != getattr(other, name):
+                    return -1
+        except AttributeError:
+            return -1
+        return 0
+    def __repr__(self):
+        return "Link(base_url=%r, url=%r, text=%r, tag=%r, attrs=%r)" % (
+            self.base_url, self.url, self.text, self.tag, self.attrs)
+
+
+def clean_url(url, encoding):
+    # percent-encode illegal URL characters
+    if type(url) == type(""):
+        url = url.decode(encoding, "replace")
+    return urllib.quote(url.encode(encoding), URLQUOTE_SAFE_URL_CHARS)
+
+class LinksFactory:
+
+    def __init__(self,
+                 link_parser_class=None,
+                 link_class=Link,
+                 urltags=None,
+                 ):
+        import pullparser
+        if link_parser_class is None:
+            link_parser_class = pullparser.TolerantPullParser
+        self.link_parser_class = link_parser_class
+        self.link_class = link_class
+        if urltags is None:
+            urltags = {
+                "a": "href",
+                "area": "href",
+                "frame": "src",
+                "iframe": "src",
+                }
+        self.urltags = urltags
+
+    def links(self, fh, base_url, encoding=None):
+        """Return an iterator that provides links of the document."""
+        import pullparser
+        p = self.link_parser_class(fh, encoding=encoding)
+
+        for token in p.tags(*(self.urltags.keys()+["base"])):
+            if token.data == "base":
+                base_url = dict(token.attrs).get("href")
+                continue
+            if token.type == "endtag":
+                continue
+            attrs = dict(token.attrs)
+            tag = token.data
+            name = attrs.get("name")
+            text = None
+            # XXX use attr_encoding for ref'd doc if that doc does not provide
+            #  one by other means
+            #attr_encoding = attrs.get("charset")
+            url = attrs.get(self.urltags[tag])  # XXX is "" a valid URL?
+            if not url:
+                # Probably an <A NAME="blah"> link or <AREA NOHREF...>.
+                # For our purposes a link is something with a URL, so ignore
+                # this.
+                continue
+
+            url = clean_url(url, encoding)
+            if tag == "a":
+                if token.type != "startendtag":
+                    # hmm, this'd break if end tag is missing
+                    text = p.get_compressed_text(("endtag", tag))
+                # but this doesn't work for eg. <a href="blah"><b>Andy</b></a>
+                #text = p.get_compressed_text()
+
+            yield Link(base_url, url, text, tag, token.attrs)
+
+class FormsFactory:
+
+    """Makes a sequence of objects satisfying ClientForm.HTMLForm interface.
+
+    For constructor argument docs, see ClientForm.ParseResponse
+    argument docs.
+
+    """
+
+    def __init__(self,
+                 select_default=False,
+                 form_parser_class=None,
+                 request_class=None,
+                 backwards_compat=False,
+                 encoding="latin-1",  # deprecated
+                 ):
+        import ClientForm
+        self.select_default = select_default
+        if form_parser_class is None:
+            form_parser_class = ClientForm.FormParser
+        self.form_parser_class = form_parser_class
+        if request_class is None:
+            request_class = ClientCookie.Request
+        self.request_class = request_class
+        self.backwards_compat = backwards_compat
+        self.encoding = encoding
+
+    def parse_response(self, response, encoding=None):
+        import ClientForm
+        if encoding is None:
+            encoding = self.encoding
+        return ClientForm.ParseResponse(
+            response,
+            select_default=self.select_default,
+            form_parser_class=self.form_parser_class,
+            request_class=self.request_class,
+            backwards_compat=self.backwards_compat,
+            encoding=encoding,
+            )
+
+    def parse_file(self, file_obj, base_url, encoding=None):
+        import ClientForm
+        if encoding is None:
+            encoding = self.encoding
+        return ClientForm.ParseFile(
+            file_obj,
+            base_url,
+            select_default=self.select_default,
+            form_parser_class=self.form_parser_class,
+            request_class=self.request_class,
+            backwards_compat=self.backwards_compat,
+            encoding=encoding,
+            )
+
+def pp_get_title(response, encoding):
+    import pullparser
+    p = pullparser.TolerantPullParser(response, encoding=encoding)
+    try:
+        p.get_tag("title")
+    except pullparser.NoMoreTokensError:
+        return None
+    else:
+        return p.get_text()
+
+
+def unescape(data, entities, encoding):
+    if data is None or "&" not in data:
+        return data
+
+    def replace_entities(match):
+        ent = match.group()
+        if ent[1] == "#":
+            return unescape_charref(ent[2:-1], encoding)
+
+        repl = entities.get(ent[1:-1])
+        if repl is not None:
+            repl = unichr(repl)
+            if type(repl) != type(""):
+                try:
+                    repl = repl.encode(encoding)
+                except UnicodeError:
+                    repl = ent
+        else:
+            repl = ent
+        return repl
+
+    return re.sub(r"&#?[A-Za-z0-9]+?;", replace_entities, data)
+
+def unescape_charref(data, encoding):
+    name, base = data, 10
+    if name.startswith("x"):
+        name, base= name[1:], 16
+    uc = unichr(int(name, base))
+    if encoding is None:
+        return uc
+    else:
+        try:
+            repl = uc.encode(encoding)
+        except UnicodeError:
+            repl = "&#%s;" % data
+        return repl
+
+def get_entitydefs():
+    try:
+        htmlentitydefs.name2codepoint
+    except AttributeError:
+        entitydefs = {}
+        for name, char in htmlentitydefs.entitydefs.items():
+            uc = char.decode("latin-1")
+            if uc.startswith("&#") and uc.endswith(";"):
+                uc = unescape_charref(uc[2:-1], None)
+            codepoint = ord(uc)
+            entitydefs[name] = codepoint
+    else:
+        entitydefs = htmlentitydefs.name2codepoint
+    return entitydefs
+
+
+try:
+    import BeautifulSoup
+except ImportError:
+    pass
+else:
+    import sgmllib
+    # monkeypatch to fix http://www.python.org/sf/803422 :-(
+    sgmllib.charref = re.compile("&#(x?[0-9a-fA-F]+)[^0-9a-fA-F]")
+    class MechanizeBs(BeautifulSoup.BeautifulSoup):
+        _entitydefs = get_entitydefs()
+        def __init__(self, encoding, text=None, avoidParserProblems=True,
+                     initialTextIsEverything=True):
+            self._encoding = encoding
+            BeautifulSoup.BeautifulSoup.__init__(
+                self, text, avoidParserProblems, initialTextIsEverything)
+
+        def handle_charref(self, ref):
+            t = unescape("&#%s;"%ref, self._entitydefs, self._encoding)
+            self.handle_data(t)
+        def handle_entityref(self, ref):
+            t = unescape("&%s;"%ref, self._entitydefs, self._encoding)
+            self.handle_data(t)
+        def unescape_attrs(self, attrs):
+            escaped_attrs = []
+            for key, val in attrs:
+                val = unescape(val, self._entitydefs, self._encoding)
+                escaped_attrs.append((key, val))
+            return escaped_attrs
+
+class RobustLinksFactory:
+
+    compress_re = re.compile(r"\s+")
+
+    def __init__(self,
+                 link_parser_class=None,
+                 link_class=Link,
+                 urltags=None,
+                 ):
+        import BeautifulSoup
+        if link_parser_class is None:
+            link_parser_class = MechanizeBs
+        self.link_parser_class = link_parser_class
+        self.link_class = link_class
+        if urltags is None:
+            urltags = {
+                "a": "href",
+                "area": "href",
+                "frame": "src",
+                "iframe": "src",
+                }
+        self.urltags = urltags
+
+    def links(self, fh, base_url, encoding=None):
+        import BeautifulSoup
+        data = fh.read()
+        bs = self.link_parser_class(encoding, data)
+        gen = bs.recursiveChildGenerator()
+        for ch in bs.recursiveChildGenerator():
+            if (isinstance(ch, BeautifulSoup.Tag) and
+                ch.name in self.urltags.keys()+["base"]):
+                link = ch
+                attrs = bs.unescape_attrs(link.attrs)
+                attrs_dict = dict(attrs)
+                if link.name == "base":
+                    base_url = attrs_dict.get("href")
+                    continue
+                url_attr = self.urltags[link.name]
+                url = attrs_dict.get(url_attr)
+                if not url:
+                    continue
+                url = clean_url(url, encoding)
+                text = link.firstText(lambda t: True)
+                if text is BeautifulSoup.Null:
+                    # follow pullparser's weird behaviour rigidly
+                    if link.name == "a":
+                        text = ""
+                    else:
+                        text = None
+                else:
+                    text = self.compress_re.sub(" ", text.strip())
+                yield Link(base_url, url, text, link.name, attrs)
+
+
+class RobustFormsFactory(FormsFactory):
+    def __init__(self, *args, **kwds):
+        import ClientForm
+        args = form_parser_args(*args, **kwds)
+        if args.form_parser_class is None:
+            args.form_parser_class = ClientForm.RobustFormParser
+        FormsFactory.__init__(self, **args.dictionary)
+
+def bs_get_title(response, encoding):
+    import BeautifulSoup
+    # XXXX encoding
+    bs = BeautifulSoup.BeautifulSoup(response.read())
+    title = bs.first("title")
+    if title == BeautifulSoup.Null:
+        return None
+    else:
+        return title.firstText(lambda t: True)
+
+
+class Factory:
+    """Factory for forms, links, etc.
+
+    The interface of this class may expand in future.
+
+    """
+
+    def __init__(self, forms_factory, links_factory, get_title):
+        """
+
+        Pass keyword
+        arguments only.
+
+        """
+        self._forms_factory = forms_factory
+        self._links_factory = links_factory
+        self._get_title = get_title
+
+    def set_request_class(self, request_class):
+        """Set urllib2.Request class.
+
+        ClientForm.HTMLForm instances returned by .forms() will return
+        instances of this class when .click()ed.
+
+        """
+        self._forms_factory.request_class = request_class
+
+    def forms(self, response, encoding):
+        """Return iterable over ClientForm.HTMLForm-like objects."""
+        return self._forms_factory.parse_response(response, encoding)
+
+    def links(self, response, encoding):
+        """Return iterable over mechanize.Link-like objects."""
+        return self._links_factory.links(response, response.geturl(), encoding)
+
+    def title(self, response, encoding):
+        """Return page title."""
+        return self._get_title(response, encoding)
+
+class DefaultFactory(Factory):
+    def __init__(self):
+        Factory.__init__(self,
+                         forms_factory=FormsFactory(),
+                         links_factory=LinksFactory(),
+                         get_title=pp_get_title,
+                         )
+
+class RobustFactory(Factory):
+    def __init__(self):
+        Factory.__init__(self,
+                         forms_factory=RobustFormsFactory(),
+                         links_factory=RobustLinksFactory(),
+                         get_title=bs_get_title,
+                         )

Modified: wwwsearch/mechanize/trunk/mechanize/_mechanize.py
==============================================================================
--- wwwsearch/mechanize/trunk/mechanize/_mechanize.py	(original)
+++ wwwsearch/mechanize/trunk/mechanize/_mechanize.py	Fri Apr 21 00:02:25 2006
@@ -16,16 +16,14 @@
 # The stuff on web page's todo list.
 # Moof's emails about response object, .back(), etc.
 
-from __future__ import generators
-
-import urllib2, socket, urlparse, urllib, re, sys, htmlentitydefs, copy
-from urlparse import urljoin
+import urllib2, urlparse, sys, copy
 
 import ClientCookie
 from ClientCookie._Util import response_seek_wrapper
 from ClientCookie._HeadersUtil import split_header_words, is_html
 
 from _useragent import UserAgent
+from _html import DefaultFactory
 
 __version__ = (0, 0, 12, "a", None)  # 0.0.12a
 
@@ -33,400 +31,6 @@
 class LinkNotFoundError(Exception): pass
 class FormNotFoundError(Exception): pass
 
-## def chr_range(a, b):
-##     return "".join(map(chr, range(ord(a), ord(b)+1)))
-
-## RESERVED_URL_CHARS = ("ABCDEFGHIJKLMNOPQRSTUVWXYZ"
-##                       "abcdefghijklmnopqrstuvwxyz"
-##                       "-_.~")
-## UNRESERVED_URL_CHARS = "!*'();:@&=+$,/?%#[]"
-# we want (RESERVED_URL_CHARS+UNRESERVED_URL_CHARS), minus those
-# 'safe'-by-default characters that urllib.urlquote never quotes
-URLQUOTE_SAFE_URL_CHARS = "!*'();:@&=+$,/?%#[]~"
-
-## # XXXX miserable hack
-## def urljoin(base, url):
-##     if url.startswith("?"):
-##         return base+url
-##     else:
-##         return urlparse.urljoin(base, url)
-
-# idea for this argument-processing trick is from Peter Otten
-class Args:
-    def __init__(self, args_map):
-        self.dictionary = dict(args_map)
-    def __getattr__(self, key):
-        try:
-            return self.dictionary[key]
-        except KeyError:
-            return getattr(self.__class__, key)
-
-def form_parser_args(
-    select_default=False,
-    form_parser_class=None,
-    request_class=None,
-    backwards_compat=False,
-    encoding="latin-1",  # deprecated
-    ):
-    return Args(locals())
-
-
-class Link:
-    def __init__(self, base_url, url, text, tag, attrs):
-        assert None not in [url, tag, attrs]
-        self.base_url = base_url
-        self.absolute_url = urljoin(base_url, url)
-        self.url, self.text, self.tag, self.attrs = url, text, tag, attrs
-    def __cmp__(self, other):
-        try:
-            for name in "url", "text", "tag", "attrs":
-                if getattr(self, name) != getattr(other, name):
-                    return -1
-        except AttributeError:
-            return -1
-        return 0
-    def __repr__(self):
-        return "Link(base_url=%r, url=%r, text=%r, tag=%r, attrs=%r)" % (
-            self.base_url, self.url, self.text, self.tag, self.attrs)
-
-
-def clean_url(url, encoding):
-    # percent-encode illegal URL characters
-    if type(url) == type(""):
-        url = url.decode(encoding, "replace")
-    return urllib.quote(url.encode(encoding), URLQUOTE_SAFE_URL_CHARS)
-
-class LinksFactory:
-
-    def __init__(self,
-                 link_parser_class=None,
-                 link_class=Link,
-                 urltags=None,
-                 ):
-        import pullparser
-        if link_parser_class is None:
-            link_parser_class = pullparser.TolerantPullParser
-        self.link_parser_class = link_parser_class
-        self.link_class = link_class
-        if urltags is None:
-            urltags = {
-                "a": "href",
-                "area": "href",
-                "frame": "src",
-                "iframe": "src",
-                }
-        self.urltags = urltags
-
-    def links(self, fh, base_url, encoding=None):
-        """Return an iterator that provides links of the document."""
-        import pullparser
-        p = self.link_parser_class(fh, encoding=encoding)
-
-        for token in p.tags(*(self.urltags.keys()+["base"])):
-            if token.data == "base":
-                base_url = dict(token.attrs).get("href")
-                continue
-            if token.type == "endtag":
-                continue
-            attrs = dict(token.attrs)
-            tag = token.data
-            name = attrs.get("name")
-            text = None
-            # XXX use attr_encoding for ref'd doc if that doc does not provide
-            #  one by other means
-            #attr_encoding = attrs.get("charset")
-            url = attrs.get(self.urltags[tag])  # XXX is "" a valid URL?
-            if not url:
-                # Probably an <A NAME="blah"> link or <AREA NOHREF...>.
-                # For our purposes a link is something with a URL, so ignore
-                # this.
-                continue
-
-            url = clean_url(url, encoding)
-            if tag == "a":
-                if token.type != "startendtag":
-                    # hmm, this'd break if end tag is missing
-                    text = p.get_compressed_text(("endtag", tag))
-                # but this doesn't work for eg. <a href="blah"><b>Andy</b></a>
-                #text = p.get_compressed_text()
-
-            yield Link(base_url, url, text, tag, token.attrs)
-
-class FormsFactory:
-
-    """Makes a sequence of objects satisfying ClientForm.HTMLForm interface.
-
-    For constructor argument docs, see ClientForm.ParseResponse
-    argument docs.
-
-    """
-
-    def __init__(self,
-                 select_default=False,
-                 form_parser_class=None,
-                 request_class=None,
-                 backwards_compat=False,
-                 encoding="latin-1",  # deprecated
-                 ):
-        import ClientForm
-        self.select_default = select_default
-        if form_parser_class is None:
-            form_parser_class = ClientForm.FormParser
-        self.form_parser_class = form_parser_class
-        if request_class is None:
-            request_class = ClientCookie.Request
-        self.request_class = request_class
-        self.backwards_compat = backwards_compat
-        self.encoding = encoding
-
-    def parse_response(self, response, encoding=None):
-        import ClientForm
-        if encoding is None:
-            encoding = self.encoding
-        return ClientForm.ParseResponse(
-            response,
-            select_default=self.select_default,
-            form_parser_class=self.form_parser_class,
-            request_class=self.request_class,
-            backwards_compat=self.backwards_compat,
-            encoding=encoding,
-            )
-
-    def parse_file(self, file_obj, base_url, encoding=None):
-        import ClientForm
-        if encoding is None:
-            encoding = self.encoding
-        return ClientForm.ParseFile(
-            file_obj,
-            base_url,
-            select_default=self.select_default,
-            form_parser_class=self.form_parser_class,
-            request_class=self.request_class,
-            backwards_compat=self.backwards_compat,
-            encoding=encoding,
-            )
-
-def pp_get_title(response, encoding):
-    import pullparser
-    p = pullparser.TolerantPullParser(response, encoding=encoding)
-    try:
-        p.get_tag("title")
-    except pullparser.NoMoreTokensError:
-        return None
-    else:
-        return p.get_text()
-
-
-def unescape(data, entities, encoding):
-    if data is None or "&" not in data:
-        return data
-
-    def replace_entities(match):
-        ent = match.group()
-        if ent[1] == "#":
-            return unescape_charref(ent[2:-1], encoding)
-
-        repl = entities.get(ent[1:-1])
-        if repl is not None:
-            repl = unichr(repl)
-            if type(repl) != type(""):
-                try:
-                    repl = repl.encode(encoding)
-                except UnicodeError:
-                    repl = ent
-        else:
-            repl = ent
-        return repl
-
-    return re.sub(r"&#?[A-Za-z0-9]+?;", replace_entities, data)
-
-def unescape_charref(data, encoding):
-    name, base = data, 10
-    if name.startswith("x"):
-        name, base= name[1:], 16
-    uc = unichr(int(name, base))
-    if encoding is None:
-        return uc
-    else:
-        try:
-            repl = uc.encode(encoding)
-        except UnicodeError:
-            repl = "&#%s;" % data
-        return repl
-
-def get_entitydefs():
-    try:
-        htmlentitydefs.name2codepoint
-    except AttributeError:
-        entitydefs = {}
-        for name, char in htmlentitydefs.entitydefs.items():
-            uc = char.decode("latin-1")
-            if uc.startswith("&#") and uc.endswith(";"):
-                uc = unescape_charref(uc[2:-1], None)
-            codepoint = ord(uc)
-            entitydefs[name] = codepoint
-    else:
-        entitydefs = htmlentitydefs.name2codepoint
-    return entitydefs
-
-
-try:
-    import BeautifulSoup
-except ImportError:
-    pass
-else:
-    import sgmllib
-    # monkeypatch to fix http://www.python.org/sf/803422 :-(
-    sgmllib.charref = re.compile("&#(x?[0-9a-fA-F]+)[^0-9a-fA-F]")
-    class MechanizeBs(BeautifulSoup.BeautifulSoup):
-        _entitydefs = get_entitydefs()
-        def __init__(self, encoding, text=None, avoidParserProblems=True,
-                     initialTextIsEverything=True):
-            self._encoding = encoding
-            BeautifulSoup.BeautifulSoup.__init__(
-                self, text, avoidParserProblems, initialTextIsEverything)
-
-        def handle_charref(self, ref):
-            t = unescape("&#%s;"%ref, self._entitydefs, self._encoding)
-            self.handle_data(t)
-        def handle_entityref(self, ref):
-            t = unescape("&%s;"%ref, self._entitydefs, self._encoding)
-            self.handle_data(t)
-        def unescape_attrs(self, attrs):
-            escaped_attrs = []
-            for key, val in attrs:
-                val = unescape(val, self._entitydefs, self._encoding)
-                escaped_attrs.append((key, val))
-            return escaped_attrs
-
-class RobustLinksFactory:
-
-    compress_re = re.compile(r"\s+")
-
-    def __init__(self,
-                 link_parser_class=None,
-                 link_class=Link,
-                 urltags=None,
-                 ):
-        import BeautifulSoup
-        if link_parser_class is None:
-            link_parser_class = MechanizeBs
-        self.link_parser_class = link_parser_class
-        self.link_class = link_class
-        if urltags is None:
-            urltags = {
-                "a": "href",
-                "area": "href",
-                "frame": "src",
-                "iframe": "src",
-                }
-        self.urltags = urltags
-
-    def links(self, fh, base_url, encoding=None):
-        import BeautifulSoup
-        data = fh.read()
-        bs = self.link_parser_class(encoding, data)
-        gen = bs.recursiveChildGenerator()
-        for ch in bs.recursiveChildGenerator():
-            if (isinstance(ch, BeautifulSoup.Tag) and
-                ch.name in self.urltags.keys()+["base"]):
-                link = ch
-                attrs = bs.unescape_attrs(link.attrs)
-                attrs_dict = dict(attrs)
-                if link.name == "base":
-                    base_url = attrs_dict.get("href")
-                    continue
-                url_attr = self.urltags[link.name]
-                url = attrs_dict.get(url_attr)
-                if not url:
-                    continue
-                url = clean_url(url, encoding)
-                text = link.firstText(lambda t: True)
-                if text is BeautifulSoup.Null:
-                    # follow pullparser's weird behaviour rigidly
-                    if link.name == "a":
-                        text = ""
-                    else:
-                        text = None
-                else:
-                    text = self.compress_re.sub(" ", text.strip())
-                yield Link(base_url, url, text, link.name, attrs)
-
-
-class RobustFormsFactory(FormsFactory):
-    def __init__(self, *args, **kwds):
-        import ClientForm
-        args = form_parser_args(*args, **kwds)
-        if args.form_parser_class is None:
-            args.form_parser_class = ClientForm.RobustFormParser
-        FormsFactory.__init__(self, **args.dictionary)
-
-def bs_get_title(response, encoding):
-    import BeautifulSoup
-    # XXXX encoding
-    bs = BeautifulSoup.BeautifulSoup(response.read())
-    title = bs.first("title")
-    if title == BeautifulSoup.Null:
-        return None
-    else:
-        return title.firstText(lambda t: True)
-
-
-class Factory:
-    """Factory for forms, links, etc.
-
-    The interface of this class may expand in future.
-
-    """
-
-    def __init__(self, forms_factory, links_factory, get_title):
-        """
-
-        Pass keyword
-        arguments only.
-
-        """
-        self._forms_factory = forms_factory
-        self._links_factory = links_factory
-        self._get_title = get_title
-
-    def set_request_class(self, request_class):
-        """Set urllib2.Request class.
-
-        ClientForm.HTMLForm instances returned by .forms() will return
-        instances of this class when .click()ed.
-
-        """
-        self._forms_factory.request_class = request_class
-
-    def forms(self, response, encoding):
-        """Return iterable over ClientForm.HTMLForm-like objects."""
-        return self._forms_factory.parse_response(response, encoding)
-
-    def links(self, response, encoding):
-        """Return iterable over mechanize.Link-like objects."""
-        return self._links_factory.links(response, response.geturl(), encoding)
-
-    def title(self, response, encoding):
-        """Return page title."""
-        return self._get_title(response, encoding)
-
-class DefaultFactory(Factory):
-    def __init__(self):
-        Factory.__init__(self,
-                         forms_factory=FormsFactory(),
-                         links_factory=LinksFactory(),
-                         get_title=pp_get_title,
-                         )
-
-class RobustFactory(Factory):
-    def __init__(self):
-        Factory.__init__(self,
-                         forms_factory=RobustFormsFactory(),
-                         links_factory=RobustLinksFactory(),
-                         get_title=bs_get_title,
-                         )
-
 
 class History:
     """

Modified: wwwsearch/mechanize/trunk/test.py
==============================================================================
--- wwwsearch/mechanize/trunk/test.py	(original)
+++ wwwsearch/mechanize/trunk/test.py	Fri Apr 21 00:02:25 2006
@@ -20,7 +20,7 @@
 class UnescapeTests(TestCase):
 
     def test_unescape_charref(self):
-        from mechanize._mechanize import unescape_charref, get_entitydefs
+        from mechanize._html import unescape_charref, get_entitydefs
         mdash_utf8 = u"\u2014".encode("utf-8")
         for ref, codepoint, utf8, latin1 in [
             ("38", 38, u"&".encode("utf-8"), "&"),
@@ -32,7 +32,7 @@
             self.assertEqual(unescape_charref(ref, 'utf-8'), utf8)
 
     def test_get_entitydefs(self):
-        from mechanize._mechanize import get_entitydefs
+        from mechanize._html import get_entitydefs
         ed = get_entitydefs()
         for name, codepoint in [
             ("amp", ord(u"&")),
@@ -45,7 +45,7 @@
 
     def test_unescape(self):
         import htmlentitydefs
-        from mechanize._mechanize import unescape, get_entitydefs
+        from mechanize._html import unescape, get_entitydefs
         data = "&amp; &lt; &mdash; &#8212; &#x2014;"
         mdash_utf8 = u"\u2014".encode("utf-8")
         ue = unescape(data, get_entitydefs(), "utf-8")
@@ -407,7 +407,7 @@
     def _test_link_encoding(self, factory):
         import urllib
         import mechanize
-        from mechanize._mechanize import clean_url
+        from mechanize._html import clean_url
         url = "http://example.com/"
         for encoding in ["UTF-8", "latin-1"]:
             encoding_decl = "; charset=%s" % encoding
From jjlee at codespeak.net  Sun Apr 23 18:18:22 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sun Apr 23 18:18:23 2006
Subject: [wwwsearch-commits] r26195 -
	wwwsearch/mechanize/branch/mechanize-0.1.0-devel
Message-ID: <20060423161822.104F31009D@code0.codespeak.net>

Author: jjlee
Date: Sun Apr 23 18:18:21 2006
New Revision: 26195

Added:
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/
      - copied from r26194, wwwsearch/mechanize/trunk/
Log:
Branch to make backwards-incompatible changes, heading towards beta / final release
From jjlee at codespeak.net  Sun Apr 23 19:11:45 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sun Apr 23 19:11:46 2006
Subject: [wwwsearch-commits] r26198 - in
	wwwsearch/mechanize/branch/mechanize-0.1.0-devel: . mechanize
Message-ID: <20060423171145.B4E5B1009D@code0.codespeak.net>

Author: jjlee
Date: Sun Apr 23 19:11:43 2006
New Revision: 26198

Modified:
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/README.html.in
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_html.py
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_mechanize.py
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py
Log:
Interface changes and cleanup for beta release (not finished yet: want to use single BeautifulSoup instance for RobustFactory): Changed Factory interface to make it easier to avoid re-parsing (principally: add .set_response() method and make factory methods take no args); Moved some c'tor args from Browser to Factory; Browser.encoding method (which was new anyway) now takes no args; Simplify links code and remove .get_links_iter(); .forms() and .links() now both return iterators (in fact, generators), not sequences (not really an interface change: these were always documented to return iterables)

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/README.html.in
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/README.html.in	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/README.html.in	Sun Apr 23 19:11:43 2006
@@ -159,16 +159,14 @@
 
 <h3>Specific to mechanize</h3>
 <ul>
-  <li>Apply Titus' patch to move stuff into separate file and change
-    Factory interface.
-  <li>Kill off <code>.get_links_iter()</code>.
+  <li>Make encoding_finder public, I guess.
+  <li>Test BeautifulSoup support better / fix encoding issue.
   <li>Support Mark Pilgrim's universal encoding detector?
   <li>Add another History implementation or two and finalise interface.
   <li>History cache expiration.
   <li>Investigate possible leak (see Balazs Ree's list posting).
   <li>Add <code>Browser.form_as_string()</code> and
     <code>Browser.__str__()</code> methods.
-  <li>Test BeautifulSoup support better / fix encoding issue.
   <li>Add two-way links between BeautifulSoup & ClientForm object models.
   <li>Add basic proxy support.  I hope somebody else does this!
 </ul>

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_html.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_html.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_html.py	Sun Apr 23 19:11:43 2006
@@ -1,9 +1,10 @@
 from __future__ import generators
 
-import re, urllib, htmlentitydefs
+import re, copy, urllib, htmlentitydefs
 from urlparse import urljoin
 
 import ClientCookie
+from ClientCookie._HeadersUtil import split_header_words, is_html as _is_html
 
 ## # XXXX miserable hack
 ## def urljoin(base, url):
@@ -23,6 +24,41 @@
 # 'safe'-by-default characters that urllib.urlquote never quotes
 URLQUOTE_SAFE_URL_CHARS = "!*'();:@&=+$,/?%#[]~"
 
+DEFAULT_ENCODING = "latin-1"
+
+class CachingGeneratorFunction(object):
+    """Caching wrapper around a no-arguments iterable."""
+    def __init__(self, iterable):
+        self._iterable = iterable
+        self._cache = []
+    def __call__(self):
+        cache = self._cache
+        for item in cache:
+            yield item
+        for item in self._iterable:
+            cache.append(item)
+            yield item
+
+def encoding_finder(default_encoding):
+    def encoding(response):
+        # HTTPEquivProcessor may be in use, so both HTTP and HTTP-EQUIV
+        # headers may be in the response.  HTTP-EQUIV headers come last,
+        # so try in order from first to last.
+        for ct in response.info().getheaders("content-type"):
+            for k, v in split_header_words([ct])[0]:
+                if k == "charset":
+                    return v
+        return default_encoding
+    return encoding
+
+def make_is_html(allow_xhtml):
+    def is_html(response, encoding):
+        ct_hdrs = response.info().getheaders("content-type")
+        url = response.geturl()
+        # XXX encoding
+        return _is_html(ct_hdrs, url, allow_xhtml)
+    return is_html
+
 # idea for this argument-processing trick is from Peter Otten
 class Args:
     def __init__(self, args_map):
@@ -38,7 +74,7 @@
     form_parser_class=None,
     request_class=None,
     backwards_compat=False,
-    encoding="latin-1",  # deprecated
+    encoding=DEFAULT_ENCODING,  # deprecated
     ):
     return Args(locals())
 
@@ -138,7 +174,7 @@
                  form_parser_class=None,
                  request_class=None,
                  backwards_compat=False,
-                 encoding="latin-1",  # deprecated
+                 encoding=DEFAULT_ENCODING,  # deprecated
                  ):
         import ClientForm
         self.select_default = select_default
@@ -347,20 +383,48 @@
 class Factory:
     """Factory for forms, links, etc.
 
-    The interface of this class may expand in future.
+    This interface may expand in future.
+
+    Public methods:
+
+    set_request_class(request_class)
+    set_response(response)
+    forms()
+    links()
+
+    Public attributes:
+
+    encoding: string specifying the encoding of response if it contains a text
+     document (this value is left unspecified for documents that do not have
+     an encoding, e.g. an image file)
+    is_html: true if response contains an HTML document (XHTML may be
+     regarded as HTML too)
+    title: page title, or None if no title or not HTML
 
     """
 
-    def __init__(self, forms_factory, links_factory, get_title):
+    def __init__(self, forms_factory, links_factory, get_title,
+                 get_encoding=encoding_finder(DEFAULT_ENCODING),
+                 is_html_p=make_is_html(allow_xhtml=False),
+                 ):
         """
 
-        Pass keyword
-        arguments only.
+        Pass keyword arguments only.
+
+        default_encoding: character encoding to use if encoding cannot be
+         determined (or guessed) from the response.  You should turn on
+         HTTP-EQUIV handling if you want the best chance of getting this right
+         without resorting to this default.  The default value of this
+         parameter (currently latin-1) may change in future.
 
         """
         self._forms_factory = forms_factory
         self._links_factory = links_factory
         self._get_title = get_title
+        self._get_encoding = get_encoding
+        self._is_html_p = is_html_p
+
+        self.set_response(None)
 
     def set_request_class(self, request_class):
         """Set urllib2.Request class.
@@ -371,30 +435,73 @@
         """
         self._forms_factory.request_class = request_class
 
-    def forms(self, response, encoding):
+    def set_response(self, response):
+        """Set response.
+
+        The response must implement the same interface as objects returned by
+        urllib2.urlopen().
+
+        """
+        self._response = response
+        self._forms_genf = self._links_genf = None
+        for name in ["encoding", "is_html", "title"]:
+            try:
+                delattr(self, name)
+            except AttributeError:
+                pass
+
+    def __getattr__(self, name):
+        if name not in ["encoding", "is_html", "title"]:
+            return getattr(self.__class__, name)
+
+        try:
+            if name == "encoding":
+                self.encoding = self._get_encoding(self._response)
+                return self.encoding
+            elif name == "is_html":
+                self.is_html = self._is_html_p(self._response, self.encoding)
+                return self.is_html
+            elif name == "title":
+                if self.is_html:
+                    self.title = self._get_title(self._response, self.encoding)
+                else:
+                    self.title = None
+                return self.title
+        finally:
+            self._response.seek(0)
+
+    def forms(self):
         """Return iterable over ClientForm.HTMLForm-like objects."""
-        return self._forms_factory.parse_response(response, encoding)
+        if self._forms_genf is None:
+            forms_gen = self._forms_factory.parse_response(
+                copy.copy(self._response), self.encoding)
+            self._forms_genf = CachingGeneratorFunction(forms_gen)
+        return self._forms_genf()
 
-    def links(self, response, encoding):
+    def links(self):
         """Return iterable over mechanize.Link-like objects."""
-        return self._links_factory.links(response, response.geturl(), encoding)
-
-    def title(self, response, encoding):
-        """Return page title."""
-        return self._get_title(response, encoding)
+        if self._links_genf is None:
+            links_gen = self._links_factory.links(
+                copy.copy(self._response), self._response.geturl(), self.encoding)
+            self._links_genf = CachingGeneratorFunction(links_gen)
+        return self._links_genf()
 
 class DefaultFactory(Factory):
-    def __init__(self):
-        Factory.__init__(self,
-                         forms_factory=FormsFactory(),
-                         links_factory=LinksFactory(),
-                         get_title=pp_get_title,
-                         )
+    def __init__(self, i_want_broken_xhtml_support=False):
+        Factory.__init__(
+            self,
+            forms_factory=FormsFactory(),
+            links_factory=LinksFactory(),
+            get_title=pp_get_title,
+            is_html_p=make_is_html(allow_xhtml=i_want_broken_xhtml_support),
+            )
 
 class RobustFactory(Factory):
-    def __init__(self):
-        Factory.__init__(self,
-                         forms_factory=RobustFormsFactory(),
-                         links_factory=RobustLinksFactory(),
-                         get_title=bs_get_title,
-                         )
+    def __init__(self, i_want_broken_xhtml_support=False):
+        Factory.__init__(
+            self,
+            forms_factory=RobustFormsFactory(),
+            links_factory=RobustLinksFactory(),
+            get_title=bs_get_title,
+            is_html_p=make_is_html(allow_xhtml=i_want_broken_xhtml_support),
+            )

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_mechanize.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_mechanize.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_mechanize.py	Sun Apr 23 19:11:43 2006
@@ -10,17 +10,19 @@
 """
 
 # XXXX
+# spaces in URLs
+# clean_url(): test Moz behaviour against Apache rather than File->Open!
 # test referer bugs (frags and don't add in redirect unless orig req had Referer)
 
 # XXX
 # The stuff on web page's todo list.
 # Moof's emails about response object, .back(), etc.
 
+from __future__ import generators
+
 import urllib2, urlparse, sys, copy
 
 import ClientCookie
-from ClientCookie._Util import response_seek_wrapper
-from ClientCookie._HeadersUtil import split_header_words, is_html
 
 from _useragent import UserAgent
 from _html import DefaultFactory
@@ -76,66 +78,47 @@
 
     request: current request (ClientCookie.Request or urllib2.Request)
     form: currently selected form (see .select_form())
-    default_encoding: character encoding used if no encoding is found in the
-     response (you should turn on HTTP-EQUIV handling if you want the best
-     chance of getting this right without resorting to this default)
 
     """
 
-    def __init__(self, default_encoding="latin-1",
+    def __init__(self,
                  factory=None,
                  history=None,
                  request_class=None,
-                 i_want_broken_xhtml_support=False,
-                 forms_factory=None,  # deprecated
-                 links_factory=None,  # deprecated
-                 get_title=None,  # deprecated
                  ):
         """
 
         Only named arguments should be passed to this constructor.
 
-        default_encoding: See class docs.
+        factory: object implementing the mechanize.Factory interface.
+        history: object implementing the mechanize.History interface.  Note this
+         interface is still experimental and may change in future.
         request_class: Request class to use.  Defaults to ClientCookie.Request
          by default for Pythons older than 2.4, urllib2.Request otherwise.
-        factory: mechanize.Factory
+
+        The Factory and History objects passed in are 'owned' by the Browser,
+        so they should not be shared across Browsers.  In particular,
+        factory.set_response() should not be called except by the owning
+        Browser itself.
 
         Note that the supplied factory's request_class is overridden by this
         constructor, to ensure only one Request class is used.
 
-
-        Deprecated arguments:
-
-        forms_factory: Object supporting the mechanize.FormsFactory interface.
-        links_factory: Object supporting the mechanize.LinksFactory interface.
-        get_title: callable taking a response object and an encoding string,
-         and returning the page title.
-
         """
-        self.default_encoding = default_encoding
-        self._allow_xhtml = i_want_broken_xhtml_support
         if history is None:
             history = History()
         self._history = history
         self.request = self._response = None
         self.form = None
-        self._forms = None
-        self._links = None
-        self._title = None
 
         if request_class is None:
             if not hasattr(urllib2.Request, "add_unredirected_header"):
                 request_class = ClientCookie.Request
             else:
-                request_class = urllib2.Request  # Python 2.4
+                request_class = urllib2.Request  # Python >= 2.4
 
         if factory is None:
-            if (forms_factory is None and
-                links_factory is None and
-                get_title is None):
-                factory = DefaultFactory()
-            else:
-                factory = Factory(forms_factory, links_factory, get_title)
+            factory = DefaultFactory()
         factory.set_request_class(request_class)
         self._factory = factory
         self.request_class = request_class
@@ -149,7 +132,6 @@
         if self._history is not None:
             self._history.close()
             self._history = None
-        self._forms = self._title = self._links = None
         self.request = self._response = None
 
     def open(self, url, data=None):
@@ -180,10 +162,10 @@
 
         success = True
         try:
-            self._response = UserAgent.open(self, self.request, data)
+            response = UserAgent.open(self, self.request, data)
         except urllib2.HTTPError, error:
             success = False
-            self._response = error
+            response = error
 ##         except (IOError, socket.error, OSError), error:
 ##             # Yes, urllib2 really does raise all these :-((
 ##             # See test_urllib2.py in stdlib and in ClientCookie for examples
@@ -196,9 +178,7 @@
 ##             # Python core, a fix would need some backwards-compat. hack to be
 ##             # acceptable.
 ##             raise
-        if not hasattr(self._response, "seek"):
-            self._response = response_seek_wrapper(self._response)
-        self._parse_html(self._response)
+        self.set_response(response)
         if not success:
             raise error
         return copy.copy(self._response)
@@ -214,10 +194,11 @@
 
     def set_response(self, response):
         """Replace current response with response."""
+        self.form = None
         if not hasattr(response, "seek"):
-            response = response_seek_wrapper(self._response)
+            response = ClientCookie.response_seek_wrapper(response)
         self._response = response
-        self._parse_html(self._response)
+        self._factory.set_response(self._response)
 
     def geturl(self):
         """Get URL of current document."""
@@ -241,9 +222,9 @@
         """
         if self._response is not None:
             self._response.close()
-        self.request, self._response = self._history.back(n, self._response)
-        self._parse_html(self._response)
-        return self._response
+        self.request, response = self._history.back(n, self._response)
+        self.set_response(response)
+        return response
 
     def clear_history(self):
         self._history.clear()
@@ -252,31 +233,11 @@
         """Return iterable over links (mechanize.Link objects)."""
         if not self.viewing_html():
             raise BrowserStateError("not viewing HTML")
+        links = self._factory.links()
         if kwds:
-            return self._find_links(False, **kwds)
-        if self._links is None:
-            try:
-                self._links = list(self.get_links_iter())
-            finally:
-                self._response.seek(0)
-        return self._links
-
-    def get_links_iter(self):
-        """Return an iterator that provides links of the document.
-
-        This method is provided in addition to .links() to allow lazy iteration
-        over links, while still keeping .links() safe against somebody
-        .seek()ing on a response "behind your back".  When response objects are
-        fixed to have independent seek positions, this method will be
-        deprecated in favour of .links().
-
-        """
-        if not self.viewing_html():
-            raise BrowserStateError("not viewing HTML")
-        base_url = self._response.geturl()
-        self._response.seek(0)
-        return self._factory.links(
-            self._response, self.encoding(self._response))
+            return self._filter_links(links, **kwds)
+        else:
+            return links
 
     def forms(self):
         """Return iterable over forms.
@@ -286,33 +247,19 @@
         """
         if not self.viewing_html():
             raise BrowserStateError("not viewing HTML")
-        if self._forms is None:
-            response = self._response
-            response.seek(0)
-            try:
-                self._forms = self._factory.forms(
-                    response, self.encoding(self._response))
-            finally:
-                response.seek(0)
-        return self._forms
+        return self._factory.forms()
 
     def viewing_html(self):
         """Return whether the current response contains HTML data."""
         if self._response is None:
             raise BrowserStateError("not viewing any document")
-        ct_hdrs = self._response.info().getheaders("content-type")
-        url = self._response.geturl()
-        return is_html(ct_hdrs, url, self._allow_xhtml)
-
-    def encoding(self, response):
-        # HTTPEquivProcessor may be in use, so both HTTP and HTTP-EQUIV
-        # headers may be in the response.  HTTP-EQUIV headers come last,
-        # so try in order from first to last.
-        for ct in response.info().getheaders("content-type"):
-            for k, v in split_header_words([ct])[0]:
-                if k == "charset":
-                    return v
-        return self.default_encoding
+        return self._factory.is_html
+
+    def encoding(self):
+        """"""
+        if self._response is None:
+            raise BrowserStateError("not viewing any document")
+        return self._factory.encoding
 
     def title(self):
         """Return title, or None if there is no title element in the document.
@@ -323,10 +270,7 @@
         """
         if not self.viewing_html():
             raise BrowserStateError("not viewing HTML")
-        if self._title is None:
-            self._title = self._factory.title(
-                self._response, self.encoding(self._response))
-        return self._title
+        return self._factory.title
 
     def select_form(self, name=None, predicate=None, nr=None):
         """Select an HTML form for input.
@@ -489,7 +433,10 @@
         nr: matches the nth link that matches all other criteria (default 0)
 
         """
-        return self._find_links(True, **kwds)
+        try:
+            return self._filter_links(self._factory.links(), **kwds).next()
+        except StopIteration:
+            raise LinkNotFoundError()
 
     def __getattr__(self, name):
         # pass through ClientForm / DOMForm methods and attributes
@@ -503,7 +450,7 @@
 #---------------------------------------------------
 # Private methods.
 
-    def _find_links(self, single,
+    def _filter_links(self, links,
                     text=None, text_regex=None,
                     name=None, name_regex=None,
                     url=None, url_regex=None,
@@ -517,19 +464,7 @@
         found_links = []
         orig_nr = nr
 
-        # An optimization, so that if we look for a single link we do not have
-        # to necessarily parse the entire file.
-        if self._links is None and single:
-            all_links = self.get_links_iter()
-        else:
-            if self._links is None:
-                try:
-                    self._links = list(self.get_links_iter())
-                finally:
-                    self._response.seek(0)
-            all_links = self._links
-
-        for link in all_links:
+        for link in links:
             if url is not None and url != link.url:
                 continue
             if url_regex is not None and not url_regex.search(link.url):
@@ -553,18 +488,5 @@
             if nr:
                 nr -= 1
                 continue
-            if single:
-                return link
-            else:
-                found_links.append(link)
-                nr = orig_nr
-        if not found_links:
-            raise LinkNotFoundError()
-        return found_links
-
-    def _parse_html(self, response):
-        # this is now lazy, so we just reset the various attributes that
-        # result from parsing
-        self.form = None
-        self._title = None
-        self._forms = self._links = None
+            yield link
+            nr = orig_nr

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py	Sun Apr 23 19:11:43 2006
@@ -17,6 +17,46 @@
     FACTORY_CLASSES.append(mechanize.RobustFactory)
 
 
+class CachingGeneratorFunctionTests(TestCase):
+
+    def _get_simple_cgenf(self, log):
+        from mechanize._html import CachingGeneratorFunction
+        todo = []
+        for ii in range(2):
+            def work(ii=ii):
+                log.append(ii)
+                return ii
+            todo.append(work)
+        def genf():
+            for a in todo:
+                yield a()
+        return CachingGeneratorFunction(genf())
+
+    def test_cache(self):
+        log = []
+        cgenf = self._get_simple_cgenf(log)
+        for repeat in range(2):
+            for ii, jj in zip(cgenf(), range(2)):
+                self.assertEqual(ii, jj)
+            self.assertEqual(log, range(2))  # work only done once
+
+    def test_interleaved(self):
+        log = []
+        cgenf = self._get_simple_cgenf(log)
+        cgen = cgenf()
+        self.assertEqual(cgen.next(), 0)
+        self.assertEqual(log, [0])
+        cgen2 = cgenf()
+        self.assertEqual(cgen2.next(), 0)
+        self.assertEqual(log, [0])
+        self.assertEqual(cgen.next(), 1)
+        self.assertEqual(log, [0, 1])
+        self.assertEqual(cgen2.next(), 1)
+        self.assertEqual(log, [0, 1])
+        self.assertRaises(StopIteration, cgen.next)
+        self.assertRaises(StopIteration, cgen2.next)
+
+
 class UnescapeTests(TestCase):
 
     def test_unescape_charref(self):
@@ -199,11 +239,12 @@
         import mechanize
         from StringIO import StringIO
         import urllib, mimetools
-        # always take first encoding, since that's the one
+        # always take first encoding, since that's the one from the real HTTP
+        # headers, rather than from HTTP-EQUIV
         b = mechanize.Browser()
-        for s, ct in [("", b.default_encoding),
+        for s, ct in [("", mechanize._html.DEFAULT_ENCODING),
 
-                      ("Foo: Bar\r\n\r\n", b.default_encoding),
+                      ("Foo: Bar\r\n\r\n", mechanize._html.DEFAULT_ENCODING),
 
                       ("Content-Type: text/html; charset=UTF-8\r\n\r\n",
                        "UTF-8"),
@@ -214,7 +255,8 @@
                       ]:
             msg = mimetools.Message(StringIO(s))
             r = urllib.addinfourl(StringIO(""), msg, "http://www.example.com/")
-            self.assertEqual(b.encoding(r), ct)
+            b.set_response(r)
+            self.assertEqual(b.encoding(), ct)
 
     def test_history(self):
         import mechanize
@@ -281,7 +323,8 @@
                 ("text/html; charset=blah", True),
                 (" text/html ; charset=ook ", True),
                 ]:
-                b = TestBrowser(i_want_broken_xhtml_support=allow_xhtml)
+                b = TestBrowser(mechanize.DefaultFactory(
+                    i_want_broken_xhtml_support=allow_xhtml))
                 hdrs = {}
                 if ct is not None:
                     hdrs["Content-Type"] = ct
@@ -303,7 +346,8 @@
                 (".xml", False),
                 ("", False),
                 ]:
-                b = TestBrowser(i_want_broken_xhtml_support=allow_xhtml)
+                b = TestBrowser(mechanize.DefaultFactory(
+                    i_want_broken_xhtml_support=allow_xhtml))
                 url = "http://example.com/foo"+ext
                 b.add_handler(MockHandler(
                     [("http_open", MockResponse(url, "", {}))]))
@@ -378,7 +422,7 @@
         b.add_handler(MockHandler([("http_open", r)]))
         r = b.open(url)
 
-        forms = b.forms()
+        forms = list(b.forms())
         self.assertEqual(len(forms), 2)
         for got, expect in zip([f.name for f in forms], [
             "form1", "form2"]):
@@ -489,7 +533,7 @@
             Link(url, "foo", None, "iframe",
                  [("src", "foo")]),
             ]
-        links = b.links()
+        links = list(b.links())
         self.assertEqual(len(links), len(exp_links))
         for got, expect in zip(links, exp_links):
             self.assertEqual(got, expect)
@@ -579,6 +623,7 @@
 
 class ResponseTests(TestCase):
     def test_set_response(self):
+        import copy
         from ClientCookie import response_seek_wrapper
 
         br = TestBrowser()
@@ -591,7 +636,8 @@
         r = br.open(url)
         self.assertEqual(r.read(), html)
         r.seek(0)
-        self.assertEqual(br.links()[0].url, "spam")
+        self.assertEqual(copy.copy(r).read(), html)
+        self.assertEqual(list(br.links())[0].url, "spam")
 
         newhtml = """<html><body><a href="eggs">click me</a></body></html>"""
 
@@ -600,11 +646,12 @@
         self.assertEqual(br.response().read(), html)
         br.response().set_data(newhtml)
         self.assertEqual(br.response().read(), html)
-        self.assertEqual(br.links()[0].url, "spam")
+        self.assertEqual(list(br.links())[0].url, "spam")
+        r.seek(0)
 
         br.set_response(r)
         self.assertEqual(br.response().read(), newhtml)
-        self.assertEqual(br.links()[0].url, "eggs")
+        self.assertEqual(list(br.links())[0].url, "eggs")
 
 
 class UserAgentTests(TestCase):
From jjlee at codespeak.net  Sun Apr 23 19:18:31 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sun Apr 23 19:18:32 2006
Subject: [wwwsearch-commits] r26199 -
	wwwsearch/mechanize/branch/mechanize-0.1.0-devel
Message-ID: <20060423171831.6A2901009D@code0.codespeak.net>

Author: jjlee
Date: Sun Apr 23 19:18:30 2006
New Revision: 26199

Modified:
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py
Log:
Add missing __future__ import for 2.2

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py	Sun Apr 23 19:18:30 2006
@@ -1,5 +1,7 @@
 #!/usr/bin/env python
 
+from __future__ import generators
+
 import sys, random
 from unittest import TestCase
 import StringIO, re, UserDict, urllib2
From jjlee at codespeak.net  Mon Apr 24 22:33:33 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Mon Apr 24 22:33:34 2006
Subject: [wwwsearch-commits] r26277 -
	wwwsearch/ClientCookie/trunk/ClientCookie
Message-ID: <20060424203333.F3F5F1007C@code0.codespeak.net>

Author: jjlee
Date: Mon Apr 24 22:33:33 2006
New Revision: 26277

Modified:
   wwwsearch/ClientCookie/trunk/ClientCookie/_Util.py
Log:
Allow mechanize to tell if a response can be safely .close()d -- if not, mechanize will pick it apart and re-wrap the pieces with _Util.closeable_response

Modified: wwwsearch/ClientCookie/trunk/ClientCookie/_Util.py
==============================================================================
--- wwwsearch/ClientCookie/trunk/ClientCookie/_Util.py	(original)
+++ wwwsearch/ClientCookie/trunk/ClientCookie/_Util.py	Mon Apr 24 22:33:33 2006
@@ -613,6 +613,8 @@
     it: http://python.org/sf/1144636).
 
     """
+    # presence of this attr indicates is useable after .close()
+    closeable_response = None
 
     def __init__(self, fp, headers, url, code, msg):
         self._set_fp(fp)
From jjlee at codespeak.net  Tue Apr 25 00:17:30 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Tue Apr 25 00:17:32 2006
Subject: [wwwsearch-commits] r26278 - in
	wwwsearch/mechanize/branch/mechanize-0.1.0-devel: . mechanize
Message-ID: <20060424221730.A415410077@code0.codespeak.net>

Author: jjlee
Date: Tue Apr 25 00:17:29 2006
New Revision: 26278

Modified:
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_mechanize.py
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py
Log:
Work around problem that not all responses Browser has to deal with are instances of ClientCookie._Util.closeable_response

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_mechanize.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_mechanize.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_mechanize.py	Tue Apr 25 00:17:29 2006
@@ -193,10 +193,42 @@
         return copy.copy(self._response)
 
     def set_response(self, response):
-        """Replace current response with response."""
+        """Replace current response with (a copy of) response."""
+        from ClientCookie._Util import closeable_response
+        # sanity check, necessary but far from sufficient
+        if not (hasattr(response, "info") and hasattr(response, "geturl") and
+                hasattr(response, "read")):
+            raise ValueError("not a response object")
+
         self.form = None
+
+        # XXX bleah!!
+
+        if not hasattr(response, 'closeable_response'):
+            # we expect to get here if a urllib2 handler constructed the
+            # response, i.e. the response is an urllib.addinfourl, instead of a
+            # ClientCookie._Util.closeable_response as returned by
+            # e.g. ClientCookie.HTTPHandler
+            try:
+                code = response.code
+            except AttributeError:
+                code = None
+            try:
+                msg = response.msg
+            except AttributeError:
+                msg = None
+            # assume response has an .fp attribute, the socket fileobject
+            # (i.e. is an urllib.addinfourl, really).
+            response = closeable_response(
+                response.fp, response.info(), response.geturl(), code, msg)
         if not hasattr(response, "seek"):
             response = ClientCookie.response_seek_wrapper(response)
+        # 0) don't want to copy here, but
+        # 1) don't want to copy some of the time and not other times
+        # 2) need response to be .close()able and .seek()able
+        # 3) 2) and 1) imply must always be copy.copy()ed
+        response = copy.copy(response)
+
         self._response = response
         self._factory.set_response(self._response)
 

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py	Tue Apr 25 00:17:29 2006
@@ -116,18 +116,19 @@
         return self.data.values()
 
 class MockResponse:
+    closeable_response = None
     def __init__(self, url="http://example.com/", data=None, info=None):
         self.url = url
-        self._f = StringIO.StringIO(data)
+        self.fp = StringIO.StringIO(data)
         if info is None: info = {}
         self._info = MockHeaders(info)
         self.source = "%d%d" % (id(self), random.randint(0, sys.maxint-1))
     def info(self): return self._info
     def geturl(self): return self.url
-    def read(self, size=-1): return self._f.read(size)
+    def read(self, size=-1): return self.fp.read(size)
     def seek(self, whence):
         assert whence == 0
-        self._f.seek(0)
+        self.fp.seek(0)
     def close(self): pass
     def __getstate__(self):
         state = self.__dict__
From jjlee at codespeak.net  Sun Apr 30 17:30:46 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sun Apr 30 17:30:47 2006
Subject: [wwwsearch-commits] r26606 -
	wwwsearch/mechanize/branch/mechanize-0.1.0-devel
Message-ID: <20060430153046.467B410081@code0.codespeak.net>

Author: jjlee
Date: Sun Apr 30 17:30:44 2006
New Revision: 26606

Modified:
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/functional_tests.py
Log:
Fix bad API usage in functional test

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/functional_tests.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/functional_tests.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/functional_tests.py	Sun Apr 30 17:30:44 2006
@@ -74,11 +74,11 @@
         self.assertEqual(br.response().read(), html)
         br.response().set_data(newhtml)
         self.assertEqual(br.response().read(), html)
-        self.assertEqual(br.links()[0].url, 'http://sourceforge.net')
+        self.assertEqual(list(br.links())[0].url, 'http://sourceforge.net')
 
         br.set_response(r)
         self.assertEqual(br.response().read(), newhtml)
-        self.assertEqual(br.links()[0].url, "spam")
+        self.assertEqual(list(br.links())[0].url, "spam")
 
     def test_close_pickle_load(self):
         print ("Test test_close_pickle_load is expected to fail unless Python "
From jjlee at codespeak.net  Sun Apr 30 17:46:25 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sun Apr 30 17:46:27 2006
Subject: [wwwsearch-commits] r26607 - in
	wwwsearch/mechanize/branch/mechanize-0.1.0-devel: . mechanize
Message-ID: <20060430154625.843C510081@code0.codespeak.net>

Author: jjlee
Date: Sun Apr 30 17:46:22 2006
New Revision: 26607

Modified:
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/__init__.py
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_useragent.py
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py
Log:
Improve auth support

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/__init__.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/__init__.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/__init__.py	Sun Apr 30 17:46:22 2006
@@ -1,4 +1,4 @@
-from _useragent import UserAgent
+from _useragent import UserAgent, HTTPProxyPasswordMgr
 from _mechanize import Browser, \
      BrowserStateError, LinkNotFoundError, FormNotFoundError, \
      __version__

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_useragent.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_useragent.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/mechanize/_useragent.py	Sun Apr 30 17:46:22 2006
@@ -34,6 +34,51 @@
     https_request = http_request
 
 
+class HTTPProxyPasswordMgr(urllib2.HTTPPasswordMgr):
+    # has default realm and host/port
+    def add_password(self, realm, uri, user, passwd):
+        # uri could be a single URI or a sequence
+        if uri is None or isinstance(uri, basestring):
+            uris = [uri]
+        else:
+            uris = uri
+        passwd_by_domain = self.passwd.setdefault(realm, {})
+        for uri in uris:
+            uri = self.reduce_uri(uri)
+            passwd_by_domain[uri] = (user, passwd)
+
+    def find_user_password(self, realm, authuri):
+        perms = [(realm, authuri), (None, authuri)]
+        # bleh, want default realm to take precedence over default
+        # URI/authority, hence this outer loop
+        for default_uri in False, True:
+            for realm, authuri in perms:
+                authinfo_by_domain = self.passwd.get(realm, {})
+                reduced_authuri = self.reduce_uri(authuri)
+                for uri, authinfo in authinfo_by_domain.iteritems():
+                    if uri is None and not default_uri:
+                        continue
+                    if self.is_suburi(uri, reduced_authuri):
+                        return authinfo
+                user, password = None, None
+
+                if user is not None:
+                    break
+        return user, password
+
+    def reduce_uri(self, uri):
+        if uri is None:
+            return None
+        return urllib2.HTTPPasswordMgr.reduce_uri(self, uri)
+
+    def is_suburi(self, base, test):
+        if base is None:
+            # default to the proxy's host/port
+            hostport, path = test
+            base = (hostport, "/")
+        return urllib2.HTTPPasswordMgr.is_suburi(self, base, test)
+
+
 class UserAgent(OpenerDirector):
     """Convenient user-agent class.
 
@@ -58,7 +103,6 @@
         "ftp": urllib2.FTPHandler,  # CacheFTPHandler is buggy in 2.3
         "file": urllib2.FileHandler,
         "gopher": urllib2.GopherHandler,
-        # XXX etc.
 
         # other handlers
         "_unknown": urllib2.UnknownHandler,
@@ -68,8 +112,8 @@
         "_http_default_error": urllib2.HTTPDefaultErrorHandler,
 
         # feature handlers
-        "_authen": urllib2.HTTPBasicAuthHandler,
-        # XXX rest of authentication stuff
+        "_basicauth": urllib2.HTTPBasicAuthHandler,
+        "_digestauth": urllib2.HTTPBasicAuthHandler,
         "_redirect": ClientCookie.HTTPRedirectHandler,
         "_cookies": ClientCookie.HTTPCookieProcessor,
         "_refresh": ClientCookie.HTTPRefreshProcessor,
@@ -77,7 +121,8 @@
         "_equiv": ClientCookie.HTTPEquivProcessor,
         "_seek": ClientCookie.SeekableProcessor,
         "_proxy": urllib2.ProxyHandler,
-        # XXX there's more to proxies, too
+        "_proxy_basicauth": urllib2.ProxyBasicAuthHandler,
+        "_proxy_digestauth": urllib2.ProxyDigestAuthHandler,
 
         # debug handlers
         "_debug_redirect": ClientCookie.HTTPRedirectDebugProcessor,
@@ -86,9 +131,14 @@
 
     default_schemes = ["http", "ftp", "file", "gopher"]
     default_others = ["_unknown", "_http_error", "_http_request_upgrade",
-                      "_http_default_error"]
-    default_features = ["_authen", "_redirect", "_cookies", "_refresh",
-                        "_referer", "_equiv", "_seek", "_proxy"]
+                      "_http_default_error",
+                      ]
+    default_features = ["_redirect", "_cookies", "_referer",
+                        "_refresh", "_equiv",
+                        "_basicauth", "_digestauth",
+                        "_proxy", "_proxy_basicauth", "_proxy_digestauth",
+                        "_seek",
+                        ]
     if hasattr(httplib, 'HTTPS'):
         handler_classes["https"] = ClientCookie.HTTPSHandler
         default_schemes.append("https")
@@ -99,21 +149,31 @@
     def __init__(self):
         OpenerDirector.__init__(self)
 
-        self._ua_handlers = {}
+        ua_handlers = self._ua_handlers = {}
         for scheme in (self.default_schemes+
                        self.default_others+
                        self.default_features):
             klass = self.handler_classes[scheme]
-            self._ua_handlers[scheme] = klass()
-        for handler in self._ua_handlers.itervalues():
+            ua_handlers[scheme] = klass()
+        for handler in ua_handlers.itervalues():
             self.add_handler(handler)
 
+        # Yuck.
         # Ensure correct default constructor args were passed to
-        # HTTPRefererProcessor and HTTPEquivProcessor.  Yuck.
-        if '_refresh' in self._ua_handlers:
+        # HTTPRefererProcessor and HTTPEquivProcessor.
+        if "_refresh" in ua_handlers:
             self.set_handle_refresh(True)
-        if '_equiv' in self._ua_handlers:
+        if "_equiv" in ua_handlers:
             self.set_handle_equiv(True)
+        # Ensure default password managers are installed.
+        pm = ppm = None
+        if "_basicauth" in ua_handlers or "_digestauth" in ua_handlers:
+            pm = urllib2.HTTPPasswordMgrWithDefaultRealm()
+        if ("_proxy_basicauth" in ua_handlers or
+            "_proxy_digestauth" in ua_handlers):
+            ppm = HTTPProxyPasswordMgr()
+        self.set_credentials(pm)
+        self.set_proxy_credentials(ppm)
 
         # special case, requires extra support from mechanize.Browser
         self._handle_referer = True
@@ -165,10 +225,35 @@
     def set_cookiejar(self, cookiejar):
         """Set a ClientCookie.CookieJar, or None."""
         self._set_handler("_cookies", obj=cookiejar)
+
+    # XXX could use Greg Stein's httpx for some of this instead?
+    # or httplib2??
+    def set_proxies(self, proxies):
+        """Set a dictionary mapping URL scheme to proxy specification, or None.
+
+        e.g. {'http': 'myproxy.example.com',
+              'ftp': 'joe:password@proxy.example.com:8080'}
+
+        """
+        self._set_handler("_proxy", obj=proxies)
+
+    def add_password(self, url, user, password, realm=None):
+        self._credentials.add_password(realm, url, user, password)
+    def add_proxy_password(self, user, password, hostport=None, realm=None):
+        self._proxy_credentials.add_password(realm, hostport, user, password)
+
+    # the following are rarely useful -- use add_password / add_proxy_password
+    # instead
     def set_credentials(self, credentials):
-        """Set a urllib2.HTTPPasswordMgr, or None."""
-        # XXX use Greg Stein's httpx instead?
-        self._set_handler("_authen", obj=credentials)
+        """Set a urllib2.HTTPPasswordMgrWithDefaultRealm, or None."""
+        self._credentials = credentials
+        self._set_handler("_basicauth", obj=credentials)
+        self._set_handler("_digestauth", obj=credentials)
+    def set_proxy_credentials(self, credentials):
+        """Set a mechanize.HTTPProxyPasswordMgr, or None."""
+        self._proxy_credentials = credentials
+        self._set_handler("_proxy_basicauth", obj=credentials)
+        self._set_handler("_proxy_digestauth", obj=credentials)
 
     # these methods all take a boolean parameter
     def set_handle_robots(self, handle):

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/test.py	Sun Apr 30 17:46:22 2006
@@ -19,6 +19,104 @@
     FACTORY_CLASSES.append(mechanize.RobustFactory)
 
 
+def test_password_manager(self):
+    """
+    >>> mgr = mechanize.HTTPProxyPasswordMgr()
+    >>> add = mgr.add_password
+
+    >>> add("Some Realm", "http://example.com/", "joe", "password")
+    >>> add("Some Realm", "http://example.com/ni", "ni", "ni")
+    >>> add("c", "http://example.com/foo", "foo", "ni")
+    >>> add("c", "http://example.com/bar", "bar", "nini")
+    >>> add("b", "http://example.com/", "first", "blah")
+    >>> add("b", "http://example.com/", "second", "spam")
+    >>> add("a", "http://example.com", "1", "a")
+    >>> add("Some Realm", "http://c.example.com:3128", "3", "c")
+    >>> add("Some Realm", "d.example.com", "4", "d")
+    >>> add("Some Realm", "e.example.com:3128", "5", "e")
+
+    >>> mgr.find_user_password("Some Realm", "example.com")
+    ('joe', 'password')
+    >>> mgr.find_user_password("Some Realm", "http://example.com")
+    ('joe', 'password')
+    >>> mgr.find_user_password("Some Realm", "http://example.com/")
+    ('joe', 'password')
+    >>> mgr.find_user_password("Some Realm", "http://example.com/spam")
+    ('joe', 'password')
+    >>> mgr.find_user_password("Some Realm", "http://example.com/spam/spam")
+    ('joe', 'password')
+    >>> mgr.find_user_password("c", "http://example.com/foo")
+    ('foo', 'ni')
+    >>> mgr.find_user_password("c", "http://example.com/bar")
+    ('bar', 'nini')
+
+    Currently, we use the highest-level path where more than one match:
+
+    >>> mgr.find_user_password("Some Realm", "http://example.com/ni")
+    ('joe', 'password')
+
+    Use latest add_password() in case of conflict:
+
+    >>> mgr.find_user_password("b", "http://example.com/")
+    ('second', 'spam')
+
+    No special relationship between a.example.com and example.com:
+
+    >>> mgr.find_user_password("a", "http://example.com/")
+    ('1', 'a')
+    >>> mgr.find_user_password("a", "http://a.example.com/")
+    (None, None)
+
+    Ports:
+
+    >>> mgr.find_user_password("Some Realm", "c.example.com")
+    (None, None)
+    >>> mgr.find_user_password("Some Realm", "c.example.com:3128")
+    ('3', 'c')
+    >>> mgr.find_user_password("Some Realm", "http://c.example.com:3128")
+    ('3', 'c')
+    >>> mgr.find_user_password("Some Realm", "d.example.com")
+    ('4', 'd')
+    >>> mgr.find_user_password("Some Realm", "e.example.com:3128")
+    ('5', 'e')
+
+
+    Now features specific to HTTPProxyPasswordMgr.
+
+    Default realm:
+
+    >>> mgr.find_user_password("d", "f.example.com")
+    (None, None)
+    >>> add(None, "f.example.com", "6", "f")
+    >>> mgr.find_user_password("d", "f.example.com")
+    ('6', 'f')
+
+    Default host/port:
+
+    >>> mgr.find_user_password("e", "g.example.com")
+    (None, None)
+    >>> add("e", None, "7", "g")
+    >>> mgr.find_user_password("e", "g.example.com")
+    ('7', 'g')
+
+    Default realm and host/port:
+
+    >>> mgr.find_user_password("f", "h.example.com")
+    (None, None)
+    >>> add(None, None, "8", "h")
+    >>> mgr.find_user_password("f", "h.example.com")
+    ('8', 'h')
+
+    Default realm beats default host/port:
+
+    >>> add("d", None, "9", "i")
+    >>> mgr.find_user_password("d", "f.example.com")
+    ('6', 'f')
+
+    """
+    pass
+
+
 class CachingGeneratorFunctionTests(TestCase):
 
     def _get_simple_cgenf(self, log):
@@ -703,5 +801,7 @@
         ua._set_handler("_blah", True)
 
 if __name__ == "__main__":
+    import doctest
+    doctest.testmod()
     import unittest
     unittest.main()
From jjlee at codespeak.net  Sun Apr 30 17:58:09 2006
From: jjlee at codespeak.net (jjlee@codespeak.net)
Date: Sun Apr 30 17:58:10 2006
Subject: [wwwsearch-commits] r26608 -
	wwwsearch/mechanize/branch/mechanize-0.1.0-devel
Message-ID: <20060430155809.A72BA10081@code0.codespeak.net>

Author: jjlee
Date: Sun Apr 30 17:58:08 2006
New Revision: 26608

Added:
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/0.1.0-changes.txt
Modified:
   wwwsearch/mechanize/branch/mechanize-0.1.0-devel/README.html.in
Log:
Update docs re recent API changes &c.

Added: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/0.1.0-changes.txt
==============================================================================
--- (empty file)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/0.1.0-changes.txt	Sun Apr 30 17:58:08 2006
@@ -0,0 +1,16 @@
+Recent public API changes:
+
+- ATM, auth support requires Python 2.5 (I hope to fix this soon for
+  older Pythons).  As a result, the tests are failing for 2.2, 2.3,
+  and 2.4.
+- mechanize.Browser.default_encoding is gone
+- Some mechanize.Browser constructor args have been removed (default_encoding, ...)
+- mechanize.Factory has changed completely.
+- .links no longer raises LinkNotFoundError (was accidental -- only
+  .click_link() / .find_link() should raise this)
+- .get_links_iter is gone (use .links() instead)
+- _authen handler name renamed --> _basicauth
+
+Changed in SVN fairly recently (since last mechanize release):
+ - .encoding() now takes no parameters
+ - i_want_broken_xhtml_support moved from Browser to Factory()

Modified: wwwsearch/mechanize/branch/mechanize-0.1.0-devel/README.html.in
==============================================================================
--- wwwsearch/mechanize/branch/mechanize-0.1.0-devel/README.html.in	(original)
+++ wwwsearch/mechanize/branch/mechanize-0.1.0-devel/README.html.in	Sun Apr 30 17:58:08 2006
@@ -160,6 +160,8 @@
 <h3>Specific to mechanize</h3>
 <ul>
   <li>Make encoding_finder public, I guess.
+  <li>Fi