From trac at ftputil.sschwarzer.net Thu Oct 21 00:19:42 2010 From: trac at ftputil.sschwarzer.net (ftputil) Date: Wed, 20 Oct 2010 22:19:42 -0000 Subject: [ftputil] #53: FTPHost.walk fails when the argument is a unicode string and the tree contains non-ASCII characters Message-ID: <046.c131001bbf3f163c5d12a0fe2912047a@ftputil.sschwarzer.net> #53: FTPHost.walk fails when the argument is a unicode string and the tree contains non-ASCII characters -----------------------------------------------+---------------------------- Reporter: schwa | Owner: schwa Type: defect | Status: new Priority: major | Milestone: 2.5 Component: Library | Version: 2.4.2 Keywords: unicode, walk, UnicodeEncodeError | -----------------------------------------------+---------------------------- When `FTPHost.walk` is used to examine a filesystem tree which somewhere contains a non-7-bit-ASCII character in a name ''and'' the argument passed in is any unicode string, the `walk` method implicitly will raise a `UnicodeEncodeError`. Imagine this directory structure: {{{ some_dir some_str?nge_file }}} then {{{ ftp_host.walk(u"some_dir") }}} will cause a `UnicodeEncodeError` in `posixpath` like: {{{ File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/posixpath.py", line 70, in join path += '/' + b UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 41: ordinal not in range(128) }}} (original [http://codespeak.net/pipermail/ftputil/2010q4/000329.html report] by Henning Hraban Ramm - thanks!) -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Thu Oct 21 00:36:13 2010 From: trac at ftputil.sschwarzer.net (ftputil) Date: Wed, 20 Oct 2010 22:36:13 -0000 Subject: [ftputil] #53: FTPHost.walk fails when the argument is a unicode string and the tree contains non-ASCII characters In-Reply-To: <046.c131001bbf3f163c5d12a0fe2912047a@ftputil.sschwarzer.net> References: <046.c131001bbf3f163c5d12a0fe2912047a@ftputil.sschwarzer.net> Message-ID: <055.e18ff69b1f511dae12edc55abf192177@ftputil.sschwarzer.net> #53: FTPHost.walk fails when the argument is a unicode string and the tree contains non-ASCII characters ----------------------+----------------------------------------------------- Reporter: schwa | Owner: schwa Type: defect | Status: assigned Priority: major | Milestone: 2.5 Component: Library | Version: 2.4.2 Resolution: | Keywords: unicode, walk, UnicodeEncodeError ----------------------+----------------------------------------------------- Changes (by schwa): * status: new => assigned Comment: As pointed out in [http://codespeak.net/pipermail/ftputil/2010q4/000332.html this mail], FTP has no concept of encodings. As the encoding of the directories and files on the remote side is unknown, there's no convenient solution. At the moment, I think the most appropriate approach is to have a method fail as early as possible if it accepts remote paths and gets a unicode string for them. A solution might be something like: {{{ def walk(self, root): # If `root` isn't ASCII, fail now instead of later. # Otherwise, continue with a byte string. root = str(root) ... }}} -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Thu Oct 21 13:35:17 2010 From: trac at ftputil.sschwarzer.net (ftputil) Date: Thu, 21 Oct 2010 11:35:17 -0000 Subject: [ftputil] #53: FTPHost.walk fails when the argument is a unicode string and the tree contains non-ASCII characters In-Reply-To: <046.c131001bbf3f163c5d12a0fe2912047a@ftputil.sschwarzer.net> References: <046.c131001bbf3f163c5d12a0fe2912047a@ftputil.sschwarzer.net> Message-ID: <055.7a6664761b684da4bfbe3341441597a1@ftputil.sschwarzer.net> #53: FTPHost.walk fails when the argument is a unicode string and the tree contains non-ASCII characters ----------------------+----------------------------------------------------- Reporter: schwa | Owner: schwa Type: defect | Status: assigned Priority: major | Milestone: 2.5.1 Component: Library | Version: 2.4.2 Resolution: | Keywords: unicode, walk, UnicodeEncodeError ----------------------+----------------------------------------------------- Changes (by schwa): * milestone: 2.5 => 2.5.1 Comment: I put this off until ftputil 2.5.1. I'll have to go through all the methods and see how they're affected, and I don't want to delay the release of ftputil 2.5 final even more after the apparently long beta phase. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sun Oct 24 17:28:29 2010 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sun, 24 Oct 2010 15:28:29 -0000 Subject: [ftputil] #54: Auto-resizing functionality for stat cache Message-ID: <046.ebc5bbcaac08ae35093137780a752e21@ftputil.sschwarzer.net> #54: Auto-resizing functionality for stat cache -------------------------------------------------+-------------------------- Reporter: schwa | Owner: schwa Type: defect | Status: new Priority: major | Milestone: 2.6 Component: Library | Version: 2.5 Keywords: cache, flushing, default size, stat | -------------------------------------------------+-------------------------- (For some background on stat caching read the [wiki:Documentation documentation].) The default stat cache size is 1000 entries. Though this usually is enough for most purposes, people from time to time run into the problem of directories with more items than the default cache size. In this case, old entries (further towards the top of the directory) are flushed out to make place for newer stat items. Then, by the time the directory is iterated over, the older entries are gone, so the directory listing is fetched again and again. I've been thinking about just increasing the default stat cache size, but of course this costs memory which possibly isn't necessary. Maybe an auto- resize feature is a better approach. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sun Oct 24 17:29:30 2010 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sun, 24 Oct 2010 15:29:30 -0000 Subject: [ftputil] #54: Auto-resizing functionality for stat cache In-Reply-To: <046.ebc5bbcaac08ae35093137780a752e21@ftputil.sschwarzer.net> References: <046.ebc5bbcaac08ae35093137780a752e21@ftputil.sschwarzer.net> Message-ID: <055.c907239eb7b7277931cef45db0dd1ac7@ftputil.sschwarzer.net> #54: Auto-resizing functionality for stat cache --------------------------+------------------------------------------------- Reporter: schwa | Owner: schwa Type: enhancement | Status: assigned Priority: major | Milestone: 2.6 Component: Library | Version: 2.5 Resolution: | Keywords: cache, flushing, default size, stat --------------------------+------------------------------------------------- Changes (by schwa): * status: new => assigned * type: defect => enhancement -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sun Oct 24 17:32:59 2010 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sun, 24 Oct 2010 15:32:59 -0000 Subject: [ftputil] #54: Auto-resizing functionality for stat cache In-Reply-To: <046.ebc5bbcaac08ae35093137780a752e21@ftputil.sschwarzer.net> References: <046.ebc5bbcaac08ae35093137780a752e21@ftputil.sschwarzer.net> Message-ID: <055.87eaf591293b0857be244c784774c4d2@ftputil.sschwarzer.net> #54: Auto-resizing functionality for stat cache --------------------------+------------------------------------------------- Reporter: schwa | Owner: schwa Type: enhancement | Status: assigned Priority: major | Milestone: 2.6 Component: Library | Version: 2.5 Resolution: | Keywords: cache, flushing, default size, stat --------------------------+------------------------------------------------- Description changed by schwa: Old description: > (For some background on stat caching read the [wiki:Documentation > documentation].) > > The default stat cache size is 1000 entries. Though this usually is > enough for most purposes, people from time to time run into the problem > of directories with more items than the default cache size. In this case, > old entries (further towards the top of the directory) are flushed out to > make place for newer stat items. Then, by the time the directory is > iterated over, the older entries are gone, so the directory listing is > fetched again and again. > > I've been thinking about just increasing the default stat cache size, but > of course this costs memory which possibly isn't necessary. Maybe an > auto-resize feature is a better approach. New description: (For some background on stat caching read the [wiki:Documentation#local- caching-of-file-system-information documentation].) The default stat cache size is 1000 entries. Though this is enough for most purposes, people from time to time run into the problem of directories with more items than the default cache size. In this case, old entries (further towards the top of the directory) are flushed out to make place for newer stat items. Then, by the time the directory is iterated over, the older entries are gone, so the directory listing is fetched again for each stat call (`isdir`, `getmtime` etc.). I've been thinking about just increasing the default stat cache size, but of course this costs memory which possibly isn't necessary. Maybe an auto- resize feature is a better approach. -- -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sun Nov 14 22:56:42 2010 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sun, 14 Nov 2010 21:56:42 -0000 Subject: [ftputil] #53: FTPHost.walk fails when the argument is a unicode string and the tree contains non-ASCII characters In-Reply-To: <046.c131001bbf3f163c5d12a0fe2912047a@ftputil.sschwarzer.net> References: <046.c131001bbf3f163c5d12a0fe2912047a@ftputil.sschwarzer.net> Message-ID: <055.db7528322c367f617516055fe2a12c57@ftputil.sschwarzer.net> #53: FTPHost.walk fails when the argument is a unicode string and the tree contains non-ASCII characters ----------------------+----------------------------------------------------- Reporter: schwa | Owner: schwa Type: defect | Status: closed Priority: major | Milestone: 2.5.1 Component: Library | Version: 2.4.2 Resolution: fixed | Keywords: unicode, walk, UnicodeEncodeError ----------------------+----------------------------------------------------- Changes (by schwa): * status: assigned => closed * resolution: => fixed Comment: Fixed in commits [7ee81a2ca43a] and [8c59e4da5479]. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sat Dec 25 17:08:52 2010 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sat, 25 Dec 2010 16:08:52 -0000 Subject: [ftputil] #54: Auto-resizing functionality for stat cache In-Reply-To: <046.ebc5bbcaac08ae35093137780a752e21@ftputil.sschwarzer.net> References: <046.ebc5bbcaac08ae35093137780a752e21@ftputil.sschwarzer.net> Message-ID: <055.b116cf5cb664140b1912a163d7019288@ftputil.sschwarzer.net> #54: Auto-resizing functionality for stat cache --------------------------+------------------------------------------------- Reporter: schwa | Owner: schwa Type: enhancement | Status: closed Priority: major | Milestone: 2.6 Component: Library | Version: 2.5 Resolution: fixed | Keywords: cache, flushing, default size, stat --------------------------+------------------------------------------------- Changes (by schwa): * status: assigned => closed * resolution: => fixed Comment: Added in [0fb499033581]. -- Ticket URL: ftputil Python FTP client library