From trac at ftputil.sschwarzer.net Wed Jan 4 15:27:29 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Wed, 04 Jan 2012 14:27:29 -0000 Subject: [ftputil] #59: support MLSD and MLST Message-ID: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> #59: support MLSD and MLST ------------------------------+--------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: enhancement | Status: new Priority: major | Milestone: Component: Library | Version: Keywords: MLSD, MLST, stat | ------------------------------+--------------------------------------------- I'd like to second the request for supporting the MLSD and MLST commands if available as discussed at http://thread.gmane.org/gmane.comp.python.ftputil/440 Even though the LIST parser works fine, time resolution is very restricted to days for all files that weren't modified today. The MLS* command solve that problem. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Thu Jan 5 23:44:18 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Thu, 05 Jan 2012 22:44:18 -0000 Subject: [ftputil] #59: support MLSD and MLST In-Reply-To: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> References: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> Message-ID: <061.cba6854641dc611e7bd89f5600985c23@ftputil.sschwarzer.net> #59: support MLSD and MLST ------------------------------+--------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: enhancement | Status: new Priority: major | Milestone: Component: Library | Version: Keywords: MLSD, MLST, stat | ------------------------------+--------------------------------------------- Comment(by schwa): Hello Andi, Thanks for entering the ticket. Given what I wrote in [http://codespeak.net/pipermail/ftputil/2005q2/000035.html my reply] to the [http://codespeak.net/pipermail/ftputil/2005q2/000034.html mail you mention] I couldn't yet make up my mind on whether I want to implement MLSD/MLST support for `stat`/`lstat`. How badly are you affected by the current admittedly rather limited precision for older directories and files? What workaround(s) do you use? I'd like to get a feeling for how important the feature is in practice. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sat Jan 7 00:06:43 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Fri, 06 Jan 2012 23:06:43 -0000 Subject: [ftputil] #59: support MLSD and MLST In-Reply-To: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> References: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> Message-ID: <061.5da6af0e0a92bf9dc58d591fdcc48fa0@ftputil.sschwarzer.net> #59: support MLSD and MLST ------------------------------+--------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: enhancement | Status: new Priority: major | Milestone: Component: Library | Version: Keywords: MLSD, MLST, stat | ------------------------------+--------------------------------------------- Comment(by schwa): I've looked a bit more into this. From now on, I'm always talking about getting information for a complete directory per request because that's the basis for the stat caching. The response to MLSD for the PureFTPd server installed here looks like this: {{{ type=cdir;sizd=4096;modify=20111120085556;UNIX.mode=0755;UNIX.uid=1004;UNIX.gid=1004;unique=805g41401c2; . type=pdir;sizd=4096;modify=20111120085556;UNIX.mode=0755;UNIX.uid=1004;UNIX.gid=1004;unique=805g41401c2; .. type=file;size=641;modify=20090722070212;UNIX.mode=0644;UNIX.uid=0;UNIX.gid=0;unique=805g41401d4; CONTENTS type=OS.unix=slink:;size=11;modify=20090722070039;UNIX.mode=0777;UNIX.uid=0;UNIX.gid=0;unique=805g41401d2; broken_link type=file;size=13705975;modify=20060817084836;UNIX.mode=0644;UNIX.uid=1004;UNIX.gid=1004;unique=805g41401d3; debian-keyring.tar.gz type=dir;sizd=4096;modify=20070603143530;UNIX.mode=0755;UNIX.uid=1004;UNIX.gid=1004;unique=805g41401e3; dir with spaces type=dir;sizd=4096;modify=20090722080040;UNIX.mode=0755;UNIX.uid=0;UNIX.gid=0;unique=805g41401e9; dir_with_broken_link type=dir;sizd=4096;modify=20060128191352;UNIX.mode=0755;UNIX.uid=0;UNIX.gid=0;unique=805g41401e7; rootdir1 type=dir;sizd=4096;modify=20060128191443;UNIX.mode=0755;UNIX.uid=0;UNIX.gid=0;unique=805g41401d0; rootdir2 type=OS.unix=slink:;size=21;modify=20090722070029;UNIX.mode=0777;UNIX.uid=0;UNIX.gid=0;unique=805g41401e8; valid_link type=dir;sizd=4096;modify=20090405163514;UNIX.mode=0755;UNIX.uid=1004;UNIX.gid=1004;unique=805g41401d5; walk_test }}} The corresponding DIR response is {{{ -rw-r--r-- 1 0 0 641 Jul 22 2009 CONTENTS lrwxrwxrwx 1 0 0 11 Jul 22 2009 broken_link -> nonexistent -rw-r--r-- 1 1004 1004 13705975 Aug 17 2006 debian- keyring.tar.gz drwxr-xr-x 3 1004 1004 4096 Jun 3 2007 dir with spaces drwxr-xr-x 2 0 0 4096 Jul 22 2009 dir_with_broken_link drwxr-xr-x 2 0 0 4096 Jan 28 2006 rootdir1 drwxr-xr-x 3 0 0 4096 Jan 28 2006 rootdir2 lrwxrwxrwx 1 0 0 21 Jul 22 2009 valid_link -> debian-keyring.tar.gz drwxr-xr-x 5 1004 1004 4096 Apr 5 2009 walk_test }}} Notable differences between the MLSD and the DIR response: * The [http://tools.ietf.org/rfc/rfc3659.txt specification] for the MLSD and MLST commands doesn't seem to specify which facts (the term is from the RFC) should or even may be included in the response. * I was unable to find out whether the user id in the MLSD response is always or even usually the same as the corresponding column in the DIR output. More precisely, I wonder whether the user id may be a number in the MLSD response but a string in the DIR response. * At least in this specific example, the MLSD response doesn't contain link targets which are required for `FTPHost.stat` (which follows links) to work. These differences mean that even if the remote server supports the MLSD command, the DIR request and the parsing of its response probably are still required. In other words, ftputil needs to send ''two'' directory requests and retrieve and process their responses. If the code using ftputil does relatively little file transfer in comparison to getting stat information this will slow down the FTP-related code. So the use of MLSD, even if it's supported by the server, may have to be optional. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sat Jan 7 00:31:44 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Fri, 06 Jan 2012 23:31:44 -0000 Subject: [ftputil] #59: Support MLSD and MLST (was: support MLSD and MLST) In-Reply-To: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> References: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> Message-ID: <061.1512870920cbcc25577efee499902633@ftputil.sschwarzer.net> #59: Support MLSD and MLST ------------------------------+--------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: enhancement | Status: new Priority: major | Milestone: Component: Library | Version: Keywords: MLSD, MLST, stat | ------------------------------+--------------------------------------------- Old description: > I'd like to second the request for supporting the MLSD and MLST commands > if available as discussed at > http://thread.gmane.org/gmane.comp.python.ftputil/440 > > Even though the LIST parser works fine, time resolution is very > restricted to days for all files that weren't modified today. The MLS* > command solve that problem. New description: I'd like to second the request for supporting the MLSD and MLST commands if available as discussed at http://thread.gmane.org/gmane.comp.python.ftputil/440 Even though the LIST parser works fine, time resolution is very restricted to days for all files that weren't modified today. The MLS* commands solve that problem. -- Comment(by schwa): ''If'' MLSD support is to be implemented, it will most likely be done as follows: * Addition of an MLSD parser in `ftp_stat.py`. * Extend `ftp_stat._Stat._stat_results_from_dir` so that after using DIR it tries to use MLSD to get more precise values for the time of the item's last modification. If the MLSD command turns out to be unsupported by the server or the MLSD response doesn't contain the modification time, set a flag so that MLSD isn't even tried again for this connection. * Add unit tests (usually in form of test-driven development during the implementation). * If the use of MLSD should be configurable by the user, extend `FTPHost` accordingly and describe the API in the documentation. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sat Jan 7 14:24:14 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sat, 07 Jan 2012 13:24:14 -0000 Subject: [ftputil] #59: Support MLSD and MLST In-Reply-To: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> References: <052.3b4029eed45ba30eb1563ae73ef9815f@ftputil.sschwarzer.net> Message-ID: <061.6d0c619dc6c2bd110de826d7146b8dff@ftputil.sschwarzer.net> #59: Support MLSD and MLST --------------------------+------------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: enhancement | Status: closed Priority: major | Milestone: Component: Library | Version: Resolution: wontfix | Keywords: MLSD, MLST, stat --------------------------+------------------------------------------------- Changes (by schwa): * status: new => closed * resolution: => wontfix Comment: Meanwhile I got some feedback by the original ticket author that he can work around the absence of MLSD/MLST with sufficient ease. I close the ticket as "wontfix" ''for now''. If someone runs into actual problems because of the lack of precision in the modification timestamps, please reopen the ticket and describe your problem in a comment. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sat Jan 7 14:33:34 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sat, 07 Jan 2012 13:33:34 -0000 Subject: [ftputil] #58: Ability to access source text line for file stats. In-Reply-To: <052.27a32e4feed7b55c6b26e0333ca25faf@ftputil.sschwarzer.net> References: <052.27a32e4feed7b55c6b26e0333ca25faf@ftputil.sschwarzer.net> Message-ID: <061.32c1cd4ea52fd7cd0ddf4ceb8c31818d@ftputil.sschwarzer.net> #58: Ability to access source text line for file stats. --------------------------+------------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: enhancement | Status: closed Priority: trivial | Milestone: Component: Library | Version: 2.6 Resolution: wontfix | Keywords: --------------------------+------------------------------------------------- Changes (by schwa): * status: assigned => closed * resolution: => wontfix Comment: On second thought I decided against adding the source line to every stat object just for debugging. I think it's not appropriate to generally use the extra memory. If necessary you can always use the approach described in the previous comment. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sat Jan 7 14:38:26 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sat, 07 Jan 2012 13:38:26 -0000 Subject: [ftputil] #40: ftp_sync misses path separator from target directory In-Reply-To: <052.dc42011419624f4e3fe45d974d1bd111@ftputil.sschwarzer.net> References: <052.dc42011419624f4e3fe45d974d1bd111@ftputil.sschwarzer.net> Message-ID: <061.1ccdfcd509db765467aefac7db882d4e@ftputil.sschwarzer.net> #40: ftp_sync misses path separator from target directory --------------------------+------------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: defect | Status: closed Priority: minor | Milestone: Component: Library | Version: Resolution: worksforme | Keywords: sync, target directory, path separator --------------------------+------------------------------------------------- Changes (by schwa): * status: assigned => closed * resolution: => worksforme Comment: I didn't receive any reply from the ticket author, so I'll close this ticket as "worksforme" because I couldn't reproduce the problem. Please reopen the ticket if you have a code example which shows the bug. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Wed Mar 21 15:46:13 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Wed, 21 Mar 2012 14:46:13 -0000 Subject: [ftputil] #60: 'listdir' strips whitespace from beginning of filenames on FTP site Message-ID: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> #60: 'listdir' strips whitespace from beginning of filenames on FTP site --------------------------------+------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: defect | Status: new Priority: minor | Milestone: 2.7 Component: Library | Version: 2.6 Keywords: listdir whitespace | --------------------------------+------------------------------------------- A call to 'listdir' will return a list of filenames in a particular directory on an FTP server. However, these filenames appear to be stripped of any whitespace on the beginning of the filename. Thus, the following code will fail if one of the files on the FTP server has a name with a leading space, throwing a ftputil.ftp_error.PermanentError (Not Found): ftplist = host.listdir(host.curdir) for ftpfile in ftplist: host.remove(ftpfile) 'ftpfile' contains the filename without the leading space, leading to the conclusion that listdir is faulty. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Wed Mar 21 15:56:16 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Wed, 21 Mar 2012 14:56:16 -0000 Subject: [ftputil] #60: 'listdir' strips whitespace from beginning of filenames on FTP site In-Reply-To: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> References: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> Message-ID: <061.34714fdcc8b5b4b587c5ee955798a8c6@ftputil.sschwarzer.net> #60: 'listdir' strips whitespace from beginning of filenames on FTP site --------------------------------+------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: defect | Status: new Priority: minor | Milestone: 2.7 Component: Library | Version: 2.6 Keywords: listdir whitespace | --------------------------------+------------------------------------------- Comment(by ftputiluser): Just to attach a couple more details to this, I'm using: * FTPutil 2.6 * Python 2.7 * ProFTPD 1.3.3 Server (Hddn FTP Server01) * Client OS: Ubuntu 11.10, Server OS unknown -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Thu Mar 22 21:19:26 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Thu, 22 Mar 2012 20:19:26 -0000 Subject: [ftputil] #60: 'listdir' strips whitespace from beginning of filenames on FTP site In-Reply-To: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> References: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> Message-ID: <061.e54c8c8162a03663e422702f968260a0@ftputil.sschwarzer.net> #60: 'listdir' strips whitespace from beginning of filenames on FTP site --------------------------------+------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: defect | Status: assigned Priority: minor | Milestone: 2.7 Component: Library | Version: 2.6 Keywords: listdir whitespace | --------------------------------+------------------------------------------- Changes (by schwa): * status: new => assigned Old description: > A call to 'listdir' will return a list of filenames in a particular > directory on an FTP server. However, these filenames appear to be > stripped of any whitespace on the beginning of the filename. Thus, the > following code will fail if one of the files on the FTP server has a name > with a leading space, throwing a ftputil.ftp_error.PermanentError (Not > Found): > > ftplist = host.listdir(host.curdir) > for ftpfile in ftplist: > host.remove(ftpfile) > > 'ftpfile' contains the filename without the leading space, leading to the > conclusion that listdir is faulty. New description: A call to 'listdir' will return a list of filenames in a particular directory on an FTP server. However, these filenames appear to be stripped of any whitespace on the beginning of the filename. Thus, the following code will fail if one of the files on the FTP server has a name with a leading space, throwing a `ftputil.ftp_error.PermanentError` (Not Found): {{{ ftplist = host.listdir(host.curdir) for ftpfile in ftplist: host.remove(ftpfile) }}} 'ftpfile' contains the filename without the leading space, leading to the conclusion that listdir is faulty. -- Comment: Thanks for the report. The names are extracted from the listing data that is retrieved with an FTP `DIR` command. This gives a listing similar to the output of `ls -l` on Posix system or `dir` on Windows systems. Both the Unix format parser and the Microsoft format parser in ftputil use `str.split()` which splits at sequences of whitespace. Hence there's no distinction between a space which is part of a filename and a space for column alignment in the directory listing. I have one or two ideas on how to handle this to allow names with whitespace. However, this would have to use some heuristics and won't be a robust approach by any means. Additionally, I'm not sure if I could handle filenames with leading/trailing spaces in them correctly even if I managed to extract them correctly from the listings. `ftputil` uses Python's `ftplib` module, so I have to rely on `ftplib` doing the right thing. That doesn't always seem to be the case. For example, I noticed that if I use `ftplib`'s `mkd` command with a directory name with a leading space, the leading space is converted to an underscore. I very much tend to close this ticket as "wontfix": The "fix", if there's any, would be brittle, and there's no guarantee either that the names will be processed correctly in following FTP commands. My advice is to avoid leading and trailing whitespace (or if possible any whitespace) in directory or file names if you need to process them with a protocol which relies on parsing that's not strictly defined. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Fri Mar 23 16:26:44 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Fri, 23 Mar 2012 15:26:44 -0000 Subject: [ftputil] #60: 'listdir' strips whitespace from beginning of filenames on FTP site In-Reply-To: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> References: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> Message-ID: <061.3f92cfb21dcaeac4353768027d4984d2@ftputil.sschwarzer.net> #60: 'listdir' strips whitespace from beginning of filenames on FTP site --------------------------------+------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: defect | Status: assigned Priority: minor | Milestone: 2.7 Component: Library | Version: 2.6 Keywords: listdir whitespace | --------------------------------+------------------------------------------- Comment(by schwa): Replying to [comment:2 schwa]: > `ftputil` uses Python's `ftplib` module, so I have to rely on `ftplib` doing the right thing. That doesn't always seem to be the case. For example, I noticed that if I use `ftplib`'s `mkd` command with a directory name with a leading space, the leading space is converted to an underscore. I looked at the `ftplib` source code. It seems the FTP server put in the leading underscore, not the `mkd` method. -- Ticket URL: ftputil Python FTP client library From trac at ftputil.sschwarzer.net Sat Mar 31 15:07:41 2012 From: trac at ftputil.sschwarzer.net (ftputil) Date: Sat, 31 Mar 2012 13:07:41 -0000 Subject: [ftputil] #60: 'listdir' strips whitespace from beginning of filenames on FTP site In-Reply-To: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> References: <052.be9bc65591e43a1ae0fa2829f3bfcdcf@ftputil.sschwarzer.net> Message-ID: <061.ac1e1081884960affa23b4b50a1b91e9@ftputil.sschwarzer.net> #60: 'listdir' strips whitespace from beginning of filenames on FTP site --------------------------+------------------------------------------------- Reporter: ftputiluser | Owner: schwa Type: defect | Status: closed Priority: minor | Milestone: 2.7 Component: Library | Version: 2.6 Resolution: wontfix | Keywords: listdir whitespace --------------------------+------------------------------------------------- Changes (by schwa): * status: assigned => closed * resolution: => wontfix Comment: I decided not to "solve" the problem for the reasons given in my first comment. Sorry. -- Ticket URL: ftputil Python FTP client library