From sschwarzer at sschwarzer.net Thu Aug 5 21:24:08 2010 From: sschwarzer at sschwarzer.net (Stefan Schwarzer) Date: Thu, 05 Aug 2010 21:24:08 +0200 Subject: [ftputil] ftputil website restructured Message-ID: <4C5B0FD8.2070605@sschwarzer.net> Hello, During the last few days I've restructured the ftputil website at http://ftputil.sschwarzer.net . The main issue with the old structure was that the default menu of Trac made it necessary to put links to the documentation, mailing lists and other topics on the bottom of the home page. I'm quite sure that this drove some people away who just looked at the main menu and didn't find a link to download a release or a link to the documentation. Please let me know what you think about the new site structure and how it may be improved further. Thanks! :-) Stefan From hraban at fiee.net Fri Aug 20 12:52:11 2010 From: hraban at fiee.net (Henning Hraban Ramm) Date: Fri, 20 Aug 2010 12:52:11 +0200 Subject: [ftputil] ftputil.ftp_error.TemporaryError: 421 No Transfer Timeout Message-ID: Ahoi, sorry I couldn't find a public searchable archive of this list, so maybe that's a FAQ: After up- or downloading big files (>10 MB) I often get the error in the subject. Here's a traceback: Traceback (most recent call last): File "./ftpsync.py", line 388, in host.rename(sourcepath, host.path.join(root, abgeholtdir, f)) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ftputil-2.4.2-py2.6.egg/ftputil/ftputil.py", line 746, in rename self._check_inaccessible_login_directory() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ftputil-2.4.2-py2.6.egg/ftputil/ftputil.py", line 535, in _check_inaccessible_login_directory self.chdir(presumable_login_dir) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ftputil-2.4.2-py2.6.egg/ftputil/ftputil.py", line 587, in chdir ftp_error._try_with_oserror(self._session.cwd, path) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ftputil-2.4.2-py2.6.egg/ftputil/ftp_error.py", line 146, in _try_with_oserror raise TemporaryError(*exc.args) ftputil.ftp_error.TemporaryError: 421 No Transfer Timeout (300 seconds): closing control connection. Debugging info: ftputil 2.4.2, Python 2.6.4 (darwin) The mentioned line (host.rename) is right after: host.download_if_newer(sourcepath, targetpath, mode='b') with host being a ftputil.FTPHost Is there anything I can do to prevent the session from timing out? I don't even understand how a session can time out while transferring data. (But the last file before the error is always transferred completely.) Something like host.reopen() would help, I guess... Greetlings, Hraban From sschwarzer at sschwarzer.net Sat Aug 21 13:45:03 2010 From: sschwarzer at sschwarzer.net (Stefan Schwarzer) Date: Sat, 21 Aug 2010 13:45:03 +0200 Subject: [ftputil] ftputil.ftp_error.TemporaryError: 421 No Transfer Timeout In-Reply-To: References: Message-ID: <4C6FBC3F.8020106@sschwarzer.net> Hello Henning, Thanks for your report! On 2010-08-20 12:52, Henning Hraban Ramm wrote: > sorry I couldn't find a public searchable archive of this list, I just noticed that you can search the mirror archive at gmane [1]. :-) > so maybe that's a FAQ: There's an FAQ list at the end of the documentation on the website [2], but it doesn't contain your question. > After up- or downloading big files (>10 MB) I often get the error in > the subject. > Here's a traceback: > [...] > The mentioned line (host.rename) is right after: > host.download_if_newer(sourcepath, targetpath, mode='b') > with host being a ftputil.FTPHost > > Is there anything I can do to prevent the session from timing out? I > don't even understand how a session can time out while transferring > data. (But the last file before the error is always transferred > completely.) The current behavior of ftputil is to open a "child" FTP session for every file transfer. From the point of view of the FTP server, this is just another client connecting. So while a child session can run an upload or download the "parent" may remain idle and eventually time out. I guess the idea behind the child sessions was to be able to have one or multiple concurrent uploads/downloads from different threads spawned from the same `FTPHost` object. Moreover, because of the way FTP works, without the child sessions you can't use the main session while a file transfer is in progress. I filed a bug report at [3]. To prevent the timeouts, you could use another thread which executes ftphost_object.chdir(ftphost_object.getcwd()) in intervals shorter than the timeout interval. (I know that's cumbersome.) You have to use the `chdir` call as the current directory is cached locally in the `FTPHost` object when invoking `chdir`, so a plain `getcwd` alone won't access the FTP server. Maybe it would be easier to catch the `TemporaryError` and in this case open a new `FTPHost` instance. The downside is that (re-)establishing an FTP connection will take more time than re-using an already open connection. > Something like host.reopen() would help, I guess... If you're really brave ;-) you can apply the following workaround patch. Beware: Even though all the unit tests in my development version still pass, I haven't tested it for the main purpose of resetting the connection nor written any unit tests for that. diff --git a/ftputil.py b/ftputil.py --- a/ftputil.py +++ b/ftputil.py @@ -163,8 +163,31 @@ # Store arguments for later operations self._args = args self._kwargs = kwargs - #XXX Maybe put the following in a `reset` method. Probably - # the time shift setting shouldn't be reset though. + # Initialize connection. + self.reset() + # Set curdir, pardir etc. for the remote host. RFC 959 states + # that this is, strictly speaking, dependent on the server OS + # but it seems to work at least with Unix and Windows + # servers. + self.curdir, self.pardir, self.sep = '.', '..', '/' + # Set default time shift (used in `upload_if_newer` and + # `download_if_newer`). + self.set_time_shift(0.0) + + def reset(self): + """ + Reset the connection. + + This closes the current connection and all still-open + child sessions. + """ + try: + self.closed + except AttributeError: + # The instance wasn't initialized before, continue. + pass + else: + self.close() # Make a session according to these arguments self._session = self._make_session() # Simulate os.path. @@ -181,14 +204,6 @@ self._file = None # Now opened self.closed = False - # Set curdir, pardir etc. for the remote host. RFC 959 states - # that this is, strictly speaking, dependent on the server OS - # but it seems to work at least with Unix and Windows - # servers. - self.curdir, self.pardir, self.sep = '.', '..', '/' - # Set default time shift (used in `upload_if_newer` and - # `download_if_newer`). - self.set_time_shift(0.0) # # Dealing with child sessions and file-like objects [1] http://blog.gmane.org/gmane.comp.python.ftputil [2] http://ftputil.sschwarzer.net/trac/wiki/Documentation#faq-tips-and-tricks [3] http://ftputil.sschwarzer.net/trac/ticket/52 Stefan From sschwarzer at sschwarzer.net Sat Aug 21 14:31:00 2010 From: sschwarzer at sschwarzer.net (Stefan Schwarzer) Date: Sat, 21 Aug 2010 14:31:00 +0200 Subject: [ftputil] ftputil.ftp_error.TemporaryError: 421 No Transfer Timeout In-Reply-To: <4C6FBC3F.8020106@sschwarzer.net> References: <4C6FBC3F.8020106@sschwarzer.net> Message-ID: <4C6FC704.1010304@sschwarzer.net> Hi, On 2010-08-21 13:45, Stefan Schwarzer wrote: > I guess the idea behind the child sessions was to be able to > have one or multiple concurrent uploads/downloads from > different threads spawned from the same `FTPHost` object. Not only that, but the current design allows to do something like: # Copy a remote file to another remote file on the same server. host = ftp_host.FTPHost(server, userid, password) source_file = host.open("source", "b") target_file = host.open("target", "b") host.copyfileobj(source_file, target_file) source_file.close() target_file.close() This is actually possible with the current ftputil version. Without the child sessions, it wouldn't be possible to have more than one remote file open at the same time. Reminder: Note that the file objects returned by `FTPHost.open` are Python file objects (although missing `seek`, `tell` etc.), so you can do things like with host.open("logfile") as remote_file: for line in remote_file: if some_test_on_line(line): ... break That is, you don't have to get the whole file from the FTP server if the `some_test_on_line` result becomes true before reaching the end of the file. Stefan From hraban at fiee.net Sat Aug 21 14:53:46 2010 From: hraban at fiee.net (Henning Hraban Ramm) Date: Sat, 21 Aug 2010 14:53:46 +0200 Subject: [ftputil] ftputil.ftp_error.TemporaryError: 421 No Transfer Timeout In-Reply-To: <4C6FC704.1010304@sschwarzer.net> References: <4C6FBC3F.8020106@sschwarzer.net> <4C6FC704.1010304@sschwarzer.net> Message-ID: <4963EABD-BFCF-4133-9666-E4027AA1955B@fiee.net> Thank you for your extensive response! I'll try the patch soon... Am 2010-08-21 um 14:31 schrieb Stefan Schwarzer: > Not only that, but the current design allows to do something > like: > > # Copy a remote file to another remote file on the same server. > host = ftp_host.FTPHost(server, userid, password) > source_file = host.open("source", "b") > target_file = host.open("target", "b") > host.copyfileobj(source_file, target_file) > source_file.close() > target_file.close() > > This is actually possible with the current ftputil version. Yes, I had that in my application and wanted to tell you next that it produced corrupted files - but maybe I only forgot the binary mode (I used "r" and "w" modes), will check... And I'll try to make another thread to keep the connection open. Greetlings from Lake Constance! Hraban --- http://www.fiee.net https://www.cacert.org (I'm an assurer) From sschwarzer at sschwarzer.net Tue Aug 24 22:19:00 2010 From: sschwarzer at sschwarzer.net (Stefan Schwarzer) Date: Tue, 24 Aug 2010 22:19:00 +0200 Subject: [ftputil] ftputil.ftp_error.TemporaryError: 421 No Transfer Timeout In-Reply-To: <4963EABD-BFCF-4133-9666-E4027AA1955B@fiee.net> References: <4C6FBC3F.8020106@sschwarzer.net> <4C6FC704.1010304@sschwarzer.net> <4963EABD-BFCF-4133-9666-E4027AA1955B@fiee.net> Message-ID: <4C742934.6000108@sschwarzer.net> Hi Henning, On 2010-08-21 14:53, Henning Hraban Ramm wrote: > Am 2010-08-21 um 14:31 schrieb Stefan Schwarzer: >> Not only that, but the current design allows to do something >> like: >> >> # Copy a remote file to another remote file on the same server. >> host = ftp_host.FTPHost(server, userid, password) >> source_file = host.open("source", "b") >> target_file = host.open("target", "b") By the way, that should be "rb" and "wb", respectively. >> host.copyfileobj(source_file, target_file) >> source_file.close() >> target_file.close() >> >> This is actually possible with the current ftputil version. > > Yes, I had that in my application and wanted to tell you next that it > produced corrupted files - but maybe I only forgot the binary mode (I > used "r" and "w" modes), will check... What's the outcome? :) > Greetlings from Lake Constance! Greetings from Leipzig! ;-) Stefan From hraban at fiee.net Tue Aug 24 22:53:15 2010 From: hraban at fiee.net (Henning Hraban Ramm) Date: Tue, 24 Aug 2010 22:53:15 +0200 Subject: [ftputil] ftputil.ftp_error.TemporaryError: 421 No Transfer Timeout In-Reply-To: <4C742934.6000108@sschwarzer.net> References: <4C6FBC3F.8020106@sschwarzer.net> <4C6FC704.1010304@sschwarzer.net> <4963EABD-BFCF-4133-9666-E4027AA1955B@fiee.net> <4C742934.6000108@sschwarzer.net> Message-ID: <1FBCC8BD-E7C6-4D3C-B7EB-E5A7CCDD61FC@fiee.net> Am 2010-08-24 um 22:19 schrieb Stefan Schwarzer: >> Yes, I had that in my application and wanted to tell you next that it >> produced corrupted files - but maybe I only forgot the binary mode (I >> used "r" and "w" modes), will check... > > What's the outcome? :) Sorry, I was working long hours the last days for a magazine deadline, so had no time and didn't want to risk more transmission problems. (I'm using ftputil mostly in an semi-automated prepress workflow - pictures from and PDFs for correction to the customer.) I hope I can check tomorrow (wednesday). Greetlings from Lake Constance! Hraban --- http://www.fiee.net https://www.cacert.org (I'm an assurer) From hraban at fiee.net Wed Aug 25 10:38:39 2010 From: hraban at fiee.net (Henning Hraban Ramm) Date: Wed, 25 Aug 2010 10:38:39 +0200 Subject: [ftputil] ftputil.ftp_error.TemporaryError: 421 No Transfer Timeout In-Reply-To: <1FBCC8BD-E7C6-4D3C-B7EB-E5A7CCDD61FC@fiee.net> References: <4C6FBC3F.8020106@sschwarzer.net> <4C6FC704.1010304@sschwarzer.net> <4963EABD-BFCF-4133-9666-E4027AA1955B@fiee.net> <4C742934.6000108@sschwarzer.net> <1FBCC8BD-E7C6-4D3C-B7EB-E5A7CCDD61FC@fiee.net> Message-ID: 2010/8/24 Henning Hraban Ramm : > Am 2010-08-24 um 22:19 schrieb Stefan Schwarzer: > >>> Yes, I had that in my application and wanted to tell you next that it >>> produced corrupted files - but maybe I only forgot the binary mode (I >>> used "r" and "w" modes), will check... >> >> What's the outcome? :) Ok, copying works in binary mode! Great! Now I must clean up my spaghetti code to try the multithreaded approach to keep the connection open... Thank you again for a great little library! Greetlings from Lake Constance! Hraban --- http://www.fiee.net https://www.cacert.org (I'm an assurer) From sschwarzer at sschwarzer.net Wed Sep 8 13:56:14 2010 From: sschwarzer at sschwarzer.net (Stefan Schwarzer) Date: Wed, 08 Sep 2010 13:56:14 +0200 Subject: [ftputil] Callback API for upload/download Message-ID: <4C8779DE.5010901@sschwarzer.net> Hi everyone, as you may have seen [1] one of the features of ftputil 2.5 will be the possibility to specify callback functions for uploads and downloads [2]. The current API design allows an additional argument `callback` which is a callable accepting one argument, `callback_info`. When the callback function is called, `callback_info` will be an object with the attributes chunk: the data read from the source in this iteration transferred_chunks: the number of transferred chunks so far actual_chunk_size: the size of the read chunk in bytes transferred_bytes: number of bytes transferred so far after starting the upload/download Meanwhile I wonder if I'm focusing too much an this upload/ download tracking. Maybe the callback parameter should be just the chunk data (as in `ftplib.FTP.retrbinary`, for example) and the user of the library can derive all the other data from that. What do you think? Is the current design overly complicated? What other uses can you imagine for a callback function other than showing the progress of the transfer? Do you have any other thoughts on the callback topic? It's important for me to get a good interface now if possible; after the first official release it's always much worse to change the API than before the release. Thanks for your input! :-) [1] http://ftputil.sschwarzer.net/trac/timeline?daysback=100&changeset=on&update=Update [2] http://ftputil.sschwarzer.net/trac/ticket/20 Stefan From markus.falb at fasel.at Wed Sep 15 10:11:33 2010 From: markus.falb at fasel.at (Markus Falb) Date: Wed, 15 Sep 2010 10:11:33 +0200 Subject: [ftputil] rpm spec file Message-ID: <52FFB088-BA12-4E89-8910-99066FC2C9E7@fasel.at> Hello, I built a rpm for CentOS/RHEL 5. I am asking if there is any interest in including the specfile in the tarball. Ideally it would exist in the top level directory of the tarball, then to build a rpm one could do: #$ rpmbuild -ta ftputil-2.4.2.tar.gz Regards, Markus -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 195 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/ftputil/attachments/20100915/b2f1cd9f/attachment-0001.pgp From sschwarzer at sschwarzer.net Sat Sep 18 11:23:28 2010 From: sschwarzer at sschwarzer.net (Stefan Schwarzer) Date: Sat, 18 Sep 2010 11:23:28 +0200 Subject: [ftputil] rpm spec file In-Reply-To: <52FFB088-BA12-4E89-8910-99066FC2C9E7@fasel.at> References: <52FFB088-BA12-4E89-8910-99066FC2C9E7@fasel.at> Message-ID: <4C948510.2040809@sschwarzer.net> Hello Markus, sorry for not replying earlier; I was at a conference. On 2010-09-15 10:11, Markus Falb wrote: > I built a rpm for CentOS/RHEL 5. I am asking if there is any interest > in including the specfile in the tarball. Ideally it would exist in > the top level directory of the tarball, then to build a rpm one could > do: > > #$ rpmbuild -ta ftputil-2.4.2.tar.gz Thanks for making the effort to build the rpm file! I have only very little experience with rpm files. That said, following are some thoughts I had when/after reading your email ... First, I don't think a file to build a platform-specific package should be in the regular release tarball, especially not in the root directory of the release. (My Debian stuff isn't in the release either.) I _may_ include the specfile somewhere in the repository though. But even here I have some doubts: Wouldn't it be much better to get ftputil into the regular package repositories for CentOS/RHEL? I think many users prefer to install a program/library from their Linux distro's packages if they can. For example, if I were to look for a Python FTP library in my distro's packages, I'd do something along the lines of $ aptitude search ftp | grep python Additionally, a package in a Linux distro repository has the huge advantage that it can be easily included as a dependency of some other package. Maybe you've seen the Debian packaging code in the ftputil repository and thought the rpm stuff could go in there as well. But the only reason the Debian stuff is there is that I worked on it myself and wanted it versioned. After getting the Debian stuff almost right (there still were some small issues), I made some efforts to get ftputil into Debian which actually has happened recently [1]. :-) I think about removing the Debian package stuff from the ftputil repository as soon as ftputil 2.5 (in development) gets into the Debian repositories, so I can check first how smoothly official package updates work. I guess, before sending your rpm/specfile to the CentOS/RHEL package maintainers, you should check how it differs from an rpm package you can build with $ python setup.py bdist --formats=rpm Moreover, I've seen that ftputil is included in the OpenSUSE repositories [2] (it's an older version though). I think you should look at this as well for comparison. Maybe you can get the OpenSUSE package updated, too. I have no idea how platform-specific rpm files are. If you don't want to make the effort to get ftputil in the regular CentOS/RHEL repository, I suggest you to make a page on the ftputil wiki regarding the rpm file (login as usual with ftputiluser / ftputil ). After that, I would add a pointer to the wiki page in the ftputil documentation, most likely in the "tips & tricks" section. However, I'd very much prefer if ftputil got into the regular CentOS/RHEL repositories where it would benefit many more users including other package maintainers of the platform. [1] http://svn.debian.org/viewsvn/python-modules/packages/ftputil/ [2] http://software.opensuse.org/search?q=ftputil&baseproject=openSUSE%3A11.3&lang=en&exclude_filter=home%3A&exclude_debug=true Stefan From sschwarzer at sschwarzer.net Sat Sep 18 11:42:11 2010 From: sschwarzer at sschwarzer.net (Stefan Schwarzer) Date: Sat, 18 Sep 2010 11:42:11 +0200 Subject: [ftputil] FTPHost.reset method Message-ID: <4C948973.6050906@sschwarzer.net> Hello, After the thread started by Henning [1], I had thought about adding a `reset` method to the `FTPHost` class. When I was writing the first test case (I prefer test-driven development) it appeared to me that this wasn't as trivial as I had thought. The issues I ran into: TODO: What to do about these? - `FTPHost` instance had been closed already. - Re-use cached current directory or reset to login directory? - Re-use time shift settings or reset to 0? - Re-use set parser or reset to automated parser setting? I tend to just do a "hard reset", that is re-setting everything in a way to resemble a freshly instantiated `FTPHost` object. But, if that's `reset`'s interface, you can just close a "faulty" `FTPHost` object and re-assign it: def my_host(): return ftputil.FTPHost(SERVER, USER, PASSWD) host = my_host() try: # bad things happening with `host` ... ... except ftp_error.TemporaryError: host = my_host() As the only reason I can think of for a `reset` method is timed out FTP sessions, I think it might be better to just add a `keep_alive` method to the class which fetches the current directory from the FTP server. (Note that this isn't fool-proof; it won't help with file transfer connections in progress which have timed out. - But these can't be rescued by a `reset` method either.) Any other thoughts/suggestions? [1] http://codespeak.net/pipermail/ftputil/2010q3/000317.html Stefan From sschwarzer at sschwarzer.net Sun Sep 19 13:09:36 2010 From: sschwarzer at sschwarzer.net (Stefan Schwarzer) Date: Sun, 19 Sep 2010 13:09:36 +0200 Subject: [ftputil] [ANN] ftputil 2.5b released Message-ID: <4C95EF70.6060307@sschwarzer.net> ftputil 2.5b is now available from http://ftputil.sschwarzer.net/download . Changes since version 2.4.2 --------------------------- - As announced over a year ago [1], the `xreadlines` method for FTP file objects has been removed, and exceptions can no longer be accessed via the `ftputil` namespace. Only use `ftp_error` to access the exceptions. The distribution contains a small tool `find_deprecated_code.py` to scan a directory tree for the deprecated uses. Invoke the program with the `--help` option to see a description. - Upload and download methods now accept a `callback` argument to do things during a transfer. Modification time comparisons in `upload_if_newer` and `download_if_newer` now consider the timestamp precision of the remote file which may lead to some unnecessary transfers. These can be avoided by waiting at least a minute between calls of `upload_if_newer` (or `download_if_newer`) for the same file. See the documentation for details [2]. - The `FTPHost` class got a `keep_alive` method. It should be used carefully though, not routinely. Please read the description [3] in the documentation. - Several bugs were fixed [4-7]. - The source code was restructured. The tests are now in a `test` subdirectory and are no longer part of the release archive. You can still get them via the source repository. Licensing matters have been moved to a common `LICENSE` file. What is ftputil? ---------------- ftputil is a high-level FTP client library for the Python programming language. ftputil implements a virtual file system for accessing FTP servers, that is, it can generate file-like objects for remote files. The library supports many functions similar to those in the os, os.path and shutil modules. ftputil has convenience functions for conditional uploads and downloads, and handles FTP clients and servers in different timezones. Read the documentation at http://ftputil.sschwarzer.net/documentation . License ------- ftputil is Open Source software, released under the revised BSD license (see http://www.opensource.org/licenses/bsd-license.php ). [1] http://codespeak.net/pipermail/ftputil/2009q1/000256.html [2] http://ftputil.sschwarzer.net/trac/wiki/Documentation#uploading-and-downloading-files [3] http://ftputil.sschwarzer.net/trac/wiki/Documentation#keep-alive [4] http://ftputil.sschwarzer.net/trac/ticket/44 [5] http://ftputil.sschwarzer.net/trac/ticket/46 [6] http://ftputil.sschwarzer.net/trac/ticket/47 [7] http://ftputil.sschwarzer.net/trac/ticket/51 Stefan