[ftputil] dealing with server timeout + GMANE

[Terrence Brannon]

There was a trick in your code I had never seen before. The comma was
necessary to create a sequence here:

  >>> t = threading.Thread(target=poll, args=(host,))

Anyway, the program seems to head once I initiate the thread. I created a
class for your code and then called it like below. I think throwing a simple
chdir() in between each downloaded file should prevent the timeout, but I'd
like to see if I can get the thread thing working anyway:

import ftputil
import time
import threading

class busyloop(object):

    def __init__(self,host):
        self.host = host

    def poll(self, host):
        while True:
            host.chdir(host.curdir)
            time.sleep(300)

    def loop(self):
        t = threading.Thread(target=self.poll, args=(self.host,))
        t.start()
        t.join()


    def fetch_file(my):
        #print "host = ftputil.FTPHost",  my.cnf.dl.host, my.cnf.dl.user,
my.c\
nf.dl.password

        host = ftputil.FTPHost(my.cnf.dl.host, my.cnf.dl.user,
                               my.cnf.dl.password)

#        t = data.localpy.ftputil.busyloop(host)
#        t.loop()

        print host
        host.chdir(my.cnf.dl.root_cwd)

        expected_files = ['diseasecategories_export.txt',
                'fdadruglistings_export.txt',
                'medicalareas_export.txt',
                'pipelinedrugs_export.txt',
                'pipelinedrugscategories_export.txt',
                'studylistings_export.txt',
                'studyresults_export.txt']
        for f in host.listdir(host.curdir):
            if re.compile(my.cnf.dl.filepattern).search(f):
                print "Downloading", f
                host.chdir(host.curdir)
                host.download(f, f, 't')

                try:
                    expected_files.remove(path(f).basename())
                except:
                    pass
                shutil.copy(f, my.storage.input)
                shutil.move(f, my.storage.zip)

        if len(expected_files) != 0:
            print "Expected files", expected_files, "not on", my.cnf.dl.host
            sys.exit()


On 12/5/06, Stefan Schwarzer <sschwarzer at sschwarzer.net> wrote:
>
> Hi Terrence,
>
> On 2006-12-05 17:25, Terrence Brannon wrote:
> > Hi, I filled out the subscription form at GMANE for this list so that it
> > will be archived there as well as mail-archive.com... it is not easy to
> > search mailman archives.
>
> You are quite right, thanks for adding the Gmane interface for
> the mailing list!
>
> > Anyway, I have an issue where I am downloading some large files from a
> > server and I am getting at "Connection timed out" error.
>
> Do you get the error _during_ the download or after using the
> FTPHost object _after_ the download?
>
> I assume the latter. Ftputil opens an FTP session for the "main"
> session when you construct an FTPHost object. For each file-like
> object (i. e. also for up/downloads) a new FTP session is opened
> behind the scenes. So I guess that after the download your main
> session has timed out.
>
> You could prevent this by giving the main session something to do
> during the download. If your Python is compiled with threading
> support, the following should work:
>
> >>> import ftputil
> >>> import time
> >>> import threading
> >>>
> >>> def poll(host):
> ...   while True:
> ...     # getcwd won't work because it only fetches a cached value
> ...     host.chdir(host.curdir)
> ...     # must be less seconds than a server timeout
> ...     time.sleep(300)
> ...
> >>> host = ftputil.FTPHost("ftp.debian.org", 'anonymous',
> ...                        'sschwarzer at sschwarzer.net')`
> >>> t = threading.Thread(target=poll, args=(host,))
> >>> t.start()
> >>> t.join()
>
> This isn't production-ready code (e. g. the thread isn't stopped
> when the FTPHost instance is closed). Also note that server
> timeouts are there for a reason, so be very careful about the
> duration you maintain the connection.
>
> Please let me know if that helps.
>
> I had thought about "timeout safety" a while ago, see
> http://codespeak.net/pipermail/ftputil/2006q1/000073.html . Since
> I didn't get any responses, I removed the timeout functionality
> in the face of the described implementation difficulty.
>
> Stefan
> _______________________________________________
> ftputil mailing list
> ftputil at codespeak.net
> http://codespeak.net/mailman/listinfo/ftputil
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20061206/d2467ec0/attachment-0001.htm