[ftputil] How to correctlry close a ftp connection

[Stefan Schwarzer]

Hi Yvan,

On 2008-12-12 13:40, Yvan Strahm wrote:
> I get this error:
> 
> Traceback (most recent call last):
>   File "./map_taxo.py", line 186, in ?
>     Download_me(ftp,ftp_dir,file,target)
>   File "./map_taxo.py", line 95, in Download_me
>     host=ftputil.FTPHost(ftp,user,password)
>   File "/usr/lib/python2.4/site-packages/ftputil/ftputil.py", line 136,
> in __init__
>     self._session = self._make_session()
>   File "/usr/lib/python2.4/site-packages/ftputil/ftputil.py", line 174,
> in _make_session
>     return ftp_error._try_with_oserror(factory, *args, **kwargs)
>   File "/usr/lib/python2.4/site-packages/ftputil/ftp_error.py", line 86,
> in _try_with_oserror
>     raise PermanentError(obj)
> ftputil.ftp_error.PermanentError: 530 Sorry, the maximum number clients
> (32) from your host are already connected.
> Debugging info: ftputil 2.2.3, Python 2.4.3 (linux2)

Does the FTPHost construction here refer to the first in the try
clause or the second in the except clause?

> I have a list of file s to be downloaded and open/close for every items.
> Is it the correct way of doing it?
> 
> def Download_me(ftp,ftp_dir,file,target):
>   user = 'anonymous'
>   password = 'anonymous'
>   try:
>     print "CONNECTING TO "+ ftp + ftp_dir+file+"\n"

Which actual values did you use for ftp, ftp_dir, file and
target which caused the exception, if I may ask?

>     host=ftputil.FTPHost(ftp,user,password)
>     host.chdir(ftp_dir+file)

Does ftp_dir contain a trailing slash here? I'd recommend to use
FTPHost.path.join(ftp_dir, file) and ftp_dir without a trailing
slash.

>     names=host.listdir(host.curdir)
>     for name in names:
>       if host.path.isfile(name):
>         p="pep"
>         m_pep=re.search(p,name)
>         if m_pep:
>           returned=name
>           host.download_if_newer(name,target,'b')
>     host.close()
>   except:

I suggest you avoid a bare "except"; use "except
ftp_error.PermanentError" (or rather the exceptions you actually
expect). In your current code, _any_ exception would trigger the
except clause, thus catching exceptions you perhaps don't want to
be caught.

>     print "could not be dowloaded try KAAS"
>     ftp='ftp.genome.jp <http://ftp.genome.jp>'

If you use this for the host argument of the FTPHost constructor,
it will probably fail. Use just the full name of the FTP server
(ftp.genome.jp), without the part in angle brackets.

>     ftp_dir='/pub/kegg/genes/organisms_kaas/'
>     print "CONNECTING TO "+ ftp + ftp_dir+file+"\n"
>     host=ftputil.FTPHost(ftp,user,password)
>     host.chdir(ftp_dir+file)
>     names=host.listdir(host.curdir)
>     for name in names:
>       if host.path.isfile(name):
>         p="pep"
>         m_pep=re.search(p,name)
>         if m_pep:
>           returned=name
>           host.download_if_newer(name,target,'b')
>     host.close()

Possibly you should put the common parts of the try and except
clauses into a function.

Regarding the symmetry of the FTPHost construction and calling
the close method, at first sight this looks well. However, if the
constructor fails, the close method won't be called, so that may
be the reason why you exhaust the possible number of connections.
(On the other hand, this could be a genuine error message if
enough people are logged into the FTP server at the same time.)
I think you should use a try ... finally construct:

try:
    print "CONNECTING TO "+ ftp + ftp_dir+file+"\n"
    host=ftputil.FTPHost(ftp,user,password)
    try:
        host.chdir(ftp_dir+file)
        names=host.listdir(host.curdir)
        for name in names:
            if host.path.isfile(name):
                p="pep"
                m_pep=re.search(p,name)
                if m_pep:
                    returned=name
                    host.download_if_newer(name,target,'b')
    finally:
        host.close()
except:
    ...

In this way, the connection will be closed, whether there's an
exception or not.

Does that help?

Best regards,
Stefan