[ftputil] mirror script + listing files with dot inside

[Stefan Schwarzer]

Hi Benjamin,

Thanks a lot for your contributions.

On 2006-12-27 23:12, Benjamin Sergeant wrote:
> http://www.unixwiz.net/techtips/wget-dotfiles.html
> Here they explain how to get the . files : one have to issue the LIST
> with -a as a parameter (as for ls)

As far as I know, whether a directory listing contains files with
a leading dot depends on the FTP server configuration.

>>>> ftp.retrlines('LIST')

ftp is an ftplib.FTP instance here, isn't it?

> drwxr-xr-x   3 bsergean bsergean      176 Dec 16 03:40 httpd
> -rwxr-xr-x   1 bsergean bsergean     4222 Dec 16 03:40 index_flanders.html
> -rwxr-xr-x   1 bsergean bsergean      571 Dec 12 20:32 redirection.html
> -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:17 svn-commit.2.tmp~
> -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:16 svn-commit.tmp
> -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:16 svn-commit.tmp~
> -rw-r--r--   1 bsergean bsergean      353 Nov  8  2005 vpn.pcf
> -rw-r--r--   1 bsergean bsergean      355 Nov  8  2005 vpn.pcf~
> '226-Transfer complete.\n226 Down: 0 Files (0mb)  Up: 0 Files (0mb)
> 10,000,000:1  CR: LEECH'
>>>> ftp.retrlines('LIST -a')
> drwxr-xr-x   5 bsergean bsergean      728 Dec 26 20:28 .
> drwxr-xr-x 115 bsergean bsergean     7992 Dec 27 21:29 ..
> -rw-r--r--   1 bsergean bsergean     2471 Nov 17 19:43 .addressbook
> -rw-r--r--   1 bsergean bsergean     3100 Dec 12 23:16 .bash_profile
> -rw-r--r--   1 bsergean bsergean     3074 Dec 12 20:32 .bash_profile~
> -rw-r--r--   1 bsergean bsergean     1287 Dec 16 03:40 .bashrc_adm424
> -rw-r--r--   1 bsergean bsergean     4019 Dec 26 20:28 .bashrc_bsergean
> -rw-r--r--   1 bsergean bsergean     4019 Dec 26 18:43 .bashrc_bsergean~
> lrwxrwxrwx   1 bsergean bsergean       16 Nov 21 19:52 .bashrc_build
> -> .bashrc_bsergean
> lrwxrwxrwx   1 bsergean bsergean       16 Dec 12 22:36 .bashrc_ttf ->
> .bashrc_bsergean
> drwxr-xr-x   4 bsergean bsergean      872 Nov 16 21:27 .emacs
> -rw-r--r--   1 bsergean bsergean     3229 Dec 12 22:36 .emacs.el
> -rw-r--r--   1 bsergean bsergean      613 Nov 17 19:43 .fetchmailrc
> -rw-r--r--   1 bsergean bsergean    19847 Nov 17 19:43 .pinerc
> -rw-r--r--   1 bsergean bsergean     1706 Dec 16 03:40 .procmailrc
> drwxr-xr-x   7 bsergean bsergean      296 Dec 19 01:05 .svn
> drwxr-xr-x   3 bsergean bsergean      176 Dec 16 03:40 httpd
> -rwxr-xr-x   1 bsergean bsergean     4222 Dec 16 03:40 index_flanders.html
> -rwxr-xr-x   1 bsergean bsergean      571 Dec 12 20:32 redirection.html
> -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:17 svn-commit.2.tmp~
> -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:16 svn-commit.tmp
> -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:16 svn-commit.tmp~
> -rw-r--r--   1 bsergean bsergean      353 Nov  8  2005 vpn.pcf
> -rw-r--r--   1 bsergean bsergean      355 Nov  8  2005 vpn.pcf~
> '226-Transfer complete.\n226 Down: 0 Files (0mb)  Up: 0 Files (0mb)
> 10,000,000:1  CR: LEECH'
>
> By default ftputils doesn't use that.
> I tried hard to understand the code but didn't see any DIR or try.

ftputil internally uses so-called "sessions" which by
default are ftplib.FTP instances. For directory listings,
ftputil.FTPHost.listdir calls ftplib.FTP.dir, which in turn
calls FTP's LIST command without arguments or options.

> The closest I found was
>
> def _dir(self, path):
> in the ftputil function ...

_dir is called by various methods, e. g. FTPHost.listdir, but
also implicitly by stat, isdir and lots more.

The RFC 959 which defines the file transfer protocol tells about
LIST but says nothing about a parameter -a , so the effect of
this option probably depends on the server or its configuration.
A LIST -a _may_ also cause an error and thus an exception, I
assume. Therefore, I'm not sure if the complexity of trying
LIST -a and possibly falling back to a plain LIST justifies the
result of having dirs and files with leading dots included.

Even more important may the backward incompatibility if LIST -a
is/would be used by default and lists files which "weren't there"
with "previous" versions of ftputil. The answer to that may be to
make the usage of "LIST -a" optional, which again adds complexity
and design decisions regarding if LIST -a should also be used for
stat calls and such.

What do the others think?

> And here is a mirror tool I'm working on:
>
> import logging
> logger = logging.getLogger("ftpUploader")
> logger.setLevel(logging.DEBUG)
> #create console handler and set level to debug
> ch = logging.StreamHandler()
> ch.setLevel(logging.DEBUG)
> #create formatter
> formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s
> - %(message)s")
> #add formatter to ch
> ch.setFormatter(formatter)
> #add ch to logger
> logger.addHandler(ch)
>
> from ftputil import FTPHost
> from glob import glob
>
> class ftpUploader(FTPHost):
>
>     def exists(self, fn):
>         ''' Test whether file exist in the current remote dir '''
>         return basename(fn) in self.listdir('.')
>
>     def mirror(self, dirname):
>         logger.info('mirror (%s)' % dirname)
>         remoteDirname = basename(dirname)
>
>         if self.exists(remoteDirname):
>             logger.info('Remove dir %s' % remoteDirname)
>             self.rmdir(remoteDirname)
>
>         logger.info('Create remote dir %s' % remoteDirname)
>         self.mkdir(remoteDirname)
>
>         logger.info('chdir to remote dir %s' % remoteDirname)
>         self.chdir(remoteDirname)
>
>         walkdir = dirname
>         for path, subdirs, files in os.walk(walkdir):
>             print path, subdirs, files
>
>             os.chdir(path)
>
>             if path != walkdir:
>                 self.chdir(path.split(walkdir + os.sep)[1])
>
>             for d in subdirs:
>                 if not self.exists(d):
>                     self.mkdir(d)
>
>             for f in files:
>                 self.upload(f, f)
>
> ftp = ftpUploader('localhost', 'bsergean', 'secret')

I don't know if you sent this in response to issue #6 at
http://ftputil.sschwarzer.net/trac/ticket/6 :-) If you assert
that I can use your code under the revised BSD license used by
ftputil (see http://www.opensource.org/licenses/bsd-license.php ),
I'll look into your code. :)

> Hope you can do some stuff with that,

Thanks again!

Best wishes
Stefan