[ftputil] mirror script + listing files with dot inside

Benjamin Sergeant bsergean at gmail.com
Thu Dec 28 02:18:14 CET 2006 

> Hi Benjamin,
>
> Thanks a lot for your contributions.
>

You're very welcome !

> On 2006-12-27 23:12, Benjamin Sergeant wrote:
> > http://www.unixwiz.net/techtips/wget-dotfiles.html
> > Here they explain how to get the . files : one have to issue the LIST
> > with -a as a parameter (as for ls)
>
> As far as I know, whether a directory listing contains files with
> a leading dot depends on the FTP server configuration.
>
> >>>> ftp.retrlines('LIST')
>
> ftp is an ftplib.FTP instance here, isn't it?

It is indeed.

> > drwxr-xr-x   3 bsergean bsergean      176 Dec 16 03:40 httpd
> > -rwxr-xr-x   1 bsergean bsergean     4222 Dec 16 03:40 index_flanders.html
> > -rwxr-xr-x   1 bsergean bsergean      571 Dec 12 20:32 redirection.html
> > -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:17 svn-commit.2.tmp~
> > -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:16 svn-commit.tmp
> > -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:16 svn-commit.tmp~
> > -rw-r--r--   1 bsergean bsergean      353 Nov  8  2005 vpn.pcf
> > -rw-r--r--   1 bsergean bsergean      355 Nov  8  2005 vpn.pcf~
> > '226-Transfer complete.\n226 Down: 0 Files (0mb)  Up: 0 Files (0mb)
> > 10,000,000:1  CR: LEECH'
> >>>> ftp.retrlines('LIST -a')
> > drwxr-xr-x   5 bsergean bsergean      728 Dec 26 20:28 .
> > drwxr-xr-x 115 bsergean bsergean     7992 Dec 27 21:29 ..
> > -rw-r--r--   1 bsergean bsergean     2471 Nov 17 19:43 .addressbook
> > -rw-r--r--   1 bsergean bsergean     3100 Dec 12 23:16 .bash_profile
> > -rw-r--r--   1 bsergean bsergean     3074 Dec 12 20:32 .bash_profile~
> > -rw-r--r--   1 bsergean bsergean     1287 Dec 16 03:40 .bashrc_adm424
> > -rw-r--r--   1 bsergean bsergean     4019 Dec 26 20:28 .bashrc_bsergean
> > -rw-r--r--   1 bsergean bsergean     4019 Dec 26 18:43 .bashrc_bsergean~
> > lrwxrwxrwx   1 bsergean bsergean       16 Nov 21 19:52 .bashrc_build
> > -> .bashrc_bsergean
> > lrwxrwxrwx   1 bsergean bsergean       16 Dec 12 22:36 .bashrc_ttf ->
> > .bashrc_bsergean
> > drwxr-xr-x   4 bsergean bsergean      872 Nov 16 21:27 .emacs
> > -rw-r--r--   1 bsergean bsergean     3229 Dec 12 22:36 .emacs.el
> > -rw-r--r--   1 bsergean bsergean      613 Nov 17 19:43 .fetchmailrc
> > -rw-r--r--   1 bsergean bsergean    19847 Nov 17 19:43 .pinerc
> > -rw-r--r--   1 bsergean bsergean     1706 Dec 16 03:40 .procmailrc
> > drwxr-xr-x   7 bsergean bsergean      296 Dec 19 01:05 .svn
> > drwxr-xr-x   3 bsergean bsergean      176 Dec 16 03:40 httpd
> > -rwxr-xr-x   1 bsergean bsergean     4222 Dec 16 03:40 index_flanders.html
> > -rwxr-xr-x   1 bsergean bsergean      571 Dec 12 20:32 redirection.html
> > -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:17 svn-commit.2.tmp~
> > -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:16 svn-commit.tmp
> > -rw-r--r--   1 bsergean bsergean       69 Dec 12 23:16 svn-commit.tmp~
> > -rw-r--r--   1 bsergean bsergean      353 Nov  8  2005 vpn.pcf
> > -rw-r--r--   1 bsergean bsergean      355 Nov  8  2005 vpn.pcf~
> > '226-Transfer complete.\n226 Down: 0 Files (0mb)  Up: 0 Files (0mb)
> > 10,000,000:1  CR: LEECH'
> >
> > By default ftputils doesn't use that.
> > I tried hard to understand the code but didn't see any DIR or try.
>
> ftputil internally uses so-called "sessions" which by
> default are ftplib.FTP instances. For directory listings,
> ftputil.FTPHost.listdir calls ftplib.FTP.dir, which in turn
> calls FTP's LIST command without arguments or options.
>

Thanks for the explanaition.

> > The closest I found was
> >
> > def _dir(self, path):
> > in the ftputil function ...
>
> _dir is called by various methods, e. g. FTPHost.listdir, but
> also implicitly by stat, isdir and lots more.
>
> The RFC 959 which defines the file transfer protocol tells about
> LIST but says nothing about a parameter -a , so the effect of
> this option probably depends on the server or its configuration.
> A LIST -a _may_ also cause an error and thus an exception, I
> assume. Therefore, I'm not sure if the complexity of trying
> LIST -a and possibly falling back to a plain LIST justifies the
> result of having dirs and files with leading dots included.
>
> Even more important may the backward incompatibility if LIST -a
> is/would be used by default and lists files which "weren't there"
> with "previous" versions of ftputil. The answer to that may be to
> make the usage of "LIST -a" optional, which again adds complexity
> and design decisions regarding if LIST -a should also be used for
> stat calls and such.
>

I understand that it's non standard and not part of the RFC ...

The other problem I got was this one:
When trying to remove a dir with rmtree, if the dir contains files
with dot inside the rmdir fails since the dir is not really empty.

Benjamin.

ps:
You can of course use the small mirror function code under the new BSD
license (it will be probably released under GPL in my pet project,
http://code.google.com/p/pytof/


> What do the others think?
>
> > And here is a mirror tool I'm working on:
> >
> > import logging
> > logger = logging.getLogger("ftpUploader")
> > logger.setLevel(logging.DEBUG)
> > #create console handler and set level to debug
> > ch = logging.StreamHandler()
> > ch.setLevel(logging.DEBUG)
> > #create formatter
> > formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s
> > - %(message)s")
> > #add formatter to ch
> > ch.setFormatter(formatter)
> > #add ch to logger
> > logger.addHandler(ch)
> >
> > from ftputil import FTPHost
> > from glob import glob
> >
> > class ftpUploader(FTPHost):
> >
> >     def exists(self, fn):
> >         ''' Test whether file exist in the current remote dir '''
> >         return basename(fn) in self.listdir('.')
> >
> >     def mirror(self, dirname):
> >         logger.info('mirror (%s)' % dirname)
> >         remoteDirname = basename(dirname)
> >
> >         if self.exists(remoteDirname):
> >             logger.info('Remove dir %s' % remoteDirname)
> >             self.rmdir(remoteDirname)
> >
> >         logger.info('Create remote dir %s' % remoteDirname)
> >         self.mkdir(remoteDirname)
> >
> >         logger.info('chdir to remote dir %s' % remoteDirname)
> >         self.chdir(remoteDirname)
> >
> >         walkdir = dirname
> >         for path, subdirs, files in os.walk(walkdir):
> >             print path, subdirs, files
> >
> >             os.chdir(path)
> >
> >             if path != walkdir:
> >                 self.chdir(path.split(walkdir + os.sep)[1])
> >
> >             for d in subdirs:
> >                 if not self.exists(d):
> >                     self.mkdir(d)
> >
> >             for f in files:
> >                 self.upload(f, f)
> >
> > ftp = ftpUploader('localhost', 'bsergean', 'secret')
>
> I don't know if you sent this in response to issue #6 at
> http://ftputil.sschwarzer.net/trac/ticket/6 :-) If you assert
> that I can use your code under the revised BSD license used by
> ftputil (see http://www.opensource.org/licenses/bsd-license.php ),
> I'll look into your code. :)
>
> > Hope you can do some stuff with that,
>
> Thanks again!
>
> Best wishes
> Stefan
>
>

More information about the ftputil mailing list