[ftputil] Stat'ing a Whole Directory

[Dan Milstein]

a) BSD license -- would be delighted.  Very happy to contribute.

b) My code -- I'll clean up what I've got and post it today or tomorrow.

c) Low-level/high-lvel

A thought: yes, returning a list of stat results feels a bit low- 
level.  Esp since the reason I need the stat results and not just the  
names is to do things like isfile() and isdir(), which I can then  
only do by explicitly calling stat.S_ISDIR(stat_result.st_mode).  Blech.

What might be nice, but would move just a bit away from the os.path  
analogy, would be to make ftphost.path be able to run isfile()/isdir 
()/etc on both path names and also FTPFile objects (or something like  
FTPFile objects -- see below).

Would that be something you might be interested in seeing?  I'd be  
happy to write it.  Before I dive in, I wanted to see if you (or  
anyone else on the list) thinks that might belong in the distribution.

If you are interested, what I'd like to do is add a method to ftphost:

getfiles(path)

Which returns a list of FTPFile-like objects (with stat and path  
information hidden in them).  They would lazily support the file  
interface, but would only connect to the remote server if someone  
actually reads or writes their contents.  And then I would extend  
ftp_path.isdir()/isfile()/etc to handle those objects as well as path  
names.

So that, if you need to operate on a directory, to say, copy all  
files, and recurse over all subdirectories, you could do something like:

def copy_all_files(path):
     for f in ftphost.getfiles(path):
         if ftphost.path.isdir(f):
            copy_all_files(f.path)
         else:
            ftphost.copyfileobj(f, file(f.name))

Which feels fairly clean and intuitive, and would behave pretty  
nicely in terms of how it hits the remote server (whereas, with the  
current implementation, the lib is fetching the entire directory  
listing over and over again for every call to isdir()).

Does that make sense?  If not, should I just code it up so you can  
play with it? ;-)

-Dan

On Jul 19, 2006, at 6:42 PM, Stefan Schwarzer wrote:

> Hi Dan,
>
> On 2006-07-19 21:57, Dan Milstein wrote:
>> I've been using the lovely ftputil lib, and, basically, it's fab.
>
> thanks :-)
>
>> However,
>
> aah, now ... ;-)
>
>> for my app, I need to produce detailed directory listings
>> for the files on the remote server.  With the current interface, that
>> means getting the list of all files (one remote call), and then
>> fetching that entire list over and over in order to stat each
>> individual file (a remote call for every file).
>>
>> There's already a ticket for caching stat results, but what works
>> better for me is to add a new call:
>>
>> host.statdir(path)
>>
>> (or host.lstatdir(path), if that makes more sense -- I'm not the most
>> perfectly clear on how symbolic links interact with ftp).
>>
>> The call returns a list of stat results, making a single remote call
>> and then parsing the entire list.  This gets around the need for
>> caching (which I'd prefer to avoid, because of synchronization
>> issues), and has very nice performance.
>
> This is a good idea! Nonetheless I'm a bit hesitant to use it in
> ftputil, because this is somewhat "low-level", compared with the
> usual ftputil mindset.
>
> I'm _very_ interested what the others on the list think!
>
>> I've actually patched ftputil and ftp_stat with the necessary code to
>> do this.  I'd be happy to send along patches for those files, or just
>> include the code in an email.
>
> _Please_ send the patch(es) to the mailing list(*), so anyone who
> wants to try the code (and how its usage "feels") can do it. If
> you can, make a single patch for all affected files. I'm not 100%
> sure if the list accepts attachments. If your mail bounces,
> please include the patch into the normal text and resend it.
>
> (*) Before you post, think about whether you can and want to put
> the code under the BSD license that ftputil uses. Strictly
> speaking, this is only relevant for inclusion of the code into
> ftputil but I also read the list, of course, and I wouldn't want
> to accidently put code from my memory into ftputil which doesn't
> belong there.
>
> Best wishes
> Stefan