[ftputil] host.download_if_newer problems

[Stefan Schwarzer]

Hi Yvan,

On 2008-05-22 08:54, Yvan Strahm wrote:
> Yes, not problem for sending the mails to the list I thought the reply
> went to the mailing list, sorry.

No problem, this has happened to almost anybody, I guess. :)

> I guess i will just compare the date of the compressed file on the
> server with the date of the uncompressed file on the client, then if the
> server file is newer then it will be downloaded.

Which timestamp will the uncompressed file have after
uncompressing? The timestamp of the original file before it was
compressed or the date and time of the uncompression process (i.
e. when the uncompressed file was generated)? Probably none of
these timestamps is that of the compressed file on the server (or
is it?). Be careful not to compare apples and oranges. You may
set the timestamp of the uncompressed local file to the timestamp
of the corresponding compressed file on the server though.

> This is how I thought it can be done
>
> 1. getting a dictionary (key:filename ,value:date) from the server:
>
> 	host=ftputil.FTPHost(ftp,user,password)
> 	host.chdir(ftp_dir)
> 	files=host.listdir(host.curdir)
> 	for f in files:
> 		mtime=host.path.getmtime(f)
> 		dict={f:mtime}

Here, you overwrite the dict every time. Did you actually write
"dict[f] = mtime"?

By the way, it's error-prone to introduce a name which is the
same as a Python builtin (in this case, "dict"). Unless you use
some trick, this will prevent you from using the builtin dict
object until your object's name goes out of scope or the local
name is deleted.

> 	close(ftp)
>
> 2. getting a equivalent dictionary from the client

Again, think of which timestamps you store and compare.

> 3. comparing both list
> 	if not present on the client, add the file to a list
> 	if server date > client date, add the file to a list

Seems sensible.

> 4. reopen ftp and download the list to_be_download
>
> but it doesn't work at all!

Can you show some output to demonstrate the problem(s)?

Does your code work for small directories and becomes only
slow for large directories? Or does it even fail when small
directories are involved?

> Problems are size of the dictionaries, more than 50'000 files

How much RAM can you use on the client? Even if you use 100 bytes
per dictionary entry, this will amount to 5 MB which is not much
nowadays.

Perhaps the bottleneck is elsewhere. If you have many files in
the directory it may be sensible to increase ftputil's stat cache
size (see the documentation), so ftputil will have to fetch the
directory listing from the server only once.

> and how to compare efficiently files. And the main problem is the ftp
> connection dying.

Do you mean dying in the sense that the server encounters a
timeout and closes the connection "consciously" or do you speak
of an instable connection where the time until it breaks varies
rather randomly?

Please explain a bit more in which way your code fails (see
above), and we might be able to get to a working solution. :-)

Best regards,
Stefan