From rick at kromedesignstudio.com  Tue Apr  5 19:18:25 2011
From: rick at kromedesignstudio.com (Krome Design Studio)
Date: Tue, 5 Apr 2011 10:18:25 -0700
Subject: [ftputil] ftp download issue
Message-ID: <76103FAA-5B52-4A37-A4CC-7BDEFA77072A@kromedesignstudio.com>

Hi All,

I am having an issue trying to download files from our remote FTP  
server using ftputils-2.2.3.

  In local development, we can connect to the remote FTP server and  
download the files to our local box, no problem. When we deploy the  
codebase to our development webserver and try, the code just hangs,  
throwing no exception. Our webhosting company, Webfaction, says it  
doesn't look like an issue from their end.

Basically the user selects the required files from the page via  
checkboxes ('download' parameter).

Code that downloads the files:

c = Context({})
     if request.method == 'POST' and request.is_ajax():

         cloud_user = get_object_or_404(User,username=username)
         user_profile = UserProfile.objects.get(user=cloud_user)
         data = {}
         client =  
FTPHost 
(settings 
.CLOUD_SERVER,user_profile.cloud_username,user_profile.cloud_password)
         adir = os.path.dirname(user_profile.download_dir)
         if not os.path.isabs(adir):
             adir = ('%s%s' % ("/",adir))

         if os.path.isdir(adir):

             downloads = request.POST.getlist('download')
             for download in downloads:

                 filename = os.path.split(download)[1]
                 localfile = os.path.join(adir,filename)
                 client.download(download, localfile)

             data['outcome'] = True
             data['results_text'] = ('%s%s' % ("All files ",  
"successfully downloaded."))

             client.close()


         else:

             data['outcome'] = False
             data['results_text'] = ('%s%s%s' % ("The download  
directory ",user_profile.download_dir, " does not exist on your local  
system. No files downloaded."))

         return _json_response(data)



     return render_to_response('cloud/directory.html', c,  
context_instance=RequestContext(request))

Any help much appreciated!

Cheers
Rick

From sschwarzer at sschwarzer.net  Thu Apr  7 20:51:15 2011
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Thu, 07 Apr 2011 20:51:15 +0200
Subject: [ftputil] Fwd: Re: throttled upload with ftputil?
Message-ID: <4D9E07A3.5080809@sschwarzer.net>

Hello,

Here's an application of the callback parameter for
uploads and downloads which I didn't think of when
I added callbacks.

Forwarded with consent from Alec. :)

Stefan

-------- Original Message --------
Subject: Re: throttled upload with ftputil?
Date: Sun, 27 Mar 2011 11:28:56 +0200
From: Stefan Schwarzer <sschwarzer at sschwarzer.net>
To: Alec Bennett <wrybread at gmail.com>

Hello Alec,

Thanks for your feedback!

On 2011-03-27 06:53, Alec Bennett wrote:
> I was wondering if its possible to throttle the upload speed when uploading
> a file through ftputil?
> 
> I'm currently uploading like this, if its a factor:
> 
> ftp = ftputil.FTPHost(hostname, username, password)
> ftp.upload(fname, target, mode = "b")

ftputil doesn't have a parameter to specifically manipulate
the upload or download speed. Usually such limiting is done
on the operating system level.

That said, I guess you could achieve something useful by
using a callback parameter [1] in the `upload` method.
(Callbacks were added in ftputil 2.5.) I think you could
just use

  import time

  import ftputil

  def throttle_callback(chunk):
      """Insert some sleep time to decrease the transfer speed."""
      # Use whatever interval you find appropriate.
      time.sleep(1)

  ftp = ftputil.FTPHost(hostname, username, password)
  ftp.upload(fname, target, mode='b', callback=throttle_callback)

If you need more control, e. g. want to throttle down to a
specific transfer rate, you could count the bytes
transferred so far (the sum of the lengths of the chunks)
and the elapsed time (`time.time()` minus start time of
transfer) and insert corresponding sleep intervals if the
rate is too high. If you need more concrete help, let me
know.

> And thanks for the spectacular module, by the way.

Thanks! You're welcome. :)

Although you wrote me in private, I think the question and
the approach to it are interesting. I had never had such an
application in mind when I added the callback support. I'd
like to forward this e-mail to the ftputil mailing list.
Would that be ok for you?

[1] http://ftputil.sschwarzer.net/trac/wiki/Documentation#uploading-and-downloading-files

Stefan

From sschwarzer at sschwarzer.net  Thu Apr  7 20:48:28 2011
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Thu, 07 Apr 2011 20:48:28 +0200
Subject: [ftputil] ftp download issue
In-Reply-To: <76103FAA-5B52-4A37-A4CC-7BDEFA77072A@kromedesignstudio.com>
References: <76103FAA-5B52-4A37-A4CC-7BDEFA77072A@kromedesignstudio.com>
Message-ID: <4D9E06FC.7050303@sschwarzer.net>

Hi Rick,

On 2011-04-05 19:18, Krome Design Studio wrote:
> I am having an issue trying to download files from our remote FTP  
> server using ftputils-2.2.3.
> 
>   In local development, we can connect to the remote FTP server and  
> download the files to our local box, no problem. When we deploy the  
> codebase to our development webserver and try, the code just hangs,  
> throwing no exception. Our webhosting company, Webfaction, says it  
> doesn't look like an issue from their end.
> [...]

Before we dive further into this, I'd recommend two things:

- Try to isolate the FTP stuff in its own script. In other
  words, try to reproduce the problem without all the other
  network stuff seemingly going on. Perhaps the latter is
  the actual cause of the hangs. Try to use the same host
  configuration as in the context of your web service.

- Do the above with the latest ftputil release if at all
  possible. From time to time there are changes to make
  ftputil more robust, and ftputil 2.2.3 is almost four(!)
  years old by now. I wouldn't like tracking down the
  problem only to find out it has been fixed in a newer
  ftputil release. Tarballs of ftputil can be found at
  http://ftputil.sschwarzer.net/trac/wiki/Download .

Stefan

From zechs.marquie at gmail.com  Sun May  1 18:48:44 2011
From: zechs.marquie at gmail.com (Niall OConnor)
Date: Sun, 1 May 2011 17:48:44 +0100
Subject: [ftputil] FTPOSError
Message-ID: <BANLkTikxd_1eYuzoEVoXnw9-sdHpB6SLsw@mail.gmail.com>

I get this error with ftputil.  I thought it was because the version (
ftp2.6) was too great for python 2.5.2 so I installed ftp2.5 .  I still had
no luck.  Can you tell me waht might be wrong here?

Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ftputil
>>> pem_user = {'PEM_password': xxx', 'PEM_account_id': u'xx',
'domain_name': xxx, 'FTP_login': xxx', 'FTP_password': xxx',
'FTP_host_name': xxx, 'PEM_password_confirm': xxx', 'PEM_subscription_id':
xxx', 'PEM_login_id': xxx', 'domain_id': 1}
>>> host = ftputil.FTPHost(pem_user['FTP_host_name'], pem_user['FTP_login'],
pem_user['FTP_password'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-i686/egg/ftputil/ftputil.py", line 104, in
__init__
  File "build/bdist.linux-i686/egg/ftputil/ftputil.py", line 158, in
_make_session
  File "build/bdist.linux-i686/egg/ftputil/ftp_error.py", line 149, in
_try_with_oserror
ftputil.ftp_error.FTPOSError: -2
Debugging info: ftputil 2.6, Python 2.5.2 (linux2)
>>>
KeyboardInterrupt
>>>
vivien:/home/www/sitebuilder.blacknight.ie/web# pip uninstall ftputil
Uninstalling ftputil:
  /usr/lib/python2.5/site-packages/ftputil-2.6-py2.5.egg
Proceed (y/n)? y
  Successfully uninstalled ftputil
vivien:/home/www/sitebuilder.blacknight.ie/web# python
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ftputil
>>> pem_user = {'PEM_password': xxx', 'PEM_account_id': u'xx',
'domain_name': xxx, 'FTP_login': xxx', 'FTP_password': xxx',
'FTP_host_name': xxx, 'PEM_password_confirm': xxx', 'PEM_subscription_id':
xxx', 'PEM_login_id': xxx', 'domain_id': 1}
>>> host = ftputil.FTPHost(pem_user['FTP_host_name'], pem_user['FTP_login'],
pem_user['FTP_password'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site-packages/ftputil/ftputil.py", line 103, in
__init__
    self._session = self._make_session()
  File "/usr/lib/python2.5/site-packages/ftputil/ftputil.py", line 157, in
_make_session
    return ftp_error._try_with_oserror(factory, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/ftputil/ftp_error.py", line 149, in
_try_with_oserror
    raise FTPOSError(*exc.args)
ftputil.ftp_error.FTPOSError: -2
Debugging info: ftputil 2.5, Python 2.5.2 (linux2)
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20110501/0b9a6b6b/attachment.htm 

From sschwarzer at sschwarzer.net  Sun May  1 20:34:09 2011
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Sun, 01 May 2011 20:34:09 +0200
Subject: [ftputil] FTPOSError
In-Reply-To: <BANLkTikxd_1eYuzoEVoXnw9-sdHpB6SLsw@mail.gmail.com>
References: <BANLkTikxd_1eYuzoEVoXnw9-sdHpB6SLsw@mail.gmail.com>
Message-ID: <4DBDA7A1.8090609@sschwarzer.net>

Hi Niall,

On 2011-05-01 18:48, Niall OConnor wrote:
> I get this error with ftputil.  I thought it was because the version (
> ftp2.6) was too great for python 2.5.2 so I installed ftp2.5 .

ftputil 2.6 is supposed to work with Python 2.4 to 2.7, so
downgrading shouldn't be necessary.

> I still had
> no luck.  Can you tell me waht might be wrong here?
>
> Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14)
> [GCC 4.3.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import ftputil
>>>> pem_user = {'PEM_password': xxx', 'PEM_account_id': u'xx',
> 'domain_name': xxx, 'FTP_login': xxx', 'FTP_password': xxx',
> 'FTP_host_name': xxx, 'PEM_password_confirm': xxx', 'PEM_subscription_id':
> xxx', 'PEM_login_id': xxx', 'domain_id': 1}
>>>> host = ftputil.FTPHost(pem_user['FTP_host_name'], pem_user['FTP_login'],
> pem_user['FTP_password'])
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "build/bdist.linux-i686/egg/ftputil/ftputil.py", line 104, in
> __init__
>   File "build/bdist.linux-i686/egg/ftputil/ftputil.py", line 158, in
> _make_session
>   File "build/bdist.linux-i686/egg/ftputil/ftp_error.py", line 149, in
> _try_with_oserror
> ftputil.ftp_error.FTPOSError: -2
> Debugging info: ftputil 2.6, Python 2.5.2 (linux2)

I can't exactly tell you what's wrong, but at least the
traceback shows that an error occurs when instantiating the
session instance (in this case an instance of ftplib.FTP)
and that the exception raised by ftplib.FTP is neither an
ftplib.error_temp nor an ftplib.error_perm. (These
exceptions are described at
http://docs.python.org/library/ftplib.html#ftplib.error_reply ).

The exception handling in ftp_error in line 147 catches
ftplib.all_errors which, according to the ftplib
documentation, consists of error_reply, error_temp,
error_perm, error_proto, socket.error and IOError.
error_temp and error_perm are handled in lines 139-146 of
ftp_error.py, so the actual exception raised by ftplib.FTP
is one of error_reply, error_proto, socket.error and
IOError.

Could you please do the following:

- Use the ftputil.FTPHost constructor as above but with the
  literal values instead of getting the values from the
  dictionary with pem_user['FTP_host_name'] etc. Do you get
  the same traceback?

- If yes, try to instantiate an ftplib.FTP object with the
  same data you use for logging in with ftputil.FTPHost
  (also with the literal values). If you get an exception,
  please post the traceback you get.

Stefan