From peter at pjstirling.plus.com  Thu Apr  2 07:40:14 2009
From: peter at pjstirling.plus.com (Peter Stirling)
Date: Thu, 02 Apr 2009 06:40:14 +0100
Subject: [ftputil] ftputil 2.4 with python 2.6.1
Message-ID: <1238650814.7579.61.camel@peter-desktop>

Hi,

I've been using ftputil to perform the upload part of a offline web-page
generation program (templating without requiring the server to cgi).

When I run it on python-2.5.2 (the default for ubuntu-intrepid) it works
as expected, however when I run it on Windows (python-2.6.1 using the
installer) I get an exception:

Traceback (most recent call last):
  File "g:\test\test.py", line 13, in <module>
    host.synchronize_times()
  File "g:\test\ftputil\ftputil.py", line 896, in __exit__
    self.close()
  File "g:\test\ftputil\ftputil.py", line 253, in close
    self.stat_cache.clear()
  File "g:\test\ftputil\ftp_stat_cache.py", line 117, in clear
    self.resize(0)
  File "g:\test\ftputil\ftp_stat_cache.py", line 100, in resize
    self._cache.size = new_size
  File "g:\test\ftputil\lrucache.py", line 186, in __setattr__
    del self.__dict[lru.key]
KeyError: '/htdocs/new/_ftputil_sync_'
done

I've attached a minimal program that demonstrates the problem (you'll
need to edit it for login credentials).

Interestingly, I don't get the exception when running on python-2.6 on
mac os X (fink package manager), but I do get a deprecation warning for:

ftputil/ftp_stat.py:58: DeprecationWarning: object.__init__() takes no
parameters
  super(StatResult, self).__init__(sequence)


-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.py
Type: text/x-python
Size: 296 bytes
Desc: not available
Url : http://codespeak.net/pipermail/ftputil/attachments/20090402/61665f69/attachment.py 

From sschwarzer at sschwarzer.net  Sun Apr  5 22:08:48 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Sun, 05 Apr 2009 22:08:48 +0200
Subject: [ftputil] ftputil 2.4 with python 2.6.1
In-Reply-To: <1238650814.7579.61.camel@peter-desktop>
References: <1238650814.7579.61.camel@peter-desktop>
Message-ID: <49D90FD0.9010800@sschwarzer.net>

Hello Peter,

Thanks a lot for your feedback!

On 2009-04-02 07:40, Peter Stirling wrote:
> I've been using ftputil to perform the upload part of a offline web-page
> generation program (templating without requiring the server to cgi).
> 
> When I run it on python-2.5.2 (the default for ubuntu-intrepid) it works
> as expected, however when I run it on Windows (python-2.6.1 using the
> installer) I get an exception:
> [...]

I have been able to reproduce your problem: It occurs here on
Windows XP but not on Ubuntu Linux (both scenarios with
Python 2.6). However, until now I haven't found the bug though
narrowed it down somewhat.

> Interestingly, I don't get the exception when running on python-2.6 on
> mac os X (fink package manager), but I do get a deprecation warning for:
> 
> ftputil/ftp_stat.py:58: DeprecationWarning: object.__init__() takes no
> parameters
>   super(StatResult, self).__init__(sequence)

I found the reason for the warning and removed the cause. I also
changed some code to remove another deprecation warning that
showed up during the unit tests when running under Python 2.6.
The current Subversion repository includes those changes already.

I'm very interested in finding out what the cause of the
traceback you got is.

Best regards,
Stefan

From peter at pjstirling.plus.com  Thu Apr  9 01:44:24 2009
From: peter at pjstirling.plus.com (Peter Stirling)
Date: Thu, 09 Apr 2009 00:44:24 +0100
Subject: [ftputil]  ftputil 2.4 with python 2.6.1
Message-ID: <1239234264.7031.253.camel@peter-desktop>

After further inspection it appears that the windows python2.6 install doesn't
include a dll of the compiled heapq module 

	/usr/lib/python2.5/lib-dynload/_heapq.so, on ubuntu python2.5

However when I tried to test the pure python version it decided to work, so there
must be a bad dll somewhere.

To test this I made a copy of the pure python heapq.py as heapq26.py, and commented
out the code that replaces pure-python with the c versions (line 313). I then altered
ftputil/lrucache.py(line 43) to import from this new module, and when I run the test
program I no longer get the exception.

I am very confused.

> Hello Peter,
> 
> Thanks a lot for your feedback!
> 
> On 2009-04-02 07:40, Peter Stirling wrote:
> > I've been using ftputil to perform the upload part of a offline web-page
> > generation program (templating without requiring the server to cgi).
> > 
> > When I run it on python-2.5.2 (the default for ubuntu-intrepid) it works
> > as expected, however when I run it on Windows (python-2.6.1 using the
> > installer) I get an exception:
> > [...]
> 
> I have been able to reproduce your problem: It occurs here on
> Windows XP but not on Ubuntu Linux (both scenarios with
> Python 2.6). However, until now I haven't found the bug though
> narrowed it down somewhat.
> 
> > Interestingly, I don't get the exception when running on python-2.6 on
> > mac os X (fink package manager), but I do get a deprecation warning for:
> > 
> > ftputil/ftp_stat.py:58: DeprecationWarning: object.__init__() takes no
> > parameters
> >   super(StatResult, self).__init__(sequence)
> 
> I found the reason for the warning and removed the cause. I also
> changed some code to remove another deprecation warning that
> showed up during the unit tests when running under Python 2.6.
> The current Subversion repository includes those changes already.
> 
> I'm very interested in finding out what the cause of the
> traceback you got is.
> 
> Best regards,
> Stefan


From sschwarzer at sschwarzer.net  Thu Apr  9 07:09:36 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Thu, 09 Apr 2009 07:09:36 +0200
Subject: [ftputil] ftputil 2.4 with python 2.6.1
In-Reply-To: <1239234264.7031.253.camel@peter-desktop>
References: <1239234264.7031.253.camel@peter-desktop>
Message-ID: <49DD8310.6040901@sschwarzer.net>

Hi Peter,

I've entered the issue at
http://ftputil.sschwarzer.net/trac/ticket/32

You can edit it if you log in as user ftputiluser with password
ftputil (this is for spam prevention).

On 2009-04-09 01:44, Peter Stirling wrote:
> After further inspection it appears that the windows python2.6
> install doesn't include a dll of the compiled heapq module 
> 
> 	/usr/lib/python2.5/lib-dynload/_heapq.so, on ubuntu python2.5
> 
> However when I tried to test the pure python version it decided
> to work, so there must be a bad dll somewhere.
> 
> To test this I made a copy of the pure python heapq.py as
> heapq26.py, and commented out the code that replaces
> pure-python with the c versions (line 313). I then altered
> ftputil/lrucache.py(line 43) to import from this new module,
> and when I run the test program I no longer get the exception.

Thanks for working further on the matter! :-) This makes my life
a bit easier. By now, I also had actually suspected the Python
installation itself. I'll add more information to the bug tracker
after more thorough investigation. (I'd like to add some code to
lrucache to see more clearly if my assumptions from the debugging
session are true.)

Best regards,
Stefan

From sschwarzer at sschwarzer.net  Fri Apr 10 12:16:03 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Fri, 10 Apr 2009 12:16:03 +0200
Subject: [ftputil] ftputil 2.4 with python 2.6.1
In-Reply-To: <49DD8310.6040901@sschwarzer.net>
References: <1239234264.7031.253.camel@peter-desktop>
	<49DD8310.6040901@sschwarzer.net>
Message-ID: <49DF1C63.1070607@sschwarzer.net>

Hello all,

On 2009-04-09 07:09, Stefan Schwarzer wrote:
> I've entered the issue at
> http://ftputil.sschwarzer.net/trac/ticket/32
> 
> You can edit it if you log in as user ftputiluser with password
> ftputil (this is for spam prevention).

Please continue working on the bug report at the URL above. Until
now, there have been some additions but no fix yet at the time I
send this mail.

For now, the safest way to avoid the traceback is to disable the
cache on Windows directly after the connection by coding
something like

    import sys

    host = ftputil.FTPHost(server, user, password)
    if sys.platform == 'win32':
        host.stat_cache.disable()

    ...

    host.close()

Because of the possible performance drawbacks of disabling the
cache, you might prefer to use other workarounds for your
specific code.

I'll notify you if I know more of the nature of the bug and thus
know a fix or at least more specific workarounds which can be
included in ftputil.

Happy Easter days! :-)

Best regards,
Stefan

From peter at pjstirling.plus.com  Wed Apr 22 01:01:07 2009
From: peter at pjstirling.plus.com (Peter Stirling)
Date: Wed, 22 Apr 2009 00:01:07 +0100
Subject: [ftputil] ftputil 2.4 with python 2.6.1
In-Reply-To: <49DD83A6.1080205@sschwarzer.net>
References: <1239234264.7031.253.camel@peter-desktop>
	<49DD83A6.1080205@sschwarzer.net>
Message-ID: <1240354867.9239.74.camel@peter-desktop>

Hi, again,

I'm pretty certain now that the error is in the code of the python-2.6.1
windows build on the python site, so they need to get a bug report.

I've done some more testing, your pointing out that the lrucache had its
own tests was quite helpful for making a smaller reproducer.

I attach a script that runs the same test code by using each of the
versions of heapq that I had available to me (py-2.5, py-2.5 with the
c-code module disabled, py-2.6, py-2.6-no-c, and the default heapq).
When it runs on linux I get the exact same results from using each
method (and the output is correct), but on windows we see the same stack
traces for each of the tests, you get different answers from each test,
and none of the different answers are right.

The archive has linux.txt and win32.txt as output from the program on my
linux and vista machines.


On Thu, 2009-04-09 at 07:12 +0200, Stefan Schwarzer wrote:
> Hello Peter,
> 
> On 2009-04-09 01:44, Peter Stirling wrote:
> > After further inspection it appears that the windows python2.6 install doesn't
> > include a dll of the compiled heapq module 
> > 
> > 	/usr/lib/python2.5/lib-dynload/_heapq.so, on ubuntu python2.5
> > 
> > However when I tried to test the pure python version it decided to work, so there
> > must be a bad dll somewhere.
> > 
> > To test this I made a copy of the pure python heapq.py as heapq26.py, and commented
> > out the code that replaces pure-python with the c versions (line 313). I then altered
> > ftputil/lrucache.py(line 43) to import from this new module, and when I run the test
> > program I no longer get the exception.
> 
> thanks a lot for your investigation. :o) Please see also my reply
> to the mailing list.
> 
> > I am very confused.
> 
> Why?
> 
> Stefan

As I said before, there doesn't appear to be a dll for the compiled
build to live in, but there definitely is SOMETHING interfering.
Possibly they compiled the code directly into the python.exe file, but I
haven't seen any reason for that to be the case (and there are external
dlls that I would have expected to be part of the executable over the
heapq module, eg ctypes).


-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.tbz
Type: application/x-bzip-compressed-tar
Size: 13932 bytes
Desc: not available
Url : http://codespeak.net/pipermail/ftputil/attachments/20090422/98d3fae5/attachment.bin 

From peter at pjstirling.plus.com  Wed Apr 22 13:20:50 2009
From: peter at pjstirling.plus.com (Peter Stirling)
Date: Wed, 22 Apr 2009 12:20:50 +0100
Subject: [ftputil] ftputil 2.4 with python 2.6.1
In-Reply-To: <1240354867.9239.74.camel@peter-desktop>
References: <1239234264.7031.253.camel@peter-desktop>
	<49DD83A6.1080205@sschwarzer.net>
	<1240354867.9239.74.camel@peter-desktop>
Message-ID: <747E3816-64E1-4E7B-9247-A10A1E405C0B@pjstirling.plus.com>

Hmm, actually I think I need to make some more tests

On 22 Apr 2009, at 12:01 am, Peter Stirling wrote:

> Hi, again,
>
> I'm pretty certain now that the error is in the code of the  
> python-2.6.1
> windows build on the python site, so they need to get a bug report.
>
> I've done some more testing, your pointing out that the lrucache had  
> its
> own tests was quite helpful for making a smaller reproducer.
>
> I attach a script that runs the same test code by using each of the
> versions of heapq that I had available to me (py-2.5, py-2.5 with the
> c-code module disabled, py-2.6, py-2.6-no-c, and the default heapq).
> When it runs on linux I get the exact same results from using each
> method (and the output is correct), but on windows we see the same  
> stack
> traces for each of the tests, you get different answers from each  
> test,
> and none of the different answers are right.
>
> The archive has linux.txt and win32.txt as output from the program  
> on my
> linux and vista machines.
>
>
> On Thu, 2009-04-09 at 07:12 +0200, Stefan Schwarzer wrote:
>> Hello Peter,
>>
>> On 2009-04-09 01:44, Peter Stirling wrote:
>>> After further inspection it appears that the windows python2.6  
>>> install doesn't
>>> include a dll of the compiled heapq module
>>>
>>> 	/usr/lib/python2.5/lib-dynload/_heapq.so, on ubuntu python2.5
>>>
>>> However when I tried to test the pure python version it decided to  
>>> work, so there
>>> must be a bad dll somewhere.
>>>
>>> To test this I made a copy of the pure python heapq.py as  
>>> heapq26.py, and commented
>>> out the code that replaces pure-python with the c versions (line  
>>> 313). I then altered
>>> ftputil/lrucache.py(line 43) to import from this new module, and  
>>> when I run the test
>>> program I no longer get the exception.
>>
>> thanks a lot for your investigation. :o) Please see also my reply
>> to the mailing list.
>>
>>> I am very confused.
>>
>> Why?
>>
>> Stefan
>
> As I said before, there doesn't appear to be a dll for the compiled
> build to live in, but there definitely is SOMETHING interfering.
> Possibly they compiled the code directly into the python.exe file,  
> but I
> haven't seen any reason for that to be the case (and there are  
> external
> dlls that I would have expected to be part of the executable over the
> heapq module, eg ctypes).
>
>
> <test.tbz>_______________________________________________
> ftputil mailing list
> ftputil at codespeak.net
> http://codespeak.net/mailman/listinfo/ftputil


From lists at svrinformatica.it  Sun Apr 26 00:55:02 2009
From: lists at svrinformatica.it (MailingList SVR)
Date: Sun, 26 Apr 2009 00:55:02 +0200
Subject: [ftputil] Ftputil seems broken with link
Message-ID: <200904260055.02403.lists@svrinformatica.it>

Hi,

ftputil seems unable to follow links, go for example to:

ftp.heanet.ie with an anonymous user,

>>> conn.path.islink('/pub/apache')                       
True 
>>> conn.listdir('/pub/apache')
['/pub/apache']

so listdir doesn't follow the link. Is there a workaround ?

thanks
Nicola 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090426/a5ee8f73/attachment.htm 

From sschwarzer at sschwarzer.net  Sun Apr 26 10:23:52 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Sun, 26 Apr 2009 10:23:52 +0200
Subject: [ftputil] Ftputil seems broken with link
In-Reply-To: <200904260055.02403.lists@svrinformatica.it>
References: <200904260055.02403.lists@svrinformatica.it>
Message-ID: <49F41A18.7080108@sschwarzer.net>

Hi Nicola,

thanks for your report.

On 2009-04-26 00:55, MailingList SVR wrote:
> ftputil seems unable to follow links, go for example to:
>
> ftp.heanet.ie with an anonymous user,
>
> >>> conn.path.islink('/pub/apache')
> True
> >>> conn.listdir('/pub/apache')
> ['/pub/apache']
>
> so listdir doesn't follow the link. Is there a workaround ?

I can't reproduce this with my local FTP test server:

>>> h = ftputil.FTPHost("localhost", 'ftptest', ...)
>>> h.path.islink("blah")
True
>>> h.listdir("blah")
['second dir', 'some file', 'some_file']

The server is PureFTPd version 1.0.21-11.2ubuntu1. This server
setup is used for all the tests in the file _test_real_ftp.py.
Here, I only added the link.

In your specific case, I can repeat the behavior, though.

Since ftputil just displays the directory contents it gets from
the server, I think the result has something to do with the
server. Let's examine the situation:

>>> h = ftputil.FTPHost("ftp.heanet.ie", "anonymous", "sschwarzer at sschwarzer.net")
>>> stat_result = h.lstat("/pub/apache")
>>> stat_result._st_name
'apache'
>>> stat_result._st_target
'./www.apache.org'

(_st_name and _st_target have a leading underscore because
"native" stat result objects don't have those attributes either.
When I introduced the "stat-ing", I wanted to keep ftputil as
similar as possible to Python's native file system interface, so
I didn't disclose the attributes. ftputil uses them internally,
though.)

If the link is resolved manually (the workaround you asked for),
this causes a traceback:

>>> stat_result = h.lstat("/pub/apache")
>>> h.path.join("/pub/apache", stat_result._st_target)
'/pub/apache/./www.apache.org'
>>> h.listdir(h.path.join("/pub/apache", stat_result._st_target))
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
  File "ftputil.py", line 805, in listdir
    return self._stat.listdir(path)
  File "ftp_stat.py", line 565, in listdir
    return self.__call_with_parser_retry(self._real_listdir, path)
  File "ftp_stat.py", line 543, in __call_with_parser_retry
    result = method(*args, **kwargs)
  File "ftp_stat.py", line 406, in _real_listdir
    path)
PermanentError: 550 /pub/apache/www.apache.org: no such directory or wrong directory parser used
Debugging info: ftputil 2.4, Python 2.5.2 (linux2)

I guess the selected parser, hinted at in the message, is ok
because I can list the root directory on the server and can get
the stat result for /pub/apache.

So I assume the link on the server points to a non-existent or
otherwise invalid directory and the server upon the listdir call
decides to just return the path of the link.

> thanks

You're welcome. :-)

Best regards,
Stefan


From lists at svrinformatica.it  Sun Apr 26 12:11:31 2009
From: lists at svrinformatica.it (MailingList SVR)
Date: Sun, 26 Apr 2009 12:11:31 +0200
Subject: [ftputil] Ftputil seems broken with link
In-Reply-To: <49F41A18.7080108@sschwarzer.net>
References: <200904260055.02403.lists@svrinformatica.it>
	<49F41A18.7080108@sschwarzer.net>
Message-ID: <200904261211.31516.lists@svrinformatica.it>

Hi Stefan,



thanks for your answer, the problem only appear when there is a link to a folder the same directory as the link and it begin with "./" that is a valid path in the unix world, so a workaround would be to replace './' with '../' if _st_target begin with "./". For now I solved with my own code, do you think you can add this workaround for the next ftputil version?



regards
Nicola



In data domenica 26 aprile 2009 10:23:52, Stefan Schwarzer ha scritto:
: > Hi Nicola,
> 
> thanks for your report.
> 
> On 2009-04-26 00:55, MailingList SVR wrote:
> > ftputil seems unable to follow links, go for example to:
> >
> > ftp.heanet.ie with an anonymous user,
> >
> > >>> conn.path.islink('/pub/apache')
> > True
> > >>> conn.listdir('/pub/apache')
> > ['/pub/apache']
> >
> > so listdir doesn't follow the link. Is there a workaround ?
> 
> I can't reproduce this with my local FTP test server:
> 
> >>> h = ftputil.FTPHost("localhost", 'ftptest', ...)
> >>> h.path.islink("blah")
> True
> >>> h.listdir("blah")
> ['second dir', 'some file', 'some_file']
> 
> The server is PureFTPd version 1.0.21-11.2ubuntu1. This server
> setup is used for all the tests in the file _test_real_ftp.py.
> Here, I only added the link.
> 
> In your specific case, I can repeat the behavior, though.
> 
> Since ftputil just displays the directory contents it gets from
> the server, I think the result has something to do with the
> server. Let's examine the situation:
> 
> >>> h = ftputil.FTPHost("ftp.heanet.ie", "anonymous", "sschwarzer at sschwarzer.net")
> >>> stat_result = h.lstat("/pub/apache")
> >>> stat_result._st_name
> 'apache'
> >>> stat_result._st_target
> './www.apache.org'
> 
> (_st_name and _st_target have a leading underscore because
> "native" stat result objects don't have those attributes either.
> When I introduced the "stat-ing", I wanted to keep ftputil as
> similar as possible to Python's native file system interface, so
> I didn't disclose the attributes. ftputil uses them internally,
> though.)
> 
> If the link is resolved manually (the workaround you asked for),
> this causes a traceback:
> 
> >>> stat_result = h.lstat("/pub/apache")
> >>> h.path.join("/pub/apache", stat_result._st_target)
> '/pub/apache/./www.apache.org'
> >>> h.listdir(h.path.join("/pub/apache", stat_result._st_target))
> Traceback (most recent call last):
>   File "<ipython console>", line 1, in <module>
>   File "ftputil.py", line 805, in listdir
>     return self._stat.listdir(path)
>   File "ftp_stat.py", line 565, in listdir
>     return self.__call_with_parser_retry(self._real_listdir, path)
>   File "ftp_stat.py", line 543, in __call_with_parser_retry
>     result = method(*args, **kwargs)
>   File "ftp_stat.py", line 406, in _real_listdir
>     path)
> PermanentError: 550 /pub/apache/www.apache.org: no such directory or wrong directory parser used
> Debugging info: ftputil 2.4, Python 2.5.2 (linux2)
> 
> I guess the selected parser, hinted at in the message, is ok
> because I can list the root directory on the server and can get
> the stat result for /pub/apache.
> 
> So I assume the link on the server points to a non-existent or
> otherwise invalid directory and the server upon the listdir call
> decides to just return the path of the link.
> 
> > thanks
> 
> You're welcome. :-)
> 
> Best regards,
> Stefan
> 
> _______________________________________________
> ftputil mailing list
> ftputil at codespeak.net
> http://codespeak.net/mailman/listinfo/ftputil
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090426/8710cfe8/attachment.htm 

From lists at svrinformatica.it  Sun Apr 26 12:14:31 2009
From: lists at svrinformatica.it (MailingList SVR)
Date: Sun, 26 Apr 2009 12:14:31 +0200
Subject: [ftputil] Ftputil seems broken with link
In-Reply-To: <200904261211.31516.lists@svrinformatica.it>
References: <200904260055.02403.lists@svrinformatica.it>
	<49F41A18.7080108@sschwarzer.net>
	<200904261211.31516.lists@svrinformatica.it>
Message-ID: <200904261214.32094.lists@svrinformatica.it>

In data domenica 26 aprile 2009 12:11:31, MailingList SVR ha scritto:
: > Hi Stefan,
> 
> 
> 
> thanks for your answer, the problem only appear when there is a link to a folder the same directory as the link and it begin with "./" that is a valid path in the unix world, so a workaround would be to replace './' with '../' if _st_target begin with "./". For now I solved with my own code, do you think you can add this workaround for the next ftputil version?
> 
> 
> 
> regards
> Nicola
> 

additionally I tested standard ftp clients (filezilla and cli linux  ftp client) and they resolve the link and then show the content

> 
> 
> In data domenica 26 aprile 2009 10:23:52, Stefan Schwarzer ha scritto:
> : > Hi Nicola,
> > 
> > thanks for your report.
> > 
> > On 2009-04-26 00:55, MailingList SVR wrote:
> > > ftputil seems unable to follow links, go for example to:
> > >
> > > ftp.heanet.ie with an anonymous user,
> > >
> > > >>> conn.path.islink('/pub/apache')
> > > True
> > > >>> conn.listdir('/pub/apache')
> > > ['/pub/apache']
> > >
> > > so listdir doesn't follow the link. Is there a workaround ?
> > 
> > I can't reproduce this with my local FTP test server:
> > 
> > >>> h = ftputil.FTPHost("localhost", 'ftptest', ...)
> > >>> h.path.islink("blah")
> > True
> > >>> h.listdir("blah")
> > ['second dir', 'some file', 'some_file']
> > 
> > The server is PureFTPd version 1.0.21-11.2ubuntu1. This server
> > setup is used for all the tests in the file _test_real_ftp.py.
> > Here, I only added the link.
> > 
> > In your specific case, I can repeat the behavior, though.
> > 
> > Since ftputil just displays the directory contents it gets from
> > the server, I think the result has something to do with the
> > server. Let's examine the situation:
> > 
> > >>> h = ftputil.FTPHost("ftp.heanet.ie", "anonymous", "sschwarzer at sschwarzer.net")
> > >>> stat_result = h.lstat("/pub/apache")
> > >>> stat_result._st_name
> > 'apache'
> > >>> stat_result._st_target
> > './www.apache.org'
> > 
> > (_st_name and _st_target have a leading underscore because
> > "native" stat result objects don't have those attributes either.
> > When I introduced the "stat-ing", I wanted to keep ftputil as
> > similar as possible to Python's native file system interface, so
> > I didn't disclose the attributes. ftputil uses them internally,
> > though.)
> > 
> > If the link is resolved manually (the workaround you asked for),
> > this causes a traceback:
> > 
> > >>> stat_result = h.lstat("/pub/apache")
> > >>> h.path.join("/pub/apache", stat_result._st_target)
> > '/pub/apache/./www.apache.org'
> > >>> h.listdir(h.path.join("/pub/apache", stat_result._st_target))
> > Traceback (most recent call last):
> >   File "<ipython console>", line 1, in <module>
> >   File "ftputil.py", line 805, in listdir
> >     return self._stat.listdir(path)
> >   File "ftp_stat.py", line 565, in listdir
> >     return self.__call_with_parser_retry(self._real_listdir, path)
> >   File "ftp_stat.py", line 543, in __call_with_parser_retry
> >     result = method(*args, **kwargs)
> >   File "ftp_stat.py", line 406, in _real_listdir
> >     path)
> > PermanentError: 550 /pub/apache/www.apache.org: no such directory or wrong directory parser used
> > Debugging info: ftputil 2.4, Python 2.5.2 (linux2)
> > 
> > I guess the selected parser, hinted at in the message, is ok
> > because I can list the root directory on the server and can get
> > the stat result for /pub/apache.
> > 
> > So I assume the link on the server points to a non-existent or
> > otherwise invalid directory and the server upon the listdir call
> > decides to just return the path of the link.
> > 
> > > thanks
> > 
> > You're welcome. :-)
> > 
> > Best regards,
> > Stefan
> > 
> > _______________________________________________
> > ftputil mailing list
> > ftputil at codespeak.net
> > http://codespeak.net/mailman/listinfo/ftputil
> > 
> 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090426/0e051ec5/attachment-0001.htm 

From lists at svrinformatica.it  Mon Apr 27 16:56:45 2009
From: lists at svrinformatica.it (MailingList SVR)
Date: Mon, 27 Apr 2009 16:56:45 +0200
Subject: [ftputil] Ftputil and ftp.gnome.org
Message-ID: <200904271656.45507.lists@svrinformatica.it>

Hi Stefan,

seems there are problems with ftp.gnome.org too

with ftputil (only a directory is found):

import ftputil
conn=ftputil.FTPHost('ftp.gnome.org','anonymous','pippo at pippo.com')
conn.listdir('/')
['pub']


with standard python ftplib (all directory are listed)

ftp = ftplib.FTP('ftp.gnome.org')

ftp.login()
'230 Login successful.'

ftp.retrlines('LIST')
-rw-r--r--    1 ftp      ftp          1001 May 07  2007 HEADER.html
lrwxrwxrwx    1 ftp      ftp             3 Sep 18  2008 Public -> pub
drwxr-xr-x    2 ftp      ftp             8 Apr 15 18:43 about
drwxrwxr-x   11 ftp      ftp            11 Apr 14 20:19 cdimage
drwxr-xr-x    2 ftp      ftp             3 Jun 14  2006 conspiracy
lrwxrwxrwx    1 ftp      ftp            22 Sep 18  2008 debian -> cdimage/.debian-mirror
lrwxrwxrwx    1 ftp      ftp            16 Sep 18  2008 debian-cd -> cdimage/release/
lrwxrwxrwx    1 ftp      ftp            20 Sep 18  2008 debian-non-US -> mirror/debian-non-US
-rw-r--r--    1 ftp      ftp           318 Mar 21  2005 favicon.ico
drwxr-xr-x   27 ftp      ftp            32 Apr 15 18:43 mirror
drwxr-xr-x    4 ftp      ftp            15 Dec 19 14:31 pub
lrwxrwxrwx    1 ftp      ftp            23 Sep 18  2008 releases -> mirror/ubuntu-releases/
-rw-r--r--    1 ftp      ftp           445 Mar 06  2007 robots.txt
lrwxrwxrwx    1 ftp      ftp            13 Sep 18  2008 ubuntu -> mirror/ubuntu
-rw-r--r--    1 ftp      ftp          1037 Aug 22  2001 welcome.msg
-rw-r--r--    1 ftp      ftp          1136 Apr 19  2001 welcome2.msg

'226 Directory send OK.'

ftp.dir()

-rw-r--r--    1 ftp      ftp          1001 May 07  2007 HEADER.html
lrwxrwxrwx    1 ftp      ftp             3 Sep 18  2008 Public -> pub
drwxr-xr-x    2 ftp      ftp             8 Apr 15 18:43 about
drwxrwxr-x   11 ftp      ftp            11 Apr 14 20:19 cdimage
drwxr-xr-x    2 ftp      ftp             3 Jun 14  2006 conspiracy
lrwxrwxrwx    1 ftp      ftp            22 Sep 18  2008 debian -> cdimage/.debian-mirror
lrwxrwxrwx    1 ftp      ftp            16 Sep 18  2008 debian-cd -> cdimage/release/
lrwxrwxrwx    1 ftp      ftp            20 Sep 18  2008 debian-non-US -> mirror/debian-non-US
-rw-r--r--    1 ftp      ftp           318 Mar 21  2005 favicon.ico
drwxr-xr-x   27 ftp      ftp            32 Apr 15 18:43 mirror
drwxr-xr-x    4 ftp      ftp            15 Dec 19 14:31 pub
lrwxrwxrwx    1 ftp      ftp            23 Sep 18  2008 releases -> mirror/ubuntu-releases/
-rw-r--r--    1 ftp      ftp           445 Mar 06  2007 robots.txt
lrwxrwxrwx    1 ftp      ftp            13 Sep 18  2008 ubuntu -> mirror/ubuntu
-rw-r--r--    1 ftp      ftp          1037 Aug 22  2001 welcome.msg
-rw-r--r--    1 ftp      ftp          1136 Apr 19  2001 welcome2.msg


however 

ftp.dir('/')
drwxr-xr-x    4 ftp      ftp            15 Dec 19 14:31 pub

give the same results as ftpdir, a workaroud is

ftp.cwd(path)
ftp.dir()  --- without specify path,

can you apply this to ftputil?

regards
Nicola



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090427/ba0a8f13/attachment.htm 

From lists at svrinformatica.it  Tue Apr 28 10:20:53 2009
From: lists at svrinformatica.it (MailingList SVR)
Date: Tue, 28 Apr 2009 10:20:53 +0200
Subject: [ftputil] Ftputil and ftp.sunfreeware.com
Message-ID: <200904281020.53530.lists@svrinformatica.it>

Hi Stefan,

ftp.sunfreeware.com need another parser I think:

ftp> ls
227 Entering Passive Mode (66,193,208,66,41,119)
150 Opening ASCII mode data connection for /bin/ls.
total 10
lrwxrwxrwx   1 staff          7 Aug 13  2003 bin -> usr/bin
d--x--x--x   2 staff        512 Sep 24  2000 dev
d--x--x--x   3 staff        512 Sep 25  2000 etc
dr-xr-xr-x   3 staff        512 Oct  3  2000 pub
d--x--x--x   5 staff        512 Oct  3  2000 usr
226 Transfer complete.

conn=ftputil.FTPHost('ftp.sunfreeware.com','anonymous','pippo at pippo.com')

In [6]: conn
Out[6]: <FtpManager.ftputil.ftputil.FTPHost object at 0x1ad3610>

In [7]: conn.listdir('/')
---------------------------------------------------------------------------
ParserError                               Traceback (most recent call last)

/home/nicola/workspace/FtpManager/<ipython console> in <module>()

/home/nicola/workspace/FtpManager/ftputil/ftputil.py in listdir(self, path)
    764         any of the available parsers raise a `ParserError`.        
    765         """                                                        
--> 766         return self._stat.listdir(path)                            
    767                                                                    
    768     def lstat(self, path, _exception_for_missing_path=True):       

/home/nicola/workspace/FtpManager/ftputil/ftp_stat.py in listdir(self, path)
    533                                                                     
    534     def listdir(self, path):                                        
--> 535         return self.__call_with_parser_retry(self._real_listdir, path)
    536                                                                       
    537     def lstat(self, path, _exception_for_missing_path=True):          

/home/nicola/workspace/FtpManager/ftputil/ftp_stat.py in __call_with_parser_retry(self, method, *args, **kwargs)                                                                              
    528                 self._allow_parser_switching = False                                   
    529                 self._parser = MSParser()
--> 530                 return method(*args, **kwargs)
    531             else:
    532                 raise

/home/nicola/workspace/FtpManager/ftputil/ftp_stat.py in _real_listdir(self, path)
    397             #  correct timestamp values in the cache

    398             stat_result = self._parser.parse_line(line,
--> 399                                                   self._host.time_shift())
    400             loop_path = self._path.join(path, stat_result._st_name)
    401             self._lstat_cache[loop_path] = stat_result

/home/nicola/workspace/FtpManager/ftputil/ftp_stat.py in parse_line(self, line, time_shift)
    327                 st_size = int(dir_or_size)
    328             except ValueError:
--> 329                 raise ftp_error.ParserError("invalid size %s" % dir_or_size)
    330         else:
    331             st_size = None

ParserError: invalid size staff

do you have something already done?

regards
Nicola






-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090428/4236a670/attachment.htm 

From sschwarzer at sschwarzer.net  Tue Apr 28 20:57:31 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Tue, 28 Apr 2009 20:57:31 +0200
Subject: [ftputil] Ftputil and ftp.sunfreeware.com
In-Reply-To: <200904281020.53530.lists@svrinformatica.it>
References: <200904281020.53530.lists@svrinformatica.it>
Message-ID: <49F7519B.7010405@sschwarzer.net>

Hello Nicola,

Thanks a lot for your feedback. :)

On 2009-04-28 10:20, MailingList SVR wrote:
> ftp.sunfreeware.com need another parser I think:
> [...]

> do you have something already done?

Not yet. Hopefully I get around to work on it at the weekend.
(ftputil is a spare-time project for me.)

Best regards,
Stefan

From lists at svrinformatica.it  Wed Apr 29 22:46:56 2009
From: lists at svrinformatica.it (MailingList SVR)
Date: Wed, 29 Apr 2009 22:46:56 +0200
Subject: [ftputil] Ftputil seems really slow to distinguish file and folders
Message-ID: <200904292246.56413.lists@svrinformatica.it>

Hi Stefan,

I need to distinguish beetween file and folder, I'm using host.path.isdir and host.path.isfile but these seems really slow.

Here is an example code:

host=ftputil.FTPHost('ftp.nluug.nl','anonymous','pippo at pippo.com')

folder='/vol/1/centoslinux/5.3/os/SRPMS/'

lista=host.listdir(folder)

# host.listdir is fast, good!

for l in lista:
        print folder+l
        print datetime.datetime.now()
       #host.path.isdir and isfile are slow :-(
        if host.path.isdir(folder+l):
                print 'dir:'
                print datetime.datetime.now()
        if host.path.isfile(folder+l):
                print 'file:'


and this is the output:

/vol/1/centoslinux/5.3/os/SRPMS/Cluster_Administration-5.2-1.el5.centos.src.rpm
2009-04-29 22:39:53.646165                                                     
file:                                                                          
2009-04-29 22:39:54.917435                                                     
/vol/1/centoslinux/5.3/os/SRPMS/Deployment_Guide-5.2-11.el5.centos.src.rpm     
2009-04-29 22:39:54.917775
file:
2009-04-29 22:39:56.248630
/vol/1/centoslinux/5.3/os/SRPMS/ElectricFence-2.2.2-20.2.2.src.rpm
2009-04-29 22:39:56.249002
file:
2009-04-29 22:39:57.563621
/vol/1/centoslinux/5.3/os/SRPMS/GConf2-2.14.0-9.el5.src.rpm
2009-04-29 22:39:57.563988
file:
2009-04-29 22:39:58.832466
/vol/1/centoslinux/5.3/os/SRPMS/Global_File_System-5.2-1.el5.centos.src.rpm
2009-04-29 22:39:58.832842
file:
2009-04-29 22:40:00.104433
/vol/1/centoslinux/5.3/os/SRPMS/ImageMagick-6.2.8.0-4.el5_1.1.src.rpm
2009-04-29 22:40:00.104776

as you can see this method is really slow. I think I'm missing something ... distinguish files and folders is the main function of every ftp frontend, can you point me towards the right approach?

thanks
Nicola


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090429/ebd3c829/attachment-0001.htm 

From sschwarzer at sschwarzer.net  Thu Apr 30 21:54:11 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Thu, 30 Apr 2009 21:54:11 +0200
Subject: [ftputil] Ftputil seems really slow to distinguish file and
	folders
In-Reply-To: <200904292246.56413.lists@svrinformatica.it>
References: <200904292246.56413.lists@svrinformatica.it>
Message-ID: <49FA01E3.6030605@sschwarzer.net>

Hello Nicola,

What I like about your reports is that you always provide
concrete examples with working test code. Great! :-)

On 2009-04-29 22:46, MailingList SVR wrote:
> I need to distinguish beetween file and folder, I'm using
> host.path.isdir and host.path.isfile but these seems really slow.
> 
> Here is an example code:
> 
> host=ftputil.FTPHost('ftp.nluug.nl','anonymous','pippo at pippo.com')
> folder='/vol/1/centoslinux/5.3/os/SRPMS/'
> lista=host.listdir(folder)
> 
> # host.listdir is fast, good!
>
> for l in lista:
>         print folder+l
>         print datetime.datetime.now()
>        #host.path.isdir and isfile are slow :-(
>         if host.path.isdir(folder+l):
>                 print 'dir:'
>                 print datetime.datetime.now()
>         if host.path.isfile(folder+l):
>                 print 'file:'
> 
> and this is the output:
> [...]

Running this with ftputil 2.4 on my computer takes about
25 minutes. When increasing the cache size to 2000, the code
runs in about 40 seconds. :)

For the details, please read this section of the ftputil
documentation:

http://ftputil.sschwarzer.net/trac/wiki/Documentation#local-caching-of-file-system-information

(context: the directory you work on has 1193 items in it, and
ftputil's default cache size is 1000)

Best regards,
Stefan

From lists at svrinformatica.it  Thu Apr 30 23:50:03 2009
From: lists at svrinformatica.it (MailingList SVR)
Date: Thu, 30 Apr 2009 23:50:03 +0200
Subject: [ftputil] Ftputil seems really slow to distinguish file and
	folders
In-Reply-To: <49FA01E3.6030605@sschwarzer.net>
References: <200904292246.56413.lists@svrinformatica.it>
	<49FA01E3.6030605@sschwarzer.net>
Message-ID: <200904302350.03918.lists@svrinformatica.it>

In data gioved? 30 aprile 2009 21:54:11, Stefan Schwarzer ha scritto:
: > Hello Nicola,
> 
> What I like about your reports is that you always provide
> concrete examples with working test code. Great! :-)

and your answers are ever rich of quality valuable infos, thanks!

> 
> On 2009-04-29 22:46, MailingList SVR wrote:
> > I need to distinguish beetween file and folder, I'm using
> > host.path.isdir and host.path.isfile but these seems really slow.
> > 
> > Here is an example code:
> > 
> > host=ftputil.FTPHost('ftp.nluug.nl','anonymous','pippo at pippo.com')
> > folder='/vol/1/centoslinux/5.3/os/SRPMS/'
> > lista=host.listdir(folder)
> > 
> > # host.listdir is fast, good!
> >
> > for l in lista:
> >         print folder+l
> >         print datetime.datetime.now()
> >        #host.path.isdir and isfile are slow :-(
> >         if host.path.isdir(folder+l):
> >                 print 'dir:'
> >                 print datetime.datetime.now()
> >         if host.path.isfile(folder+l):
> >                 print 'file:'
> > 
> > and this is the output:
> > [...]
> 
> Running this with ftputil 2.4 on my computer takes about
> 25 minutes. When increasing the cache size to 2000, the code
> runs in about 40 seconds. :)
>

works in about 40 seconds on my box too, this time is acceptable, however on the same directory a standard ftpclient give the directory listing in few seconds. I haven't look at ftputil code but host.listdir is fast such as filezilla and company (2-3 seconds) what other tasks ftputil does in about 35 seconds? It is possible to have a fastest listing or the code is already optimized?

thanks
Nicola




> For the details, please read this section of the ftputil
> documentation:
> 
> http://ftputil.sschwarzer.net/trac/wiki/Documentation#local-caching-of-file-system-information
> 
> (context: the directory you work on has 1193 items in it, and
> ftputil's default cache size is 1000)
>
> Best regards,
> Stefan
> _______________________________________________
> ftputil mailing list
> ftputil at codespeak.net
> http://codespeak.net/mailman/listinfo/ftputil
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090430/1168d89a/attachment.htm 

From sschwarzer at sschwarzer.net  Fri May  1 10:52:54 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Fri, 01 May 2009 10:52:54 +0200
Subject: [ftputil] Ftputil seems really slow to distinguish file
	and	folders
In-Reply-To: <200904302350.03918.lists@svrinformatica.it>
References: <200904292246.56413.lists@svrinformatica.it>	<49FA01E3.6030605@sschwarzer.net>
	<200904302350.03918.lists@svrinformatica.it>
Message-ID: <49FAB866.4040106@sschwarzer.net>

Hi Nicola,

On 2009-04-30 23:50, MailingList SVR wrote:
>> What I like about your reports is that you always provide
>> concrete examples with working test code. Great! :-)
> 
> and your answers are ever rich of quality valuable infos, thanks!

Thanks a lot :-)

>> Running this with ftputil 2.4 on my computer takes about
>> 25 minutes. When increasing the cache size to 2000, the code
>> runs in about 40 seconds. :)
> 
> works in about 40 seconds on my box too, this time is acceptable,
> however on the same directory a standard ftpclient give the directory
> listing in few seconds. I haven't look at ftputil code but host.listdir
> is fast such as filezilla and company (2-3 seconds) what other tasks
> ftputil does in about 35 seconds?

First, I'd like to add that if I leave the print statements
out of the loop, i. e.

>>> def f():
...   for i in lista:
...     isd = host.path.isdir(folder+i)
...     isf = host.path.isfile(folder+i)

I'm down to 25 seconds (with a cache size of 2000). :)

Using only one cache access per name (as listdir does) in the
loop, I get the loop done in 11 seconds. The following function,
including connecting to the server, runs in about 13 seconds:

>>> def h():
...   host = ftputil.FTPHost('ftp.nluug.nl','anonymous','pippo at pippo.com')
...   host.stat_cache.resize(2000)
...   for i in host.listdir(folder):
...     s = host.lstat(folder + i)
>>> %time h()  # built into IPython
CPU times: user 9.38 s, sys: 0.08 s, total: 9.46 s
Wall time: 12.59 s

However, listdir (needing 1.2 seconds for the directory) only
_stores_ values in the cache while lstat _retrieves_ the info. So
I suspect the retrieval is significantly slower than the storage.

A little test:

>>> def g():
...   for i in lista:
...     s = host.lstat(folder+i)
>>> %prun g()  # built into IPython
         11358799 function calls in 54.934 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  5661503   25.974    0.000   38.745    0.000 lrucache.py:110(__cmp__)
     2386   15.896    0.007   54.641    0.023 {_heapq.heapify}
  5661503   12.771    0.000   12.771    0.000 {cmp}
...

The total time of the other calls seems negligible, so ftputil's
own code seems rather innocent. ;-)

If you're writing an end-user client and need to give the user a
directory listing fast, including the stat'ed information for
each file/dir, the following - untested - approach _might_ work:

- Write a cache class with an interface like that in
  ftp_stat_cache.py. However, use a raw Python dictionary to
  store the stat results. This cache will have no (automatic)
  means to prune old entries, but see below. You can use the
  present cache class a template.

- Derive a stat class from ftp_stat._Stat:

  class MyStat(ftp_stat._Stat):
      def __init__(self, *args, **kwargs):
          super(MyStat, self).__init__(*args, **kwargs)
          self._lstat_cache = MyCache()

- Derive a class from FTPHost:

  class MyFTPHost(ftputil.FTPHost):
      def __init__(self, *args, **kwargs):
          super(MyFTPHost, self).__init__(*args, **kwargs)
          self._stat = MyStat(self)
          self.stat_cache = self._stat._lstat_cache

- To get a listing, instantiate your custom FTPHost class. Clear
  the cache with host.stat_cache.clear() before retrieving a
  directory listing. Also clear the cache explicitly after you no
  longer need the directory data.

Keep in mind that if your software isn't interactive, you most
probably don't need to worry about tuning at all. [1] ... And
_with_ interactivity, I just measured: Nautilus 2.24.1 needs
about 8 seconds to show the directory listing, so I think the
13 seconds I got with the pure lstat calls (see above) are not so
bad! Also remember that most users won't have so many directory
items frequently. For most directories, you won't notice any
difference.

If you nevertheless tried out the above idea, I (and maybe other
readers of the list) would be very thankful if you shared your
results. :-)

> It is possible to have a fastest
> listing or the code is already optimized?

As far as I remember, I haven't optimized the code at all because
I haven't had a use case yet where the code was too slow for me. I
don't know how much tuning has gone into lrucache [2], though.

[1] Please read
    http://sschwarzer.com/download/optimization_europython2006.pdf
    if you haven't already done so. :)

[2] http://pypi.python.org/pypi/lrucache/0.2 (It's listed there
    as alpha code, but to my question over two years ago the
    author replied it were used successfully in several projects
    and that it had comprehensive unit tests. I haven't had any
    complaints either.)

Best regards,
Stefan

From lists at svrinformatica.it  Fri May  1 17:44:20 2009
From: lists at svrinformatica.it (MailingList SVR)
Date: Fri, 1 May 2009 17:44:20 +0200
Subject: [ftputil]
 =?iso-8859-1?q?Ftputil_seems_really_slow_to_distinguish?=
 =?iso-8859-1?q?_file_and=09folders?=
In-Reply-To: <49FAB866.4040106@sschwarzer.net>
References: <200904292246.56413.lists@svrinformatica.it>
	<200904302350.03918.lists@svrinformatica.it>
	<49FAB866.4040106@sschwarzer.net>
Message-ID: <200905011744.20820.lists@svrinformatica.it>

Hi Stefan,



thanks for your suggestions I'll try them this weekend,



what I mean for fast mode is use only host.listdir to provide the distintion beetween files and directory. Is it possible to retrieve this basic info only with listdir and so in 1.2 seconds? This way we can give a very fast directory listing and for example give the whole informations only if len(host.listdir) < x



regards 
Nicola



In data venerd? 01 maggio 2009 10:52:54, Stefan Schwarzer ha scritto:
: > Hi Nicola,
> 
> On 2009-04-30 23:50, MailingList SVR wrote:
> >> What I like about your reports is that you always provide
> >> concrete examples with working test code. Great! :-)
> > 
> > and your answers are ever rich of quality valuable infos, thanks!
> 
> Thanks a lot :-)
> 
> >> Running this with ftputil 2.4 on my computer takes about
> >> 25 minutes. When increasing the cache size to 2000, the code
> >> runs in about 40 seconds. :)
> > 
> > works in about 40 seconds on my box too, this time is acceptable,
> > however on the same directory a standard ftpclient give the directory
> > listing in few seconds. I haven't look at ftputil code but host.listdir
> > is fast such as filezilla and company (2-3 seconds) what other tasks
> > ftputil does in about 35 seconds?
> 
> First, I'd like to add that if I leave the print statements
> out of the loop, i. e.
> 
> >>> def f():
> ...   for i in lista:
> ...     isd = host.path.isdir(folder+i)
> ...     isf = host.path.isfile(folder+i)
> 
> I'm down to 25 seconds (with a cache size of 2000). :)
> 
> Using only one cache access per name (as listdir does) in the
> loop, I get the loop done in 11 seconds. The following function,
> including connecting to the server, runs in about 13 seconds:
> 
> >>> def h():
> ...   host = ftputil.FTPHost('ftp.nluug.nl','anonymous','pippo at pippo.com')
> ...   host.stat_cache.resize(2000)
> ...   for i in host.listdir(folder):
> ...     s = host.lstat(folder + i)
> >>> %time h()  # built into IPython
> CPU times: user 9.38 s, sys: 0.08 s, total: 9.46 s
> Wall time: 12.59 s
> 
> However, listdir (needing 1.2 seconds for the directory) only
> _stores_ values in the cache while lstat _retrieves_ the info. So
> I suspect the retrieval is significantly slower than the storage.
> 
> A little test:
> 
> >>> def g():
> ...   for i in lista:
> ...     s = host.lstat(folder+i)
> >>> %prun g()  # built into IPython
>          11358799 function calls in 54.934 CPU seconds
> 
>    Ordered by: internal time
> 
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>   5661503   25.974    0.000   38.745    0.000 lrucache.py:110(__cmp__)
>      2386   15.896    0.007   54.641    0.023 {_heapq.heapify}
>   5661503   12.771    0.000   12.771    0.000 {cmp}
> ...
> 
> The total time of the other calls seems negligible, so ftputil's
> own code seems rather innocent. ;-)
> 
> If you're writing an end-user client and need to give the user a
> directory listing fast, including the stat'ed information for
> each file/dir, the following - untested - approach _might_ work:
> 
> - Write a cache class with an interface like that in
>   ftp_stat_cache.py. However, use a raw Python dictionary to
>   store the stat results. This cache will have no (automatic)
>   means to prune old entries, but see below. You can use the
>   present cache class a template.
> 
> - Derive a stat class from ftp_stat._Stat:
> 
>   class MyStat(ftp_stat._Stat):
>       def __init__(self, *args, **kwargs):
>           super(MyStat, self).__init__(*args, **kwargs)
>           self._lstat_cache = MyCache()
> 
> - Derive a class from FTPHost:
> 
>   class MyFTPHost(ftputil.FTPHost):
>       def __init__(self, *args, **kwargs):
>           super(MyFTPHost, self).__init__(*args, **kwargs)
>           self._stat = MyStat(self)
>           self.stat_cache = self._stat._lstat_cache
> 
> - To get a listing, instantiate your custom FTPHost class. Clear
>   the cache with host.stat_cache.clear() before retrieving a
>   directory listing. Also clear the cache explicitly after you no
>   longer need the directory data.
> 
> Keep in mind that if your software isn't interactive, you most
> probably don't need to worry about tuning at all. [1] ... And
> _with_ interactivity, I just measured: Nautilus 2.24.1 needs
> about 8 seconds to show the directory listing, so I think the
> 13 seconds I got with the pure lstat calls (see above) are not so
> bad! Also remember that most users won't have so many directory
> items frequently. For most directories, you won't notice any
> difference.
> 
> If you nevertheless tried out the above idea, I (and maybe other
> readers of the list) would be very thankful if you shared your
> results. :-)
> 
> > It is possible to have a fastest
> > listing or the code is already optimized?
> 
> As far as I remember, I haven't optimized the code at all because
> I haven't had a use case yet where the code was too slow for me. I
> don't know how much tuning has gone into lrucache [2], though.
> 
> [1] Please read
>     http://sschwarzer.com/download/optimization_europython2006.pdf
>     if you haven't already done so. :)
> 
> [2] http://pypi.python.org/pypi/lrucache/0.2 (It's listed there
>     as alpha code, but to my question over two years ago the
>     author replied it were used successfully in several projects
>     and that it had comprehensive unit tests. I haven't had any
>     complaints either.)
> 
> Best regards,
> Stefan
> _______________________________________________
> ftputil mailing list
> ftputil at codespeak.net
> http://codespeak.net/mailman/listinfo/ftputil
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090501/567e5e01/attachment-0001.htm 

From sschwarzer at sschwarzer.net  Fri May  1 19:22:26 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Fri, 01 May 2009 19:22:26 +0200
Subject: [ftputil] Ftputil seems really slow to distinguish file and
	folders
In-Reply-To: <200905011744.20820.lists@svrinformatica.it>
References: <200904292246.56413.lists@svrinformatica.it>	<200904302350.03918.lists@svrinformatica.it>	<49FAB866.4040106@sschwarzer.net>
	<200905011744.20820.lists@svrinformatica.it>
Message-ID: <49FB2FD2.6040802@sschwarzer.net>

Hi Nicola,

On 2009-05-01 17:44, MailingList SVR wrote:
> what I mean for fast mode is use only host.listdir to provide the
> distintion beetween files and directory. Is it possible to retrieve this
> basic info only with listdir and so in 1.2 seconds? This way we can give
> a very fast directory listing and for example give the whole
> informations only if len(host.listdir) < x

Yes and no. Actually, listdir saves a stat object for each item
in the directory it retrieves (see method _real_list_dir in the
module ftp_stat.py). This is the cache data you would access if
you implemented the approach I showed in my last mail. However,
there's currently no way (and most probably won't be) of getting
this stat data directly as a return value of the listdir call
(or rather not without modifying ftputil's code; you'll have to
patch the code for this yourself).

If the experiment I pointed at proves to be a great success, I
_may_ add a way to ftputil to plug in the "dumb" ;) but fast
cache.

Best regards,
Stefan



From peter at pjstirling.plus.com  Sun May  3 05:38:47 2009
From: peter at pjstirling.plus.com (Peter Stirling)
Date: Sun, 03 May 2009 04:38:47 +0100
Subject: [ftputil] lrucache-related bug #32
In-Reply-To: <49FBFCCA.4000407@sschwarzer.net>
References: <49FBFCCA.4000407@sschwarzer.net>
Message-ID: <1241321927.21459.2.camel@peter-desktop>

Hi,

The vista box is host running virtualbox with ubuntu as guest rather
than the converse :)

I've put a patch on the issue tracker that should fix the issue (it does
for my tests at least).


 On Sat, 2009-05-02 at 09:56 +0200, Stefan Schwarzer wrote:
> Hello Peter,
> 
> I've been busy investigating the ftputil bug you thankfully
> reported. :-)
> 
> Please read my recent comments in the bug tracker at
> 
>     http://ftputil.sschwarzer.net/trac/ticket/32
> 
> and see how you can help. In particular, I'm interested in
> whether your Windows Vista runs natively or in a virtual
> machine. If the latter, on which host OS and in which kind
> of virtual machine?
> 
> I'm still trying to find out where's the first line where
> the code on Windows and Linux behaves differently.
> 
> For now, it seems that using time.clock instead of time.time
> in the lrucache module seemingly makes the bug go away but as
> long as I don't have a clue _why_ this is a problem I can't
> consider the bug fixed. :-/
> 
> Best regards,
> Stefan


From sschwarzer at sschwarzer.net  Sun May  3 11:54:12 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Sun, 03 May 2009 11:54:12 +0200
Subject: [ftputil] Mailing list vs. bug tracker
Message-ID: <49FD69C4.6030706@sschwarzer.net>

Hello everybody :-)

To make both the mailing list and the bug tracker more helpful
and avoid some confusion when tracking bugs, I'd like to make
these suggestions:

Apart from general discussions on ftputil, which are ok here
anyway, you can of course post on the mailing list if you aren't
sure if you use ftputil correctly or aren't sure if some behavior
is a bug or not.

Once a bug is identified as one, please enter it into the bug
tracker at http://ftputil.sschwarzer.net/trac/newticket . As a
spam prevention, you have to log in first via the "Login" link
with the user name ftputiluser and the password ftputil. Provide
any information that might be helpful in resolving the bug.

After that, let's work on the bug only via the bug tracker.
Please do _not_ send the same information also (or only) to the
mailing list. You won't miss any of my comments in the bug
tracker if you add your address to the Cc list or if you
subscribe to the RSS feed of the ticket (given you also check the
feed ;) ).

If you are certain that some behavior is a bug, you may enter it
into the tracker without sending anything to the mailing list. I
automatically get a mail for each addition or change of a ticket,
so I'll see your report anyway. By the way, my mail client shows
me if there are new mails on the list, so there's no need to send
a mail to the list with my personal address in the To or Cc
field. :)

I hope all these guidelines don't repel you from reporting bugs
in the first place. I like it better if you send all your bug
stuff to the mailing list than if you don't report something at
all! If in doubt, feel free to ignore everything I asked for in
this mail. :-) I'm thankful for bug reports and follow-ups, no
matter how they reach me.

If you think that something of the above doesn't make sense or
have suggestions yourself, please reply accordingly. :-)

Best regards,
Stefan

From aciddeath at gmail.com  Mon May  4 18:29:28 2009
From: aciddeath at gmail.com (Richard Holden)
Date: Mon, 04 May 2009 10:29:28 -0600
Subject: [ftputil] Mailing list vs. bug tracker
In-Reply-To: <49FD69C4.6030706@sschwarzer.net>
References: <49FD69C4.6030706@sschwarzer.net>
Message-ID: <49FF17E8.7010404@gmail.com>

Stefan Schwarzer wrote:
> Hello everybody :-)
>
> After that, let's work on the bug only via the bug tracker.
> Please do _not_ send the same information also (or only) to the
> mailing list. You won't miss any of my comments in the bug
> tracker if you add your address to the Cc list or if you
> subscribe to the RSS feed of the ticket (given you also check the
> feed ;) ).

Can we set it up so that either bug activity is sent to the mailing list
or there is a separate mailing list for bug activity. I know I don't
check bugs regularly but I always check my email from the list. For me
it would make it easier to track what is happening and flag any bugs I
need to be aware of.

> If you think that something of the above doesn't make sense or
> have suggestions yourself, please reply accordingly. :-)
Thanks for all your work on this.

> Best regards,
> Stefan

-Richard Holden

From sschwarzer at sschwarzer.net  Wed May  6 21:08:20 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Wed, 06 May 2009 21:08:20 +0200
Subject: [ftputil] [ANN] ftputil tickets mailing list
Message-ID: <4A01E024.4050007@sschwarzer.net>

Hello,

we now have a dedicated mailing list which automatically
receives the entries/comments into the issue tracker at
http://ftputil.sschwarzer.net/trac/report/6 by e-mail.

To subscribe, visit
http://codespeak.net/mailman/listinfo/ftputil-tickets .

Note that the list is read-only. Anything else would defeat
the purpose of an extra list because then we had _three_
potential targets for bug-related mail (ticket system on
the web, general mailing list and ticket mailing list).

I hope I've everything set up correctly. If something seems
to be wrong, please send me an e-mail. Thanks a lot in
advance. :)

Stefan

From sschwarzer at sschwarzer.net  Sat May  9 14:27:47 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Sat, 09 May 2009 14:27:47 +0200
Subject: [ftputil] Converting 12am and 12pm to 24-clock format
Message-ID: <4A0576C3.9030901@sschwarzer.net>

Hello,

I've always used to think that if you have a 12-hour time with
"PM" at the end, you could add 12 hours to get to 24-hour clock
(e. g. 01:23PM -> 13:23). In ftputil, the MS parser for directory
lines assumes this.

Now, reading the Wikipedia article on 12-hour clock [1], it seems
that things are different for the hour 12:

"""
Time in digital equipment

If a 12-hour time format is set, Microsoft Windows and Office
applications denote noon by "12 pm" and midnight by "12 am".
"""

So "12:23PM" -> 12:23 and "12:23AM" -> 00:23, right? Is that also
true for the listings of Microsoft FTP server products? I hope
there aren't any changes to the date to apply at 12:xxAM. So I'd
have these conversions from date/time strings to Python time
tuples:

04-17-09 12:08AM -> (2009, 4, 17, 0, 8, 0, ...)
04-17-09 12:08PM -> (2009, 4, 17, 12, 8, 0, ...)

Can anyone confirm this (or not), please?

Assuming that I understand the Wikipedia article correctly and
things are also like this for Microsoft's FTP server, I'd patch
ftp_stat.py thusly:

$ svn di ftp_stat.py
Index: ftp_stat.py
===================================================================
--- ftp_stat.py (revision 861)
+++ ftp_stat.py (working copy)
@@ -232,7 +232,9 @@
             hour, minute = int(hour), int(minute)
         except (ValueError, IndexError):
             raise ftp_error.ParserError("invalid time string '%s'" % time_)
-        if am_pm == 'P':
+        if am_pm == 'A' and hour == 12:
+            hour = 0
+        if am_pm == 'P' and hour != 12:
             hour = hour + 12
         st_mtime = time.mktime( (year, month, day,
                                  hour, minute, 0, 0, 0, -1) )

Do you agree?

It would be nice if I got an answer soon, so I could include this
fix in an ftputil 2.4.1 release on this weekend. :-)

[1] http://en.wikipedia.org/wiki/12-hour_clock#Time_in_digital_equipment

Best regards,
Stefan


From aciddeath at gmail.com  Sun May 10 01:06:12 2009
From: aciddeath at gmail.com (Richard Holden)
Date: Sat, 9 May 2009 17:06:12 -0600
Subject: [ftputil] Converting 12am and 12pm to 24-clock format
In-Reply-To: <4A0576C3.9030901@sschwarzer.net>
References: <4A0576C3.9030901@sschwarzer.net>
Message-ID: <06E46AFB-C84D-4606-B63A-8BEB994A3047@gmail.com>

The code looks correct, I don't know if Microsoft FTP handles it all  
correctly.

Sent from my iPhone

On May 9, 2009, at 6:27 AM, Stefan Schwarzer  
<sschwarzer at sschwarzer.net> wrote:

> Hello,
>
> I've always used to think that if you have a 12-hour time with
> "PM" at the end, you could add 12 hours to get to 24-hour clock
> (e. g. 01:23PM -> 13:23). In ftputil, the MS parser for directory
> lines assumes this.
>
> Now, reading the Wikipedia article on 12-hour clock [1], it seems
> that things are different for the hour 12:
>
> """
> Time in digital equipment
>
> If a 12-hour time format is set, Microsoft Windows and Office
> applications denote noon by "12 pm" and midnight by "12 am".
> """
>
> So "12:23PM" -> 12:23 and "12:23AM" -> 00:23, right? Is that also
> true for the listings of Microsoft FTP server products? I hope
> there aren't any changes to the date to apply at 12:xxAM. So I'd
> have these conversions from date/time strings to Python time
> tuples:
>
> 04-17-09 12:08AM -> (2009, 4, 17, 0, 8, 0, ...)
> 04-17-09 12:08PM -> (2009, 4, 17, 12, 8, 0, ...)
>
> Can anyone confirm this (or not), please?
>
> Assuming that I understand the Wikipedia article correctly and
> things are also like this for Microsoft's FTP server, I'd patch
> ftp_stat.py thusly:
>
> $ svn di ftp_stat.py
> Index: ftp_stat.py
> ===================================================================
> --- ftp_stat.py (revision 861)
> +++ ftp_stat.py (working copy)
> @@ -232,7 +232,9 @@
>             hour, minute = int(hour), int(minute)
>         except (ValueError, IndexError):
>             raise ftp_error.ParserError("invalid time string '%s'" %  
> time_)
> -        if am_pm == 'P':
> +        if am_pm == 'A' and hour == 12:
> +            hour = 0
> +        if am_pm == 'P' and hour != 12:
>             hour = hour + 12
>         st_mtime = time.mktime( (year, month, day,
>                                  hour, minute, 0, 0, 0, -1) )
>
> Do you agree?
>
> It would be nice if I got an answer soon, so I could include this
> fix in an ftputil 2.4.1 release on this weekend. :-)
>
> [1] http://en.wikipedia.org/wiki/12-hour_clock#Time_in_digital_equipment
>
> Best regards,
> Stefan
>
> _______________________________________________
> ftputil mailing list
> ftputil at codespeak.net
> http://codespeak.net/mailman/listinfo/ftputil

From sschwarzer at sschwarzer.net  Sun May 10 16:23:08 2009
From: sschwarzer at sschwarzer.net (Stefan Schwarzer)
Date: Sun, 10 May 2009 16:23:08 +0200
Subject: [ftputil] [ANN] ftputil 2.4.1 released
Message-ID: <4A06E34C.8010108@sschwarzer.net>

ftputil 2.4.1 is now available from
http://ftputil.sschwarzer.net/download .

Changes since version 2.4
-------------------------

Several bugs were fixed:

- On Windows, some accesses to the stat cache caused it to become
  inconsistent, which could also trigger exceptions (report and patch
  by Peter Stirling).

- In ftputil 2.4, the use of ``super`` in the exception base class
  caused ftputil to fail on Python <2.5 (reported by Nicola Murino).
  ftputil is supposed to run with Python 2.3+.

- The conversion of 12-hour clock times to 24-hour clock in the MS
  format parser was wrong for 12 AM and 12 PM.

Upgrading is strongly recommended.

What is ftputil?
----------------

ftputil is a high-level FTP client library for the Python programming
language. ftputil implements a virtual file system for accessing FTP
servers, that is, it can generate file-like objects for remote files.
The library supports many functions similar to those in the os,
os.path and shutil modules. ftputil has convenience functions for
conditional uploads and downloads, and handles FTP clients and servers
in different timezones.

Read the documentation at
http://ftputil.sschwarzer.net/documentation .

License
-------

ftputil is Open Source software, released under the revised BSD
license (see http://www.opensource.org/licenses/bsd-license.php ).

Stefan

From Neue  Tue May 19 20:42:07 2009
From: Neue (Neue)
Date: Tue, 19 May 2009 20:42:07 +0200
Subject: [ftputil] Auf Dauer Kannst du Hier Richtig Sparen!
Message-ID: <01c9d8c2$46367980$9722b04e@pljeseyonline-msn>

Jetzt legale Software ohne Ende runterladen. Europaische Hersteller bieten Ihnen die Moeglichkeit beim Kauf zu sparen.

http://hallmamape81.tk

* Adobe Photoshop CS4 Extended: &euro;119.95
* Nero 9: &euro;39.95
* iLife '09: &euro;59.95
* Adobe Creative Suite 4 Master Collection: &euro;329.95 
* Office 2003 Professional: &euro;59.95
* iWork ?09: &euro;59.95

http://hallmamape81.tk

Geben Sie bei Ihrer Bestellund den Werbe Code D33W-3003 und schon erhalten Sie einen Rabbat von 30%.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/ftputil/attachments/20090519/f6302a8a/attachment.htm