From SridharR at activestate.com  Tue Jul  7 01:20:24 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Mon, 06 Jul 2009 16:20:24 -0700
Subject: [py-dev] stdout encoding is set to None
Message-ID: <op.uwn5sad6brrvlq@double>

I noticed that from within py.test .. `sys.stdout.encoding` is set to  
None. This makes it impossible to print non-ascii unicode strings. To  
workaround this error, I wrapped the stdout stream using codecs.getwriter  
as explained in the PrintFails[1] python wiki page. Here's my snippet:

	def _message(self, msg, *args):
	    with _good_terminal_stream() as stream:
	        msg = u'[ui] {0}'.format(msg)
	        print >>stream, msg % args

	@contextmanager
	def _good_terminal_stream():
	    """Return a proper terminal stream"""
	    preferredencoding = locale.getpreferredencoding()
	    assert preferredencoding not in (None, 'ascii'), \
		'improper locale encoding: {0}'.format(
		preferredencoding)
	    stream = codecs.getwriter(preferredencoding)(sys.stdout)
	    try:
		yield stream
	    finally:
		stream.close()

However, this lead to py.test internal error:

	    def done(self):
		""" unpatch and clean up, returns the self.tmpfile (file object)
		    """
		os.dup2(self._savefd, self.targetfd)
		self.unsetfiles()
		os.close(self._savefd)
	>       self.tmpfile.seek(0)
	E       ValueError: I/O operation on closed file

	/home/sridharr/as/pypm-txtui/eggs/py-1.0.0b7-py2.6.egg/py/io/fdcapture.py:39:  
ValueError

I've made available the entire log file here:  
http://files.getdropbox.com/u/87045/tmp/pytesterror

-srid

*****
[1] http://wiki.python.org/moin/PrintFails

From fdg001 at gmx.net  Tue Jul  7 11:30:17 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Tue, 07 Jul 2009 10:30:17 +0100
Subject: [py-dev] using docstrings for test descriptions
Message-ID: <4A5315A9.5030401@gmx.net>

Hello all,

Holger's presentations at EuroPython last week convinced me that I 
should finally start migrating to py.test.

There's only one thing I'm missing: I'm used to providing fairly 
detailed descriptions for individual tests ("foo returns bar if baz").
While one could use the assert statement's second argument for this, 
that seems cumbersome and has some undesired side-effects (e.g. 
reporting "E    AssertionError: <description>" instead of displaying the 
respective values).
Docstrings appear to be the obvious (and pythonic) solution.

So I imagine a plugin could add docstring support, which might result in 
output as described here:
     http://gist.github.com/141977
(Being new to py.test, it is possible my expectations are misguided - 
don't hesitate to point out where that's the case.)

I'd be willing to look into how this might be implemented; it shouldn't 
be too hard (e.g. using a decorator), but some pointers on how to get 
started would be appreciated.


Thanks,

Fred

From holger at merlinux.eu  Tue Jul  7 12:11:39 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 7 Jul 2009 12:11:39 +0200
Subject: [py-dev] stdout encoding is set to None
In-Reply-To: <op.uwn5sad6brrvlq@double>
References: <op.uwn5sad6brrvlq@double>
Message-ID: <20090707101138.GJ7437@trillke.net>

Hi Sridhar, 

indeed, py.test by default redirects the stdout/stderr file
descriptors to a file, see py/io/fdcapture.py.  This leads to
the encoding error.  I guess the redirection file should
better be opened with the "preferredencoding" similar to what
you do.  Can you imagine submitting an according test and patch? 

Your context manager causes problems, btw, because it closes 
the redirection-stream which it shouldn't do.  

best,

holger

On Mon, Jul 06, 2009 at 16:20 -0700, Sridhar Ratnakumar wrote:
> I noticed that from within py.test .. `sys.stdout.encoding` is set to  
> None. This makes it impossible to print non-ascii unicode strings. To  
> workaround this error, I wrapped the stdout stream using codecs.getwriter  
> as explained in the PrintFails[1] python wiki page. Here's my snippet:
> 
> 	def _message(self, msg, *args):
> 	    with _good_terminal_stream() as stream:
> 	        msg = u'[ui] {0}'.format(msg)
> 	        print >>stream, msg % args
> 
> 	@contextmanager
> 	def _good_terminal_stream():
> 	    """Return a proper terminal stream"""
> 	    preferredencoding = locale.getpreferredencoding()
> 	    assert preferredencoding not in (None, 'ascii'), \
> 		'improper locale encoding: {0}'.format(
> 		preferredencoding)
> 	    stream = codecs.getwriter(preferredencoding)(sys.stdout)
> 	    try:
> 		yield stream
> 	    finally:
> 		stream.close()
> 
> However, this lead to py.test internal error:
> 
> 	    def done(self):
> 		""" unpatch and clean up, returns the self.tmpfile (file object)
> 		    """
> 		os.dup2(self._savefd, self.targetfd)
> 		self.unsetfiles()
> 		os.close(self._savefd)
> 	>       self.tmpfile.seek(0)
> 	E       ValueError: I/O operation on closed file
> 
> 	/home/sridharr/as/pypm-txtui/eggs/py-1.0.0b7-py2.6.egg/py/io/fdcapture.py:39:  
> ValueError
> 
> I've made available the entire log file here:  
> http://files.getdropbox.com/u/87045/tmp/pytesterror
> 
> -srid
> 
> *****
> [1] http://wiki.python.org/moin/PrintFails
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Tue Jul  7 12:28:53 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 7 Jul 2009 12:28:53 +0200
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <4A5315A9.5030401@gmx.net>
References: <4A5315A9.5030401@gmx.net>
Message-ID: <20090707102853.GL7437@trillke.net>

Hi Frederik, 

On Tue, Jul 07, 2009 at 10:30 +0100, Frederik Dohr wrote:
> Hello all,
> 
> Holger's presentations at EuroPython last week convinced me that I 
> should finally start migrating to py.test.

nice to hear :) 
 
> There's only one thing I'm missing: I'm used to providing fairly 
> detailed descriptions for individual tests ("foo returns bar if baz").
> While one could use the assert statement's second argument for this, 
> that seems cumbersome and has some undesired side-effects (e.g. 
> reporting "E    AssertionError: <description>" instead of displaying the 
> respective values).
> Docstrings appear to be the obvious (and pythonic) solution.

makes sense i think. 

> So I imagine a plugin could add docstring support, which might result in 
> output as described here:
>      http://gist.github.com/141977
> (Being new to py.test, it is possible my expectations are misguided - 
> don't hesitate to point out where that's the case.)

Python's unittest.py uses the docstrings to substitute the test
function name with the docstring.  Imitating this "py.test --verbose" 
could look like this: 

    lorem ipsum: PASS 
    dolor sit amet: PASS 
    consectetur adipisic...liqua: FAIL 

The failure tracebacks probably don't need to do anything special
as they anyway report the test function source code including
the docstring usually. 

makes senses?  Anybody else has opinions? 

> I'd be willing to look into how this might be implemented; it shouldn't 
> be too hard (e.g. using a decorator), but some pointers on how to get 
> started would be appreciated.

I think you can just patch py.test proper, more specifically 
py/test/plugin/pytest_terminal.py which is the single file
containing all the terminal reporting and also includes tests.
Please feel free to submit a patch.  

best,
holger

From fdg001 at gmx.net  Wed Jul  8 21:12:17 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Wed, 08 Jul 2009 20:12:17 +0100
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <20090707102853.GL7437@trillke.net>
References: <4A5315A9.5030401@gmx.net> <20090707102853.GL7437@trillke.net>
Message-ID: <4A54EF91.9060004@gmx.net>

Thanks for the kind response!

>>      http://gist.github.com/141977
> 
> Python's unittest.py uses the docstrings to substitute the test
> function name with the docstring.  Imitating this "py.test --verbose" 
> could look like this: 
>     lorem ipsum: PASS 

I had intentionally avoided that in my proposal, as I didn't want to 
break the standard (machine-readable) "file:line:message" format.
Not entirely sure whether that's a valid concern though.

> I think you can just patch py.test proper, more specifically 
> py/test/plugin/pytest_terminal.py

Thanks, that looks promising.
Might take me a week or two to come up with something presentable 
(things are a little busy, in part due to the post-conference backlog), 
but I'll definitely give it a shot and report back here.


-- F.

From fdg001 at gmx.net  Wed Jul  8 23:05:30 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Wed, 08 Jul 2009 22:05:30 +0100
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <4A54EF91.9060004@gmx.net>
References: <4A5315A9.5030401@gmx.net> <20090707102853.GL7437@trillke.net>
	<4A54EF91.9060004@gmx.net>
Message-ID: <4A550A1A.8080803@gmx.net>

> Thanks, that looks promising.
> Might take me a week or two to come up with something presentable

I might have spoken too soon.
After some naive hacking, I now have a simple proof of concept. In the 
spirit of "release early, release often", here's the patch:
     http://gist.github.com/141977#LID1
(also attached below)
Plus some sample output:
     http://gist.github.com/141977#LID49

While this is certainly not release-ready, I'd appreciate feedback on 
whether it's the right approach.


Thanks,

Fred


Index: py/test/plugin/pytest_terminal.py
===================================================================
--- py/test/plugin/pytest_terminal.py	(revision 66152)
+++ py/test/plugin/pytest_terminal.py	(working copy)
@@ -284,6 +284,11 @@
          except AttributeError:
              pass
          reportinfo = item.config.hook.pytest_report_iteminfo(item=item)
+        # add docstring to message
+        docstring = item.obj.__doc__ # TODO: shorten
+        reportinfo = list(reportinfo) # mutable type req'd -- XXX: hacky
+        reportinfo[-1] = "%s: %s" % (reportinfo[-1], docstring)
+        reportinfo = tuple(reportinfo)
          # cache on item
          item.__reportinfo = reportinfo
          return reportinfo

From SridharR at activestate.com  Thu Jul  9 01:19:52 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Wed, 08 Jul 2009 16:19:52 -0700
Subject: [py-dev] py.test and logging issue: ValueError: I/O operation on
	closed file
Message-ID: <op.uwru3etnbrrvlq@double>

holger and others,

do you have any comments to add to this bug?

   http://bugs.python.org/issue6333

-srid

From holger at merlinux.eu  Thu Jul  9 13:20:57 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 9 Jul 2009 13:20:57 +0200
Subject: [py-dev] py.test and logging issue: ValueError: I/O
	operation	on closed file
In-Reply-To: <op.uwru3etnbrrvlq@double>
References: <op.uwru3etnbrrvlq@double>
Message-ID: <20090709112056.GO7437@trillke.net>

On Wed, Jul 08, 2009 at 16:19 -0700, Sridhar Ratnakumar wrote:
> holger and others,
> 
> do you have any comments to add to this bug?
> 
>    http://bugs.python.org/issue6333

i added a comment there, basically suggesting to follow your suggestion.  

As py.test should nicely work on many CPython versions i also
did a general workaround in py.test proper, see here:

    http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/

I intend to release a 1.0.0b8 tomorrow with this 
and some other fixes.  

best & thanks,
holger

From holger at merlinux.eu  Thu Jul  9 13:47:09 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 9 Jul 2009 13:47:09 +0200
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <4A550A1A.8080803@gmx.net>
References: <4A5315A9.5030401@gmx.net> <20090707102853.GL7437@trillke.net>
	<4A54EF91.9060004@gmx.net> <4A550A1A.8080803@gmx.net>
Message-ID: <20090709114709.GP7437@trillke.net>

On Wed, Jul 08, 2009 at 22:05 +0100, Frederik Dohr wrote:
>> Thanks, that looks promising.
>> Might take me a week or two to come up with something presentable
>
> I might have spoken too soon.
> After some naive hacking, I now have a simple proof of concept. In the  
> spirit of "release early, release often", here's the patch:

good idea :) 

>     http://gist.github.com/141977#LID1
> (also attached below)
> Plus some sample output:
>     http://gist.github.com/141977#LID49
>
> While this is certainly not release-ready, I'd appreciate feedback on  
> whether it's the right approach.

sure, here is some quick feedback.

a) Your change presumes that all test items are python test
   functions which is not true.  With 

      if isinstance(item, py.test.collect.Function) 

   you can check that you are dealing with python test
   functions. 

b) it's probably better to do the reporting tweak in 
   "def _reportinfoline" which deals with
   exactly presenting the verbose line. 

c) please add a test to the pytest_terminal.py file 
   and i recommend to rather copy+modify the "test_verbose" 
   one from py/test/testing/acceptance_test.py 
   you can simplify it of course to just test output
   for a single test function.  

best,
holger

>
>
> Index: py/test/plugin/pytest_terminal.py
> ===================================================================
> --- py/test/plugin/pytest_terminal.py	(revision 66152)
> +++ py/test/plugin/pytest_terminal.py	(working copy)
> @@ -284,6 +284,11 @@
>          except AttributeError:
>              pass
>          reportinfo = item.config.hook.pytest_report_iteminfo(item=item)
> +        # add docstring to message
> +        docstring = item.obj.__doc__ # TODO: shorten
> +        reportinfo = list(reportinfo) # mutable type req'd -- XXX: hacky
> +        reportinfo[-1] = "%s: %s" % (reportinfo[-1], docstring)
> +        reportinfo = tuple(reportinfo)
>          # cache on item
>          item.__reportinfo = reportinfo
>          return reportinfo
>

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From SridharR at activestate.com  Thu Jul  9 22:03:01 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Thu, 09 Jul 2009 13:03:01 -0700
Subject: [py-dev] py.test and logging issue: ValueError: I/O operation
 on closed file
In-Reply-To: <20090709112056.GO7437@trillke.net>
References: <op.uwru3etnbrrvlq@double> <20090709112056.GO7437@trillke.net>
Message-ID: <op.uwtgnbnhbrrvlq@double>

On Thu, 09 Jul 2009 04:20:57 -0700, holger krekel <holger at merlinux.eu>  
wrote:

> As py.test should nicely work on many CPython versions i also
> did a general workaround in py.test proper, see here:
>    http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/

This will suppress *all* exceptions thrown by the logging module, no?

-srid

From py-dev at tolomea.com  Fri Jul 10 07:44:53 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 10 Jul 2009 15:44:53 +1000
Subject: [py-dev] execnet: redirecting stdout and stderr from remote threads
Message-ID: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>

I have execnet's stdout and stderr redirecting working and it does a
fine job for the initial thread, but I'm also creating a bunch of
helper threads on the remote end of the connection and I can't figure
out how to get the stdout and stderr for them redirected.

Below is an example of what I thought might work but doesn't.

import py
import sys

code = """
def func():
   print 'hello world'

import threading
t=threading.Thread(target=func)
t.start()
t.join()
"""

gateway = py.execnet.SshGateway("localhost")
channel = gateway.remote_exec(code, sys.stdout.write)
channel.waitclose()


Regards Gordon

From holger at merlinux.eu  Fri Jul 10 09:36:54 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 09:36:54 +0200
Subject: [py-dev] py.test and logging issue: ValueError: I/O
	operation	on closed file
In-Reply-To: <op.uwtgnbnhbrrvlq@double>
References: <op.uwru3etnbrrvlq@double> <20090709112056.GO7437@trillke.net>
	<op.uwtgnbnhbrrvlq@double>
Message-ID: <20090710073654.GQ7437@trillke.net>

On Thu, Jul 09, 2009 at 13:03 -0700, Sridhar Ratnakumar wrote:
> On Thu, 09 Jul 2009 04:20:57 -0700, holger krekel <holger at merlinux.eu>  
> wrote:
>
>> As py.test should nicely work on many CPython versions i also
>> did a general workaround in py.test proper, see here:
>>    http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/
>
> This will suppress *all* exceptions thrown by the logging module, no?

no.  It is only surpressed when the whole testing process is
about to end - so should not affect any test or teardown
functions, just the atexit-handling of logging basically. 

holger

From holger at merlinux.eu  Fri Jul 10 09:58:25 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 09:58:25 +0200
Subject: [py-dev] execnet: redirecting stdout and stderr from
	remote	threads
In-Reply-To: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
Message-ID: <20090710075825.GR7437@trillke.net>

On Fri, Jul 10, 2009 at 15:44 +1000, Gordon Wrigley wrote:
> I have execnet's stdout and stderr redirecting working and it does a
> fine job for the initial thread, but I'm also creating a bunch of
> helper threads on the remote end of the connection and I can't figure
> out how to get the stdout and stderr for them redirected.
> 
> Below is an example of what I thought might work but doesn't.
> 
> import py
> import sys
> 
> code = """
> def func():
>    print 'hello world'
> 
> import threading
> t=threading.Thread(target=func)
> t.start()
> t.join()
> """
> 
> gateway = py.execnet.SshGateway("localhost")
> channel = gateway.remote_exec(code, sys.stdout.write)
> channel.waitclose()

stdout redirection is thread-aware.  This is because 
when calling 

    gateway.remote_init_threads(num=3)

you would have three execution threads on the remote
side and stdout redirection would separate the three
execution's stdout redirections.   Can you use
maybe make use of this threading handling?  If not
then maybe we can think of introducing a way 
to configure the exact behaviour but it would complicate
the API a bit which i'd like to avoid. 

best,
holger

From py-dev at tolomea.com  Fri Jul 10 12:21:27 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 10 Jul 2009 20:21:27 +1000
Subject: [py-dev] execnet: redirecting stdout and stderr from remote
	threads
In-Reply-To: <20090710075825.GR7437@trillke.net>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
Message-ID: <f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>

>> I have execnet's stdout and stderr redirecting working and it does a
>> fine job for the initial thread, but I'm also creating a bunch of
>> helper threads on the remote end of the connection and I can't figure
>> out how to get the stdout and stderr for them redirected.

> stdout redirection is thread-aware. ?This is because
> when calling
>
> ? ?gateway.remote_init_threads(num=3)
>
> you would have three execution threads on the remote
> side and stdout redirection would separate the three
> execution's stdout redirections. ? Can you use
> maybe make use of this threading handling? ?If not
> then maybe we can think of introducing a way
> to configure the exact behaviour but it would complicate
> the API a bit which i'd like to avoid.

I need to create the threads at the remote end and on demand so I
don't think I'm going to be able to use execnets threading support.
My usage pattern is rather bizarre, maybe a better way to approach
this would be to build my own stdout / stderr redirection on top of
execnet sockets.

Gordon

From holger at merlinux.eu  Fri Jul 10 12:53:45 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 12:53:45 +0200
Subject: [py-dev] execnet: redirecting stdout and stderr from
	remote	threads
In-Reply-To: <f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
Message-ID: <20090710105345.GS7437@trillke.net>

On Fri, Jul 10, 2009 at 20:21 +1000, Gordon Wrigley wrote:
> >> I have execnet's stdout and stderr redirecting working and it does a
> >> fine job for the initial thread, but I'm also creating a bunch of
> >> helper threads on the remote end of the connection and I can't figure
> >> out how to get the stdout and stderr for them redirected.
> 
> > stdout redirection is thread-aware. ?This is because
> > when calling
> >
> > ? ?gateway.remote_init_threads(num=3)
> >
> > you would have three execution threads on the remote
> > side and stdout redirection would separate the three
> > execution's stdout redirections. ? Can you use
> > maybe make use of this threading handling? ?If not
> > then maybe we can think of introducing a way
> > to configure the exact behaviour but it would complicate
> > the API a bit which i'd like to avoid.
> 
> I need to create the threads at the remote end and on demand so I
> don't think I'm going to be able to use execnets threading support.
> My usage pattern is rather bizarre, maybe a better way to approach
> this would be to build my own stdout / stderr redirection on top of
> execnet sockets.

yes, might make sense then.  I attached a working example
how to do general output redirection, maybe that helps you. 

Maybe also simplifying/separating the API makes sense, i.e. having a 

    remote_redirect_output(..., perthread=False)

and a simple 

    remote_exec(source) 

call.  That would cater your use case as well, i guess. 

best,
holger
 _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Fri Jul 10 13:05:01 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 13:05:01 +0200
Subject: [py-dev] execnet: redirecting stdout and stderr from
	remote	threads
In-Reply-To: <20090710105345.GS7437@trillke.net>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
	<20090710105345.GS7437@trillke.net>
Message-ID: <20090710110501.GT7437@trillke.net>

On Fri, Jul 10, 2009 at 12:53 +0200, holger krekel wrote:
> On Fri, Jul 10, 2009 at 20:21 +1000, Gordon Wrigley wrote:
> > >> I have execnet's stdout and stderr redirecting working and it does a
> > >> fine job for the initial thread, but I'm also creating a bunch of
> > >> helper threads on the remote end of the connection and I can't figure
> > >> out how to get the stdout and stderr for them redirected.
> > 
> > > stdout redirection is thread-aware. ?This is because
> > > when calling
> > >
> > > ? ?gateway.remote_init_threads(num=3)
> > >
> > > you would have three execution threads on the remote
> > > side and stdout redirection would separate the three
> > > execution's stdout redirections. ? Can you use
> > > maybe make use of this threading handling? ?If not
> > > then maybe we can think of introducing a way
> > > to configure the exact behaviour but it would complicate
> > > the API a bit which i'd like to avoid.
> > 
> > I need to create the threads at the remote end and on demand so I
> > don't think I'm going to be able to use execnets threading support.
> > My usage pattern is rather bizarre, maybe a better way to approach
> > this would be to build my own stdout / stderr redirection on top of
> > execnet sockets.
> 
> yes, might make sense then.  I attached a working example
> how to do general output redirection, maybe that helps you. 

now attached for real - let me know if there are problems with it.

holger
-------------- next part --------------
A non-text attachment was scrubbed...
Name: redirect_remote_output.py
Type: text/x-python
Size: 660 bytes
Desc: not available
Url : http://codespeak.net/pipermail/py-dev/attachments/20090710/a257fb1a/attachment.py 

From py-dev at tolomea.com  Fri Jul 10 13:07:50 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 10 Jul 2009 21:07:50 +1000
Subject: [py-dev] execnet: redirecting stdout and stderr from remote
	threads
In-Reply-To: <20090710110501.GT7437@trillke.net>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
	<20090710105345.GS7437@trillke.net>
	<20090710110501.GT7437@trillke.net>
Message-ID: <f2d961430907100407x1c7a7372r46b935294ea7d5bd@mail.gmail.com>

btw, that whole sending a channel over a channel thing is beautiful
and has proved very helpful

G

On Fri, Jul 10, 2009 at 9:05 PM, holger krekel<holger at merlinux.eu> wrote:
> On Fri, Jul 10, 2009 at 12:53 +0200, holger krekel wrote:
>> On Fri, Jul 10, 2009 at 20:21 +1000, Gordon Wrigley wrote:
>> > >> I have execnet's stdout and stderr redirecting working and it does a
>> > >> fine job for the initial thread, but I'm also creating a bunch of
>> > >> helper threads on the remote end of the connection and I can't figure
>> > >> out how to get the stdout and stderr for them redirected.
>> >
>> > > stdout redirection is thread-aware. ?This is because
>> > > when calling
>> > >
>> > > ? ?gateway.remote_init_threads(num=3)
>> > >
>> > > you would have three execution threads on the remote
>> > > side and stdout redirection would separate the three
>> > > execution's stdout redirections. ? Can you use
>> > > maybe make use of this threading handling? ?If not
>> > > then maybe we can think of introducing a way
>> > > to configure the exact behaviour but it would complicate
>> > > the API a bit which i'd like to avoid.
>> >
>> > I need to create the threads at the remote end and on demand so I
>> > don't think I'm going to be able to use execnets threading support.
>> > My usage pattern is rather bizarre, maybe a better way to approach
>> > this would be to build my own stdout / stderr redirection on top of
>> > execnet sockets.
>>
>> yes, might make sense then. ?I attached a working example
>> how to do general output redirection, maybe that helps you.
>
> now attached for real - let me know if there are problems with it.
>
> holger
>

From py-dev at tolomea.com  Fri Jul 10 13:16:30 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 10 Jul 2009 21:16:30 +1000
Subject: [py-dev] execnet: redirecting stdout and stderr from remote
	threads
In-Reply-To: <20090710105345.GS7437@trillke.net>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
	<20090710105345.GS7437@trillke.net>
Message-ID: <f2d961430907100416g4e199446u63910500ef635a04@mail.gmail.com>

> yes, might make sense then. ?I attached a working example
> how to do general output redirection, maybe that helps you.

I gave the example a look over and it all makes sense and is nice and
straight forward. So far I've maintained a 1 to 1 relation ship
between threads and channels, in the context of the example are there
any threading implications to having multiple threads calling into the
channel file that has been assigned to sys.stdout?

From holger at merlinux.eu  Fri Jul 10 13:23:52 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 13:23:52 +0200
Subject: [py-dev] execnet: redirecting stdout and stderr from
	remote	threads
In-Reply-To: <f2d961430907100416g4e199446u63910500ef635a04@mail.gmail.com>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
	<20090710105345.GS7437@trillke.net>
	<f2d961430907100416g4e199446u63910500ef635a04@mail.gmail.com>
Message-ID: <20090710112352.GU7437@trillke.net>

On Fri, Jul 10, 2009 at 21:16 +1000, Gordon Wrigley wrote:
> > yes, might make sense then. ?I attached a working example
> > how to do general output redirection, maybe that helps you.
> 
> I gave the example a look over and it all makes sense and is nice and
> straight forward. So far I've maintained a 1 to 1 relation ship
> between threads and channels, in the context of the example are there
> any threading implications to having multiple threads calling into the
> channel file that has been assigned to sys.stdout?

no problems i am aware off.  It *should* be safe to have
multiple threads using channel.send() concurrently. 
It depends on write()s to pipes (or sockets) being 
atomic eventually which i think they are.  

One issue that can pop up is tearing down the whole
machinery cleanly - that can sometimes be more
involved and i believe execnet still might need 
some more work there. 

holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From fdg001 at gmx.net  Sat Jul 11 09:42:02 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Sat, 11 Jul 2009 08:42:02 +0100
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <20090709114709.GP7437@trillke.net>
References: <4A5315A9.5030401@gmx.net> <20090707102853.GL7437@trillke.net>
	<4A54EF91.9060004@gmx.net> <4A550A1A.8080803@gmx.net>
	<20090709114709.GP7437@trillke.net>
Message-ID: <4A58424A.3050400@gmx.net>

> sure, here is some quick feedback [...]

Thanks Holger, much appreciated.
Those suggestions are very helpful; I'll see about incorporating them soon.


-- F.

From py-dev at tolomea.com  Mon Jul 13 03:40:10 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Mon, 13 Jul 2009 11:40:10 +1000
Subject: [py-dev] execnet: shutdown issues in a heavily threaded environment
Message-ID: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>

> One issue that can pop up is tearing down the whole
> machinery cleanly - that can sometimes be more
> involved and i believe execnet still might need
> some more work there.

On that note can you shed any light on this message:

Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
open>> ignored

Currently I'm only seeing them on my test machine, but they are
occurring very frequently, they drop out at the end of the py.test
invocation, I will paste the tail end of the py.test output below.

I'm not sure exactly what is causing this, naively it seems to
indicate a circular chain of references involving one or more
channels. However immediately prior to the conclusion of each test
exit gc.garbage is empty and there are none of my objects in
gc.get_objects.

Also this only started when I reworked our remote invocation system
recently, the most notable change in the new system is that it makes
more use of threads and particularly it has daemon threads waiting on
channel receive calls.


lab at okum:~$ uname -a
Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
2009 i686 GNU/Linux
lab at okum:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0b1'


============================================== 35 passed in 231.71
seconds ===============================================
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in Unhandled exception in thread started by <bound
method Thread.__bootstrap of <Thread(receiver, stopped daemon
1177635728)>>
<bound method Channel.__del__ of <Channel id=7 open>>Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/apport_python_hook.py", line
38, in apport_excepthook
    from apport.packaging_impl import impl as packaging
ImportError: No module named apport.packaging_impl

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.6/threading.py", line 538, in __bootstrap_inner
    (self.name, _format_exc()))
TypeError: 'NoneType' object is not callable
 ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored

From pedronis at openend.se  Mon Jul 13 19:39:06 2009
From: pedronis at openend.se (Samuele Pedroni)
Date: Mon, 13 Jul 2009 19:39:06 +0200
Subject: [py-dev] ANN: oejskit 0.8.5 JavaScript in-browser testing and
 utility kit with py.test plugin
Message-ID: <4A5B713A.9010704@openend.se>

jskit contains infrastructure and in particular a py.test plugin to
enable running unit tests for JavaScript code inside browsers. The
plugin requires py.test 1.0

The approach also enables to write integration tests such that the
JavaScript code is tested against server-side Python code mocked as
necessary. Any server-side framework that can already be exposed through
WSGI can play.

More information and downloading at:

http://pypi.python.org/pypi/oejskit

including documentation and the talk I gave at Europython.

jskit was initially developed by Open End AB and is released under the
MIT license.

In various incarnations it has been in use and useful at Open End for
more than a year, we are quite happy to share it.

Samuele Pedroni for Open End



From holger at merlinux.eu  Tue Jul 14 11:10:04 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 14 Jul 2009 11:10:04 +0200
Subject: [py-dev] execnet: shutdown issues in a heavily
	threaded	environment
In-Reply-To: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>
References: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>
Message-ID: <20090714091004.GY7437@trillke.net>

Hi Gordon,

On Mon, Jul 13, 2009 at 11:40 +1000, Gordon Wrigley wrote:
> > One issue that can pop up is tearing down the whole
> > machinery cleanly - that can sometimes be more
> > involved and i believe execnet still might need
> > some more work there.
> 
> On that note can you shed any light on this message:
> 
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
> open>> ignored
>
> Currently I'm only seeing them on my test machine, but they are
> occurring very frequently, they drop out at the end of the py.test
> invocation, I will paste the tail end of the py.test output below.

> I'm not sure exactly what is causing this, naively it seems to
> indicate a circular chain of references involving one or more
> channels. However immediately prior to the conclusion of each test
> exit gc.garbage is empty and there are none of my objects in
> gc.get_objects.

Yes, some cycle is probably there.  Your data point hints that
there is some cycle in the execnet code itself.  

> Also this only started when I reworked our remote invocation system
> recently, the most notable change in the new system is that it makes
> more use of threads and particularly it has daemon threads waiting on
> channel receive calls.

Hum, that might be related - GC-finalization issues in
MT-environments are not easy to debug :( 
Might also be python-version dependent.  What are 
the differences between your test and production system? 

I am a bit at a loss on how to best proceed at the moment. 
I think that adding more debugging information
to execnet and systematically implementing and
checking scenarios is due - but quite a bit of effort.  
Could you maybe try coming up with an self-contained 
example test-script leading to these messages? 

On a sidenote, probably around 60% of the execnet core
programming effort revolve around teardown/finalization issues
- i wonder if it would be better to avoid usage of __del__
alltogether, only have a process-wide atexit handler and
otherwise recommend explicit calling of
gateway.exit()/channel.close methods for proper resource
handling.  

best,

holger
 
> lab at okum:~$ uname -a
> Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
> 2009 i686 GNU/Linux
> lab at okum:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import py
> >>> py.version
> '1.0.0b1'
> 
> 
> ============================================== 35 passed in 231.71
> seconds ===============================================
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in Unhandled exception in thread started by <bound
> method Thread.__bootstrap of <Thread(receiver, stopped daemon
> 1177635728)>>
> <bound method Channel.__del__ of <Channel id=7 open>>Error in sys.excepthook:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/dist-packages/apport_python_hook.py", line
> 38, in apport_excepthook
>     from apport.packaging_impl import impl as packaging
> ImportError: No module named apport.packaging_impl
> 
> Original exception was:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
>     self.__bootstrap_inner()
>   File "/usr/lib/python2.6/threading.py", line 538, in __bootstrap_inner
>     (self.name, _format_exc()))
> TypeError: 'NoneType' object is not callable
>  ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From py-dev at tolomea.com  Tue Jul 14 11:56:37 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Tue, 14 Jul 2009 19:56:37 +1000
Subject: [py-dev] execnet: shutdown issues in a heavily threaded
	environment
In-Reply-To: <20090714091004.GY7437@trillke.net>
References: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>
	<20090714091004.GY7437@trillke.net>
Message-ID: <f2d961430907140256l484790d5i5a6500a23a432ec@mail.gmail.com>

Hi Holger

>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
>> open>> ignored

> Yes, some cycle is probably there. ?Your data point hints that
> there is some cycle in the execnet code itself.

A bit of debugging revealed that I didn't really understand how my own
teardown setup was supposed to work and subsequently that it didn't
actually work. I have subsequently improved my teardown process and
this have greatly improved the situation, where I was previously
getting nearly 2 dozen of these per test run I now see only one, I
haven't dug into that one yet, but I'd be willing to bet that it is
also my problem.

>> and particularly it has daemon threads waiting on channel receive calls.

> Hum, that might be related -

It was.

One thing I did notice is that under the ipython shell having a daemon
thread waiting on a channel receive is enough to keep the both the
thread and execnet live and subsequently prevent the VM from exiting.
Strangely this isn't the case under the normal python shell, but still
I think there might be something worth investigating there.

import py
import threading
g=py.execnet.SshGateway("localhost")
c=g.remote_exec("while True:\n pass")
def bob():
    c.receive()
t=threading.Thread(target=bob)
t.daemon=True
t.start()
exit()

Obviously this is a retarded test case, but the only "application"
thread in the system is a daemon so I would naively expect it to shut
down.

> GC-finalization issues in MT-environments are not easy to debug :(

>From recent experience I can confirm that  that is very true.

> Might also be python-version dependent. ?What are
> the differences between your test and production system?

I will check this in the morning.

> I am a bit at a loss on how to best proceed at the moment.
> I think that adding more debugging information
> to execnet and systematically implementing and
> checking scenarios is due - but quite a bit of effort.
> Could you maybe try coming up with an self-contained
> example test-script leading to these messages?

For now I'm happy to consider this my problem and keep picking at it
on my end, although since the VM does come down on both ends of the
ssh link (albeit not always cleanly) this isn't a high priority for
me, just a nuisance. If I do find anything that is definitely execnet
misbehaving I will post it here.

> On a sidenote, probably around 60% of the execnet core
> programming effort revolve around teardown/finalization issues
> - i wonder if it would be better to avoid usage of __del__
> alltogether, only have a process-wide atexit handler and
> otherwise recommend explicit calling of
> gateway.exit()/channel.close methods for proper resource
> handling.

I don't mind either way as long as there is some relatively straight
forward way of getting the whole thing shut down, so that a longer
running system that opens and closes multiple execnet connections can
be written in a way that doesn't accumulate cruft over time.

Regards Gordon

From py-dev at tolomea.com  Wed Jul 15 08:05:13 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Wed, 15 Jul 2009 16:05:13 +1000
Subject: [py-dev] execnet: shutdown issues in a heavily threaded
	environment
In-Reply-To: <20090714091004.GY7437@trillke.net>
References: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>
	<20090714091004.GY7437@trillke.net>
Message-ID: <f2d961430907142305h33c8c2aeq7877cb0368dfc1be@mail.gmail.com>

>> Also this only started when I reworked our remote invocation system
>> recently, the most notable change in the new system is that it makes
>> more use of threads and particularly it has daemon threads waiting on
>> channel receive calls.

> Might also be python-version dependent.  What are
> the differences between your test and production system?

development machine:

gordonw at gohma:~$ uname -a
Linux gohma 2.6.28-13-generic #45-Ubuntu SMP Tue Jun 30 19:49:51 UTC
2009 i686 GNU/Linux
gordonw at gohma:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0b7'

test machine:

lab at okum:~$ uname -a
Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
2009 i686 GNU/Linux
lab at okum:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0b1'

On Tue, Jul 14, 2009 at 7:10 PM, holger krekel<holger at merlinux.eu> wrote:
> Hi Gordon,
>
> On Mon, Jul 13, 2009 at 11:40 +1000, Gordon Wrigley wrote:
>> > One issue that can pop up is tearing down the whole
>> > machinery cleanly - that can sometimes be more
>> > involved and i believe execnet still might need
>> > some more work there.
>>
>> On that note can you shed any light on this message:
>>
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
>> open>> ignored
>>
>> Currently I'm only seeing them on my test machine, but they are
>> occurring very frequently, they drop out at the end of the py.test
>> invocation, I will paste the tail end of the py.test output below.
>
>> I'm not sure exactly what is causing this, naively it seems to
>> indicate a circular chain of references involving one or more
>> channels. However immediately prior to the conclusion of each test
>> exit gc.garbage is empty and there are none of my objects in
>> gc.get_objects.
>
> Yes, some cycle is probably there. ?Your data point hints that
> there is some cycle in the execnet code itself.
>
>> Also this only started when I reworked our remote invocation system
>> recently, the most notable change in the new system is that it makes
>> more use of threads and particularly it has daemon threads waiting on
>> channel receive calls.
>
> Hum, that might be related - GC-finalization issues in
> MT-environments are not easy to debug :(
> Might also be python-version dependent. ?What are
> the differences between your test and production system?
>
> I am a bit at a loss on how to best proceed at the moment.
> I think that adding more debugging information
> to execnet and systematically implementing and
> checking scenarios is due - but quite a bit of effort.
> Could you maybe try coming up with an self-contained
> example test-script leading to these messages?
>
> On a sidenote, probably around 60% of the execnet core
> programming effort revolve around teardown/finalization issues
> - i wonder if it would be better to avoid usage of __del__
> alltogether, only have a process-wide atexit handler and
> otherwise recommend explicit calling of
> gateway.exit()/channel.close methods for proper resource
> handling.
>
> best,
>
> holger
>
>> lab at okum:~$ uname -a
>> Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
>> 2009 i686 GNU/Linux
>> lab at okum:~$ python
>> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
>> [GCC 4.3.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import py
>> >>> py.version
>> '1.0.0b1'
>>
>>
>> ============================================== 35 passed in 231.71
>> seconds ===============================================
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in Unhandled exception in thread started by <bound
>> method Thread.__bootstrap of <Thread(receiver, stopped daemon
>> 1177635728)>>
>> <bound method Channel.__del__ of <Channel id=7 open>>Error in sys.excepthook:
>> Traceback (most recent call last):
>> ? File "/usr/lib/python2.6/dist-packages/apport_python_hook.py", line
>> 38, in apport_excepthook
>> ? ? from apport.packaging_impl import impl as packaging
>> ImportError: No module named apport.packaging_impl
>>
>> Original exception was:
>> Traceback (most recent call last):
>> ? File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
>> ? ? self.__bootstrap_inner()
>> ? File "/usr/lib/python2.6/threading.py", line 538, in __bootstrap_inner
>> ? ? (self.name, _format_exc()))
>> TypeError: 'NoneType' object is not callable
>> ?ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> _______________________________________________
>> py-dev mailing list
>> py-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/py-dev
>>
>
> --
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu
>

From py-dev at tolomea.com  Wed Jul 15 11:05:28 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Wed, 15 Jul 2009 19:05:28 +1000
Subject: [py-dev] execnet py.test teardown issue
Message-ID: <f2d961430907150205x7f44f9d4w6d22fb3e9db9d7e9@mail.gmail.com>

Hi Holger

I have produced a minimal reproduction for my one remaining issue.

To reproduce put the following into a file and run it with py.test

import py
a = None
def test_function():
    global a
    a = py.execnet.SshGateway("localhost").remote_exec("pass")

As before the problem only reproduces on my test machine not on my
development machine. On my test machine it produces:
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=1
open>> ignored

Additionally it only reproduces inside py.test.

Now that I know what it is it can be trivially fixed by del'ing the
global variable at the end of the test. I am however curious about the
difference in behavior on the two machines and I would also like to
know if you consider this a problem or if it is just that pushing a
channel into the global namespace is a bad idea. It is probably worth
noting that in the full system what I pushed into the global namespace
wasn't a channel but a utility class that happened to have a reference
to a channel, so it wasn't quite as braindead as it looks.

Gordon

> development machine:
>
> gordonw at gohma:~$ uname -a
> Linux gohma 2.6.28-13-generic #45-Ubuntu SMP Tue Jun 30 19:49:51 UTC
> 2009 i686 GNU/Linux
> gordonw at gohma:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import py
>>>> py.version
> '1.0.0b7'

> test machine:
>
> lab at okum:~$ uname -a
> Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
> 2009 i686 GNU/Linux
> lab at okum:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import py
>>>> py.version
> '1.0.0b1'

From vinjvinj at gmail.com  Wed Jul 22 05:31:25 2009
From: vinjvinj at gmail.com (Vineet Jain)
Date: Tue, 21 Jul 2009 23:31:25 -0400
Subject: [py-dev] Problem with py.lib and heapy
Message-ID: <70a7f84f0907212031k39bb085eg53f6420b24b3153c@mail.gmail.com>

I was trying to use heapy
(http://guppy-pe.sourceforge.net/)<http://guppy-pe.sourceforge.net/>
on
my application (which uses py.log and py.test)  to try and find some memory
leaks. I can't seem to get it to work with py.lib (see exception trace at
the end of the email). Can someone please guide me on how to just install
py.log (and it's minimum requirements)? I tried downloading the tar and
removed all the extra packages in setup.py, however, that failed. Currently
I have py.log commented out while I'm debugging my memory leaks.

hp = hpy()

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/__init__.py",
line 37, in hpy

return r.guppy.heapy.Use

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 45, in __getattr__

return self._share.getattr(self, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 195, in getattr

d = self.getattr2(inter, cache, owner, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 213, in getattr2

x = self.getattr_package(inter, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 261, in getattr_package

x = self.makeModule(x, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 321, in makeModule

return Share(module, self, module.__name__, Clamp)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 184, in __init__

getattr(inter, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 45, in __getattr__

return self._share.getattr(self, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 195, in getattr

d = self.getattr2(inter, cache, owner, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 215, in getattr2

x = self.getattr3(inter, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 283, in getattr3

pa = getattr(pa, at)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 45, in __getattr__

return self._share.getattr(self, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 195, in getattr

d = self.getattr2(inter, cache, owner, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 213, in getattr2

x = self.getattr_package(inter, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 250, in getattr_package

x = __import__(self.makeName(name), globals(), locals())

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/heapy/View.py",
line 555, in <module>

prime_builtin_types()

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/heapy/View.py",
line 546, in prime_builtin_types

for t in mod.__dict__.values():

File "/usr/lib/python2.5/site-packages/py-0.9.1-py2.5.egg/py/initpkg.py",
line 216, in getdict

assert not self.__map__, "%r not empty" % self.__map__

AssertionError: {'greenlet': ('./magic/greenlet.py', 'greenlet')} not empty
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/py-dev/attachments/20090721/9da9e671/attachment.htm 

From py-dev at tolomea.com  Wed Jul 22 08:16:48 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Wed, 22 Jul 2009 16:16:48 +1000
Subject: [py-dev] py.test and logging issue: ValueError: I/O operation
	on closed file
In-Reply-To: <20090709112056.GO7437@trillke.net>
References: <op.uwru3etnbrrvlq@double> <20090709112056.GO7437@trillke.net>
Message-ID: <f2d961430907212316l434c36cegaef6da2d92c7aa1d@mail.gmail.com>

When do you expect 1.0.0b8 to be available on easy-install?

I ask because I am also experiencing this logging issue and it is
starting to get quite tedious.

Regards Gordon

On Thu, Jul 9, 2009 at 9:20 PM, holger krekel<holger at merlinux.eu> wrote:
> On Wed, Jul 08, 2009 at 16:19 -0700, Sridhar Ratnakumar wrote:
>> holger and others,
>>
>> do you have any comments to add to this bug?
>>
>> ? ?http://bugs.python.org/issue6333
>
> i added a comment there, basically suggesting to follow your suggestion.
>
> As py.test should nicely work on many CPython versions i also
> did a general workaround in py.test proper, see here:
>
> ? ?http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/
>
> I intend to release a 1.0.0b8 tomorrow with this
> and some other fixes.
>
> best & thanks,
> holger
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
>

From holger at merlinux.eu  Wed Jul 22 08:47:29 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 08:47:29 +0200
Subject: [py-dev] py.test and logging issue: ValueError: I/O
	operation	on closed file
In-Reply-To: <f2d961430907212316l434c36cegaef6da2d92c7aa1d@mail.gmail.com>
References: <op.uwru3etnbrrvlq@double> <20090709112056.GO7437@trillke.net>
	<f2d961430907212316l434c36cegaef6da2d92c7aa1d@mail.gmail.com>
Message-ID: <20090722064729.GV7437@trillke.net>

On Wed, Jul 22, 2009 at 16:16 +1000, Gordon Wrigley wrote:
> When do you expect 1.0.0b8 to be available on easy-install?
> 
> I ask because I am also experiencing this logging issue and it is
> starting to get quite tedious.

will see to get it out today. 

holger
 
> On Thu, Jul 9, 2009 at 9:20 PM, holger krekel<holger at merlinux.eu> wrote:
> > On Wed, Jul 08, 2009 at 16:19 -0700, Sridhar Ratnakumar wrote:
> >> holger and others,
> >>
> >> do you have any comments to add to this bug?
> >>
> >> ? ?http://bugs.python.org/issue6333
> >
> > i added a comment there, basically suggesting to follow your suggestion.
> >
> > As py.test should nicely work on many CPython versions i also
> > did a general workaround in py.test proper, see here:
> >
> > ? ?http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/
> >
> > I intend to release a 1.0.0b8 tomorrow with this
> > and some other fixes.
> >
> > best & thanks,
> > holger
> > _______________________________________________
> > py-dev mailing list
> > py-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/py-dev
> >
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Wed Jul 22 08:51:44 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 08:51:44 +0200
Subject: [py-dev] Problem with py.lib and heapy
In-Reply-To: <70a7f84f0907212031k39bb085eg53f6420b24b3153c@mail.gmail.com>
References: <70a7f84f0907212031k39bb085eg53f6420b24b3153c@mail.gmail.com>
Message-ID: <20090722065144.GW7437@trillke.net>

Hi Vineet,

seems you are using a relatively old version (0.9.1). 

could you try to upgrade to 1.0.0b7 by doing "easy_install -U py"? 
This will probably overwrite what your system-wide package system installed
but it likely fixes the below problem as greenlets are not
part of the 1.0 series anymore. 

best,

holger

On Tue, Jul 21, 2009 at 23:31 -0400, Vineet Jain wrote:
> I was trying to use heapy
> (http://guppy-pe.sourceforge.net/)<http://guppy-pe.sourceforge.net/>
> on
> my application (which uses py.log and py.test)  to try and find some memory
> leaks. I can't seem to get it to work with py.lib (see exception trace at
> the end of the email). Can someone please guide me on how to just install
> py.log (and it's minimum requirements)? I tried downloading the tar and
> removed all the extra packages in setup.py, however, that failed. Currently
> I have py.log commented out while I'm debugging my memory leaks.
> 
> hp = hpy()
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/__init__.py",
> line 37, in hpy
> 
> return r.guppy.heapy.Use
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 45, in __getattr__
> 
> return self._share.getattr(self, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 195, in getattr
> 
> d = self.getattr2(inter, cache, owner, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 213, in getattr2
> 
> x = self.getattr_package(inter, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 261, in getattr_package
> 
> x = self.makeModule(x, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 321, in makeModule
> 
> return Share(module, self, module.__name__, Clamp)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 184, in __init__
> 
> getattr(inter, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 45, in __getattr__
> 
> return self._share.getattr(self, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 195, in getattr
> 
> d = self.getattr2(inter, cache, owner, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 215, in getattr2
> 
> x = self.getattr3(inter, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 283, in getattr3
> 
> pa = getattr(pa, at)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 45, in __getattr__
> 
> return self._share.getattr(self, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 195, in getattr
> 
> d = self.getattr2(inter, cache, owner, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 213, in getattr2
> 
> x = self.getattr_package(inter, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 250, in getattr_package
> 
> x = __import__(self.makeName(name), globals(), locals())
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/heapy/View.py",
> line 555, in <module>
> 
> prime_builtin_types()
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/heapy/View.py",
> line 546, in prime_builtin_types
> 
> for t in mod.__dict__.values():
> 
> File "/usr/lib/python2.5/site-packages/py-0.9.1-py2.5.egg/py/initpkg.py",
> line 216, in getdict
> 
> assert not self.__map__, "%r not empty" % self.__map__
> 
> AssertionError: {'greenlet': ('./magic/greenlet.py', 'greenlet')} not empty

> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev


-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From Ronny.Pfannschmidt at gmx.de  Wed Jul 22 09:51:31 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Wed, 22 Jul 2009 09:51:31 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs and distribution
Message-ID: <1248249091.8033.8.camel@localhost>

Hi,

i propose 2 new hooks to deal with more intresting aspects of test
distribution and txspecs

1. pytest_generate_gateways(config/session) -> list of execnet gateways

this hook is supposed to generate the execnet gateways for test
execution

the default implementation is supposed to just do what py.test currently
does

howevers users should be free to generate own gateways to custom set up
things like virtualenvs on local or remote computers and be able to
combine txspecs with own options

2. pytest_select_gateways(test, list of gateways) -> list of gateways

this hook is supposed to select the gateways each test is supposed to
run on

the default implementation should pass a test to all of them or
distribute them reasonable

the api for this one probably needs a few more idea since i think my
initial proposal is not adequate for all uses

Regards
Ronny


From holger at merlinux.eu  Wed Jul 22 11:55:58 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 11:55:58 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs
	and	distribution
In-Reply-To: <1248249091.8033.8.camel@localhost>
References: <1248249091.8033.8.camel@localhost>
Message-ID: <20090722095558.GZ7437@trillke.net>

Hey Ronny, 

maybe it's good to also state your overall goal: running 
tests in virtualenv-environments with different packages
pre-installed. 

On Wed, Jul 22, 2009 at 09:51 +0200, Ronny Pfannschmidt wrote:
> Hi,
> 
> i propose 2 new hooks to deal with more intresting aspects of test
> distribution and txspecs
> 
> 1. pytest_generate_gateways(config/session) -> list of execnet gateways
> 
> this hook is supposed to generate the execnet gateways for test
> execution
> 
> the default implementation is supposed to just do what py.test currently
> does

probably rather a hook for setting a single gateway

    pytest_makegateway(txspec) -> execnet gateway 

because management of multiple nodes and their setup
is its own concern - some nodes might not come up etc.

In the --tx specifciation one could say e.g. 

    popen//python=python2.5//virtualenv=docutils-0.5,mercurial-1.2

and a pytest_virtualenv plugin could take it, call virtualenv
and eventually start up a gateway with the venv/bin/python. 
 
> howevers users should be free to generate own gateways to custom set up
> things like virtualenvs on local or remote computers and be able to
> combine txspecs with own options

To get started, i would first try to get things working with
popen, then extend to handle remote places - would you want
to automatically install virtualenv there and/or require 
setuptools, btw?   
 
> 2. pytest_select_gateways(test, list of gateways) -> list of gateways
> 
> this hook is supposed to select the gateways each test is supposed to
> run on

something like this could make sense.  It's a second step, i'd think, 
though. 

Can you imagine working on a branch to try implement these hooks? 
I am willing to help/review.  

> the default implementation should pass a test to all of them or
> distribute them reasonable
> the api for this one probably needs a few more idea since i think my
> initial proposal is not adequate for all uses

yes, working from concrete uses (a pytest_virtualenv
plugin) makes sense.  

best,
holger

From Ronny.Pfannschmidt at gmx.de  Wed Jul 22 12:33:27 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Wed, 22 Jul 2009 12:33:27 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs and
 distribution
In-Reply-To: <20090722095558.GZ7437@trillke.net>
References: <1248249091.8033.8.camel@localhost>
	<20090722095558.GZ7437@trillke.net>
Message-ID: <1248258807.8033.12.camel@localhost>

On Wed, 2009-07-22 at 11:55 +0200, holger krekel wrote:
> Hey Ronny, 
> 
> maybe it's good to also state your overall goal: running 
> tests in virtualenv-environments with different packages
> pre-installed. 
> 
> On Wed, Jul 22, 2009 at 09:51 +0200, Ronny Pfannschmidt wrote:
> > Hi,
> > 
> > i propose 2 new hooks to deal with more intresting aspects of test
> > distribution and txspecs
> > 
> > 1. pytest_generate_gateways(config/session) -> list of execnet gateways
> > 
> > this hook is supposed to generate the execnet gateways for test
> > execution
> > 
> > the default implementation is supposed to just do what py.test currently
> > does
> 
> probably rather a hook for setting a single gateway
> 
>     pytest_makegateway(txspec) -> execnet gateway 

indeed seperation of generating specifications and turning them into
gateways makes sense

> 
> because management of multiple nodes and their setup
> is its own concern - some nodes might not come up etc.
> 
> In the --tx specifciation one could say e.g. 
> 
>     popen//python=python2.5//virtualenv=docutils-0.5,mercurial-1.2
> 
> and a pytest_virtualenv plugin could take it, call virtualenv
> and eventually start up a gateway with the venv/bin/python. 

ok, how would i generate sets of txspec declarations, 

cause i want to combine multiple arguments in order to generate the python versions and the package versions


>  
> > howevers users should be free to generate own gateways to custom set up
> > things like virtualenvs on local or remote computers and be able to
> > combine txspecs with own options
> 
> To get started, i would first try to get things working with
> popen, then extend to handle remote places - would you want
> to automatically install virtualenv there and/or require 
> setuptools, btw?   
>  
> > 2. pytest_select_gateways(test, list of gateways) -> list of gateways
> > 
> > this hook is supposed to select the gateways each test is supposed to
> > run on
> 
> something like this could make sense.  It's a second step, i'd think, 
> though. 
i would certainly need it
for example tests for mercurial would hardly work in a virtualenv thats
only only set up for bazaar

> 
> Can you imagine working on a branch to try implement these hooks? 
> I am willing to help/review. 
sure
>  
> 
> > the default implementation should pass a test to all of them or
> > distribute them reasonable
> > the api for this one probably needs a few more idea since i think my
> > initial proposal is not adequate for all uses
> 
> yes, working from concrete uses (a pytest_virtualenv
> plugin) makes sense.  
> 



From holger at merlinux.eu  Wed Jul 22 13:03:06 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 13:03:06 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs
	and	distribution
In-Reply-To: <1248258807.8033.12.camel@localhost>
References: <1248249091.8033.8.camel@localhost>
	<20090722095558.GZ7437@trillke.net>
	<1248258807.8033.12.camel@localhost>
Message-ID: <20090722110306.GA7437@trillke.net>

On Wed, Jul 22, 2009 at 12:33 +0200, Ronny Pfannschmidt wrote:
> > In the --tx specifciation one could say e.g. 
> > 
> >     popen//python=python2.5//virtualenv=docutils-0.5,mercurial-1.2
> > 
> > and a pytest_virtualenv plugin could take it, call virtualenv
> > and eventually start up a gateway with the venv/bin/python. 
> 
> ok, how would i generate sets of txspec declarations, 

> cause i want to combine multiple arguments in order to generate the python versions and the package versions
 
i guess by implementing a ``pytest_configure(config)`` hook to
add a few txspecs.  We need to extend the API to the
config object for that, currently it's readonly. Maybe: 

    def pytest_configure(config):
        # access your custom plugin options to 
        # determine which combis to generate 
        config.setvalue('tx', ['spec1', 'spec2', ...])
 
> > > howevers users should be free to generate own gateways to custom set up
> > > things like virtualenvs on local or remote computers and be able to
> > > combine txspecs with own options
> > 
> > To get started, i would first try to get things working with
> > popen, then extend to handle remote places - would you want
> > to automatically install virtualenv there and/or require 
> > setuptools, btw?   

i presume it makes sense to you to first try popen. 

> > > 2. pytest_select_gateways(test, list of gateways) -> list of gateways
> > > 
> > > this hook is supposed to select the gateways each test is supposed to
> > > run on
> > 
> > something like this could make sense.  It's a second step, i'd think, 
> > though. 
> i would certainly need it
> for example tests for mercurial would hardly work in a virtualenv thats
> only only set up for bazaar

you could still skip the tests late-bound but i can see you want
to avoid even trying tests in unfitting environments. 
The issue here is to have a somewhat higher level API for 
what you call "list of gateways" because the raw low-level 
execnet-gateways are not fitting.  you rather want something
where you have structured access to the "test nodes", their
txspecs etc. and this requires a bit of refactoring. 

> > Can you imagine working on a branch to try implement these hooks? 
> > I am willing to help/review. 
> sure

fine.  please add a bitbucket diff-email 
service to py-svn at codespeak.net then. 

best,
holger

> >  
> > 
> > > the default implementation should pass a test to all of them or
> > > distribute them reasonable
> > > the api for this one probably needs a few more idea since i think my
> > > initial proposal is not adequate for all uses
> > 
> > yes, working from concrete uses (a pytest_virtualenv
> > plugin) makes sense.  
> > 
> 
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From Ronny.Pfannschmidt at gmx.de  Wed Jul 22 13:17:20 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Wed, 22 Jul 2009 13:17:20 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs and
 distribution
In-Reply-To: <20090722110306.GA7437@trillke.net>
References: <1248249091.8033.8.camel@localhost>
	<20090722095558.GZ7437@trillke.net>
	<1248258807.8033.12.camel@localhost>
	<20090722110306.GA7437@trillke.net>
Message-ID: <1248261440.8033.20.camel@localhost>

On Wed, 2009-07-22 at 13:03 +0200, holger krekel wrote:
> On Wed, Jul 22, 2009 at 12:33 +0200, Ronny Pfannschmidt wrote:
> > > In the --tx specifciation one could say e.g. 
> > > 
> > >     popen//python=python2.5//virtualenv=docutils-0.5,mercurial-1.2
> > > 
> > > and a pytest_virtualenv plugin could take it, call virtualenv
> > > and eventually start up a gateway with the venv/bin/python. 
> > 
> > ok, how would i generate sets of txspec declarations, 
> 
> > cause i want to combine multiple arguments in order to generate the python versions and the package versions
>  
> i guess by implementing a ``pytest_configure(config)`` hook to
> add a few txspecs.  We need to extend the API to the
> config object for that, currently it's readonly. Maybe: 
> 
>     def pytest_configure(config):
>         # access your custom plugin options to 
>         # determine which combis to generate 
>         config.setvalue('tx', ['spec1', 'spec2', ...])
>  
> > > > howevers users should be free to generate own gateways to custom set up
> > > > things like virtualenvs on local or remote computers and be able to
> > > > combine txspecs with own options
> > > 
> > > To get started, i would first try to get things working with
> > > popen, then extend to handle remote places - would you want
> > > to automatically install virtualenv there and/or require 
> > > setuptools, btw?   
> 
> i presume it makes sense to you to first try popen. 
> 
> > > > 2. pytest_select_gateways(test, list of gateways) -> list of gateways
> > > > 
> > > > this hook is supposed to select the gateways each test is supposed to
> > > > run on
> > > 
> > > something like this could make sense.  It's a second step, i'd think, 
> > > though. 
> > i would certainly need it
> > for example tests for mercurial would hardly work in a virtualenv thats
> > only only set up for bazaar
> 
> you could still skip the tests late-bound but i can see you want
> to avoid even trying tests in unfitting environments. 
> The issue here is to have a somewhat higher level API for 
> what you call "list of gateways" because the raw low-level 
> execnet-gateways are not fitting.  you rather want something
> where you have structured access to the "test nodes", their
> txspecs etc. and this requires a bit of refactoring. 

ok, lets make a small resume about my tasks

1. a hook for turning a txspec to a gateway
2. a nice hook for taking options and generating a set of txspecs
  (the default being just to pass along the ones from options)
3. a virtualenv plugin

   a. support for setting up a virtualenv 
      with a custom python version + versions of packages
   b. support for executing the virtualenv setup via networked gateways
      ie the socket one and the ssh one
   

regards
Ronny



From holger at merlinux.eu  Wed Jul 22 16:23:31 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 16:23:31 +0200
Subject: [py-dev] 1.0.0b8 out
Message-ID: <20090722142331.GC7437@trillke.net>

Hi all, 

i just put a new 1.0.0.b8 release to PyPI, find below
a list of changes.  

Documentation is also updated majorly - there now is an 
auto-generated list of pages for each plugin, just go to 

    http://pytest.org 

and look around.  And let me know of any issues. 

have fun, and thanks to all who helped.

holger


list of changes 1.0.0b7 to 1.0.0b8:

* pytest_unittest-plugin is now enabled by default

* introduced pytest_keyboardinterrupt hook and 
  refined pytest_sessionfinish hook, added tests. 

* workaround a buggy logging module interaction ("closing already closed
  files").  Thanks to Sridhar Ratnakumar for triggering. 

* if plugins use "py.test.importorskip" for importing 
  a dependency only a warning will be issued instead 
  of exiting the testing process. 

* many improvements to docs: 
  - refined funcargs doc , use the term "factory" instead of "provider"
  - added a new talk/tutorial doc page 
  - better download page
  - better plugin docstrings
  - added new plugins page and automatic doc generation script

* fixed teardown problem related to partially failing funcarg setups 
  (thanks MrTopf for reporting), "pytest_runtest_teardown" is now 
  always invoked even if the "pytest_runtest_setup" failed. 

* tweaked doctest output for docstrings in py modules, 
  thanks Radomir. 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From py-dev at tolomea.com  Thu Jul 23 04:04:30 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Thu, 23 Jul 2009 12:04:30 +1000
Subject: [py-dev] 1.0.0b8 out
In-Reply-To: <20090722142331.GC7437@trillke.net>
References: <20090722142331.GC7437@trillke.net>
Message-ID: <f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>

Alas I'm still getting logging issues, specifically I get a big slab
of this every time a test fails

Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file

Regards Gordon

On Thu, Jul 23, 2009 at 12:23 AM, holger krekel<holger at merlinux.eu> wrote:
> Hi all,
>
> i just put a new 1.0.0.b8 release to PyPI, find below
> a list of changes.
>
> Documentation is also updated majorly - there now is an
> auto-generated list of pages for each plugin, just go to
>
> ? ?http://pytest.org
>
> and look around. ?And let me know of any issues.
>
> have fun, and thanks to all who helped.
>
> holger
>
>
> list of changes 1.0.0b7 to 1.0.0b8:
>
> * pytest_unittest-plugin is now enabled by default
>
> * introduced pytest_keyboardinterrupt hook and
> ?refined pytest_sessionfinish hook, added tests.
>
> * workaround a buggy logging module interaction ("closing already closed
> ?files"). ?Thanks to Sridhar Ratnakumar for triggering.
>
> * if plugins use "py.test.importorskip" for importing
> ?a dependency only a warning will be issued instead
> ?of exiting the testing process.
>
> * many improvements to docs:
> ?- refined funcargs doc , use the term "factory" instead of "provider"
> ?- added a new talk/tutorial doc page
> ?- better download page
> ?- better plugin docstrings
> ?- added new plugins page and automatic doc generation script
>
> * fixed teardown problem related to partially failing funcarg setups
> ?(thanks MrTopf for reporting), "pytest_runtest_teardown" is now
> ?always invoked even if the "pytest_runtest_setup" failed.
>
> * tweaked doctest output for docstrings in py modules,
> ?thanks Radomir.
>
> --
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
>

From py-dev at tolomea.com  Thu Jul 23 05:23:54 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Thu, 23 Jul 2009 13:23:54 +1000
Subject: [py-dev] 1.0.0b8 out
In-Reply-To: <f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
References: <20090722142331.GC7437@trillke.net>
	<f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
Message-ID: <f2d961430907222023y1cb548aay267a55e464e7ca5c@mail.gmail.com>

Actually on further use it appears that I am getting these on test
passes as well.

Also the test set I'm using doesn't use execnet, so that isn't a
complicating factor.

Regards Gordon

On Thu, Jul 23, 2009 at 12:04 PM, Gordon Wrigley<py-dev at tolomea.com> wrote:
> Alas I'm still getting logging issues, specifically I get a big slab
> of this every time a test fails
>
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
>
> Regards Gordon
>
> On Thu, Jul 23, 2009 at 12:23 AM, holger krekel<holger at merlinux.eu> wrote:
>> Hi all,
>>
>> i just put a new 1.0.0.b8 release to PyPI, find below
>> a list of changes.
>>
>> Documentation is also updated majorly - there now is an
>> auto-generated list of pages for each plugin, just go to
>>
>> ? ?http://pytest.org
>>
>> and look around. ?And let me know of any issues.
>>
>> have fun, and thanks to all who helped.
>>
>> holger
>>
>>
>> list of changes 1.0.0b7 to 1.0.0b8:
>>
>> * pytest_unittest-plugin is now enabled by default
>>
>> * introduced pytest_keyboardinterrupt hook and
>> ?refined pytest_sessionfinish hook, added tests.
>>
>> * workaround a buggy logging module interaction ("closing already closed
>> ?files"). ?Thanks to Sridhar Ratnakumar for triggering.
>>
>> * if plugins use "py.test.importorskip" for importing
>> ?a dependency only a warning will be issued instead
>> ?of exiting the testing process.
>>
>> * many improvements to docs:
>> ?- refined funcargs doc , use the term "factory" instead of "provider"
>> ?- added a new talk/tutorial doc page
>> ?- better download page
>> ?- better plugin docstrings
>> ?- added new plugins page and automatic doc generation script
>>
>> * fixed teardown problem related to partially failing funcarg setups
>> ?(thanks MrTopf for reporting), "pytest_runtest_teardown" is now
>> ?always invoked even if the "pytest_runtest_setup" failed.
>>
>> * tweaked doctest output for docstrings in py modules,
>> ?thanks Radomir.
>>
>> --
>> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
>> Python, PyPy, pytest contracting: http://merlinux.eu
>> _______________________________________________
>> py-dev mailing list
>> py-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/py-dev
>>
>

From holger at merlinux.eu  Thu Jul 23 19:52:55 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 23 Jul 2009 19:52:55 +0200
Subject: [py-dev] 1.0.0b8 out
In-Reply-To: <f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
References: <20090722142331.GC7437@trillke.net>
	<f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
Message-ID: <20090723175255.GD7437@trillke.net>

On Thu, Jul 23, 2009 at 12:04 +1000, Gordon Wrigley wrote:
> Alas I'm still getting logging issues, specifically I get a big slab
> of this every time a test fails
> 
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
>     stream.write(fs % msg)
> ValueError: I/O operation on closed file

hum, py.test does not do anything with logging itself. 
But it captures stdout/stderr during setup, test-call
and teardown to temporary files/StringIOs.  It properly 
closes the temporary files/streams at each of the
three phases.  Do you have a single logging configuration
during setup/call/teardown, mabe? 

if you disable output capturing with "-s" you don't 
see these error message, right? 

holger

From holger at merlinux.eu  Thu Jul 23 21:23:29 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 23 Jul 2009 21:23:29 +0200
Subject: [py-dev] 1.0.0b8 out
In-Reply-To: <20090723175255.GD7437@trillke.net>
References: <20090722142331.GC7437@trillke.net>
	<f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
	<20090723175255.GD7437@trillke.net>
Message-ID: <20090723192329.GE7437@trillke.net>

On Thu, Jul 23, 2009 at 19:52 +0200, holger krekel wrote:
> On Thu, Jul 23, 2009 at 12:04 +1000, Gordon Wrigley wrote:
> > Alas I'm still getting logging issues, specifically I get a big slab
> > of this every time a test fails
> > 
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> >     stream.write(fs % msg)
> > ValueError: I/O operation on closed file
> 
> hum, py.test does not do anything with logging itself. 
> But it captures stdout/stderr during setup, test-call
> and teardown to temporary files/StringIOs.  It properly 
> closes the temporary files/streams at each of the
> three phases.  Do you have a single logging configuration
> during setup/call/teardown, mabe? 

that seems to be the issue, here is one way to reproduce it: 

        import logging
        def setup_module(mod):
            logging.warn("hello world")

        def test_logging():
            logging.warn("hello world")
            assert 0

need to think some more about this and maybe look more
into the logging module and its assumtions. 

holger

From holger at merlinux.eu  Fri Jul 31 15:49:49 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 31 Jul 2009 15:49:49 +0200
Subject: [py-dev] pylib/py.test 1.0.0b9 out
Message-ID: <20090731134948.GL31589@trillke.net>

Hi all, 

i've just released a 1.0.0b9 to PyPI which should fix all
interactions with logging and has generally improved capturing
and reporting.  Below is the full changelog. 

If you like to help with some proof-reading, you are welcome to do
a mercurial checkout (see http://codespeak.net/py/dist/download.html) 
go to the 

    doc/  
    py/test/plugin/pytest_*.py 

directories and look through the text files and module docstrings
from which all of the plugin docs are generated at 
    
    http://codespeak.net/py/dist/test/plugin/index.html

thanks & cheers,

holger


Changes between 1.0.0b8 and 1.0.0b9
=====================================

* cleanly handle and report final teardown of test setup

* fix svn-1.6 compat issue with py.path.svnwc().versioned()
  (thanks Wouter Vanden Hove)

* setup/teardown or collection problems now show as ERRORs
  or with big "E"'s in the progress lines.  they are reported
  and counted separately. 
 
* dist-testing: properly handle test items that get locally 
  collected but cannot be collected on the remote side - often 
  due to platform/dependency reasons

* simplified py.test.mark API - see keyword plugin documentation

* integrate better with logging: capturing now by default captures
  test functions and their immediate setup/teardown in a single stream 

* capsys and capfd funcargs now have a readouterr() and a close() method
  (underlyingly py.io.StdCapture/FD objects are used which grew a 
  readouterr() method as well to return snapshots of captured out/err)

* make assert-reinterpretation work better with comparisons not 
  returning bools (reported with numpy from thanks maciej fijalkowski)

* reworked per-test output capturing into the pytest_iocapture.py plugin 
  and thus removed capturing code from config object 

* item.repr_failure(excinfo) instead of item.repr_failure(excinfo, outerr)


Changes between 1.0.0b7 and 1.0.0b8
=====================================

* pytest_unittest-plugin is now enabled by default

* introduced pytest_keyboardinterrupt hook and 
  refined pytest_sessionfinish hooked, added tests. 

* workaround a buggy logging module interaction ("closing already closed
  files").  Thanks to Sridhar Ratnakumar for triggering. 

* if plugins use "py.test.importorskip" for importing 
  a dependency only a warning will be issued instead 
  of exiting the testing process. 

* many improvements to docs: 
  - refined funcargs doc , use the term "factory" instead of "provider"
  - added a new talk/tutorial doc page 
  - better download page
  - better plugin docstrings
  - added new plugins page and automatic doc generation script

* fixed teardown problem related to partially failing funcarg setups 
  (thanks MrTopf for reporting), "pytest_runtest_teardown" is now 
  always invoked even if the "pytest_runtest_setup" failed. 

* tweaked doctest output for docstrings in py modules, 
  thanks Radomir. 

Changes between 1.0.0b3 and 1.0.0b7
=============================================

* renamed py.test.xfail back to py.test.mark.xfail to avoid 
  two ways to decorate for xfail

* re-added py.test.mark decorator for setting keywords on functions 
  (it was actually documented so removing it was not nice) 

* remove scope-argument from request.addfinalizer() because 
  request.cached_setup has the scope arg. TOOWTDI. 

* perform setup finalization before reporting failures

* apply modified patches from Andreas Kloeckner to allow 
  test functions to have no func_code (#22) and to make 
  "-k" and function keywords work  (#20) 

* apply patch from Daniel Peolzleithner (issue #23) 

* resolve issue #18, multiprocessing.Manager() and 
  redirection clash 

* make __name__ == "__channelexec__" for remote_exec code

Changes between 1.0.0b1 and 1.0.0b3
=============================================

* plugin classes are removed: one now defines 
  hooks directly in conftest.py or global pytest_*.py 
  files. 

* added new pytest_namespace(config) hook that allows 
  to inject helpers directly to the py.test.* namespace. 

* documented and refined many hooks 

* added new style of generative tests via 
  pytest_generate_tests hook that integrates 
  well with function arguments. 
 

Changes between 0.9.2 and 1.0.0b1
=============================================

* introduced new "funcarg" setup method, 
  see doc/test/funcarg.txt 

* introduced plugin architecuture and many 
  new py.test plugins, see 
  doc/test/plugins.txt

* teardown_method is now guaranteed to get 
  called after a test method has run. 
 
* new method: py.test.importorskip(mod,minversion)
  will either import or call py.test.skip()

* completely revised internal py.test architecture

* new py.process.ForkedFunc object allowing to 
  fork execution of a function to a sub process
  and getting a result back. 

XXX lots of things missing here XXX

Changes between 0.9.1 and 0.9.2
===============================

* refined installation and metadata, created new setup.py, 
  now based on setuptools/ez_setup (thanks to Ralf Schmitt 
  for his support).

* improved the way of making py.* scripts available in 
  windows environments, they are now added to the 
  Scripts directory as ".cmd" files. 

* py.path.svnwc.status() now is more complete and 
  uses xml output from the 'svn' command if available
  (Guido Wesdorp)

* fix for py.path.svn* to work with svn 1.5
  (Chris Lamb)

* fix path.relto(otherpath) method on windows to 
  use normcase for checking if a path is relative.

* py.test's traceback is better parseable from editors 
  (follows the filenames:LINENO: MSG convention)
  (thanks to Osmo Salomaa)

* fix to javascript-generation, "py.test --runbrowser" 
  should work more reliably now

* removed previously accidentally added 
  py.test.broken and py.test.notimplemented helpers. 

* there now is a py.__version__ attribute

Changes between 0.9.0 and 0.9.1
===============================

This is a fairly complete list of changes between 0.9 and 0.9.1, which can
serve as a reference for developers.

* allowing + signs in py.path.svn urls [39106]
* fixed support for Failed exceptions without excinfo in py.test [39340]
* added support for killing processes for Windows (as well as platforms that
  support os.kill) in py.misc.killproc [39655]
* added setup/teardown for generative tests to py.test [40702]
* added detection of FAILED TO LOAD MODULE to py.test [40703, 40738, 40739]
* fixed problem with calling .remove() on wcpaths of non-versioned files in
  py.path [44248]
* fixed some import and inheritance issues in py.test [41480, 44648, 44655]
* fail to run greenlet tests when pypy is available, but without stackless
  [45294]
* small fixes in rsession tests [45295]
* fixed issue with 2.5 type representations in py.test [45483, 45484]
* made that internal reporting issues displaying is done atomically in py.test
  [45518]
* made that non-existing files are igored by the py.lookup script [45519]
* improved exception name creation in py.test [45535]
* made that less threads are used in execnet [merge in 45539]
* removed lock required for atomical reporting issue displaying in py.test
  [45545]
* removed globals from execnet [45541, 45547]
* refactored cleanup mechanics, made that setDaemon is set to 1 to make atexit
  get called in 2.5 (py.execnet) [45548]
* fixed bug in joining threads in py.execnet's servemain [45549]
* refactored py.test.rsession tests to not rely on exact output format anymore
  [45646]
* using repr() on test outcome [45647]
* added 'Reason' classes for py.test.skip() [45648, 45649]
* killed some unnecessary sanity check in py.test.collect [45655]
* avoid using os.tmpfile() in py.io.fdcapture because on Windows it's only
  usable by Administrators [45901]
* added support for locking and non-recursive commits to py.path.svnwc [45994]
* locking files in py.execnet to prevent CPython from segfaulting [46010]
* added export() method to py.path.svnurl
* fixed -d -x in py.test [47277]
* fixed argument concatenation problem in py.path.svnwc [49423]
* restore py.test behaviour that it exits with code 1 when there are failures
  [49974]
* don't fail on html files that don't have an accompanying .txt file [50606]
* fixed 'utestconvert.py < input' [50645]
* small fix for code indentation in py.code.source [50755]
* fix _docgen.py documentation building [51285]
* improved checks for source representation of code blocks in py.test [51292]
* added support for passing authentication to py.path.svn* objects [52000,
  52001]
* removed sorted() call for py.apigen tests in favour of [].sort() to support
  Python 2.3 [52481]
-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Tue Aug  4 12:08:39 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 4 Aug 2009 12:08:39 +0200
Subject: [py-dev] pylib-1.0.0 released ...
Message-ID: <20090804100839.GP31589@trillke.net>

Hi folks,

thanks to you all i've just released a py.test 1.0.0 final.
actually i ran out of Beta numbers and thought i should
probably rather go for a 1.0.1 if it comes to it :) 

anyway, hope everything is well ... see also the blog post: 

    http://tetamap.wordpress.com/2009/08/04/pylib-1-0-0

let me know of any problems or - even better - success stories :)

cheers & best,
holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From pedronis at openend.se  Tue Aug  4 18:38:15 2009
From: pedronis at openend.se (Samuele Pedroni)
Date: Tue, 04 Aug 2009 18:38:15 +0200
Subject: [py-dev] oejskit javascript testing 0.8.6 released Re: pylib-1.0.0
 released ...
In-Reply-To: <20090804100839.GP31589@trillke.net>
References: <20090804100839.GP31589@trillke.net>
Message-ID: <4A7863F7.4010704@openend.se>

holger krekel wrote:
> Hi folks,
>
> thanks to you all i've just released a py.test 1.0.0 final.
> actually i ran out of Beta numbers and thought i should
> probably rather go for a 1.0.1 if it comes to it :) 
>
> anyway, hope everything is well ... see also the blog post: 
>
>     http://tetamap.wordpress.com/2009/08/04/pylib-1-0-0
>
> let me know of any problems or - even better - success stories :)
>   
there were some last minute compatibility issues between oejskit and 
py.test 1.0 final,
hpk provided some patches to fix them (thanks)

I have now pushed a 0.8.6 release including them to PyPI, see also the 
changelog:

http://pypi.python.org/pypi/oejskit


Samuele Pedroni for Open End

From py-dev at tolomea.com  Fri Aug  7 10:53:58 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 7 Aug 2009 18:53:58 +1000
Subject: [py-dev] issues with stdout redirection, ctypes and py.test
Message-ID: <f2d961430908070153m772bf312s40a7a89cf688d4f0@mail.gmail.com>

My python code talks to a C library via ctypes and it in turn prints
debug to stdout.
And we test this with py.test, which by itself works just fine.
But if at the bash prompt we then redirect the output of py.test to a
file suddenly the C debug starts propagating out of py.test.

We generally run py.test with -v and -x, but removing those didn't
change the behavior at all.

I'm at a bit of loss as to where to begin with this one, any suggestions?

gordonw at gohma:~$ uname -a
Linux gohma 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25 00:28:35 UTC
2009 i686 GNU/Linux
gordonw at gohma:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0'

Regards,
Gordon

From holger at merlinux.eu  Fri Aug  7 12:02:51 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 7 Aug 2009 12:02:51 +0200
Subject: [py-dev] issues with stdout redirection, ctypes and py.test
In-Reply-To: <f2d961430908070153m772bf312s40a7a89cf688d4f0@mail.gmail.com>
References: <f2d961430908070153m772bf312s40a7a89cf688d4f0@mail.gmail.com>
Message-ID: <20090807100250.GD31589@trillke.net>

Hi Gordon, 

On Fri, Aug 07, 2009 at 18:53 +1000, Gordon Wrigley wrote:
> My python code talks to a C library via ctypes and it in turn prints
> debug to stdout.
> And we test this with py.test, which by itself works just fine.

by this you mean that it looks roughly like this:

    http://paste.pocoo.org/show/132982/

right? 

> But if at the bash prompt we then redirect the output of py.test to a
> file suddenly the C debug starts propagating out of py.test.

hum, redirecting the above example with "py.test x.py >log" works fine. 

The following may be interesting to observe (i just did): 

    def test_hello():
        myfile = os.fdopen(1, "w")
        myfile.write("hello\n")
        myfile.flush()
        assert 0

This works fine but if i leave out the flush() the write
will *not* be captured appropriately.  I guess that it is because
of the missing buffering ("os.fdopen(1, 'w', 1)" and no flush works) 

FYI the pytest_capture plugin resumes capturing for calling
into test methods, setup/teardown code and collection
respectively and otherwise suspends capturing.  Even if this
was modified (i.e. capturing would always happen) i imagine
that the missing flush/close could associate the actual write 
into a different test. 

Maybe your behaviour is still a different issue or even a bug, though. 

> We generally run py.test with -v and -x, but removing those didn't
> change the behavior at all.

that shouldn't change anything related to capturing, indeed. 

> I'm at a bit of loss as to where to begin with this one, any suggestions?

maybe my above considerations help a bit and you can maybe
come up with a breaking example or fix things on your side? 

I want to do a 1.0.1 soon with some capturing fixes (related
to unicode handling, though) and i'd like to include any other
related improvements or add to the docs. 

best,
holger 

> gordonw at gohma:~$ uname -a
> Linux gohma 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25 00:28:35 UTC
> 2009 i686 GNU/Linux
> gordonw at gohma:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import py
> >>> py.version
> '1.0.0'
> 
> Regards,
> Gordon
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From py-dev at tolomea.com  Fri Aug  7 12:31:45 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 7 Aug 2009 20:31:45 +1000
Subject: [py-dev] issues with stdout redirection, ctypes and py.test
In-Reply-To: <20090807100250.GD31589@trillke.net>
References: <f2d961430908070153m772bf312s40a7a89cf688d4f0@mail.gmail.com>
	<20090807100250.GD31589@trillke.net>
Message-ID: <f2d961430908070331h4699527eua33deab1150b71c4@mail.gmail.com>

I need to have a think about what you've said and look at producing a
minimal reproduction. However I did forget to mention that the prints
from the python code are fine, it's only the ones from C that are
affected, in the good case the two sets are interleaved in the py.test
capture, in the bad case the python prints are in the py.test capture
and the C prints come out of py.test's stdout.
Now that I think about it some more I'm inclined to wonder from where
(and when) the C library acquires it's stdout handle.

On Fri, Aug 7, 2009 at 8:02 PM, holger krekel<holger at merlinux.eu> wrote:
> Hi Gordon,
>
> On Fri, Aug 07, 2009 at 18:53 +1000, Gordon Wrigley wrote:
>> My python code talks to a C library via ctypes and it in turn prints
>> debug to stdout.
>> And we test this with py.test, which by itself works just fine.
>
> by this you mean that it looks roughly like this:
>
> ? ?http://paste.pocoo.org/show/132982/
>
> right?
>
>> But if at the bash prompt we then redirect the output of py.test to a
>> file suddenly the C debug starts propagating out of py.test.
>
> hum, redirecting the above example with "py.test x.py >log" works fine.
>
> The following may be interesting to observe (i just did):
>
> ? ?def test_hello():
> ? ? ? ?myfile = os.fdopen(1, "w")
> ? ? ? ?myfile.write("hello\n")
> ? ? ? ?myfile.flush()
> ? ? ? ?assert 0
>
> This works fine but if i leave out the flush() the write
> will *not* be captured appropriately. ?I guess that it is because
> of the missing buffering ("os.fdopen(1, 'w', 1)" and no flush works)
>
> FYI the pytest_capture plugin resumes capturing for calling
> into test methods, setup/teardown code and collection
> respectively and otherwise suspends capturing. ?Even if this
> was modified (i.e. capturing would always happen) i imagine
> that the missing flush/close could associate the actual write
> into a different test.
>
> Maybe your behaviour is still a different issue or even a bug, though.
>
>> We generally run py.test with -v and -x, but removing those didn't
>> change the behavior at all.
>
> that shouldn't change anything related to capturing, indeed.
>
>> I'm at a bit of loss as to where to begin with this one, any suggestions?
>
> maybe my above considerations help a bit and you can maybe
> come up with a breaking example or fix things on your side?
>
> I want to do a 1.0.1 soon with some capturing fixes (related
> to unicode handling, though) and i'd like to include any other
> related improvements or add to the docs.
>
> best,
> holger
>
>> gordonw at gohma:~$ uname -a
>> Linux gohma 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25 00:28:35 UTC
>> 2009 i686 GNU/Linux
>> gordonw at gohma:~$ python
>> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
>> [GCC 4.3.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import py
>> >>> py.version
>> '1.0.0'
>>
>> Regards,
>> Gordon
>> _______________________________________________
>> py-dev mailing list
>> py-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/py-dev
>>
>
> --
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu
>

From py-dev at tolomea.com  Sat Aug  8 06:43:24 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Sat, 8 Aug 2009 14:43:24 +1000
Subject: [py-dev] issues with stdout redirection, ctypes and py.test
In-Reply-To: <f2d961430908070331h4699527eua33deab1150b71c4@mail.gmail.com>
References: <f2d961430908070153m772bf312s40a7a89cf688d4f0@mail.gmail.com>
	<20090807100250.GD31589@trillke.net>
	<f2d961430908070331h4699527eua33deab1150b71c4@mail.gmail.com>
Message-ID: <f2d961430908072143r1abca3b5g859f6f376538bb21@mail.gmail.com>

Since I have no clear concept of what is supposed to happen when a C
function called via ctypes does a printf, I have instead gone through
the C library and arranged to have all the printing routed through a
function pointer which I now setup as a python callback that does a
sys.stdout.write(). Problem solved.

On Fri, Aug 7, 2009 at 8:31 PM, Gordon Wrigley<py-dev at tolomea.com> wrote:
> I need to have a think about what you've said and look at producing a
> minimal reproduction. However I did forget to mention that the prints
> from the python code are fine, it's only the ones from C that are
> affected, in the good case the two sets are interleaved in the py.test
> capture, in the bad case the python prints are in the py.test capture
> and the C prints come out of py.test's stdout.
> Now that I think about it some more I'm inclined to wonder from where
> (and when) the C library acquires it's stdout handle.
>
> On Fri, Aug 7, 2009 at 8:02 PM, holger krekel<holger at merlinux.eu> wrote:
>> Hi Gordon,
>>
>> On Fri, Aug 07, 2009 at 18:53 +1000, Gordon Wrigley wrote:
>>> My python code talks to a C library via ctypes and it in turn prints
>>> debug to stdout.
>>> And we test this with py.test, which by itself works just fine.
>>
>> by this you mean that it looks roughly like this:
>>
>> ? ?http://paste.pocoo.org/show/132982/
>>
>> right?
>>
>>> But if at the bash prompt we then redirect the output of py.test to a
>>> file suddenly the C debug starts propagating out of py.test.
>>
>> hum, redirecting the above example with "py.test x.py >log" works fine.
>>
>> The following may be interesting to observe (i just did):
>>
>> ? ?def test_hello():
>> ? ? ? ?myfile = os.fdopen(1, "w")
>> ? ? ? ?myfile.write("hello\n")
>> ? ? ? ?myfile.flush()
>> ? ? ? ?assert 0
>>
>> This works fine but if i leave out the flush() the write
>> will *not* be captured appropriately. ?I guess that it is because
>> of the missing buffering ("os.fdopen(1, 'w', 1)" and no flush works)
>>
>> FYI the pytest_capture plugin resumes capturing for calling
>> into test methods, setup/teardown code and collection
>> respectively and otherwise suspends capturing. ?Even if this
>> was modified (i.e. capturing would always happen) i imagine
>> that the missing flush/close could associate the actual write
>> into a different test.
>>
>> Maybe your behaviour is still a different issue or even a bug, though.
>>
>>> We generally run py.test with -v and -x, but removing those didn't
>>> change the behavior at all.
>>
>> that shouldn't change anything related to capturing, indeed.
>>
>>> I'm at a bit of loss as to where to begin with this one, any suggestions?
>>
>> maybe my above considerations help a bit and you can maybe
>> come up with a breaking example or fix things on your side?
>>
>> I want to do a 1.0.1 soon with some capturing fixes (related
>> to unicode handling, though) and i'd like to include any other
>> related improvements or add to the docs.
>>
>> best,
>> holger
>>
>>> gordonw at gohma:~$ uname -a
>>> Linux gohma 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25 00:28:35 UTC
>>> 2009 i686 GNU/Linux
>>> gordonw at gohma:~$ python
>>> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
>>> [GCC 4.3.3] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>> >>> import py
>>> >>> py.version
>>> '1.0.0'
>>>
>>> Regards,
>>> Gordon
>>> _______________________________________________
>>> py-dev mailing list
>>> py-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/py-dev
>>>
>>
>> --
>> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
>> Python, PyPy, pytest contracting: http://merlinux.eu
>>
>

From holger at merlinux.eu  Sat Aug  8 16:24:04 2009
From: holger at merlinux.eu (holger krekel)
Date: Sat, 8 Aug 2009 16:24:04 +0200
Subject: [py-dev] issues with stdout redirection, ctypes and py.test
In-Reply-To: <f2d961430908072143r1abca3b5g859f6f376538bb21@mail.gmail.com>
References: <f2d961430908070153m772bf312s40a7a89cf688d4f0@mail.gmail.com>
	<20090807100250.GD31589@trillke.net>
	<f2d961430908070331h4699527eua33deab1150b71c4@mail.gmail.com>
	<f2d961430908072143r1abca3b5g859f6f376538bb21@mail.gmail.com>
Message-ID: <20090808142404.GH31589@trillke.net>

On Sat, Aug 08, 2009 at 14:43 +1000, Gordon Wrigley wrote:
> Since I have no clear concept of what is supposed to happen when a C
> function called via ctypes does a printf, I have instead gone through
> the C library and arranged to have all the printing routed through a
> function pointer which I now setup as a python callback that does a
> sys.stdout.write(). Problem solved.

if you can try to ensure open the file (look for "fdopen(2)" or so) 
in unbuffered mode that should work.  Would be good to know. 
I have played on the command line and it appeared to me that
this buffering mode is very likely the culprit. 

cheers,
holger


> On Fri, Aug 7, 2009 at 8:31 PM, Gordon Wrigley<py-dev at tolomea.com> wrote:
> > I need to have a think about what you've said and look at producing a
> > minimal reproduction. However I did forget to mention that the prints
> > from the python code are fine, it's only the ones from C that are
> > affected, in the good case the two sets are interleaved in the py.test
> > capture, in the bad case the python prints are in the py.test capture
> > and the C prints come out of py.test's stdout.
> > Now that I think about it some more I'm inclined to wonder from where
> > (and when) the C library acquires it's stdout handle.
> >
> > On Fri, Aug 7, 2009 at 8:02 PM, holger krekel<holger at merlinux.eu> wrote:
> >> Hi Gordon,
> >>
> >> On Fri, Aug 07, 2009 at 18:53 +1000, Gordon Wrigley wrote:
> >>> My python code talks to a C library via ctypes and it in turn prints
> >>> debug to stdout.
> >>> And we test this with py.test, which by itself works just fine.
> >>
> >> by this you mean that it looks roughly like this:
> >>
> >> ? ?http://paste.pocoo.org/show/132982/
> >>
> >> right?
> >>
> >>> But if at the bash prompt we then redirect the output of py.test to a
> >>> file suddenly the C debug starts propagating out of py.test.
> >>
> >> hum, redirecting the above example with "py.test x.py >log" works fine.
> >>
> >> The following may be interesting to observe (i just did):
> >>
> >> ? ?def test_hello():
> >> ? ? ? ?myfile = os.fdopen(1, "w")
> >> ? ? ? ?myfile.write("hello\n")
> >> ? ? ? ?myfile.flush()
> >> ? ? ? ?assert 0
> >>
> >> This works fine but if i leave out the flush() the write
> >> will *not* be captured appropriately. ?I guess that it is because
> >> of the missing buffering ("os.fdopen(1, 'w', 1)" and no flush works)
> >>
> >> FYI the pytest_capture plugin resumes capturing for calling
> >> into test methods, setup/teardown code and collection
> >> respectively and otherwise suspends capturing. ?Even if this
> >> was modified (i.e. capturing would always happen) i imagine
> >> that the missing flush/close could associate the actual write
> >> into a different test.
> >>
> >> Maybe your behaviour is still a different issue or even a bug, though.
> >>
> >>> We generally run py.test with -v and -x, but removing those didn't
> >>> change the behavior at all.
> >>
> >> that shouldn't change anything related to capturing, indeed.
> >>
> >>> I'm at a bit of loss as to where to begin with this one, any suggestions?
> >>
> >> maybe my above considerations help a bit and you can maybe
> >> come up with a breaking example or fix things on your side?
> >>
> >> I want to do a 1.0.1 soon with some capturing fixes (related
> >> to unicode handling, though) and i'd like to include any other
> >> related improvements or add to the docs.
> >>
> >> best,
> >> holger
> >>
> >>> gordonw at gohma:~$ uname -a
> >>> Linux gohma 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25 00:28:35 UTC
> >>> 2009 i686 GNU/Linux
> >>> gordonw at gohma:~$ python
> >>> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> >>> [GCC 4.3.3] on linux2
> >>> Type "help", "copyright", "credits" or "license" for more information.
> >>> >>> import py
> >>> >>> py.version
> >>> '1.0.0'
> >>>
> >>> Regards,
> >>> Gordon
> >>> _______________________________________________
> >>> py-dev mailing list
> >>> py-dev at codespeak.net
> >>> http://codespeak.net/mailman/listinfo/py-dev
> >>>
> >>
> >> --
> >> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> >> Python, PyPy, pytest contracting: http://merlinux.eu
> >>
> >
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Sat Aug  8 16:50:18 2009
From: holger at merlinux.eu (holger krekel)
Date: Sat, 8 Aug 2009 16:50:18 +0200
Subject: [py-dev] issues with stdout redirection, ctypes and py.test
In-Reply-To: <20090808142404.GH31589@trillke.net>
References: <f2d961430908070153m772bf312s40a7a89cf688d4f0@mail.gmail.com>
	<20090807100250.GD31589@trillke.net>
	<f2d961430908070331h4699527eua33deab1150b71c4@mail.gmail.com>
	<f2d961430908072143r1abca3b5g859f6f376538bb21@mail.gmail.com>
	<20090808142404.GH31589@trillke.net>
Message-ID: <20090808145018.GJ31589@trillke.net>

On Sat, Aug 08, 2009 at 16:24 +0200, holger krekel wrote:
> On Sat, Aug 08, 2009 at 14:43 +1000, Gordon Wrigley wrote:
> > Since I have no clear concept of what is supposed to happen when a C
> > function called via ctypes does a printf, I have instead gone through
> > the C library and arranged to have all the printing routed through a
> > function pointer which I now setup as a python callback that does a
> > sys.stdout.write(). Problem solved.
> 
> if you can try to ensure open the file (look for "fdopen(2)" or so) 

probably not the right suggestion - i didn't do C for a long time - 
but fsync(1) after each printf (maybe done in a macro anyway?) could
show if it's a buffering issue.  

best,
holger


> in unbuffered mode that should work.  Would be good to know. 
> I have played on the command line and it appeared to me that
> this buffering mode is very likely the culprit. 
> 
> cheers,
> holger
> 
> 
> > On Fri, Aug 7, 2009 at 8:31 PM, Gordon Wrigley<py-dev at tolomea.com> wrote:
> > > I need to have a think about what you've said and look at producing a
> > > minimal reproduction. However I did forget to mention that the prints
> > > from the python code are fine, it's only the ones from C that are
> > > affected, in the good case the two sets are interleaved in the py.test
> > > capture, in the bad case the python prints are in the py.test capture
> > > and the C prints come out of py.test's stdout.
> > > Now that I think about it some more I'm inclined to wonder from where
> > > (and when) the C library acquires it's stdout handle.
> > >
> > > On Fri, Aug 7, 2009 at 8:02 PM, holger krekel<holger at merlinux.eu> wrote:
> > >> Hi Gordon,
> > >>
> > >> On Fri, Aug 07, 2009 at 18:53 +1000, Gordon Wrigley wrote:
> > >>> My python code talks to a C library via ctypes and it in turn prints
> > >>> debug to stdout.
> > >>> And we test this with py.test, which by itself works just fine.
> > >>
> > >> by this you mean that it looks roughly like this:
> > >>
> > >> ? ?http://paste.pocoo.org/show/132982/
> > >>
> > >> right?
> > >>
> > >>> But if at the bash prompt we then redirect the output of py.test to a
> > >>> file suddenly the C debug starts propagating out of py.test.
> > >>
> > >> hum, redirecting the above example with "py.test x.py >log" works fine.
> > >>
> > >> The following may be interesting to observe (i just did):
> > >>
> > >> ? ?def test_hello():
> > >> ? ? ? ?myfile = os.fdopen(1, "w")
> > >> ? ? ? ?myfile.write("hello\n")
> > >> ? ? ? ?myfile.flush()
> > >> ? ? ? ?assert 0
> > >>
> > >> This works fine but if i leave out the flush() the write
> > >> will *not* be captured appropriately. ?I guess that it is because
> > >> of the missing buffering ("os.fdopen(1, 'w', 1)" and no flush works)
> > >>
> > >> FYI the pytest_capture plugin resumes capturing for calling
> > >> into test methods, setup/teardown code and collection
> > >> respectively and otherwise suspends capturing. ?Even if this
> > >> was modified (i.e. capturing would always happen) i imagine
> > >> that the missing flush/close could associate the actual write
> > >> into a different test.
> > >>
> > >> Maybe your behaviour is still a different issue or even a bug, though.
> > >>
> > >>> We generally run py.test with -v and -x, but removing those didn't
> > >>> change the behavior at all.
> > >>
> > >> that shouldn't change anything related to capturing, indeed.
> > >>
> > >>> I'm at a bit of loss as to where to begin with this one, any suggestions?
> > >>
> > >> maybe my above considerations help a bit and you can maybe
> > >> come up with a breaking example or fix things on your side?
> > >>
> > >> I want to do a 1.0.1 soon with some capturing fixes (related
> > >> to unicode handling, though) and i'd like to include any other
> > >> related improvements or add to the docs.
> > >>
> > >> best,
> > >> holger
> > >>
> > >>> gordonw at gohma:~$ uname -a
> > >>> Linux gohma 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25 00:28:35 UTC
> > >>> 2009 i686 GNU/Linux
> > >>> gordonw at gohma:~$ python
> > >>> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> > >>> [GCC 4.3.3] on linux2
> > >>> Type "help", "copyright", "credits" or "license" for more information.
> > >>> >>> import py
> > >>> >>> py.version
> > >>> '1.0.0'
> > >>>
> > >>> Regards,
> > >>> Gordon
> > >>> _______________________________________________
> > >>> py-dev mailing list
> > >>> py-dev at codespeak.net
> > >>> http://codespeak.net/mailman/listinfo/py-dev
> > >>>
> > >>
> > >> --
> > >> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> > >> Python, PyPy, pytest contracting: http://merlinux.eu
> > >>
> > >
> > 
> 
> -- 
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu 
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Wed Aug 12 20:33:42 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 12 Aug 2009 20:33:42 +0200
Subject: [py-dev] specifying testfile patterns / RFC
Message-ID: <20090812183342.GT31589@trillke.net>

Hi all, 

i am hacking on a new plugin to go into py.test-1.0.1 and i am
doing that in documentation-driven style.  So i am interested
in feedback and comments on the below soon-to-become reality
"pytest_testfiles" plugin.  

best & thanks,
holger

pytest_testfiles.py: determine test files for command line specified args. 
===========================================================================

usage example: specifying testfiles in a python module
-----------------------------------------------------

If you have a module ``mymodule.py`` you can specify test files like this::

    # content of: mymodule.py
   
    # relative paths 
    __testfiles__ = ['test/test_unittest.py', 'test/test_functional.py']

and if you then type:: 

    py.test mymodule.py 

then this will run the test files which were specified relatively 
to the directory of ``mymodule.py``.  

Ordering is significant: if you specify quick-running unit-tests 
first followed by slower running functional tests then issuing
commands like ``py.test -x`` allows to find failures quickly
and the easier-to-debug ones first. 


usage example: specifying project-specific testfile patterns
--------------------------------------------------------------

Suppose your project has this layout::

    app/
        sub1/
            module1.py 

    tests/
        unit/
            test_module1.py 
        functional/
            test_one.py 
            test_two.py 

you may specify filematching rules for finding tests like this::

    # contents of: conftest.py  
    conf_testfilepatterns = ["{topdir}/tests/unit/test_{basename}", 
                             "{topdir}/tests/functional/*.py"]

and if you then type::

    py.test app/sub1/module1.py 

the specified testfile patterns will be applied to check if there are matching test files.  In our example we will run test files in the following order:: 

    tests/unit/test_module.py 
    tests/functional/test_one.py
    tests/functional/test_two.py


default is to look for test files being close to code  
----------------------------------------------------------

If no ``conf_testfilepatterns`` are specified, py.test will use this
as a default setting::

    conf_testfilepatterns = [
        "*/test_{purebasename}.py", 
        "**/test_*.py", 
        "**/*_test.py"
    ]

Note that each matching test file will only be run at most once.  

Substitutions
-------------------

The following substitutions are performed for each file-or-directory command line argument:: 

    dirname        the dirname of the file 
    basename       basename of the file 
    purebasename   basename without the extension
    topdir         first upwards parent directory not containing __init__.py


From Ronny.Pfannschmidt at gmx.de  Tue Aug 18 13:06:38 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Tue, 18 Aug 2009 13:06:38 +0200
Subject: [py-dev] building reusable testsuites
Message-ID: <1250593598.23706.7.camel@localhost>

Hi,

while building the testsuite of anyvc
i noticed i need a way to enable others 
to use my testsuite for backends on their own code
in order to test/validate 3rd party backends.

Right now i don't think its possible,
please lets discuss possible solutions.

Regards Ronny Pfannschmidt


From holger at merlinux.eu  Tue Aug 18 18:58:37 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 18 Aug 2009 18:58:37 +0200
Subject: [py-dev] building reusable testsuites
In-Reply-To: <1250593598.23706.7.camel@localhost>
References: <1250593598.23706.7.camel@localhost>
Message-ID: <20090818165837.GA15455@trillke.net>

Hey Ronny, 

On Tue, Aug 18, 2009 at 13:06 +0200, Ronny Pfannschmidt wrote:
> while building the testsuite of anyvc
> i noticed i need a way to enable others 
> to use my testsuite for backends on their own code
> in order to test/validate 3rd party backends.
> 
> Right now i don't think its possible,
> please lets discuss possible solutions.

given that this other person has a "mybackend.py" file maybe running:

    py.test mybackend.py 

could look at __testfiles__ in the module which would
reference the tests you think each backend needs to pass? 
(see my recent RFC here on the list).  The tests would 
require a function argument that could be setup
by the mybackend.py file or within a conftest.py. 

best,
holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Wed Aug 19 18:34:43 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 19 Aug 2009 18:34:43 +0200
Subject: [py-dev] 1.0.1 pylib/py.test: improved reporting,
	nose/unittest.py support, bug fixes
Message-ID: <20090819163443.GE15455@trillke.net>

Hi all, 

pylib/py.test 1.0.1 is a bugfix release also coming with: 

* improved documentation, site navigation, configuration help
* test failure reporting improvements
* support for directly running existing nose/unittest.py style tests

visit here for more info, including quickstart and tutorials: 

    http://pytest.org and http://pylib.org

have fun, 
holger

Changelog 1.0.0 to 1.0.1 
------------------------

* added a default 'pytest_nose' plugin which handles nose.SkipTest, 
  nose-style function/method/generator setup/teardown and 
  tries to report functions correctly.  

* improved documentation, better navigation: see http://pytest.org 

* added a "--help-config" option to show conftest.py / ENV-var names for
  all longopt cmdline options, and some special conftest.py variables. 
  renamed 'conf_capture' conftest setting to 'option_capture' accordingly. 

* unicode fixes: capturing and unicode writes to sys.stdout 
  (through e.g a print statement) now work within tests, 
  they are encoded as "utf8" by default, also terminalwriting 
  was adapted and somewhat unified between windows and linux 

* fix issue #27: better reporting on non-collectable items given on commandline 
  (e.g. pyc files)

* fix issue #33: added --version flag (thanks Benjamin Peterson) 

* fix issue #32: adding support for "incomplete" paths to wcpath.status()

* "Test" prefixed classes are *not* collected by default anymore if they 
  have an __init__ method 

* monkeypatch setenv() now accepts a "prepend" parameter

* improved reporting of collection error tracebacks

* simplified multicall mechanism and plugin architecture, 
  renamed some internal methods and argnames 

From Ronny.Pfannschmidt at gmx.de  Mon Aug 24 13:51:19 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Mon, 24 Aug 2009 13:51:19 +0200
Subject: [py-dev] outlining what i intend to do with execnet + what
 additional features could be helpfull
Message-ID: <1251114679.32344.18.camel@localhost>

hi,

first let me outline my needs for

anyvc
  a vcs abstraction lib
pida
  a ide based on the idea that everything is a plugin


anyvc
------

* execute backends in a different process 
  * circumvent gpl/gpl2only of hg/dulwich/bzr
* execute the fontend on python platforms not supported by the backends
  * python 3
  * jython
  * ironpython
  * pypy

pida
----

an ide based on plugins

* run parts of plugins in different processes
  * avoid gil races for cpu-intensive tasks
* run parts of plugins on different computers/platforms
  * test execution
  * introspectiong of test/deployment environments
  * general work on remote data/projects


In order to archieve those goals 
i think i need a few conceptual ideas 
for installing the different libs on the different platforms
as well as a tool to help proxying method calls to remote objects.

regards Ronny


From holger at merlinux.eu  Mon Aug 24 14:34:14 2009
From: holger at merlinux.eu (holger krekel)
Date: Mon, 24 Aug 2009 14:34:14 +0200
Subject: [py-dev] outlining what i intend to do with execnet +
	what	additional features could be helpfull
In-Reply-To: <1251114679.32344.18.camel@localhost>
References: <1251114679.32344.18.camel@localhost>
Message-ID: <20090824123414.GY15455@trillke.net>

Hi Ronny, 

from IRC i also know that you are wondering how/if to best apply 
py.execnet for the tasks.  Can you post an example of
a frontend command/API you want to implement and some 
code of how that translates to one particular API? 
I'd then consider and suggest a way/method to apply execnet. 

cheers,
holger


On Mon, Aug 24, 2009 at 13:51 +0200, Ronny Pfannschmidt wrote:
> hi,
> 
> first let me outline my needs for
> 
> anyvc
>   a vcs abstraction lib
> pida
>   a ide based on the idea that everything is a plugin
> 
> 
> anyvc
> ------
> 
> * execute backends in a different process 
>   * circumvent gpl/gpl2only of hg/dulwich/bzr
> * execute the fontend on python platforms not supported by the backends
>   * python 3
>   * jython
>   * ironpython
>   * pypy
> 
> pida
> ----
> 
> an ide based on plugins
> 
> * run parts of plugins in different processes
>   * avoid gil races for cpu-intensive tasks
> * run parts of plugins on different computers/platforms
>   * test execution
>   * introspectiong of test/deployment environments
>   * general work on remote data/projects
> 
> 
> In order to archieve those goals 
> i think i need a few conceptual ideas 
> for installing the different libs on the different platforms
> as well as a tool to help proxying method calls to remote objects.
> 
> regards Ronny
> 
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From Ronny.Pfannschmidt at gmx.de  Mon Aug 24 15:42:45 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Mon, 24 Aug 2009 15:42:45 +0200
Subject: [py-dev] outlining what i intend to do with execnet + what
 additional features could be helpfull
In-Reply-To: <20090824123414.GY15455@trillke.net>
References: <1251114679.32344.18.camel@localhost>
	<20090824123414.GY15455@trillke.net>
Message-ID: <1251121365.32344.60.camel@localhost>

On Mon, 2009-08-24 at 14:34 +0200, holger krekel wrote:
> Hi Ronny, 
> 
> from IRC i also know that you are wondering how/if to best apply 
> py.execnet for the tasks.  Can you post an example of
> a frontend command/API you want to implement and some 
> code of how that translates to one particular API? 
> I'd then consider and suggest a way/method to apply execnet.

i think a good initial example for anyvc would be one of the unittests

def test_build_first_commit(mgr):
    repo = mgr.make_repo('repo')
    with repo.transaction(message='initial', author='test') as root:
        with root.join('test.txt').open('w') as f:
            f.write("text")

    with repo.get_default_head() as root:
        with root.join("test.txt").open() as f:
            content = f.read()
            assert content == 'text'

this test creates a repo, an initial commit, then asserts the content
mgr is a utility to manage the directories where tests dump their data
make_repo creates the repository object that should "be" in another
object when using py.execnet

i'll try to extract something semilar for pida later

> 
> cheers,
> holger
> 
> 
> On Mon, Aug 24, 2009 at 13:51 +0200, Ronny Pfannschmidt wrote:
> > hi,
> > 
> > first let me outline my needs for
> > 
> > anyvc
> >   a vcs abstraction lib
> > pida
> >   a ide based on the idea that everything is a plugin
> > 
> > 
> > anyvc
> > ------
> > 
> > * execute backends in a different process 
> >   * circumvent gpl/gpl2only of hg/dulwich/bzr
> > * execute the fontend on python platforms not supported by the backends
> >   * python 3
> >   * jython
> >   * ironpython
> >   * pypy
> > 
> > pida
> > ----
> > 
> > an ide based on plugins
> > 
> > * run parts of plugins in different processes
> >   * avoid gil races for cpu-intensive tasks
> > * run parts of plugins on different computers/platforms
> >   * test execution
> >   * introspectiong of test/deployment environments
> >   * general work on remote data/projects
> > 
> > 
> > In order to archieve those goals 
> > i think i need a few conceptual ideas 
> > for installing the different libs on the different platforms
> > as well as a tool to help proxying method calls to remote objects.
> > 
> > regards Ronny
> > 


From holger at merlinux.eu  Mon Aug 24 16:16:26 2009
From: holger at merlinux.eu (holger krekel)
Date: Mon, 24 Aug 2009 16:16:26 +0200
Subject: [py-dev] outlining what i intend to do with execnet +
	what	additional features could be helpfull
In-Reply-To: <1251121365.32344.60.camel@localhost>
References: <1251114679.32344.18.camel@localhost>
	<20090824123414.GY15455@trillke.net>
	<1251121365.32344.60.camel@localhost>
Message-ID: <20090824141626.GB15455@trillke.net>

On Mon, Aug 24, 2009 at 15:42 +0200, Ronny Pfannschmidt wrote:
> On Mon, 2009-08-24 at 14:34 +0200, holger krekel wrote:
> > Hi Ronny, 
> > 
> > from IRC i also know that you are wondering how/if to best apply 
> > py.execnet for the tasks.  Can you post an example of
> > a frontend command/API you want to implement and some 
> > code of how that translates to one particular API? 
> > I'd then consider and suggest a way/method to apply execnet.
> 
> i think a good initial example for anyvc would be one of the unittests
> 
> def test_build_first_commit(mgr):
>     repo = mgr.make_repo('repo')
>     with repo.transaction(message='initial', author='test') as root:
>         with root.join('test.txt').open('w') as f:
>             f.write("text")
> 
>     with repo.get_default_head() as root:
>         with root.join("test.txt").open() as f:
>             content = f.read()
>             assert content == 'text'
> 
> this test creates a repo, an initial commit, then asserts the content
> mgr is a utility to manage the directories where tests dump their data
> make_repo creates the repository object that should "be" in another
> object when using py.execnet

The execnet spirit is to design your own local protocol.  Let
me give and give some context by means of an example: py/path/gateway 
contains an experimental remote-path implementation, using execnet: 

* remotepath.py: a "RemotePath" class (similar to your frontend "Repo" one)
* channeltest.py: a PathServer backend implementing the fs access code 

Basically the frontend-class is connected through (sub) channels with 
a backend instance.  If you say on the frontend: 

    x = remotepath.join("hello")

this will enter this code on the caller/client/frontend side: 

    def join(self, *args):
        id = ~COUNTER.next()
        self._channel.send(('JOIN', self._id, id) + args)
        return RemotePath(self._channel, id)

which creates a prospective 'id' for the result, sends a command 
to the backend and immediately returns.  This has *zero* latency. 

If you later perform an operation on the returned instance like list()

    def listdir(self, *args):
        self._channel.send(('LIST', self._id) + args)
        return [RemotePath(self._channel, id, basename)
                for (id, basename) in self._channel.receive()]

this will send a LIST command to the backend PathServer which 
itself implements it like this: 

    def command_LIST(self, id, *args):
        path = self.C2P[id]
        answer = [(self.p2c(p), p.basename) for p in path.listdir(*args)]
        self.channel.send(answer)
  
so here we create a local path using the client-provided ID
and instantiate an answer.  C2P and p2c() help to manage the
ID/path relations.   

For implementing our own small custom "command" pattern 
protocol the backend PAthServer implements a simple dispatch
loop based on the frontend/backend connecting channel: 

    def serve(self):
        try:
            while 1:
                msg = self.channel.receive()
                meth = getattr(self, 'command_' + msg[0])
                meth(*msg[1:])
        except EOFError:
            pass

hope this makes sense to you and serves as inspiration.  To summarize:
Design your own protocol, optimized for your special purposes.
You can easily evolve your protocol on a per-need basis -
no need to have global RPC or IDL/RMI interface definitions.
There also is no need for "matching" server/client software etc.
It's a zero-install "local" protocol, also easy to test. 

best,

holger

From Ronny.Pfannschmidt at gmx.de  Thu Aug 27 10:02:06 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Thu, 27 Aug 2009 10:02:06 +0200
Subject: [py-dev] outlining some of my needs for execnet + possible solutions
Message-ID: <1251360126.24112.59.camel@localhost>

Hi,

i'd like to outline some needs i have for execnet
as well as propossing some rough sketchouts of possible solution


py.execnet between python2 and python3
--------------------------------------

This arises as i want to use anyvc in python3 while the backends are not
portable to python3 yet.

However syntax differences make normal interaction hard
and will most likely perfectly destroy the zero install mechanisms.


installing more complex dependencies in virtualenvs and reusing those
---------------------------------------------------------------------

Syntax differences between python versions as well as complex package
dependencies mandate installing some packages on the remote side.

Tools like virtualenv help with installing those separate from the
normal system as well as with limited rights.

However those tools might need some bootstraping outside of python as
the remote might not be ready to just start up a execnet slave.


a simple remote object protocol via wrappers around channels
------------------------------------------------------------

Inventing own protocols on top of channels if one want's to controll
remote-objects is a fragile task,
helper classes to ease the process are very welcome.


the proposed solutions
-----------------------

1. gw.remote_call(remote_callable, *k, **kw) as alternative to
gw.remote_exec(code)

   * import the callable remote_callable on the remote
   * call remote_callable(channel, *k, **kw)
     where channel is the usual remote_channel

2. a virtualenv based minimal install system

   it needs to do without a usable remote python
   (ie wrong platform, whatever):
   
     * create a virtualenv at a given path
     * install something like a minimal py.execnet slave,
       so the remote side can be started

   * all other install operations should be controllable
     via some kind of remote_exec/remote_call

3. RemoteCaller/RemoteHandler as wrappers around channels that represent
remote objects

   * a experiment in anyvc
   * currently lacks a way to propperly document remote methods
   * RemoteCaller is a base class for wrappers 
     that call to remote objects via a channel

     object.method(*k, **kw) translates to 
        channel.send(('method', k, kw))
        return channel.receive()

     the default handling can be overridden by implementing own methods
     the method self._call_method(name, k, kw) ma

	
   * RemoteHandler is a base class for Channel callbacks 
     once can use as convience wrapper around remote objects
     in order too ease handling messages
     for each incoming method, k, kw tuple 
     it calls the method on self, 
     unpacking the arguments
  


From holger at merlinux.eu  Thu Aug 27 12:17:57 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 27 Aug 2009 12:17:57 +0200
Subject: [py-dev] 1.0.2 released / some 1.1 planning notes
Message-ID: <20090827101757.GQ15455@trillke.net>

Hi all, 

i just released a pylib/py.test 1.0.2 fixing a number of
packaging issues, triggered by fedora packaging issues.  
I also added a link to the new pytest_django plugin. 
see the plugin index at http://pylib.org 

FYI i am currently refactoring and simplifying the code of 
pylib/py.test and aim for cleaned up mostly-compatible to 
1.1 release next month:

* featuring cpython 2.4 through to 3.1, jython + pypy compatilibty. 

* moving the tests out of the lib itself 

* introducing a way to specify what tests belong to a module/directory 
  (see recent RFC mails) 

* relicensing to LGPL

I'd also like to get the 1.1 flagged issues in the tracker resolved, see 

http://bitbucket.org/hpk42/py-trunk/issues/?status=new&status=open

cheers,

holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From Ronny.Pfannschmidt at gmx.de  Tue Sep  1 01:59:20 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Tue, 01 Sep 2009 01:59:20 +0200
Subject: [py-dev] examples for programmatically running py.test and
 collecting the results in a custom way?
Message-ID: <1251763160.25814.2.camel@localhost>

hi,

for the pida integration of py.test i could use some ideas on how to run
it and how to iteratively collect the results./changes to the results
for looponfail.

Regards Ronny Pfannschmidt



From holger at merlinux.eu  Tue Sep  1 09:46:37 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 1 Sep 2009 09:46:37 +0200
Subject: [py-dev] examples for programmatically running py.test
	and	collecting the results in a custom way?
In-Reply-To: <1251763160.25814.2.camel@localhost>
References: <1251763160.25814.2.camel@localhost>
Message-ID: <20090901074637.GC15455@trillke.net>

Hi Ronny, 

On Tue, Sep 01, 2009 at 01:59 +0200, Ronny Pfannschmidt wrote:
> hi,
> 
> for the pida integration of py.test i could use some ideas on how to run
> it and how to iteratively collect the results./changes to the results
> for looponfail.

could you provide more context and what you conceptually
or concretely need as an API? 

holger


From Ronny.Pfannschmidt at gmx.de  Tue Sep  1 10:18:47 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Tue, 01 Sep 2009 10:18:47 +0200
Subject: [py-dev] examples for programmatically running py.test and
 collecting the results in a custom way?
In-Reply-To: <20090901074637.GC15455@trillke.net>
References: <1251763160.25814.2.camel@localhost>
	<20090901074637.GC15455@trillke.net>
Message-ID: <1251793127.25814.11.camel@localhost>

On Tue, 2009-09-01 at 09:46 +0200, holger krekel wrote:
> Hi Ronny, 
> 
> On Tue, Sep 01, 2009 at 01:59 +0200, Ronny Pfannschmidt wrote:
> > hi,
> > 
> > for the pida integration of py.test i could use some ideas on how to run
> > it and how to iteratively collect the results./changes to the results
> > for looponfail.
> 
> could you provide more context and what you conceptually
> or concretely need as an API? 


Lets try to outline, what i conceptually need.

* knowdegde about what tests are run, and in what environment they are
executed, preferably as somek kind of iterator
* knowdegde about the results of those runs, their output, whatever
other metadata pytest yields that might be helpfull

is the basic need i have for running the tests once

for the looponfail stuff there are some more needs
i need to know about what tests:
* are rerun as result of a failure
* are added in a successive run
* gone missing in this run due to a error
* gone missing due to being removed from the unittests 

Ronny


From holger at merlinux.eu  Fri Sep  4 22:16:20 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 4 Sep 2009 22:16:20 +0200
Subject: [py-dev] py-trunk runs on python3 quite well now!
Message-ID: <20090904201620.GU15455@trillke.net>

Hi all, 

good news!  py-trunk on bitbucket now runs rather well on 
top of Python 3.1.1 and remains compatible to python back 
to 2.4.  Many thanks to Benjamin Peterson who contributed 
among other chunks new and simpler "assert" interpretation 
code - now enabled by default on CPython 2.5 till 3.1

Most everything works except for one functional
test: distributed testing across the 3k version barrier.  
That would obviously be interesting to get working 
because it would allow unified test runs across 
all versions. 

Apart from the compatibility work i did tons of 
removal of old or unused code, reviewed and refactored 
tests and code to live in fewer files - aiming for much
more readable source and a 1:1 mapping of namespaces
to directory/file structure.  

best, 
holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Mon Sep  7 14:38:37 2009
From: holger at merlinux.eu (holger krekel)
Date: Mon, 7 Sep 2009 14:38:37 +0200
Subject: [py-dev] jython and py.test code interaction
Message-ID: <20090907123837.GD15455@trillke.net>

Hi all, hi Benjamin, hi Frank (CCed), 

i played with Jython2.5.1rc1 and both py.test and 
py.execnet more or less work :)

Here is a simple console session opening a connection from
CPython to Jython for executing code dynamically: 

    http://paste.pocoo.org/show/138497/

This means that one can interact with Java libraries and apps
**from CPython** by firing up a Jython interpreter process and 
driving it with the "zero-install" execnet approach.  The Jython
interpreter can even be on a different machine :) 

py.test also runs on top of Jython but there are some issues with 
respect to showing tracebacks, here is a session obtained from execing

  jython2.5.1rc1 py/bin/py.test test/code/test_code.py --pastebin=all

  -->  http://paste.pocoo.org/show/138496/

The py/code directory implements assert re-intepretation and
source retrieval for dynamic code compilation which is e.g.
used by py.test.raises(EXC, 'expression') in order to show 
nice tracebacks.  Currently i see three major issues: 

* jython doesn't lookup AssertionError from the builtins
  (i filed http://bugs.jython.org/issue1461)

* dynamic compilation, i.e. py.code.Code().new method:
  After adapting it to work with jython by using the
  new.code() call i noticed it doesn't preserve the 
  co_filename object as cpython and pypy do.  Maybe it's 
  anyway better to keep a WeakValueDictionary with 
  artifical-filename -> code mappings around and deprecate
  the new() method hack.

* assert re-interpretation: no clue how this could work yet.

any comments or offers for help welcome. 

best,
holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Tue Sep  8 00:51:00 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 8 Sep 2009 00:51:00 +0200
Subject: [py-dev] jython and py.test code interaction
In-Reply-To: <1afaf6160909071209q1c551522n539b895246ba6219@mail.gmail.com>
References: <20090907123837.GD15455@trillke.net>
	<4dab5f760909071025h4d8b1088h9415d7862f277949@mail.gmail.com>
	<1afaf6160909071030u7244792du18dc1110d4c5a7a5@mail.gmail.com>
	<4dab5f760909071134h6535de3bydc3c9a64f300cc87@mail.gmail.com>
	<1afaf6160909071209q1c551522n539b895246ba6219@mail.gmail.com>
Message-ID: <20090907225100.GF15455@trillke.net>

On Mon, Sep 07, 2009 at 14:09 -0500, Benjamin Peterson wrote:
> 2009/9/7 Frank Wierzbicki <fwierzbicki at gmail.com>:
> > On Mon, Sep 7, 2009 at 1:30 PM, Benjamin Peterson<benjamin at python.org> wrote:
> >> 2009/9/7 Frank Wierzbicki <fwierzbicki at gmail.com>:
> >>> On Mon, Sep 7, 2009 at 8:38 AM, holger krekel<holger at merlinux.eu> wrote:
> >>>> * assert re-interpretation: no clue how this could work yet.
> >>> What is assert re-interpretation?
> >>
> >> When an assert statement fails, py.test tries to provide more
> >> debugging information. It does this by walking over the AST of the
> >> assert statement and reinterpreting each part to determine where
> >> exactly it fails.
> >>
> >> I'm not sure why this wouldn't work on Jython except that my
> >> implementation uses the ast module, which is not found in Python 2.5.
> > We actually put the ast.py in our 2.5 -- but it does have a few
> > incompatibilities (for example, we don't currently allow arbitrary
> > python objects in attributes -- we attempt to convert them
> > immediately)
> 
> Well, assertion reinterpretation shouldn't need to do that, so I'm not
> sure why it doesn't work.

hum, i wouldn't bet that it doesn't work, actually - there
were a number of failures in the (py/code) tests and i 
just *presumed* it wouldn't work.   Maybe it works with 
the now fixed trunk.  need to check that. 

best,
holger

From holger at merlinux.eu  Tue Sep  8 10:16:26 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 8 Sep 2009 10:16:26 +0200
Subject: [py-dev] re-licensing to LGPL
Message-ID: <20090908081625.GH15455@trillke.net>

Hi all, 

as posted earlier in the 1.1 raodmap notes i just did the
re-licensing to LGPL, also added an FAQ entry, see e.g. here:

    http://codespeak.net/py/trunk/faq.html#whygpl

Let me know if you have any issues with this - preferably
practical ones.  I think i am somewhat familiar with the usual
pro/con arguments for BSD/GPL style licenses so there is no
need to repeat those.  But i don't have a totally fixed
opinion. 

cheers,
holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From holger at merlinux.eu  Tue Sep  8 18:25:20 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 8 Sep 2009 18:25:20 +0200
Subject: [py-dev] re-licensing to LGPL
In-Reply-To: <e3e180dc0909080621m6e4f11d8q71dfccd1c427c55f@mail.gmail.com>
References: <20090908081625.GH15455@trillke.net>
	<e3e180dc0909080621m6e4f11d8q71dfccd1c427c55f@mail.gmail.com>
Message-ID: <20090908162520.GI15455@trillke.net>

Hi Harald, 

nice to hear from you!

On Tue, Sep 08, 2009 at 15:21 +0200, Massa, Harald Armin wrote:
> Holger,
> 
> the most practical reason:
> 
> py.test should become the standard testing package for Python. There is no
> better.
>
> the best of breed belongs to.... the standard library.
> 
> and therefore the Python license is required, else it will be a non-starter.
>
> IF py.test does not want to become the best of breed of testing packages,
> what is the reason to exist, anyway?

wow, thanks for the appreciation!  I heart Guido at Pycon 2008 saying that 
only "stable" software should enter the standard library.  He added 
that in biology stable means dead.  py.test is in motion and 
aims to work with many Python interpreters and environments.  

In fact, i rather believe that the python distribution and
packaging world needs to change and improve instead of trying 
to push more stuff into a centralized library released with
the top-level-version of CPython.  Thankfully, Tarek Ziade 
and his fellowship of the packaging is moving along :)

best,
holger

From lac at openend.se  Tue Sep  8 18:50:04 2009
From: lac at openend.se (Laura Creighton)
Date: Tue, 08 Sep 2009 18:50:04 +0200
Subject: [py-dev] re-licensing to LGPL
In-Reply-To: Message from holger krekel <holger@merlinux.eu> of "Tue,
	08 Sep 2009 18:25:20 +0200." <20090908162520.GI15455@trillke.net> 
References: <20090908081625.GH15455@trillke.net>
	<e3e180dc0909080621m6e4f11d8q71dfccd1c427c55f@mail.gmail.com>
	<20090908162520.GI15455@trillke.net> 
Message-ID: <200909081650.n88Go4SC021666@theraft.openend.se>

In a message of Tue, 08 Sep 2009 18:25:20 +0200, holger krekel writes:
>Hi Harald, 
>
>nice to hear from you!
>
>On Tue, Sep 08, 2009 at 15:21 +0200, Massa, Harald Armin wrote:
>> Holger,
>> 
>> the most practical reason:
>> 
>> py.test should become the standard testing package for Python. There is
> no
>> better.
>>
>> the best of breed belongs to.... the standard library.
>> 
>> and therefore the Python license is required, else it will be a non-sta
>rter.
>>
>> IF py.test does not want to become the best of breed of testing package
>s,
>> what is the reason to exist, anyway?


Actually, assuming that you wanted to put your software into the
python standard library -- I know that Holger doesn't, but this
is just to make it clear to any of you who are reading this and
might have such ambitions --

THE PSF RECOMMENDS THAT YOU DO NOT USE THE PYTHON LICENSE.
THE PSF does not accept the PSF license, either.

Indeed, the only two licenses which are currently acceptable are
the Academic Free License http://www.opensource.org/licenses/afl-3.0.php
and the Apache license 2.0. http://www.opensource.org/licenses/apache2.0.php

Read (some of the) gory details about why here:
http://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq

And why can't Python relicense itself under a better license?  Because
the legal agreement that was entered into with CNRI prohibits changing
the license.

Just thought you should know this, even though Holger won't be affected
since he doesn't want py.test to be part of the standard library.

Laura



From fdg001 at gmx.net  Wed Sep  9 11:53:03 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Wed, 09 Sep 2009 10:53:03 +0100
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <4A58424A.3050400@gmx.net>
References: <4A5315A9.5030401@gmx.net>
	<20090707102853.GL7437@trillke.net>	<4A54EF91.9060004@gmx.net>
	<4A550A1A.8080803@gmx.net>	<20090709114709.GP7437@trillke.net>
	<4A58424A.3050400@gmx.net>
Message-ID: <4AA77AFF.6080900@gmx.net>

After a few weeks of using py.test in anger, I realized that not having 
to write test descriptions made me much more productive.
So I've reversed my original position and now think that py.test's focus 
on "rapid testing with minimal effort" actually works very well for me.

Unless anyone has an actual need for this, I'm gonna treat it as YAGNI.


-- F.

From fijall at gmail.com  Wed Sep  9 15:16:03 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 9 Sep 2009 07:16:03 -0600
Subject: [py-dev] jython and py.test code interaction
In-Reply-To: <20090907123837.GD15455@trillke.net>
References: <20090907123837.GD15455@trillke.net>
Message-ID: <693bc9ab0909090616p6a094118tedacba2f08727305@mail.gmail.com>

That's awesome.

Btw, can we do the same with IronPython? This would potentially help
testing stuff
on the pypy cli backend.

Cheers,
fijal

From fijall at gmail.com  Wed Sep  9 15:20:31 2009
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 9 Sep 2009 07:20:31 -0600
Subject: [py-dev] Feature proposition
Message-ID: <693bc9ab0909090620y5c9e9ebr46773373dd77575f@mail.gmail.com>

Hi.

I have a feature proposition that I'm missing every now and then from py.test.
Basically the idea is to be able to categorize tests with some
keywords, other than
just test name or part of it. Example:

http://paste.pocoo.org/show/138853/

And I would like to be able to run py.test -k +foo for example to run
TestFoo and TestOther
(or some other syntax).

The concrete use case is for example to mark all translation tests in
pypy test suite
(or part of it).

Cheers,
fijal

From holger at merlinux.eu  Wed Sep  9 21:36:00 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 9 Sep 2009 21:36:00 +0200
Subject: [py-dev] Feature proposition
In-Reply-To: <693bc9ab0909090620y5c9e9ebr46773373dd77575f@mail.gmail.com>
References: <693bc9ab0909090620y5c9e9ebr46773373dd77575f@mail.gmail.com>
Message-ID: <20090909193600.GJ15455@trillke.net>

Hi Maciej, 

> Hi.
>
> I have a feature proposition that I'm missing every now and then from
> py.test.
> Basically the idea is to be able to categorize tests with some
> keywords, other than
> just test name or part of it. Example:
>
> http://paste.pocoo.org/show/138853/
>
> And I would like to be able to run py.test -k +foo for example to run
> TestFoo and TestOther
> (or some other syntax).

makes sense i think.  maybe as an extension to per-function
"py.test.mark" decorators: 

    class TestSomething:
        mark = ['foo']

so that you can also do:

    class TestSomething:
        mark = ['xfail']

(and eventually mark = ['skip'] - seems logical to have a declarative
'skip' and maybe an imperative 'xfail')

> The concrete use case is for example to mark all translation tests in
> pypy test suite
> (or part of it).

sure.  There a few things that could be improved regarding keywords
and keyword selection, i think. 

best,
holger

From holger at merlinux.eu  Thu Sep 10 09:36:42 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 10 Sep 2009 09:36:42 +0200
Subject: [py-dev] execnet / GPL licensing
Message-ID: <20090910073641.GO15455@trillke.net>

Hi all, 

some of you know that i am considering licensing and general
funding issues recently, see here for the current licensing status:

    http://codespeak.net/py/trunk/faq.html#whygpl

However, i am considering releasing the execnet code under the 
GPL and the the rest under the LGPL.  This would mean that execnet 
can be used in free software but not in proprietary software 
without getting a different license.  

I think execnet presents a unique and valuable approach to glueing
Python interpreters and doing rapid deployment.  It can particularly
help with managing clouds of computers and I'd like to put more efforts
into improving and extending in this area.  And i'd like to tap into
getting dual-licensing revenue.  Which, on a side note, would flow back
to me and e.g. http://merlinux.eu/people.html so we all can continue to work
on great things.  

I am open to comments, arguments or bribes in the form 
of contracts for improvements :) 

cheers,

holger

Maybe also of interest to you (currently unreachable for me, though)
http://www.fsf.org/licensing/licenses/why-not-lgpl.html

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From py-dev at tolomea.com  Thu Sep 10 11:13:36 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Thu, 10 Sep 2009 19:13:36 +1000
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <20090910073641.GO15455@trillke.net>
References: <20090910073641.GO15455@trillke.net>
Message-ID: <f2d961430909100213o2038d914w9fc4568f4b60a608@mail.gmail.com>

Disclaimer: I appreciate that no one here is an IP lawyer and that any
discussion doesn't count as legal advice.

Just so I'm clear, as I understand it, if execnet were GPL then:
1: It would still be free (beer) for us to use and abuse any way we
choose internally as long as we aren't distributing anything related
to it.
2: Any modifications we made to it and distributed would be subject to
GPL or the alternate license.
3: Any code derived from it that we distributed would be subject to
GPL or the alternate license.
4: Any code that imports it that we distribute would be subject to GPL
or the alternate license.
5: Alternatively the LGPL picture would be the same except 4 wouldn't apply.

Please correct me if any of that is wrong.

Assuming it is all correct then I have no problems with 1-3.

Point 4 could be interesting, now the obvious scenario here would seem
to be that we write some tool that uses execnet and wish to sell it
and or make it publicly available. That scenario I have no problem
with it is part and parcel of building things out of open source
components. Indeed in that vein we have an internal tool that uses
execnet that we are looking at distributing as open source. As you
said "execnet presents a unique and valuable approach to glueing
Python interpreters and doing rapid deployment". And the system we
have built around it pushes that approach further and we would like to
share that.

But there is a more subtle example that concerns me, we are an
embedded hardware company and use python (and more recently py lib) in
our toolchain. Now imagine for the sake of discussion that we wanted
to share some of that infrastructure with one of our B2B clients.
Maybe we want to give them a drop of our testing infrastructure so
they can use it to test products that contain our chips. I'm not sure
if there is anything in the test infrastructure we would consider
proprietary in that manner, but there definitely is in other areas of
our infrastructure and py lib is becoming ever more popular
internally. Under LGPL distributing this code in that manner would
seem to be fine as we aren't distributing py lib or anything derived
from it. What would be the implications of GPL licensing? and how do
you see this scenario sitting with whatever alternate license you are
considering?

Gordon.

On Thu, Sep 10, 2009 at 5:36 PM, holger krekel <holger at merlinux.eu> wrote:
> Hi all,
>
> some of you know that i am considering licensing and general
> funding issues recently, see here for the current licensing status:
>
> ? ?http://codespeak.net/py/trunk/faq.html#whygpl
>
> However, i am considering releasing the execnet code under the
> GPL and the the rest under the LGPL. ?This would mean that execnet
> can be used in free software but not in proprietary software
> without getting a different license.
>
> I think execnet presents a unique and valuable approach to glueing
> Python interpreters and doing rapid deployment. ?It can particularly
> help with managing clouds of computers and I'd like to put more efforts
> into improving and extending in this area. ?And i'd like to tap into
> getting dual-licensing revenue. ?Which, on a side note, would flow back
> to me and e.g. http://merlinux.eu/people.html so we all can continue to work
> on great things.
>
> I am open to comments, arguments or bribes in the form
> of contracts for improvements :)
>
> cheers,
>
> holger
>
> Maybe also of interest to you (currently unreachable for me, though)
> http://www.fsf.org/licensing/licenses/why-not-lgpl.html
>
> --
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
>

From holger at merlinux.eu  Thu Sep 10 11:46:29 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 10 Sep 2009 11:46:29 +0200
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <f2d961430909100213o2038d914w9fc4568f4b60a608@mail.gmail.com>
References: <20090910073641.GO15455@trillke.net>
	<f2d961430909100213o2038d914w9fc4568f4b60a608@mail.gmail.com>
Message-ID: <20090910094629.GP15455@trillke.net>

On Thu, Sep 10, 2009 at 19:13 +1000, Gordon Wrigley wrote:
> Disclaimer: I appreciate that no one here is an IP lawyer and that any
> discussion doesn't count as legal advice.

sure. Please also don't take my statements as legal advise, btw :)
I know some good lawyers knowledgable in open source stuff, 
one of them having been a programmer before, so they can 
help if it comes to it :) 
 
> Just so I'm clear, as I understand it, if execnet were GPL then:
> 1: It would still be free (beer) for us to use and abuse any way we
> choose internally as long as we aren't distributing anything related
> to it.
> 2: Any modifications we made to it and distributed would be subject to
> GPL or the alternate license.
> 3: Any code derived from it that we distributed would be subject to
> GPL or the alternate license.
> 4: Any code that imports it that we distribute would be subject to GPL
> or the alternate license.
> 5: Alternatively the LGPL picture would be the same except 4 wouldn't apply.
> 
> Please correct me if any of that is wrong.

that's also my understanding. 

> Assuming it is all correct then I have no problems with 1-3.
> 
> Point 4 could be interesting, now the obvious scenario here would seem
> to be that we write some tool that uses execnet and wish to sell it
> and or make it publicly available. That scenario I have no problem
> with it is part and parcel of building things out of open source
> components. Indeed in that vein we have an internal tool that uses
> execnet that we are looking at distributing as open source. As you
> said "execnet presents a unique and valuable approach to glueing
> Python interpreters and doing rapid deployment". And the system we
> have built around it pushes that approach further and we would like to
> share that.

cool.  Curious i am :)
 
> But there is a more subtle example that concerns me, we are an
> embedded hardware company and use python (and more recently py lib) in
> our toolchain. Now imagine for the sake of discussion that we wanted
> to share some of that infrastructure with one of our B2B clients.
> Maybe we want to give them a drop of our testing infrastructure so
> they can use it to test products that contain our chips. I'm not sure
> if there is anything in the test infrastructure we would consider
> proprietary in that manner, but there definitely is in other areas of
> our infrastructure and py lib is becoming ever more popular
> internally. Under LGPL distributing this code in that manner would
> seem to be fine as we aren't distributing py lib or anything derived
> from it. 

Are you telling your clients to pre-install py lib? 

> What would be the implications of GPL licensing? 

You might need a separate agreement then.  

> and how do you see this scenario sitting with whatever
> alternate license you are considering?

All kinds of.  I am sure we can find some fitting scheme for
your case if GPL otherwise restricts your operations in
undesirable ways. 

thanks for your clear and thoughtful response, btw. 

best,
holger

> On Thu, Sep 10, 2009 at 5:36 PM, holger krekel <holger at merlinux.eu> wrote:
> > Hi all,
> >
> > some of you know that i am considering licensing and general
> > funding issues recently, see here for the current licensing status:
> >
> > ? ?http://codespeak.net/py/trunk/faq.html#whygpl
> >
> > However, i am considering releasing the execnet code under the
> > GPL and the the rest under the LGPL. ?This would mean that execnet
> > can be used in free software but not in proprietary software
> > without getting a different license.
> >
> > I think execnet presents a unique and valuable approach to glueing
> > Python interpreters and doing rapid deployment. ?It can particularly
> > help with managing clouds of computers and I'd like to put more efforts
> > into improving and extending in this area. ?And i'd like to tap into
> > getting dual-licensing revenue. ?Which, on a side note, would flow back
> > to me and e.g. http://merlinux.eu/people.html so we all can continue to work
> > on great things.
> >
> > I am open to comments, arguments or bribes in the form
> > of contracts for improvements :)
> >
> > cheers,
> >
> > holger
> >
> > Maybe also of interest to you (currently unreachable for me, though)
> > http://www.fsf.org/licensing/licenses/why-not-lgpl.html
> >
> > --
> > Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> > Python, PyPy, pytest contracting: http://merlinux.eu
> > _______________________________________________
> > py-dev mailing list
> > py-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/py-dev
> >
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From arigo at tunes.org  Mon Sep 14 13:51:17 2009
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 14 Sep 2009 13:51:17 +0200
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <20090910073641.GO15455@trillke.net>
References: <20090910073641.GO15455@trillke.net>
Message-ID: <20090914115117.GA14463@code0.codespeak.net>

Hi,

On Thu, Sep 10, 2009 at 09:36:42AM +0200, holger krekel wrote:
> some of you know that i am considering licensing and general
> funding issues recently, see here for the current licensing status:
> 
>     http://codespeak.net/py/trunk/faq.html#whygpl
> 
> However, i am considering releasing the execnet code under the 
> GPL and the the rest under the LGPL.

I don't know if it is possible to change the license without agreement
from all parties that already contributed code.  I would be rather
opposed, in principle, to seeing my own code (12% of execnet, according
to svn blame) being re-licensed LGPL or GPL.


A bientot,

Armin.

From holger at merlinux.eu  Wed Sep 16 15:07:58 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 16 Sep 2009 15:07:58 +0200
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <20090914115117.GA14463@code0.codespeak.net>
References: <20090910073641.GO15455@trillke.net>
	<20090914115117.GA14463@code0.codespeak.net>
Message-ID: <20090916130758.GF15455@trillke.net>

Hi Armin,

On Mon, Sep 14, 2009 at 13:51 +0200, Armin Rigo wrote:
> On Thu, Sep 10, 2009 at 09:36:42AM +0200, holger krekel wrote:
> > some of you know that i am considering licensing and general
> > funding issues recently, see here for the current licensing status:
> > 
> >     http://codespeak.net/py/trunk/faq.html#whygpl
> > 
> > However, i am considering releasing the execnet code under the 
> > GPL and the the rest under the LGPL.
> 
> I don't know if it is possible to change the license without agreement
> from all parties that already contributed code.  I would be rather
> opposed, in principle, to seeing my own code (12% of execnet, according
> to svn blame) being re-licensed LGPL or GPL.

my understanding: Your contributions - e.g. most of rsync.py -
would remain MIT licensed.  Execnet distributed as a whole
package could only be used under the GPL (if i GPL my parts
and future work), however.  Does that match your understanding? 

best,
holger

From py-dev at tolomea.com  Thu Sep 17 03:07:20 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Thu, 17 Sep 2009 11:07:20 +1000
Subject: [py-dev] py.test crash
Message-ID: <f2d961430909161807h25100349j7051acd935e71bb7@mail.gmail.com>

This happened, I have no idea why or even who's fault it might be so I
thought I'd post it here in case it means anything to anyone else. The
test that was running uses execnet, it also creates a couple of
additional threads.


tests/tcp_behavior/test_basic_listen.py:3: test_function FAILTraceback
(most recent call last)
  File "/usr/local/bin/py.test", line 8, in <module>
    load_entry_point('py==1.0.0', 'console_scripts', 'py.test')()
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/cmdline/pytest.py",
line 5, in main
    py.test.cmdline.main()
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/cmdline.py",
line 16, in main
    exitstatus = session.main()
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/session.py",
line 125, in main
    self.sessionfinishes(exitstatus=exitstatus)
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/session.py",
line 93, in sessionfinishes
    exitstatus=exitstatus,
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
line 139, in __call__
    return mc.execute(firstresult=self.firstresult)
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
line 23, in execute
    res = self.execute_method(currentmethod)
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
line 50, in execute_method
    return currentmethod(self, *self.args, **self.kwargs)
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/plugin/pytest_terminal.py",
line 259, in pytest_sessionfinish
    __call__.execute()
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
line 23, in execute
    res = self.execute_method(currentmethod)
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
line 53, in execute_method
    return currentmethod(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/plugin/pytest_execnetcleanup.py",
line 30, in pytest_sessionfinish
    gw.exit()
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/register.py",
line 73, in exit
    super(PopenCmdGateway, self).exit()
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/gateway.py",
line 354, in exit
    self._cleanup.unregister(self)
  File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/gateway.py",
line 45, in unregister
    del self._activegateways[gateway]
  File "/usr/lib/python2.6/weakref.py", line 240, in __delitem__
    del self.data[ref(key)]
KeyError: <weakref at 0x86c74dc; to 'SshGateway' at 0x8f76d8c>

gordonw at okum:~$ uname -a
Linux okum 2.6.28-15-generic #51-Ubuntu SMP Mon Aug 31 13:33:16 UTC
2009 i686 GNU/Linux
gordonw at okum:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0'

From holger at merlinux.eu  Thu Sep 17 10:21:50 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 17 Sep 2009 10:21:50 +0200
Subject: [py-dev] py.test crash
In-Reply-To: <f2d961430909161807h25100349j7051acd935e71bb7@mail.gmail.com>
References: <f2d961430909161807h25100349j7051acd935e71bb7@mail.gmail.com>
Message-ID: <20090917082150.GH15455@trillke.net>

Hi Gordon,

could you check that it happens also with 1.0.2? 

holger

On Thu, Sep 17, 2009 at 11:07 +1000, Gordon Wrigley wrote:
> This happened, I have no idea why or even who's fault it might be so I
> thought I'd post it here in case it means anything to anyone else. The
> test that was running uses execnet, it also creates a couple of
> additional threads.
> 
> 
> tests/tcp_behavior/test_basic_listen.py:3: test_function FAILTraceback
> (most recent call last)
>   File "/usr/local/bin/py.test", line 8, in <module>
>     load_entry_point('py==1.0.0', 'console_scripts', 'py.test')()
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/cmdline/pytest.py",
> line 5, in main
>     py.test.cmdline.main()
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/cmdline.py",
> line 16, in main
>     exitstatus = session.main()
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/session.py",
> line 125, in main
>     self.sessionfinishes(exitstatus=exitstatus)
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/session.py",
> line 93, in sessionfinishes
>     exitstatus=exitstatus,
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> line 139, in __call__
>     return mc.execute(firstresult=self.firstresult)
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> line 23, in execute
>     res = self.execute_method(currentmethod)
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> line 50, in execute_method
>     return currentmethod(self, *self.args, **self.kwargs)
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/plugin/pytest_terminal.py",
> line 259, in pytest_sessionfinish
>     __call__.execute()
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> line 23, in execute
>     res = self.execute_method(currentmethod)
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> line 53, in execute_method
>     return currentmethod(*self.args, **self.kwargs)
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/plugin/pytest_execnetcleanup.py",
> line 30, in pytest_sessionfinish
>     gw.exit()
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/register.py",
> line 73, in exit
>     super(PopenCmdGateway, self).exit()
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/gateway.py",
> line 354, in exit
>     self._cleanup.unregister(self)
>   File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/gateway.py",
> line 45, in unregister
>     del self._activegateways[gateway]
>   File "/usr/lib/python2.6/weakref.py", line 240, in __delitem__
>     del self.data[ref(key)]
> KeyError: <weakref at 0x86c74dc; to 'SshGateway' at 0x8f76d8c>
> 
> gordonw at okum:~$ uname -a
> Linux okum 2.6.28-15-generic #51-Ubuntu SMP Mon Aug 31 13:33:16 UTC
> 2009 i686 GNU/Linux
> gordonw at okum:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import py
> >>> py.version
> '1.0.0'
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From py-dev at tolomea.com  Thu Sep 17 10:38:14 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Thu, 17 Sep 2009 18:38:14 +1000
Subject: [py-dev] py.test crash
In-Reply-To: <20090917082150.GH15455@trillke.net>
References: <f2d961430909161807h25100349j7051acd935e71bb7@mail.gmail.com>
	<20090917082150.GH15455@trillke.net>
Message-ID: <f2d961430909170138l36a796eboad972b5e9190b7c7@mail.gmail.com>

sorry, I should have been clearer, to date it has only happened once,
I wasn't doing anything unusal at the time just running the test set
the same as always.

On Thu, Sep 17, 2009 at 6:21 PM, holger krekel <holger at merlinux.eu> wrote:
> Hi Gordon,
>
> could you check that it happens also with 1.0.2?
>
> holger
>
> On Thu, Sep 17, 2009 at 11:07 +1000, Gordon Wrigley wrote:
>> This happened, I have no idea why or even who's fault it might be so I
>> thought I'd post it here in case it means anything to anyone else. The
>> test that was running uses execnet, it also creates a couple of
>> additional threads.
>>
>>
>> tests/tcp_behavior/test_basic_listen.py:3: test_function FAILTraceback
>> (most recent call last)
>> ? File "/usr/local/bin/py.test", line 8, in <module>
>> ? ? load_entry_point('py==1.0.0', 'console_scripts', 'py.test')()
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/cmdline/pytest.py",
>> line 5, in main
>> ? ? py.test.cmdline.main()
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/cmdline.py",
>> line 16, in main
>> ? ? exitstatus = session.main()
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/session.py",
>> line 125, in main
>> ? ? self.sessionfinishes(exitstatus=exitstatus)
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/session.py",
>> line 93, in sessionfinishes
>> ? ? exitstatus=exitstatus,
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
>> line 139, in __call__
>> ? ? return mc.execute(firstresult=self.firstresult)
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
>> line 23, in execute
>> ? ? res = self.execute_method(currentmethod)
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
>> line 50, in execute_method
>> ? ? return currentmethod(self, *self.args, **self.kwargs)
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/plugin/pytest_terminal.py",
>> line 259, in pytest_sessionfinish
>> ? ? __call__.execute()
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
>> line 23, in execute
>> ? ? res = self.execute_method(currentmethod)
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
>> line 53, in execute_method
>> ? ? return currentmethod(*self.args, **self.kwargs)
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/plugin/pytest_execnetcleanup.py",
>> line 30, in pytest_sessionfinish
>> ? ? gw.exit()
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/register.py",
>> line 73, in exit
>> ? ? super(PopenCmdGateway, self).exit()
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/gateway.py",
>> line 354, in exit
>> ? ? self._cleanup.unregister(self)
>> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/gateway.py",
>> line 45, in unregister
>> ? ? del self._activegateways[gateway]
>> ? File "/usr/lib/python2.6/weakref.py", line 240, in __delitem__
>> ? ? del self.data[ref(key)]
>> KeyError: <weakref at 0x86c74dc; to 'SshGateway' at 0x8f76d8c>
>>
>> gordonw at okum:~$ uname -a
>> Linux okum 2.6.28-15-generic #51-Ubuntu SMP Mon Aug 31 13:33:16 UTC
>> 2009 i686 GNU/Linux
>> gordonw at okum:~$ python
>> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
>> [GCC 4.3.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import py
>> >>> py.version
>> '1.0.0'
>> _______________________________________________
>> py-dev mailing list
>> py-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/py-dev
>>
>
> --
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu
>

From holger at merlinux.eu  Thu Sep 17 11:18:34 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 17 Sep 2009 11:18:34 +0200
Subject: [py-dev] py.test crash
In-Reply-To: <f2d961430909170138l36a796eboad972b5e9190b7c7@mail.gmail.com>
References: <f2d961430909161807h25100349j7051acd935e71bb7@mail.gmail.com>
	<20090917082150.GH15455@trillke.net>
	<f2d961430909170138l36a796eboad972b5e9190b7c7@mail.gmail.com>
Message-ID: <20090917091834.GI15455@trillke.net>

Hi Gordon, 

ok, it looks like a race condition on exiting gateways. 
Kind of known.  And should work better with the next 
release due to the various cleanups.

holger

In trunk development i plan to do some more bits regarding
On Thu, Sep 17, 2009 at 18:38 +1000, Gordon Wrigley wrote:
> sorry, I should have been clearer, to date it has only happened once,
> I wasn't doing anything unusal at the time just running the test set
> the same as always.
> 
> On Thu, Sep 17, 2009 at 6:21 PM, holger krekel <holger at merlinux.eu> wrote:
> > Hi Gordon,
> >
> > could you check that it happens also with 1.0.2?
> >
> > holger
> >
> > On Thu, Sep 17, 2009 at 11:07 +1000, Gordon Wrigley wrote:
> >> This happened, I have no idea why or even who's fault it might be so I
> >> thought I'd post it here in case it means anything to anyone else. The
> >> test that was running uses execnet, it also creates a couple of
> >> additional threads.
> >>
> >>
> >> tests/tcp_behavior/test_basic_listen.py:3: test_function FAILTraceback
> >> (most recent call last)
> >> ? File "/usr/local/bin/py.test", line 8, in <module>
> >> ? ? load_entry_point('py==1.0.0', 'console_scripts', 'py.test')()
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/cmdline/pytest.py",
> >> line 5, in main
> >> ? ? py.test.cmdline.main()
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/cmdline.py",
> >> line 16, in main
> >> ? ? exitstatus = session.main()
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/session.py",
> >> line 125, in main
> >> ? ? self.sessionfinishes(exitstatus=exitstatus)
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/session.py",
> >> line 93, in sessionfinishes
> >> ? ? exitstatus=exitstatus,
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> >> line 139, in __call__
> >> ? ? return mc.execute(firstresult=self.firstresult)
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> >> line 23, in execute
> >> ? ? res = self.execute_method(currentmethod)
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> >> line 50, in execute_method
> >> ? ? return currentmethod(self, *self.args, **self.kwargs)
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/plugin/pytest_terminal.py",
> >> line 259, in pytest_sessionfinish
> >> ? ? __call__.execute()
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> >> line 23, in execute
> >> ? ? res = self.execute_method(currentmethod)
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/_com.py",
> >> line 53, in execute_method
> >> ? ? return currentmethod(*self.args, **self.kwargs)
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/test/plugin/pytest_execnetcleanup.py",
> >> line 30, in pytest_sessionfinish
> >> ? ? gw.exit()
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/register.py",
> >> line 73, in exit
> >> ? ? super(PopenCmdGateway, self).exit()
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/gateway.py",
> >> line 354, in exit
> >> ? ? self._cleanup.unregister(self)
> >> ? File "/usr/local/lib/python2.6/dist-packages/py-1.0.0-py2.6.egg/py/execnet/gateway.py",
> >> line 45, in unregister
> >> ? ? del self._activegateways[gateway]
> >> ? File "/usr/lib/python2.6/weakref.py", line 240, in __delitem__
> >> ? ? del self.data[ref(key)]
> >> KeyError: <weakref at 0x86c74dc; to 'SshGateway' at 0x8f76d8c>
> >>
> >> gordonw at okum:~$ uname -a
> >> Linux okum 2.6.28-15-generic #51-Ubuntu SMP Mon Aug 31 13:33:16 UTC
> >> 2009 i686 GNU/Linux
> >> gordonw at okum:~$ python
> >> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> >> [GCC 4.3.3] on linux2
> >> Type "help", "copyright", "credits" or "license" for more information.
> >> >>> import py
> >> >>> py.version
> >> '1.0.0'
> >> _______________________________________________
> >> py-dev mailing list
> >> py-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/py-dev
> >>
> >
> > --
> > Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> > Python, PyPy, pytest contracting: http://merlinux.eu
> >
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From arigo at tunes.org  Fri Sep 18 14:18:48 2009
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 18 Sep 2009 14:18:48 +0200
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <20090916130758.GF15455@trillke.net>
References: <20090910073641.GO15455@trillke.net>
	<20090914115117.GA14463@code0.codespeak.net>
	<20090916130758.GF15455@trillke.net>
Message-ID: <20090918121848.GA26602@code0.codespeak.net>

Hi Holger,

On Wed, Sep 16, 2009 at 03:07:58PM +0200, holger krekel wrote:
> my understanding: Your contributions - e.g. most of rsync.py -
> would remain MIT licensed.  Execnet distributed as a whole
> package could only be used under the GPL (if i GPL my parts
> and future work), however.  Does that match your understanding? 

I'm afraid not; I'm thinking notably about the reorganization of the
execnet code I did long ago (around r13661), which still leaves e.g. 109
lines from me in channel.py (30%).  I'm not really going to enforce my
position in any way because I'm happy to stick with old versions of
execnet for my own usage, but I'm just saying that I would informally
consider this as *slightly* unfair reuse of my part of the work.  Well,
you might want to investigate the position of other people too (e.g.
cfbolz and jan, who also have lines in channel.py).


A bientot,

Armin.

From robert.kern at gmail.com  Fri Sep 18 19:05:32 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 18 Sep 2009 12:05:32 -0500
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <20090918121848.GA26602@code0.codespeak.net>
References: <20090910073641.GO15455@trillke.net>	<20090914115117.GA14463@code0.codespeak.net>	<20090916130758.GF15455@trillke.net>
	<20090918121848.GA26602@code0.codespeak.net>
Message-ID: <h90eks$7sj$1@ger.gmane.org>

On 2009-09-18 07:18 AM, Armin Rigo wrote:
> Hi Holger,
>
> On Wed, Sep 16, 2009 at 03:07:58PM +0200, holger krekel wrote:
>> my understanding: Your contributions - e.g. most of rsync.py -
>> would remain MIT licensed.  Execnet distributed as a whole
>> package could only be used under the GPL (if i GPL my parts
>> and future work), however.  Does that match your understanding?
>
> I'm afraid not; I'm thinking notably about the reorganization of the
> execnet code I did long ago (around r13661), which still leaves e.g. 109
> lines from me in channel.py (30%).  I'm not really going to enforce my
> position in any way because I'm happy to stick with old versions of
> execnet for my own usage, but I'm just saying that I would informally
> consider this as *slightly* unfair reuse of my part of the work.  Well,
> you might want to investigate the position of other people too (e.g.
> cfbolz and jan, who also have lines in channel.py).

MIT licensed code may be combined with GPLed code to make a combined work that 
is distributed under the GPL. That's one of the things that you gave permission 
for people to do when you licensed your code under the MIT license. Your MIT 
license is still attached to your code, and Holger needs to make sure that he 
maintains the correct attribution and MIT license statement for your and others' 
code. Strictly speaking, he is not relicensing your code, just incorporating it 
according to the very permissive terms that you granted him.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From holger at merlinux.eu  Mon Sep 21 17:58:10 2009
From: holger at merlinux.eu (holger krekel)
Date: Mon, 21 Sep 2009 17:58:10 +0200
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <h90eks$7sj$1@ger.gmane.org>
References: <20090910073641.GO15455@trillke.net>
	<20090914115117.GA14463@code0.codespeak.net>
	<20090916130758.GF15455@trillke.net>
	<20090918121848.GA26602@code0.codespeak.net>
	<h90eks$7sj$1@ger.gmane.org>
Message-ID: <20090921155810.GM15455@trillke.net>

Hi Robert, Armin, 

On Fri, Sep 18, 2009 at 12:05 -0500, Robert Kern wrote:
> On 2009-09-18 07:18 AM, Armin Rigo wrote:
> > Hi Holger,
> >
> > On Wed, Sep 16, 2009 at 03:07:58PM +0200, holger krekel wrote:
> >> my understanding: Your contributions - e.g. most of rsync.py -
> >> would remain MIT licensed.  Execnet distributed as a whole
> >> package could only be used under the GPL (if i GPL my parts
> >> and future work), however.  Does that match your understanding?
> >
> > I'm afraid not; I'm thinking notably about the reorganization of the
> > execnet code I did long ago (around r13661), which still leaves e.g. 109
> > lines from me in channel.py (30%).  I'm not really going to enforce my
> > position in any way because I'm happy to stick with old versions of
> > execnet for my own usage, but I'm just saying that I would informally
> > consider this as *slightly* unfair reuse of my part of the work.  Well,
> > you might want to investigate the position of other people too (e.g.
> > cfbolz and jan, who also have lines in channel.py).
> 
> MIT licensed code may be combined with GPLed code to make a combined work that 
> is distributed under the GPL. That's one of the things that you gave permission 
> for people to do when you licensed your code under the MIT license. Your MIT 
> license is still attached to your code, and Holger needs to make sure that he 
> maintains the correct attribution and MIT license statement for your and others' 
> code. Strictly speaking, he is not relicensing your code, just incorporating it 
> according to the very permissive terms that you granted him.

right, thanks for the clarification.  

However, i can see some slight unfairness because i am breaking the 
possible assumtion that not only my past but also my future 
work would be published without any restrictions. 

best,
holger

From arigo at tunes.org  Wed Sep 23 11:08:49 2009
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 23 Sep 2009 11:08:49 +0200
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <h90eks$7sj$1@ger.gmane.org>
References: <20090910073641.GO15455@trillke.net>
	<20090914115117.GA14463@code0.codespeak.net>
	<20090916130758.GF15455@trillke.net>
	<20090918121848.GA26602@code0.codespeak.net>
	<h90eks$7sj$1@ger.gmane.org>
Message-ID: <20090923090849.GA32347@code0.codespeak.net>

Hi,

On Fri, Sep 18, 2009 at 12:05:32PM -0500, Robert Kern wrote:
> MIT licensed code may be combined with GPLed code to make a combined work that 
> is distributed under the GPL.

Absolutely right.  Thanks Robert for the clarification -- I was wrong,
of course.


A bientot,

Armin.

From holger at merlinux.eu  Wed Sep 23 19:06:44 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 23 Sep 2009 19:06:44 +0200
Subject: [py-dev] execnet / GPL licensing
In-Reply-To: <e3e180dc0909160124t6ef67699i82b3acfcbe9ca500@mail.gmail.com>
References: <20090910073641.GO15455@trillke.net>
	<e3e180dc0909160124t6ef67699i82b3acfcbe9ca500@mail.gmail.com>
Message-ID: <20090923170644.GS15455@trillke.net>

Hi Massa, 

On Wed, Sep 16, 2009 at 10:24 +0200, Massa, Harald Armin wrote:
> Just to add a viewpoint ...
> 
> I stumbled upon this article:
> 
> http://www.stevestreeting.com/2009/09/15/my-evolving-view-of-open-source-licenses/

huh, this and the comments add a lot of viewpoints, not just one.
 
> BeSD wishes,

:) thanks to you and for all the feedback and advise. 

holger

> Harald
> 
> -- 
> GHUM Harald Massa
> persuadere et programmare
> Harald Armin Massa
> Spielberger Stra?e 49
> 70435 Stuttgart
> 0173/9409607
> no fx, no carrier pigeon
> -
> %s is too gigantic of an industry to bend to the whims of reality
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 

From digitalxero at gmail.com  Sat Sep 26 03:56:28 2009
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Fri, 25 Sep 2009 19:56:28 -0600
Subject: [py-dev] Remote Distribution
Message-ID: <e9764b730909251856r58a86f3fq10c1f7a6c8be57d0@mail.gmail.com>

I know py.test has distributed testing already but I have been using
Testswarm (a distributed javascript testing framework) for a while now
and thought the idea would fit nicely with py.test. This is just me
kind of brain storming the idea, so feel free to ignore me if you have
no interest in this idea.

Ok basic premise is:
    Central server for anyone to schedule test runs on
    Clients connect to the central server and run tests as they come
in for any and all apps they meet the requirements for

Goals:
    Eliminate the need for everyone to establish their own distributed
testing network by establishing a remote distributed network where
clients connect and will run tests from any app that is registered on
the server.


Brainstorm of requirements:
    Server - Needs a way for applications / libraries to register
    Server - Apps need a way to set dependencies, and a way to specify
version for that dependency
    Server - Apps need a way to set the environments to test on (os
version and python version)
    Server - Apps need a way to schedule test runs and upload the code
to test against
    Server - Need a UI to display a test metric like
http://testswarm.com/user/jquery/
    Server - Needs a method of tracking available clients, and their
environments
    Server - Needs a way to send tests to every available client that
meets the environment specs for each app.
        Needs to keep track of the test run id for results
    Server - Needs to accept results back
        Needs to identify the run and clients environment id to make
sure we can mark an environment finished for the run
    Server - Needs a fault catcher to reschedule a test run /
environment if a remote client disconnects before it finished
    Server - Needs a way of removing code once all tests have been
executed for a particular run, or after X days
    Server - Needs to ensure remote execution of apps is not possible
for security
    Client - Needs a way to connect to the central server and keep it
notified that it is still alive
    Client - Needs a way to receive Tests
    Client - Needs a way to report results back


Brainstorm implementation details:
    modified? py.execnet for the server test dispatcher
    modified? py.execnet for the client test receiver
    Client should be command line only so it can be run as a service
    Web app for managing Apps and result display
        REsT API for scheduling test runs and uploading the required code?
        "py.test schedule" command?

From Ronny.Pfannschmidt at gmx.de  Sat Sep 26 13:34:29 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Sat, 26 Sep 2009 13:34:29 +0200
Subject: [py-dev] how to test fallbacks for missing modules?
Message-ID: <1253964869.17339.18.camel@localhost>

hi,

i wonder what would be a good way to test the handling of unimportable
modules or fallbacks for missing modules.

i don't have any initial idea on how to get started with that.

Regards Ronny


From Ronny.Pfannschmidt at gmx.de  Sat Sep 26 13:53:03 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Sat, 26 Sep 2009 13:53:03 +0200
Subject: [py-dev] lack of gui/async testing examples/documentation
Message-ID: <1253965983.17339.21.camel@localhost>

Hi,

since im just right now in need to test async apps (a dbus service, and
a gtk gui app with concurrency) i wanted to remind of the notorious lack
of examples to figure how to do async testing propperly.

Regards Ronny


From holger at merlinux.eu  Sat Sep 26 17:02:59 2009
From: holger at merlinux.eu (holger krekel)
Date: Sat, 26 Sep 2009 17:02:59 +0200
Subject: [py-dev] how to test fallbacks for missing modules?
In-Reply-To: <1253964869.17339.18.camel@localhost>
References: <1253964869.17339.18.camel@localhost>
Message-ID: <20090926150258.GE15455@trillke.net>

Hi Ronny,

> i wonder what would be a good way to test the handling of unimportable
> modules or fallbacks for missing modules.
> i don't have any initial idea on how to get started with that.

what i usually do is to provide the to-be-unimportable module 
but put e.g. "raise ImportError()" into it.  Using the 'tmpdir' 
and 'monkeypatch' funcargs makes it easy to do this, e.g.:

    def test_unimportable(tmpdir, monkeypatch):
        tmpdir.join("docutils.py").write("raise ImportError()")
        monkeypatch.setattr(sys, 'path', [str(tmpdir)] + sys.path)

        ..call-and-assert..


hth,
holger

From Ronny.Pfannschmidt at gmx.de  Sat Sep 26 18:05:31 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Sat, 26 Sep 2009 18:05:31 +0200
Subject: [py-dev] how to test fallbacks for missing modules?
In-Reply-To: <20090926150258.GE15455@trillke.net>
References: <1253964869.17339.18.camel@localhost>
	<20090926150258.GE15455@trillke.net>
Message-ID: <1253981131.17339.24.camel@localhost>

On Sat, 2009-09-26 at 17:02 +0200, holger krekel wrote:
> Hi Ronny,
> 
> > i wonder what would be a good way to test the handling of unimportable
> > modules or fallbacks for missing modules.
> > i don't have any initial idea on how to get started with that.
> 
> what i usually do is to provide the to-be-unimportable module 
> but put e.g. "raise ImportError()" into it.  Using the 'tmpdir' 
> and 'monkeypatch' funcargs makes it easy to do this, e.g.:
> 
>     def test_unimportable(tmpdir, monkeypatch):
>         tmpdir.join("docutils.py").write("raise ImportError()")
>         monkeypatch.setattr(sys, 'path', [str(tmpdir)] + sys.path)
> 
>         ..call-and-assert..

unfortunately that doesnt help with modules that are already imported
unless of course i use something like propper process based sandboxing
for all tests (and i'd like to avoid that)




From holger at merlinux.eu  Sat Sep 26 18:19:56 2009
From: holger at merlinux.eu (holger krekel)
Date: Sat, 26 Sep 2009 18:19:56 +0200
Subject: [py-dev] how to test fallbacks for missing modules?
In-Reply-To: <1253981131.17339.24.camel@localhost>
References: <1253964869.17339.18.camel@localhost>
	<20090926150258.GE15455@trillke.net>
	<1253981131.17339.24.camel@localhost>
Message-ID: <20090926161956.GG15455@trillke.net>

On Sat, Sep 26, 2009 at 18:05 +0200, Ronny Pfannschmidt wrote:
> On Sat, 2009-09-26 at 17:02 +0200, holger krekel wrote:
> > Hi Ronny,
> > 
> > > i wonder what would be a good way to test the handling of unimportable
> > > modules or fallbacks for missing modules.
> > > i don't have any initial idea on how to get started with that.
> > 
> > what i usually do is to provide the to-be-unimportable module 
> > but put e.g. "raise ImportError()" into it.  Using the 'tmpdir' 
> > and 'monkeypatch' funcargs makes it easy to do this, e.g.:
> > 
> >     def test_unimportable(tmpdir, monkeypatch):
> >         tmpdir.join("docutils.py").write("raise ImportError()")
> >         monkeypatch.setattr(sys, 'path', [str(tmpdir)] + sys.path)
> > 
> >         ..call-and-assert..
> 
> unfortunately that doesnt help with modules that are already imported
> unless of course i use something like propper process based sandboxing
> for all tests (and i'd like to avoid that)

yes, either requires running things in a new process or hacking 
'sys.modules' as well - which you could all do through a nice funcarg. 

holger

From Ronny.Pfannschmidt at gmx.de  Sat Sep 26 18:36:11 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Sat, 26 Sep 2009 18:36:11 +0200
Subject: [py-dev] how to test fallbacks for missing modules?
In-Reply-To: <20090926161956.GG15455@trillke.net>
References: <1253964869.17339.18.camel@localhost>
	<20090926150258.GE15455@trillke.net>
	<1253981131.17339.24.camel@localhost>
	<20090926161956.GG15455@trillke.net>
Message-ID: <1253982971.17339.27.camel@localhost>

On Sat, 2009-09-26 at 18:19 +0200, holger krekel wrote:
> On Sat, Sep 26, 2009 at 18:05 +0200, Ronny Pfannschmidt wrote:
> > On Sat, 2009-09-26 at 17:02 +0200, holger krekel wrote:
> > > Hi Ronny,
> > > 
> > > > i wonder what would be a good way to test the handling of unimportable
> > > > modules or fallbacks for missing modules.
> > > > i don't have any initial idea on how to get started with that.
> > > 
> > > what i usually do is to provide the to-be-unimportable module 
> > > but put e.g. "raise ImportError()" into it.  Using the 'tmpdir' 
> > > and 'monkeypatch' funcargs makes it easy to do this, e.g.:
> > > 
> > >     def test_unimportable(tmpdir, monkeypatch):
> > >         tmpdir.join("docutils.py").write("raise ImportError()")
> > >         monkeypatch.setattr(sys, 'path', [str(tmpdir)] + sys.path)
> > > 
> > >         ..call-and-assert..
> > 
> > unfortunately that doesnt help with modules that are already imported
> > unless of course i use something like propper process based sandboxing
> > for all tests (and i'd like to avoid that)
> 
> yes, either requires running things in a new process or hacking 
> 'sys.modules' as well - which you could all do through a nice funcarg. 
> 
you are right, a funcarg is on the wayfor simple cases,
and im starting to see execnet as good way to wire up the more complex
cases



From holger at merlinux.eu  Sun Sep 27 10:37:03 2009
From: holger at merlinux.eu (holger krekel)
Date: Sun, 27 Sep 2009 10:37:03 +0200
Subject: [py-dev] Remote Distribution
In-Reply-To: <e9764b730909251856r58a86f3fq10c1f7a6c8be57d0@mail.gmail.com>
References: <e9764b730909251856r58a86f3fq10c1f7a6c8be57d0@mail.gmail.com>
Message-ID: <20090927083703.GJ15455@trillke.net>

Hi Dj,

On Fri, Sep 25, 2009 at 19:56 -0600, Dj Gilcrease wrote:
> I know py.test has distributed testing already but I have been using
> Testswarm (a distributed javascript testing framework) for a while now
> and thought the idea would fit nicely with py.test. This is just me
> kind of brain storming the idea, so feel free to ignore me if you have
> no interest in this idea.

that's just fine!  I am using the opportunity to add some brainstorming
and fiction on a next-generation execnet/py.test :) 

> Ok basic premise is:
>     Central server for anyone to schedule test runs on
>     Clients connect to the central server and run tests as they come
> in for any and all apps they meet the requirements for
>
> 
> Goals:
>     Eliminate the need for everyone to establish their own distributed
> testing network by establishing a remote distributed network where
> clients connect and will run tests from any app that is registered on
> the server.

i like the goal!

> Brainstorm of requirements:
>     Server - Needs a way for applications / libraries to register
>     Server - Apps need a way to set dependencies, and a way to specify
> version for that dependency
>     Server - Apps need a way to set the environments to test on (os
> version and python version)
>     Server - Apps need a way to schedule test runs and upload the code
> to test against
>     Server - Need a UI to display a test metric like
> http://testswarm.com/user/jquery/
>     Server - Needs a method of tracking available clients, and their
> environments
>     Server - Needs a way to send tests to every available client that
> meets the environment specs for each app.
>         Needs to keep track of the test run id for results
>     Server - Needs to accept results back
>         Needs to identify the run and clients environment id to make
> sure we can mark an environment finished for the run
>     Server - Needs a fault catcher to reschedule a test run /
> environment if a remote client disconnects before it finished
>     Server - Needs a way of removing code once all tests have been
> executed for a particular run, or after X days
>     Server - Needs to ensure remote execution of apps is not possible
> for security
>     Client - Needs a way to connect to the central server and keep it
> notified that it is still alive
>     Client - Needs a way to receive Tests
>     Client - Needs a way to report results back

makes sense.  I'd like to allow to configure some aspects of 
remote environments, i.e. set up virtualenvs etc. See below. 

> Brainstorm implementation details:
>     modified? py.execnet for the server test dispatcher
>     modified? py.execnet for the client test receiver
>     Client should be command line only so it can be run as a service
>     Web app for managing Apps and result display
>         REsT API for scheduling test runs and uploading the required code?
>         "py.test schedule" command?

I am currently pondering imperative schemes of setting up execnet-style
networks.  Triggered by your post i eventually wrote up some bits in
my blog, see http://tinyurl.com/yco9aua 

Applied to testing distribution, here is what i would like to have for 
remote testing conceptually: 

    py.test --tx platform=darwin//packages=django1.1,nose,...

- would *discover* a darwin host from the above execnet network
- create/use a fitting virtualenv with the given packages/versions
- run tests in a new python process in that virtualenv 
- log results back to the invoking console 

A rather decentral system so far with no reporting.  Something like 

    py.test --runweb codespeak.net:9999

could dynamically fire up a web server at the specified 
host which then receives test results and it could also fire up 
a local browser window allowing to interact with the test run. 

The network so far is rather isolated.  To build larger 
test networks there needs to be a way to "connect" two 
execnets.  Maybe to make it publically available a 
command line like this would be nice: 

    execnet publish mynet https://testrun.org/012983102938123

so that from another place people could issue 

    execnet subscribe https://testrun.org/012983102938123

and the two execnets would be connected (whatever that
means exactly). 

    execnet list 

could list my local execnet's hosts and links to other execnets. 
So one could uniformly work with a multitude of dynamic 
user-contributed execnets at the fingertip of 
command line tools.

i am happy about any feedback and already grateful for
the opportunity to clarify my thoughts and plans :) 

cheers,
holger

From digitalxero at gmail.com  Sun Sep 27 11:45:09 2009
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Sun, 27 Sep 2009 03:45:09 -0600
Subject: [py-dev] Remote Distribution
In-Reply-To: <20090927083703.GJ15455@trillke.net>
References: <e9764b730909251856r58a86f3fq10c1f7a6c8be57d0@mail.gmail.com>
	<20090927083703.GJ15455@trillke.net>
Message-ID: <e9764b730909270245q60b55754v11299fe67a6c870@mail.gmail.com>

On Sun, Sep 27, 2009 at 2:37 AM, holger krekel <holger at merlinux.eu> wrote:
> makes sense. ?I'd like to allow to configure some aspects of
> remote environments, i.e. set up virtualenvs etc. See below.

I like the idea of using virtualenv as well

> Applied to testing distribution, here is what i would like to have for
> remote testing conceptually:
>
> ? ?py.test --tx platform=darwin//packages=django1.1,nose,...
>
> - would *discover* a darwin host from the above execnet network
> - create/use a fitting virtualenv with the given packages/versions
> - run tests in a new python process in that virtualenv
> - log results back to the invoking console

maybe use a pip requirements file instead of specifying each package
individually. If I remember correctly pip is zero-install as well.
Also it would be nice to have a platforms keyword so you can specify
multiple platforms to run the test on without needing to run the
command multiple times

>
> A rather decentral system so far with no reporting. ?Something like
>
> ? ?py.test --runweb codespeak.net:9999
>
> could dynamically fire up a web server at the specified
> host which then receives test results and it could also fire up
> a local browser window allowing to interact with the test run.

I like this and it eliminates the need for py.test to get out of scope
by having and maintaining central web server where test results are
collected and leaves it more to plugin creators to create plugins that
will report back to a specified site if they want to have their one

>
> The network so far is rather isolated. ?To build larger
> test networks there needs to be a way to "connect" two
> execnets. ?Maybe to make it publically available a
> command line like this would be nice:
>
> ? ?execnet publish mynet https://testrun.org/012983102938123
>
> so that from another place people could issue
>
> ? ?execnet subscribe https://testrun.org/012983102938123
>
> and the two execnets would be connected (whatever that
> means exactly).
>
> ? ?execnet list
>
> could list my local execnet's hosts and links to other execnets.
> So one could uniformly work with a multitude of dynamic
> user-contributed execnets at the fingertip of
> command line tools.

I like this as well, just need an unsubscribe option, and a
potentially a subscribe all option that would just subscribe to all
published networks

From issues-noreply at bitbucket.org  Sun Sep 27 19:52:03 2009
From: issues-noreply at bitbucket.org (issues-noreply at bitbucket.org)
Date: Sun, 27 Sep 2009 17:52:03 -0000
Subject: [py-dev] New issue 49 in py-trunk: figleaf plugin cannot be used
Message-ID: <59244167b5f4acd997b262568d34bfde@bitbucket.org>

New issue 49: figleaf plugin cannot be used
http://bitbucket.org/hpk42/py-trunk/issue/49/figleaf-plugin-cannot-be-used

Anonymous on Sun, 27 Sep 2009 19:52:02 +0200:

Description:
  py 1.02, python 2.6, OS winxp 32 xp3

I installed py using easy_install

figleaf plugin cannot be used

tried py.test -F, no such option

tried py.test -p figleaf, could not import plugin 'pytest_figleaf', reason: "could not import 'figleaf.annotate_html' ""




-- 
This is an issue notification from bitbucket.org.
You are receiving this either because you are the
owner of the issue, or you are following the issue.

From issues-noreply at bitbucket.org  Mon Sep 28 19:49:42 2009
From: issues-noreply at bitbucket.org (issues-noreply at bitbucket.org)
Date: Mon, 28 Sep 2009 17:49:42 -0000
Subject: [py-dev] New issue 50 in py-trunk: cached_setup needs to cache
	correctly
Message-ID: <ceb9cdea833ce22cb159f0f2f455d079@bitbucket.org>

New issue 50: cached_setup needs to cache correctly
http://bitbucket.org/hpk42/py-trunk/issue/50/cached_setup-needs-to-cache-correctly

holger krekel / hpk42 on Mon, 28 Sep 2009 19:49:42 +0200:

Description:
  Ronny pointed out that cached_setup does not work how one might assume: 
{{{
#!python
def pytest_funcarg__one(request):
      return request.cached_setup(lambda: 42)
def pytest_funcarg__two(request):
      return request.cached_setup(lambda: 17)
def test_hello(one, two):
     # one and two will both be 42!
}}}

this is because the 'request' object is re-used across all function argument setups and the funcarg name is not stored on it.  Needs to be better document or fixed - likely adding 'funcargname' as a key is the best approach even if it complicates the implemnetation a bit. 

Responsible:
  hpk42
-- 
This is an issue notification from bitbucket.org.
You are receiving this either because you are the
owner of the issue, or you are following the issue.

From issues-noreply at bitbucket.org  Mon Sep 28 19:55:20 2009
From: issues-noreply at bitbucket.org (issues-noreply at bitbucket.org)
Date: Mon, 28 Sep 2009 17:55:20 -0000
Subject: [py-dev] New issue 51 in py-trunk: there should be a way to have
 tests depend on the passing of other tests
Message-ID: <e9069c17550ac4e0c6cece69ab6ae58f@bitbucket.org>

New issue 51: there should be a way to have tests depend on the passing of other tests
http://bitbucket.org/hpk42/py-trunk/issue/51/there-should-be-a-way-to-have-tests-depend-on-the-passing-of-other

RonnyPfannschmidt on Mon, 28 Sep 2009 19:55:20 +0200:

Description:
  a good example would be something like 
the execnet serializer tests 
testing the serializer on different python versions needing gateways to work

Responsible:
  hpk42
-- 
This is an issue notification from bitbucket.org.
You are receiving this either because you are the
owner of the issue, or you are following the issue.

From Ronny.Pfannschmidt at gmx.de  Tue Sep 29 00:33:00 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Tue, 29 Sep 2009 00:33:00 +0200
Subject: [py-dev] proposal: chaining pytest_generate_tests calls
Message-ID: <1254177180.16914.42.camel@localhost>

Hi,

after some refactoring in the tests for the new serializer and some chat
with Benjamin it got appearant that some form of chaining the
pytest_generate_tests is helpfull and necessart.

Since there are various details lurking around,
this is a initial try to collect the needs for propperly chaining
generate_tests calls.
I will handle each aspect in a separate section.
For each aspect i will add all items i currenly consider
intresting/usefull, that might be incomplete and/or more than
reasonable.

Necessary Scopes
----------------

conftest/plugin
    Required to allow comfortable whole-project generators
package/module
    the same goes for sub-groups of the tests
functions
    doing special cases at the module level hook tends to be a mess
    see the current serializer tests for reasons


Additional dimensions of variation that could proof usefull
------------------------------------------------------------

funcargs
  choose test generators based on the arguments a function explicitly
takes, 
  implicitly used funcargs via request.getfuncargvalue cant be
considered there
* marks
  using decorators to add values and the way to add them to the call


Chaining order
--------------

conftest plugin -> package -> subpackage -> module -> function


building the test id's
-------------------------

for just chaining the tests one can simply append to a tuple of id's for
each successive add_call invokation per hook

if variation on funcargs is considered one could show those like
additional keyword parameters

funcarg generators shouldn't support chaining themself, just use the
closest matching test generator

dealing with the param argument of addcall
-------------------------------------------

* provide the value of the most close scope to request objecs 
* previous values of params could be accessible via linking to the
precedessor of the current call or by providing a list ordered by scope
* for funcarg variation it might be helpfull to supple the funcarg
function with the param the funcarg related generator created


dealing with the funcargs argument of addcall
---------------------------------------------

* values from more close scope should override vales for less close
scope
* warnings could be printed for name-clashes/overrides


how to actually combine the generators
--------------------------------------

Note: early never released versions of the generate_tests feature
provided implied combinations of all matching test generators, however
that was fragile and got dumped for 'match most closest hook'

i thenk we should prefer a explicit combination where for each addcall
of an outer scope, the complete generate_tests function of the inner
scope has to be run

this results in a clean tree of invocations and better controll over the
actual combinations


regards Ronny Pfannschmidt


From holger at merlinux.eu  Tue Sep 29 20:46:17 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 29 Sep 2009 20:46:17 +0200
Subject: [py-dev] proposal: chaining pytest_generate_tests calls
In-Reply-To: <1254177180.16914.42.camel@localhost>
References: <1254177180.16914.42.camel@localhost>
Message-ID: <20090929184617.GW15455@trillke.net>

Hi Ronny, Benjamin,  all, 

On Tue, Sep 29, 2009 at 00:33 +0200, Ronny Pfannschmidt wrote:
> Hi,
> 
> after some refactoring in the tests for the new serializer and some chat
> with Benjamin it got appearant that some form of chaining the
> pytest_generate_tests is helpfull and necessart.

I am missing concise code examples to showcase what is missing and to
understand concretely the issues and what you propose.  From what
i understand - for a function like this: 

    def test_function(arg1, arg2):
        ... 

it's currently not easily possible to have a funcarg generator 
for arg1 and arg2 independently.  I believe that this could be 
helped through a new attribute on the request object, like: 

    for call in request.calls: 
        # use call.id, call.param, call.funcargs 
        request.addcall(...) 

If both the 'arg1' and 'arg2' generator are implenented 
this way they would collaborate to create m*n calls of
the function. 

Another bit maybe is how to explicitely set funcarg factories, something
like e.g.:

    @py.test.mark.genfuncarg(arg1=arg1factory, arg2=arg2factory)
    def test_function(arg1, arg2):
        ... 

This can be played with today through a custom pytest_generate_tests hook.  
I think the factories should be called with the request object.  
Experimentation and real-life usage is neccessary to settle details. 

It'd be great if we settled best/proven practises for 
parametrized/generated tests and did a plugin with docs on it.  
Any other comments?  Or am i missing your points entirely? 

cheers,
holger

> Since there are various details lurking around,
> this is a initial try to collect the needs for propperly chaining
> generate_tests calls.
> I will handle each aspect in a separate section.
> For each aspect i will add all items i currenly consider
> intresting/usefull, that might be incomplete and/or more than
> reasonable.
> 
> Necessary Scopes
> ----------------
> 
> conftest/plugin
>     Required to allow comfortable whole-project generators
> package/module
>     the same goes for sub-groups of the tests
> functions
>     doing special cases at the module level hook tends to be a mess
>     see the current serializer tests for reasons

note that you can also have funcargs at class level. 

> Additional dimensions of variation that could proof usefull
> ------------------------------------------------------------
> 
> funcargs
>   choose test generators based on the arguments a function explicitly
> takes, 
>   implicitly used funcargs via request.getfuncargvalue cant be
> considered there
> * marks
>   using decorators to add values and the way to add them to the call
> 
> 
> Chaining order
> --------------
> 
> conftest plugin -> package -> subpackage -> module -> function
> 
> 
> building the test id's
> -------------------------
> 
> for just chaining the tests one can simply append to a tuple of id's for
> each successive add_call invokation per hook
> 
> if variation on funcargs is considered one could show those like
> additional keyword parameters
> 
> funcarg generators shouldn't support chaining themself, just use the
> closest matching test generator
> 
> dealing with the param argument of addcall
> -------------------------------------------
> 
> * provide the value of the most close scope to request objecs 
> * previous values of params could be accessible via linking to the
> precedessor of the current call or by providing a list ordered by scope
> * for funcarg variation it might be helpfull to supple the funcarg
> function with the param the funcarg related generator created
> 
> 
> dealing with the funcargs argument of addcall
> ---------------------------------------------
> 
> * values from more close scope should override vales for less close
> scope
> * warnings could be printed for name-clashes/overrides
> 
> 
> how to actually combine the generators
> --------------------------------------
> 
> Note: early never released versions of the generate_tests feature
> provided implied combinations of all matching test generators, however
> that was fragile and got dumped for 'match most closest hook'
> 
> i thenk we should prefer a explicit combination where for each addcall
> of an outer scope, the complete generate_tests function of the inner
> scope has to be run
> 
> this results in a clean tree of invocations and better controll over the
> actual combinations
> 
> 
> regards Ronny Pfannschmidt
> 
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu