From eli at water.ca.gov Thu Jan 19 05:55:56 2012 From: eli at water.ca.gov (Ateljevich, Eli) Date: Wed, 18 Jan 2012 20:55:56 -0800 Subject: [py-dev] xdist and thread-safe resource counting Message-ID: <67CC54752B3F8D43ACD7E3AD455162630B5714DAA9@mrsbmapp20306.ad.water.ca.gov> I have a question about managing resources in a threadsafe way across xdist -n. My group is using py.test as a high-level driver for testing an mpi-based numerical code. Many of our system-level tests wrap a system call to mpirun then postprocess results. I have a decorator for the tests that hints at the number of processors needed (usually something like 1,2,8). I would like to launch as much as I can at once given the available processors. For instance, if 16 processors are available there is no reason I couldn't be doing a 12 and a 4 processor test. I was thinking of using xdist with some modest number of processors representing the maximum number of concurrent tests. The xdist test processors would launch mpi jobs when enough processors become available to satisfy the np hint for that test. This would be managed by having the tests "check out" cores and sleep if they aren't available yet. This design requires a threadsafe method to query, acquire and lock the count of available mpi cores. I could use some sort of lock or semaphore from threading, but I thought it would be good to run this by the xdist cognoscenti and find out if there might be a preferred way of doing this given how xdist itself distributes its work or manages threads. Thanks much, Eli From holger at merlinux.eu Fri Jan 20 09:50:18 2012 From: holger at merlinux.eu (holger krekel) Date: Fri, 20 Jan 2012 08:50:18 +0000 Subject: [py-dev] xdist and thread-safe resource counting In-Reply-To: <67CC54752B3F8D43ACD7E3AD455162630B5714DAA9@mrsbmapp20306.ad.water.ca.gov> References: <67CC54752B3F8D43ACD7E3AD455162630B5714DAA9@mrsbmapp20306.ad.water.ca.gov> Message-ID: <20120120085018.GL26028@merlinux.eu> Hi Eli, interesting problem. On Wed, Jan 18, 2012 at 20:55 -0800, Ateljevich, Eli wrote: > I have a question about managing resources in a threadsafe way across xdist -n. > > My group is using py.test as a high-level driver for testing an mpi-based numerical code. Many of our system-level tests wrap a system call to mpirun then postprocess results. I have a decorator for the tests that hints at the number of processors needed (usually something like 1,2,8). > > I would like to launch as much as I can at once given the available processors. For instance, if 16 processors are available there is no reason I couldn't be doing a 12 and a 4 processor test. I was thinking of using xdist with some modest number of processors representing the maximum number of concurrent tests. The xdist test processors would launch mpi jobs when enough processors become available to satisfy the np hint for that test. This would be managed by having the tests "check out" cores and sleep if they aren't available yet. > > This design requires a threadsafe method to query, acquire and lock the count of available mpi cores. I could use some sort of lock or semaphore from threading, but I thought it would be good to run this by the xdist cognoscenti and find out if there might be a preferred way of doing this given how xdist itself distributes its work or manages threads. pytest-xdist itself does not provide or use a method to query the number of available processors. Quick background of xdist: Master process starts a number of processes which collect tests (see output of py.test --collectonly) and the master sees the test ids of all those collections. It then decides the scheduling (Each or Load at the moment, "-n5" implies load-balancing) and sends test ids to the nodes to execute. It pre-loads tests with test ids and then waits for completion for sending more test ids to each node. There is no node-to-node communication for co-ordination. It might be easiest to not try to extend the xdist-mechanisms but to implement an independent method which co-ordinates the number of running MPI tests / used processors via a file or so. For example, on posix you can get read/write a file with some meta-information and use the atomic os.rename operation. Not sure about the exact semantics but this should be doable and testable without any xdist involvement. If you have such a method which helps to restrict the number of MPI-processes you can then use it from a pytest_runtest_setup which can read your decorator-attributes/markers and then make the decision if to wait or run the test. This method also makes you rather independent from the number of worker processes started with "-nNUM". HTH, holger From eli at water.ca.gov Fri Jan 20 21:56:37 2012 From: eli at water.ca.gov (Ateljevich, Eli) Date: Fri, 20 Jan 2012 12:56:37 -0800 Subject: [py-dev] xdist and thread-safe resource counting In-Reply-To: <20120120085018.GL26028@merlinux.eu> References: <67CC54752B3F8D43ACD7E3AD455162630B5714DAA9@mrsbmapp20306.ad.water.ca.gov>, <20120120085018.GL26028@merlinux.eu> Message-ID: <67CC54752B3F8D43ACD7E3AD455162630B5714DAB0@mrsbmapp20306.ad.water.ca.gov> Thanks, Holger. I appreciate the hint about where to do the testing and waiting in pytest_runtest_setup and I think the atomic file rename idea is an interesting way to set up a signal. I may not fully understand about xdist. I certainly agree it is efficient use of mpirun that is the crux of doing the job right ... so probably any load balancing offered by xdist is going to be wasted on me. The main service I was looking for out of xdist was the ability to run tests concurrently. As I think you realize, if I have a pool of 16 processors and the first four tests collected require 8, 4, 8, 4 processors, I would want this behavior: 1. the first test to start immediately 2. the second test to start immediately without the first finishing 3. the third test to either wait or start in a python sense but "sleep" before launching mpi 4. the fourth test to start immediately Is vanilla py.test able to do this kind of concurrent testing? Or would I need to tweak it to launch tests in threads according to my criterion for readiness? I think we have settled how I would allocate resources, but your idea implies I might have all the test hints in one place. If I have full control all the test launches this might allow me to do some sort of knapsack problem-ish kind of reorganization to keep everything fully utilized rather than taking the test in the order they were collected. For instance, if I had 16 processors and the first four tests take 12-12-4-4 I could do this in the order (12+4 concurrently) (12+4 concurrently). Do I have this level of control? Thanks, Eli _________________________ _______________ From: holger krekel [holger at merlinux.eu] Sent: Friday, January 20, 2012 12:50 AM To: Ateljevich, Eli Cc: py-dev at codespeak.net Subject: Re: [py-dev] xdist and thread-safe resource counting Hi Eli, interesting problem. On Wed, Jan 18, 2012 at 20:55 -0800, Ateljevich, Eli wrote: > I have a question about managing resources in a threadsafe way across xdist -n. > > My group is using py.test as a high-level driver for testing an mpi-based numerical code. Many of our system-level tests wrap a system call to mpirun then postprocess results. I have a decorator for the tests that hints at the number of processors needed (usually something like 1,2,8). > > I would like to launch as much as I can at once given the available processors. For instance, if 16 processors are available there is no reason I couldn't be doing a 12 and a 4 processor test. I was thinking of using xdist with some modest number of processors representing the maximum number of concurrent tests. The xdist test processors would launch mpi jobs when enough processors become available to satisfy the np hint for that test. This would be managed by having the tests "check out" cores and sleep if they aren't available yet. > > This design requires a threadsafe method to query, acquire and lock the count of available mpi cores. I could use some sort of lock or semaphore from threading, but I thought it would be good to run this by the xdist cognoscenti and find out if there might be a preferred way of doing this given how xdist itself distributes its work or manages threads. pytest-xdist itself does not provide or use a method to query the number of available processors. Quick background of xdist: Master process starts a number of processes which collect tests (see output of py.test --collectonly) and the master sees the test ids of all those collections. It then decides the scheduling (Each or Load at the moment, "-n5" implies load-balancing) and sends test ids to the nodes to execute. It pre-loads tests with test ids and then waits for completion for sending more test ids to each node. There is no node-to-node communication for co-ordination. It might be easiest to not try to extend the xdist-mechanisms but to implement an independent method which co-ordinates the number of running MPI tests / used processors via a file or so. For example, on posix you can get read/write a file with some meta-information and use the atomic os.rename operation. Not sure about the exact semantics but this should be doable and testable without any xdist involvement. If you have such a method which helps to restrict the number of MPI-processes you can then use it from a pytest_runtest_setup which can read your decorator-attributes/markers and then make the decision if to wait or run the test. This method also makes you rather independent from the number of worker processes started with "-nNUM". HTH, holger From holger at merlinux.eu Fri Jan 20 23:26:48 2012 From: holger at merlinux.eu (holger krekel) Date: Fri, 20 Jan 2012 22:26:48 +0000 Subject: [py-dev] xdist and thread-safe resource counting In-Reply-To: <67CC54752B3F8D43ACD7E3AD455162630B5714DAB0@mrsbmapp20306.ad.water.ca.gov> References: <67CC54752B3F8D43ACD7E3AD455162630B5714DAA9@mrsbmapp20306.ad.water.ca.gov> <20120120085018.GL26028@merlinux.eu> <67CC54752B3F8D43ACD7E3AD455162630B5714DAB0@mrsbmapp20306.ad.water.ca.gov> Message-ID: <20120120222648.GR26028@merlinux.eu> On Fri, Jan 20, 2012 at 12:56 -0800, Ateljevich, Eli wrote: > Thanks, Holger. I appreciate the hint about where to do the testing and waiting in pytest_runtest_setup and I think the atomic file rename idea is an interesting way to set up a signal. > > I may not fully understand about xdist. I certainly agree it is efficient use of mpirun that is the crux of doing the job right ... so probably any load balancing offered by xdist is going to be wasted on me. > > The main service I was looking for out of xdist was the ability to run tests concurrently. As I think you realize, if I have a pool of 16 processors and the first four tests collected require 8, 4, 8, 4 processors, I would want this behavior: > 1. the first test to start immediately > 2. the second test to start immediately without the first finishing > 3. the third test to either wait or start in a python sense but "sleep" before launching mpi > 4. the fourth test to start immediately > > Is vanilla py.test able to do this kind of concurrent testing? Or would I need to tweak it to launch tests in threads according to my criterion for readiness? A run with pytest-xdist, notably, "py.test -nNUM" allows to implement this behaviour, i think. > I think we have settled how I would allocate resources, but your idea implies I might have all the test hints in one place. If I have full control all the test launches this might allow me to do some sort of knapsack problem-ish kind of reorganization to keep everything fully utilized rather than taking the test in the order they were collected. For instance, if I had 16 processors and the first four tests take 12-12-4-4 I could do this in the order (12+4 concurrently) (12+4 concurrently). Do I have this level of control? I think so yes. IIRC pytest-xdist distributed the first four tests such that they each land at different nodes. So, given the algorithm i hinted at, and running with "py.test -n3" the first sub process would start and run on 12 processors. The second process would see that there are 12 used and wait until 12 become available. The third process would only need 4 and immediatly continue, utilizing all 16 processors at that time. When the first one finishes the second sub process would see that there now are enough and proceed with its testing. This is all fully compatible with pytest-xdist semantics and only needs code at pytest_runtest_setup time i think. best, holger From eli at water.ca.gov Thu Jan 26 00:30:42 2012 From: eli at water.ca.gov (Ateljevich, Eli) Date: Wed, 25 Jan 2012 15:30:42 -0800 Subject: [py-dev] xdist and thread-safe resource counting In-Reply-To: <20120120222648.GR26028@merlinux.eu> References: <67CC54752B3F8D43ACD7E3AD455162630B5714DAA9@mrsbmapp20306.ad.water.ca.gov> <20120120085018.GL26028@merlinux.eu> <67CC54752B3F8D43ACD7E3AD455162630B5714DAB0@mrsbmapp20306.ad.water.ca.gov> <20120120222648.GR26028@merlinux.eu> Message-ID: <67CC54752B3F8D43ACD7E3AD455162630B57114387@mrsbmapp20306.ad.water.ca.gov> Thanks. I really appreciate the help. I have one last question -- is the underlying parallel mechanism threads or mpi or something else? Do I have to worry about things like MPI communicators getting in each other's way? Cheers, Eli -----Original Message----- From: holger krekel [mailto:holger at merlinux.eu] Sent: Friday, January 20, 2012 2:27 PM To: Ateljevich, Eli Cc: holger krekel; py-dev at codespeak.net Subject: Re: [py-dev] xdist and thread-safe resource counting On Fri, Jan 20, 2012 at 12:56 -0800, Ateljevich, Eli wrote: > Thanks, Holger. I appreciate the hint about where to do the testing and waiting in pytest_runtest_setup and I think the atomic file rename idea is an interesting way to set up a signal. > > I may not fully understand about xdist. I certainly agree it is efficient use of mpirun that is the crux of doing the job right ... so probably any load balancing offered by xdist is going to be wasted on me. > > The main service I was looking for out of xdist was the ability to run tests concurrently. As I think you realize, if I have a pool of 16 processors and the first four tests collected require 8, 4, 8, 4 processors, I would want this behavior: > 1. the first test to start immediately > 2. the second test to start immediately without the first finishing > 3. the third test to either wait or start in a python sense but "sleep" before launching mpi > 4. the fourth test to start immediately > > Is vanilla py.test able to do this kind of concurrent testing? Or would I need to tweak it to launch tests in threads according to my criterion for readiness? A run with pytest-xdist, notably, "py.test -nNUM" allows to implement this behaviour, i think. > I think we have settled how I would allocate resources, but your idea implies I might have all the test hints in one place. If I have full control all the test launches this might allow me to do some sort of knapsack problem-ish kind of reorganization to keep everything fully utilized rather than taking the test in the order they were collected. For instance, if I had 16 processors and the first four tests take 12-12-4-4 I could do this in the order (12+4 concurrently) (12+4 concurrently). Do I have this level of control? I think so yes. IIRC pytest-xdist distributed the first four tests such that they each land at different nodes. So, given the algorithm i hinted at, and running with "py.test -n3" the first sub process would start and run on 12 processors. The second process would see that there are 12 used and wait until 12 become available. The third process would only need 4 and immediatly continue, utilizing all 16 processors at that time. When the first one finishes the second sub process would see that there now are enough and proceed with its testing. This is all fully compatible with pytest-xdist semantics and only needs code at pytest_runtest_setup time i think. best, holger From Ronny.Pfannschmidt at gmx.de Thu Jan 26 01:46:04 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 26 Jan 2012 01:46:04 +0100 Subject: [py-dev] xdist and thread-safe resource counting In-Reply-To: <67CC54752B3F8D43ACD7E3AD455162630B57114387@mrsbmapp20306.ad.water.ca.gov> References: <67CC54752B3F8D43ACD7E3AD455162630B5714DAA9@mrsbmapp20306.ad.water.ca.gov> <20120120085018.GL26028@merlinux.eu> <67CC54752B3F8D43ACD7E3AD455162630B5714DAB0@mrsbmapp20306.ad.water.ca.gov> <20120120222648.GR26028@merlinux.eu> <67CC54752B3F8D43ACD7E3AD455162630B57114387@mrsbmapp20306.ad.water.ca.gov> Message-ID: <4F20A24C.1020108@gmx.de> Hi, the underlying parallel mechanism is processes communicating via execnet. -- Ronny On 01/26/2012 12:30 AM, Ateljevich, Eli wrote: > Thanks. I really appreciate the help. I have one last question -- is the underlying parallel mechanism threads or mpi or something else? Do I have to worry about things like MPI communicators getting in each other's way? > > Cheers, > Eli > > -----Original Message----- > From: holger krekel [mailto:holger at merlinux.eu] > Sent: Friday, January 20, 2012 2:27 PM > To: Ateljevich, Eli > Cc: holger krekel; py-dev at codespeak.net > Subject: Re: [py-dev] xdist and thread-safe resource counting > > On Fri, Jan 20, 2012 at 12:56 -0800, Ateljevich, Eli wrote: >> Thanks, Holger. I appreciate the hint about where to do the testing and waiting in pytest_runtest_setup and I think the atomic file rename idea is an interesting way to set up a signal. >> >> I may not fully understand about xdist. I certainly agree it is efficient use of mpirun that is the crux of doing the job right ... so probably any load balancing offered by xdist is going to be wasted on me. >> >> The main service I was looking for out of xdist was the ability to run tests concurrently. As I think you realize, if I have a pool of 16 processors and the first four tests collected require 8, 4, 8, 4 processors, I would want this behavior: >> 1. the first test to start immediately >> 2. the second test to start immediately without the first finishing >> 3. the third test to either wait or start in a python sense but "sleep" before launching mpi >> 4. the fourth test to start immediately >> >> Is vanilla py.test able to do this kind of concurrent testing? Or would I need to tweak it to launch tests in threads according to my criterion for readiness? > > A run with pytest-xdist, notably, "py.test -nNUM" allows to implement this > behaviour, i think. > >> I think we have settled how I would allocate resources, but your idea implies I might have all the test hints in one place. If I have full control all the test launches this might allow me to do some sort of knapsack problem-ish kind of reorganization to keep everything fully utilized rather than taking the test in the order they were collected. For instance, if I had 16 processors and the first four tests take 12-12-4-4 I could do this in the order (12+4 concurrently) (12+4 concurrently). Do I have this level of control? > > I think so yes. IIRC pytest-xdist distributed the first four tests > such that they each land at different nodes. So, given the algorithm > i hinted at, and running with "py.test -n3" the first sub process would > start and run on 12 processors. The second process would see that > there are 12 used and wait until 12 become available. The > third process would only need 4 and immediatly continue, utilizing > all 16 processors at that time. When the first one finishes the > second sub process would see that there now are enough and proceed > with its testing. This is all fully compatible with pytest-xdist > semantics and only needs code at pytest_runtest_setup time i think. > > best, > holger > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/py-dev/attachments/20120126/63d4b4e2/attachment.pgp From eli at water.ca.gov Fri Jan 27 06:16:58 2012 From: eli at water.ca.gov (Ateljevich, Eli) Date: Thu, 26 Jan 2012 21:16:58 -0800 Subject: [py-dev] Can py.test collect a test inside a shared library? Message-ID: <67CC54752B3F8D43ACD7E3AD455162630B5714DAE2@mrsbmapp20306.ad.water.ca.gov> Hi all, If I have a test compiled from C or f2py that is called test_something and is compiled to mymod.so (using Linux as the example), would it get collected ... if you grant for the sake of conversation that I could possibly have something useful to put in it? I realize I can't use the usual recompiling of assertion magic. I tried creating an __init__.py file: from mymod import * and running py.test on the directory without success. Maybe it needs to be in test_something.py? I thought I would check in about whether this is credited with a chance before I double my efforts. Thanks, Eli From Ronny.Pfannschmidt at gmx.de Fri Jan 27 08:54:22 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 27 Jan 2012 08:54:22 +0100 Subject: [py-dev] Can py.test collect a test inside a shared library? In-Reply-To: <67CC54752B3F8D43ACD7E3AD455162630B5714DAE2@mrsbmapp20306.ad.water.ca.gov> References: <67CC54752B3F8D43ACD7E3AD455162630B5714DAE2@mrsbmapp20306.ad.water.ca.gov> Message-ID: <4F22582E.5040905@gmx.de> Hi Eli, by default py.test searches for test functions test_*.py as for test functions, they are supposed to start with test as well it might make sense to slit your c written test function into a few test util function, that way you can run the c written part step by step, and do assertions in between (for getting a more detailed idea of whats wrong, if something goes wrong) -- Ronny On 01/27/2012 06:16 AM, Ateljevich, Eli wrote: > Hi all, > If I have a test compiled from C or f2py that is called test_something and is compiled to mymod.so (using Linux as the example), would it get collected ... if you grant for the sake of conversation that I could possibly have something useful to put in it? I realize I can't use the usual recompiling of assertion magic. > > I tried creating an __init__.py file: > from mymod import * > > and running py.test on the directory without success. Maybe it needs to be in test_something.py? I thought I would check in about whether this is credited with a chance before I double my efforts. > > Thanks, > Eli > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/py-dev/attachments/20120127/43a65515/attachment.pgp From eli at water.ca.gov Fri Jan 27 16:52:43 2012 From: eli at water.ca.gov (Ateljevich, Eli) Date: Fri, 27 Jan 2012 07:52:43 -0800 Subject: [py-dev] Can py.test collect a test inside a shared library? In-Reply-To: <4F22582E.5040905@gmx.de> References: <67CC54752B3F8D43ACD7E3AD455162630B5714DAE2@mrsbmapp20306.ad.water.ca.gov>, <4F22582E.5040905@gmx.de> Message-ID: <67CC54752B3F8D43ACD7E3AD455162630B5714DAE8@mrsbmapp20306.ad.water.ca.gov> Thanks Ronny. I will wrap the C/Fortran in python, name it test_* and expect it to work. I figured if I went that far it would work, since it amounts to just using py.test the normal way on tests that happen to use C library functions to do some work. I had hoped that python namespaces were being searched for test_* rather than files. What you suggest about utility function makes sense, and it is my first choice. There is a subtle difference between the more intuitive thing you are suggesting: 1. Code to be tested is in fortran/C, tests are entirely in python, testing system is py.test and the precise thing I was asking about: 2. Code is in fortran/C, test is in fortran/C (with a helper macro to make it look easy) and the testing system is py.test That the latter can even be achieved is probably not obvious, but tools such as f2py have a callback system and I can write a little preprocessor macro for wrapping the test so that it looks like it is native but utilizes a python callback function for the assert (which will be a somewhat special one having to do with mpi). So it isn't that hard to achieve a unit testing framework that looks like it is written entirely in the native languages and has a fairly uniform way of treating parallel asserts (ie, it peforms the all-reduce so that the processors agree on pass/fail). But possible does not mean well-motivated. I am not sure that approach #2 is best even for my requirements, and it certainly would not be the best practice in general. It was more attractive if test collection would become trivially automatic rather than having to generate a test_* function to shadow each native test. Maybe I could write a custom collection hook to do that, but I don't want to bother anyone further until I run through my requirements and confirm this convoluted approach is actually better at meeting them. Thanks, Eli ________________________________________ From: Ronny Pfannschmidt [Ronny.Pfannschmidt at gmx.de] Sent: Thursday, January 26, 2012 11:54 PM To: Ateljevich, Eli Cc: py-dev at codespeak.net Subject: Re: [py-dev] Can py.test collect a test inside a shared library? Hi Eli, by default py.test searches for test functions test_*.py as for test functions, they are supposed to start with test as well it might make sense to slit your c written test function into a few test util function, that way you can run the c written part step by step, and do assertions in between (for getting a more detailed idea of whats wrong, if something goes wrong) -- Ronny On 01/27/2012 06:16 AM, Ateljevich, Eli wrote: > Hi all, > If I have a test compiled from C or f2py that is called test_something and is compiled to mymod.so (using Linux as the example), would it get collected ... if you grant for the sake of conversation that I could possibly have something useful to put in it? I realize I can't use the usual recompiling of assertion magic. > > I tried creating an __init__.py file: > from mymod import * > > and running py.test on the directory without success. Maybe it needs to be in test_something.py? I thought I would check in about whether this is credited with a chance before I double my efforts. > > Thanks, > Eli > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From Ronny.Pfannschmidt at gmx.de Sat Jan 28 06:07:35 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Sat, 28 Jan 2012 06:07:35 +0100 Subject: [py-dev] Can py.test collect a test inside a shared library? In-Reply-To: <67CC54752B3F8D43ACD7E3AD455162630B5714DAE8@mrsbmapp20306.ad.water.ca.gov> References: <67CC54752B3F8D43ACD7E3AD455162630B5714DAE2@mrsbmapp20306.ad.water.ca.gov>, <4F22582E.5040905@gmx.de> <67CC54752B3F8D43ACD7E3AD455162630B5714DAE8@mrsbmapp20306.ad.water.ca.gov> Message-ID: <4F238297.2050703@gmx.de> Hi Eli, i simply suggested, option 1, cause its much more easy and straightforward to do from my pov its entirely possible to implement implement a collector that gets to participate in finding test functions inside of a shared library (just like osjekit does for javascript files for example), and if the shared lib acts close enough to a module, it may even be possible to reuse the Module collector but you will loose some of the python integration things if you do that and i'm not sure if its reasonably simple to bring those to a c/fortran binding -- Ronny On 01/27/2012 04:52 PM, Ateljevich, Eli wrote: > Thanks Ronny. I will wrap the C/Fortran in python, name it test_* and expect it to work. I figured if I went that far it would work, since it amounts to just using py.test the normal way on tests that happen to use C library functions to do some work. > > I had hoped that python namespaces were being searched for test_* rather than files. What you suggest about utility function makes sense, and it is my first choice. There is a subtle difference between the more intuitive thing you are suggesting: > 1. Code to be tested is in fortran/C, tests are entirely in python, testing system is py.test > and the precise thing I was asking about: > 2. Code is in fortran/C, test is in fortran/C (with a helper macro to make it look easy) and the testing system is py.test > > That the latter can even be achieved is probably not obvious, but tools such as f2py have a callback system and I can write a little preprocessor macro for wrapping the test so that it looks like it is native but utilizes a python callback function for the assert (which will be a somewhat special one having to do with mpi). So it isn't that hard to achieve a unit testing framework that looks like it is written entirely in the native languages and has a fairly uniform way of treating parallel asserts (ie, it peforms the all-reduce so that the processors agree on pass/fail). > > But possible does not mean well-motivated. I am not sure that approach #2 is best even for my requirements, and it certainly would not be the best practice in general. It was more attractive if test collection would become trivially automatic rather than having to generate a test_* function to shadow each native test. Maybe I could write a custom collection hook to do that, but I don't want to bother anyone further until I run through my requirements and confirm this convoluted approach is actually better at meeting them. > > Thanks, > Eli > > > > ________________________________________ > From: Ronny Pfannschmidt [Ronny.Pfannschmidt at gmx.de] > Sent: Thursday, January 26, 2012 11:54 PM > To: Ateljevich, Eli > Cc: py-dev at codespeak.net > Subject: Re: [py-dev] Can py.test collect a test inside a shared library? > > Hi Eli, > > by default py.test searches for test functions test_*.py > as for test functions, they are supposed to start with test as well > > it might make sense to slit your c written test function into a few test > util function, that way you can run the c written part step by step, and > do assertions in between (for getting a more detailed idea of whats > wrong, if something goes wrong) > > -- Ronny > > On 01/27/2012 06:16 AM, Ateljevich, Eli wrote: >> Hi all, >> If I have a test compiled from C or f2py that is called test_something and is compiled to mymod.so (using Linux as the example), would it get collected ... if you grant for the sake of conversation that I could possibly have something useful to put in it? I realize I can't use the usual recompiling of assertion magic. >> >> I tried creating an __init__.py file: >> from mymod import * >> >> and running py.test on the directory without success. Maybe it needs to be in test_something.py? I thought I would check in about whether this is credited with a chance before I double my efforts. >> >> Thanks, >> Eli >> _______________________________________________ >> py-dev mailing list >> py-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/py-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/py-dev/attachments/20120128/6f235a4e/attachment-0001.pgp From holger at merlinux.eu Mon Feb 6 00:36:28 2012 From: holger at merlinux.eu (holger krekel) Date: Sun, 5 Feb 2012 23:36:28 +0000 Subject: [py-dev] pytest-2.2.2: bug fixes, collectonly improvements Message-ID: <20120205233628.GK26028@merlinux.eu> pytest-2.2.2 is a minor backward-compatible release of the py.test testing tool. It contains bug fixes and a few refinements particularly to reporting with "--collectonly", see below for betails. For general information see here: http://pytest.org/ To install or upgrade pytest: pip install -U pytest # or easy_install -U pytest Special thanks for helping on this release to Ronny Pfannschmidt and Ralf Schmitt and the contributors of issues. best, holger krekel Changes between 2.2.1 and 2.2.2 ---------------------------------------- - fix issue101: wrong args to unittest.TestCase test function now produce better output - fix issue102: report more useful errors and hints for when a test directory was renamed and some pyc/__pycache__ remain - fix issue106: allow parametrize to be applied multiple times e.g. from module, class and at function level. - fix issue107: actually perform session scope finalization - don't check in parametrize if indirect parameters are funcarg names - add chdir method to monkeypatch funcarg - fix crash resulting from calling monkeypatch undo a second time - fix issue115: make --collectonly robust against early failure (missing files/directories) - "-qq --collectonly" now shows only files and the number of tests in them - "-q --collectonly" now shows test ids - allow adding of attributes to test reports such that it also works with distributed testing (no upgrade of pytest-xdist needed) From basti at rotekroete.de Tue Feb 7 19:38:22 2012 From: basti at rotekroete.de (Sebastian Rahlf) Date: Tue, 7 Feb 2012 19:38:22 +0100 Subject: [py-dev] New plugin pytest-pydev Message-ID: Hello all! I've just released the first version of a new py.test plugin which will allow you to use a remote Python debug server. This will, for instance, come in especially handy if you want to find out why a particular unittest is failing in a web application. Simply fire up your debug server in your IDE (PyDev or PyCharm) e.g. on localhost:8042 and run your unittests with py.test --pydevd=your-ip:8042 If you have set a break point you should soon see the effects. Enjoy! Sebastian P.S. Feedback is always welcome! From basti at rotekroete.de Tue Feb 7 19:40:04 2012 From: basti at rotekroete.de (Sebastian Rahlf) Date: Tue, 7 Feb 2012 19:40:04 +0100 Subject: [py-dev] New plugin pytest-pydev In-Reply-To: References: Message-ID: > Hello all! > > I've just released the first version of a new py.test plugin which > will allow you to use a remote Python debug server. This will, for > instance, come in especially handy if you want to find out why a > particular unittest is failing in a web application. > > Simply fire up your debug server in your IDE (PyDev or PyCharm) e.g. > on localhost:8042 and run your unittests with > > ? ?py.test --pydevd=your-ip:8042 > > If you have set a break point you should soon see the effects. > > Enjoy! > Sebastian > > P.S. Feedback is always welcome! Forgot to mention: code lives on bitbucket.org https://bitbucket.org/basti/pytest-pydev Sorry! Sebastian From anton7811 at gmail.com Thu Feb 16 13:48:15 2012 From: anton7811 at gmail.com (Anton P) Date: Thu, 16 Feb 2012 14:48:15 +0200 Subject: [py-dev] how to access docstrings from pytest_report_teststatus or pytest_runtest_logreport Message-ID: Hi, I'm trying to write my own py.test pluging for extended junitxml reports. This pluging have to parse testcase's docstrings for some additional info for report. I've found the next code in pytest_bugzilla plugin: ??? def pytest_report_teststatus(self,report): ??????? if report.failed: ??????????? if self.analysed(report.item.function.__doc__): ??????????????? return "analysed", "A", "ANALYSED" And I've tried to do the same to access docstrings. But I'm confused, because report object does not contain "item" attribute. At the moment I've added class ExtTestReport(TestReport) and redefined pytest_runtest_makereport() in plugin. The last hook now return report with "item" object. But I think, that redefining native pytest_runtest_makereport() in pluging is not the best idea. Especially because new method is always called regardless of my pluging usage. Please, point me to right solution to access testcase's docstrings from pytest_report_teststatus() or pytest_runtest_logreport(). Thank you in advance! From Ronny.Pfannschmidt at gmx.de Thu Feb 16 14:32:01 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 16 Feb 2012 14:32:01 +0100 Subject: [py-dev] how to access docstrings from pytest_report_teststatus or pytest_runtest_logreport In-Reply-To: References: Message-ID: <4F3D0551.5070902@gmx.de> On 02/16/2012 01:48 PM, Anton P wrote: > Hi, > > I'm trying to write my own py.test pluging for extended junitxml > reports. This pluging have to parse testcase's docstrings for some > additional info for report. > > I've found the next code in pytest_bugzilla plugin: > > def pytest_report_teststatus(self,report): > if report.failed: > if self.analysed(report.item.function.__doc__): > return "analysed", "A", "ANALYSED" > > And I've tried to do the same to access docstrings. But I'm confused, > because report object does not contain "item" attribute. > > At the moment I've added class ExtTestReport(TestReport) and redefined > pytest_runtest_makereport() in plugin. The last hook now return report > with "item" object. > > But I think, that redefining native pytest_runtest_makereport() in > pluging is not the best idea. Especially because new method is always > called regardless of my pluging usage. > > Please, point me to right solution to access testcase's docstrings > from pytest_report_teststatus() or pytest_runtest_logreport(). > you can extend the native pytest_runtest_makereport think of something like def pytest_runtest_makereport(__multicall__, ...): rep = __multicall__.execute() rep._docstring = item... return rep -- Regards Ronny > Thank you in advance! > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From anton7811 at gmail.com Fri Feb 17 12:25:16 2012 From: anton7811 at gmail.com (Anton P) Date: Fri, 17 Feb 2012 13:25:16 +0200 Subject: [py-dev] how to access docstrings from pytest_report_teststatus or pytest_runtest_logreport In-Reply-To: <4F3D0551.5070902@gmx.de> References: <4F3D0551.5070902@gmx.de> Message-ID: Thank you! That works for me! -Anton 2012/2/16 Ronny Pfannschmidt : > On 02/16/2012 01:48 PM, Anton P wrote: >> >> Hi, >> >> I'm trying to write my own py.test pluging for extended junitxml >> reports. This pluging have to parse testcase's docstrings for some >> additional info for report. >> >> I've found the next code in pytest_bugzilla plugin: >> >> ? ? def pytest_report_teststatus(self,report): >> ? ? ? ? if report.failed: >> ? ? ? ? ? ? if self.analysed(report.item.function.__doc__): >> ? ? ? ? ? ? ? ? return "analysed", "A", "ANALYSED" >> >> And I've tried to do the same to access docstrings. But I'm confused, >> because report object does not contain "item" attribute. >> >> At the moment I've added class ExtTestReport(TestReport) and redefined >> pytest_runtest_makereport() in plugin. The last hook now return report >> with "item" object. >> >> But I think, that redefining native pytest_runtest_makereport() in >> pluging is not the best idea. Especially because new method is always >> called regardless of my pluging usage. >> >> Please, point me to right solution to access testcase's docstrings >> from pytest_report_teststatus() or pytest_runtest_logreport(). >> > you can extend the native pytest_runtest_makereport > > think of something like > > def pytest_runtest_makereport(__multicall__, ...): > ? rep = __multicall__.execute() > ? rep._docstring = item... > ? return rep > > -- Regards Ronny > >> Thank you in advance! >> _______________________________________________ >> py-dev mailing list >> py-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/py-dev > > From anton7811 at gmail.com Wed Feb 22 12:10:55 2012 From: anton7811 at gmail.com (Anton P) Date: Wed, 22 Feb 2012 13:10:55 +0200 Subject: [py-dev] XML Parsing Error when esc characters are in traceback Message-ID: Hello all, I have the following issue with generated junitxml reports: I'm writing test cases which use scapy functions. So I make asserts on some packet objects. As I understand scapy uses esc characters for its own objects representation, and I can see colorful output on the console. But when test is failed, and its traceback went to xml, I can't open generated xml due to the error "XML Parsing Error: not well-formed" In my case the problem with esc charecter 033 (\x1b). I resolved the issue by adding the following code to append_failure method in LogXML class: longrepr = str(report.longrepr).replace('\x1b', '\\033') fail.append(longrepr) I think that it will be good if you add some cleanup functions to avoid such situations. Thank you, -Anton From flub at devork.be Mon Feb 27 00:19:59 2012 From: flub at devork.be (Floris Bruynooghe) Date: Sun, 26 Feb 2012 23:19:59 +0000 Subject: [py-dev] New plugin: pytest-timeout Message-ID: Hi all, This is to announce the first release of pytest-timeout, a plugin which will interrupt long running (i.e. blocking) tests. This is particularly useful when running the tests on a CI host. When a test is interrupted the stacks of all threads are dumped to stderr, which helps you to locate the reason for the blocking. If the system supports SIGALRM then the test itself is interrupted using pytest.fail() and other tests continue to run, otherwise the stack(s) are dumped to stderr and the process exists immediately. The plugin is available from the cheeseshop: http://pypi.python.org/pypi/pytest-timeout and the code lives at bitbucket: https://bitbucket.org/flub/pytest-timeout/. Feel free to provide any feedback or report issues. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From lpbrac at dolby.com Tue Feb 28 19:08:40 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Tue, 28 Feb 2012 10:08:40 -0800 Subject: [py-dev] New plugin: pytest-timeout References: Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net> Hi Floris, first of all thanks for this plugin as this is something my team will be more than happy to leverage. I gave it a shot yesterday (just kicking the tires) and while the plugin was installed (see below) and registered it didn't work for me (ubuntu/python 2.6.6) This is py.test version 2.2.3, imported from /home/lpbrac/.buildout/eggs/pytest-2.2.3-py2.6.egg/pytest.pyc setuptools registered plugins: pytest-xdist-1.8 at /home/lpbrac/.buildout/eggs/pytest_xdist-1.8-py2.6.egg/xdist/plugin.pyc pytest-cov-1.5 at /home/lpbrac/.buildout/eggs/pytest_cov-1.5-py2.6.egg/pytest_cov.pyc pytest-testlink-0.0.6 at /home/lpbrac/repos/p4_qa/qa/infrastructure/taf/pytest_plugins/pytest_testlink/src/pytest_testlink/tl_plugin.py pytest-session-cfg-0.0.2 at /home/lpbrac/.buildout/eggs/pytest_session_cfg-0.0.2-py2.6.egg/pytest_session_cfg.py pytest-timeout-0.1 at /home/lpbrac/.buildout/eggs/pytest_timeout-0.1-py2.6.egg/pytest_timeout.pyc Just wrote an infinite loop which never got preempted. I didn't have time to dig any deeper. Does the test need to be executed in a separate process via xdist? Also, currently the time out applies to all tests (and defaults to 5 minutes). I think it would be useful to be able to override the timeout at the test level using markers. If something goes wrong and we have thousands of tests (we use pytest to test embedded systems), this is an awful lot of time to realize that things are DOA. Best Regards and thanks again for starting this. /Laurent -----Original Message----- From: py-dev-bounces at codespeak.net on behalf of Floris Bruynooghe Sent: Sun 2/26/2012 3:19 PM To: py-dev at codespeak.net Subject: [py-dev] New plugin: pytest-timeout Hi all, This is to announce the first release of pytest-timeout, a plugin which will interrupt long running (i.e. blocking) tests. This is particularly useful when running the tests on a CI host. When a test is interrupted the stacks of all threads are dumped to stderr, which helps you to locate the reason for the blocking. If the system supports SIGALRM then the test itself is interrupted using pytest.fail() and other tests continue to run, otherwise the stack(s) are dumped to stderr and the process exists immediately. The plugin is available from the cheeseshop: http://pypi.python.org/pypi/pytest-timeout and the code lives at bitbucket: https://bitbucket.org/flub/pytest-timeout/. Feel free to provide any feedback or report issues. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org _______________________________________________ py-dev mailing list py-dev at codespeak.net http://codespeak.net/mailman/listinfo/py-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20120228/9749e887/attachment-0001.htm From flub at devork.be Wed Feb 29 11:31:44 2012 From: flub at devork.be (Floris Bruynooghe) Date: Wed, 29 Feb 2012 10:31:44 +0000 Subject: [py-dev] New plugin: pytest-timeout In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net> References: <05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net> Message-ID: Hello Laurent, On 28 February 2012 18:08, Brack, Laurent P. wrote: > Just wrote an infinite loop which never got preempted. I didn't have time to > dig any deeper. Does the test need to be executed in a separate process via > xdist? xdist is not needed, in fact I haven't tried it but assume it will behave fine. By default SIGALM will be used which does have it's limitations, the main issue is that it needs a chance to deliver the signal to python code. So depending on how your infinite loop is written this might be an issue and you may have to consider --nosigalrm which will use a timer thread (this is more expensive obviously). Could you show me the test function you where trying out? > Also, currently the time out applies to all tests (and defaults to 5 > minutes). I think it would be useful to be able to override the timeout > at the test level using markers. If something goes wrong and we have > thousands of tests (we use pytest to test embedded systems), this is an > awful lot of time to realize that things are DOA. Yes, I agree with this and it is noted in the TODO file that individual tests should have some control on the timeout and the mechanism used (timer vs sigalrm). But we only have about 10000 tests running nightly with normally none blocking, this plugin was written on the spur of the moment when we had about a dozen hanging and this minimal functionality seemed to do the job. It does assume hanging tests are an exception rather then the norm, certainly evident from the behaviour of --nosigalrm where os._exit() is called on a hanging test. I'm happy to work with you on adding a marker which will fulfil your needs however. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From holger at merlinux.eu Fri Mar 2 15:12:58 2012 From: holger at merlinux.eu (holger krekel) Date: Fri, 2 Mar 2012 14:12:58 +0000 Subject: [py-dev] XML Parsing Error when esc characters are in traceback In-Reply-To: References: Message-ID: <20120302141258.GS26028@merlinux.eu> Hi Anton, guess you know it - Ronny opened a bug on this issue and you might want to follow it. See https://bitbucket.org/hpk42/pytest/issue/126/junitxml-doesnt-escape-everything holger On Wed, Feb 22, 2012 at 13:10 +0200, Anton P wrote: > Hello all, > > I have the following issue with generated junitxml reports: > > I'm writing test cases which use scapy functions. So I make asserts on > some packet objects. As I understand scapy uses esc characters for its > own objects representation, and I can see colorful output on the > console. But when test is failed, and its traceback went to xml, I > can't open generated xml due to the error "XML Parsing Error: not > well-formed" In my case the problem with esc charecter 033 (\x1b). > I resolved the issue by adding the following code to append_failure > method in LogXML class: > longrepr = str(report.longrepr).replace('\x1b', '\\033') > fail.append(longrepr) > > I think that it will be good if you add some cleanup functions to > avoid such situations. > > Thank you, > > -Anton > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From anton7811 at gmail.com Mon Mar 5 10:56:40 2012 From: anton7811 at gmail.com (Anton P) Date: Mon, 5 Mar 2012 11:56:40 +0200 Subject: [py-dev] XML Parsing Error when esc characters are in traceback In-Reply-To: <20120302141258.GS26028@merlinux.eu> References: <20120302141258.GS26028@merlinux.eu> Message-ID: Thank you! I've checked that. 2012/3/2 holger krekel : > Hi Anton, > > guess you know it - Ronny opened a bug on this issue and you > might want to follow it. See > > https://bitbucket.org/hpk42/pytest/issue/126/junitxml-doesnt-escape-everything > > holger > > On Wed, Feb 22, 2012 at 13:10 +0200, Anton P wrote: >> Hello all, >> >> I have the following issue with generated junitxml reports: >> >> I'm writing test cases which use scapy functions. So I make asserts on >> some packet objects. As I understand scapy uses esc characters for its >> own objects representation, and I can see colorful output on the >> console. But when test is failed, and its traceback went to xml, I >> can't open generated xml due to the error "XML Parsing Error: not >> well-formed" In my case the problem with esc charecter 033 (\x1b). >> I resolved the issue by adding the following code to append_failure >> method in LogXML class: >> longrepr = str(report.longrepr).replace('\x1b', '\\033') >> fail.append(longrepr) >> >> I think that it will be good if you add some cleanup functions to >> avoid such situations. >> >> Thank you, >> >> -Anton >> _______________________________________________ >> py-dev mailing list >> py-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/py-dev >> From lpbrac at dolby.com Wed Mar 7 19:36:14 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Wed, 7 Mar 2012 10:36:14 -0800 Subject: [py-dev] New plugin: pytest-timeout References: <05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net> Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A39E96@sfo-exch-01.dolby.net> Hello Floris, Thanks for your prompt reply (and sorry for my delayed one :). I haven't had time to play more with the plugin but my observations (and Valentin in CC) has been that neither method was working for us. I haven't done my homework in terms of investigating this further (but I will next week) however, to answer your question the infinite loop looked like this: ------------------------------------------ import time def test_time_out(): counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) ------------------------------------------ Neither with or without sigalarm worked (I am running on Ubuntu as I mentioned and Valentine tried it on Windows). Based on the code above, I think that both methods should work in that case. Don't worry about it as I will try to figure out what is going on (unless you can think of something). It is however a very nice addition to pytest, as in early development stages things do have a tendency to hang. Thanks /Laurent -----Original Message----- From: floris.bruynooghe at gmail.com on behalf of Floris Bruynooghe Sent: Wed 2/29/2012 2:31 AM To: Brack, Laurent P. Cc: py-dev at codespeak.net Subject: Re: [py-dev] New plugin: pytest-timeout Hello Laurent, On 28 February 2012 18:08, Brack, Laurent P. wrote: > Just wrote an infinite loop which never got preempted. I didn't have time to > dig any deeper. Does the test need to be executed in a separate process via > xdist? xdist is not needed, in fact I haven't tried it but assume it will behave fine. By default SIGALM will be used which does have it's limitations, the main issue is that it needs a chance to deliver the signal to python code. So depending on how your infinite loop is written this might be an issue and you may have to consider --nosigalrm which will use a timer thread (this is more expensive obviously). Could you show me the test function you where trying out? > Also, currently the time out applies to all tests (and defaults to 5 > minutes). I think it would be useful to be able to override the timeout > at the test level using markers. If something goes wrong and we have > thousands of tests (we use pytest to test embedded systems), this is an > awful lot of time to realize that things are DOA. Yes, I agree with this and it is noted in the TODO file that individual tests should have some control on the timeout and the mechanism used (timer vs sigalrm). But we only have about 10000 tests running nightly with normally none blocking, this plugin was written on the spur of the moment when we had about a dozen hanging and this minimal functionality seemed to do the job. It does assume hanging tests are an exception rather then the norm, certainly evident from the behaviour of --nosigalrm where os._exit() is called on a hanging test. I'm happy to work with you on adding a marker which will fulfil your needs however. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20120307/6f7d33a3/attachment.htm From lpbrac at dolby.com Wed Mar 7 21:56:13 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Wed, 7 Mar 2012 12:56:13 -0800 Subject: [py-dev] New plugin: pytest-timeout References: <05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net> <05BB196AB3DA6C4BBE11AB6C957581FE42A39E96@sfo-exch-01.dolby.net> Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A39E98@sfo-exch-01.dolby.net> All right, it picked my curiosity so I looked into it. There was fundamentally nothing wrong with your plugin (it does work) however none of the hooks were being called. I could see the FaultHandlerPlugin class was being instanciated during the pytest_configure hook but then None of the hooks were called afterwards. Somehow (and I don't know why), registering the plugin as 'timeout' instead of 'pytest_timeout' was the cause of the problem, so for it to work on my side, I had to do: config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout') as opposed to config.pluginmanager.register(FaultHandlerPlugin(config), 'timeout') I also played with the ability to use markers to override the default timeout at the test level (here is an excerpt of your code modified) class FaultHandlerPlugin(object): """The timeout plugin""" def __init__(self, config): self.config = config self._current_timer = None self.cancel = None self._default_timeout = self.config.getvalue('timeout') # Default time outset during cfg - Laurent self._timeout = self._default_timeout # Current timeout value @property def timeout(self): return self._timeout @timeout.setter def timeout(self, value): """ Overrides the current timeout - Laurent """ self._timeout = value def reset_timeout(self): """ Resets the timeout to default config value""" self.timeout = self._default_timeout def pytest_runtest_setup(self, item): # Attempt to retrieve a timeout override from the timeout marker # Example: # @pytest.mark.timeout(3) # def test_something(): # ... try: if isinstance(item, item.Function) and hasattr(item.obj, 'timeout'): self.timeout = getattr(item.obj, 'timeout').args[0] except Exception: pass ... def pytest_runtest_teardown(self): """Cancel the timeout trigger""" # When skipping is raised from a pytest_runtest_setup function # (as is the case when using the pytest.mark.skipif marker) we # may be called without our setup counterpart having been # called. Hence the test for self.cancel. if self.cancel: self.cancel() self.cancel = None self.reset_timeout() # reset the timeout to default config value. Here is a sample of tests "working" with the timeout. import sys import pytest import time def test_time_out(): """ Should timeout only with the default timeout. """ counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) class TestTimeout(): @pytest.mark.timeout(2) def test_time_out_2(self): """ Timeout should be overriden """ counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) assert counter <= 3 @pytest.mark.timeout(5) def test_time_out_5(self): """ Timeout should be overriden """ counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) assert counter <= 6 And the output is (I know know, I should have use funcargs but I was lazy :)) def test_time_out(): """ Should timeout only with the default timeout. """ counter = 0 while(1): print "hanging ...", counter counter += 1 > time.sleep(1) E Failed: Timeout >8s test/test_sample.py:14: Failed _________________________________________________________________________________________ TestTimeout.test_time_out_2 _________________________________________________________________________________________ self = @pytest.mark.timeout(2) def test_time_out_2(self): """ Timeout should be overriden """ counter = 0 while(1): print "hanging ...", counter counter += 1 > time.sleep(1) E Failed: Timeout >2s test/test_sample.py:25: Failed ________________________________________________________________________________________ TestTimeout.test_time_out_5 _________________________________________________________________________________________ self = @pytest.mark.timeout(5) def test_time_out_5(self): """ Timeout should be overriden """ counter = 0 while(1): print "hanging ...", counter counter += 1 > time.sleep(1) E Failed: Timeout >5s Cheers /Laurent -----Original Message----- From: py-dev-bounces at codespeak.net on behalf of Brack, Laurent P. Sent: Wed 3/7/2012 10:36 AM To: Floris Bruynooghe Cc: Brard, Valentin; py-dev at codespeak.net Subject: Re: [py-dev] New plugin: pytest-timeout Hello Floris, Thanks for your prompt reply (and sorry for my delayed one :). I haven't had time to play more with the plugin but my observations (and Valentin in CC) has been that neither method was working for us. I haven't done my homework in terms of investigating this further (but I will next week) however, to answer your question the infinite loop looked like this: ------------------------------------------ import time def test_time_out(): counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) ------------------------------------------ Neither with or without sigalarm worked (I am running on Ubuntu as I mentioned and Valentine tried it on Windows). Based on the code above, I think that both methods should work in that case. Don't worry about it as I will try to figure out what is going on (unless you can think of something). It is however a very nice addition to pytest, as in early development stages things do have a tendency to hang. Thanks /Laurent -----Original Message----- From: floris.bruynooghe at gmail.com on behalf of Floris Bruynooghe Sent: Wed 2/29/2012 2:31 AM To: Brack, Laurent P. Cc: py-dev at codespeak.net Subject: Re: [py-dev] New plugin: pytest-timeout Hello Laurent, On 28 February 2012 18:08, Brack, Laurent P. wrote: > Just wrote an infinite loop which never got preempted. I didn't have time to > dig any deeper. Does the test need to be executed in a separate process via > xdist? xdist is not needed, in fact I haven't tried it but assume it will behave fine. By default SIGALM will be used which does have it's limitations, the main issue is that it needs a chance to deliver the signal to python code. So depending on how your infinite loop is written this might be an issue and you may have to consider --nosigalrm which will use a timer thread (this is more expensive obviously). Could you show me the test function you where trying out? > Also, currently the time out applies to all tests (and defaults to 5 > minutes). I think it would be useful to be able to override the timeout > at the test level using markers. If something goes wrong and we have > thousands of tests (we use pytest to test embedded systems), this is an > awful lot of time to realize that things are DOA. Yes, I agree with this and it is noted in the TODO file that individual tests should have some control on the timeout and the mechanism used (timer vs sigalrm). But we only have about 10000 tests running nightly with normally none blocking, this plugin was written on the spur of the moment when we had about a dozen hanging and this minimal functionality seemed to do the job. It does assume hanging tests are an exception rather then the norm, certainly evident from the behaviour of --nosigalrm where os._exit() is called on a hanging test. I'm happy to work with you on adding a marker which will fulfil your needs however. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20120307/7f74e040/attachment-0001.htm From holger at merlinux.eu Wed Mar 7 22:55:49 2012 From: holger at merlinux.eu (holger krekel) Date: Wed, 7 Mar 2012 21:55:49 +0000 Subject: [py-dev] New plugin: pytest-timeout In-Reply-To: References: Message-ID: <20120307215549.GH26028@merlinux.eu> Hi Floris, thanks for the plugin! looking at version 0.1 i wonder why you didn't go with always activating the plugin so not requiring any pytest_plugins setting. The default could be "no timeout" or None which could be modified with: [pytest] timeout = 2.3 # secs or with a marker. Not using some implicit magic number is anyway a good idea i think. best, holger On Sun, Feb 26, 2012 at 23:19 +0000, Floris Bruynooghe wrote: > Hi all, > > This is to announce the first release of pytest-timeout, a plugin > which will interrupt long running (i.e. blocking) tests. This is > particularly useful when running the tests on a CI host. When a test > is interrupted the stacks of all threads are dumped to stderr, which > helps you to locate the reason for the blocking. If the system > supports SIGALRM then the test itself is interrupted using > pytest.fail() and other tests continue to run, otherwise the stack(s) > are dumped to stderr and the process exists immediately. > > The plugin is available from the cheeseshop: > http://pypi.python.org/pypi/pytest-timeout and the code lives at > bitbucket: https://bitbucket.org/flub/pytest-timeout/. Feel free to > provide any feedback or report issues. > > Regards, > Floris > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From Ronny.Pfannschmidt at gmx.de Thu Mar 8 00:35:08 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 08 Mar 2012 00:35:08 +0100 Subject: [py-dev] New plugin: pytest-timeout In-Reply-To: <20120307215549.GH26028@merlinux.eu> References: <20120307215549.GH26028@merlinux.eu> Message-ID: <4F57F0AC.7000603@gmx.de> Hi, in addition something for eventlet/gevent based timeout objects as well as a ini option to choose the timeout method between signal/thread, gevent, eventlet could be nice best, ronny On 03/07/2012 10:55 PM, holger krekel wrote: > Hi Floris, > > thanks for the plugin! > > looking at version 0.1 i wonder why you didn't go with always activating > the plugin so not requiring any pytest_plugins setting. The default > could be "no timeout" or None which could be modified with: > > [pytest] > timeout = 2.3 # secs > > or with a marker. Not using some implicit magic number is anyway a good > idea i think. > > best, > holger > > On Sun, Feb 26, 2012 at 23:19 +0000, Floris Bruynooghe wrote: >> Hi all, >> >> This is to announce the first release of pytest-timeout, a plugin >> which will interrupt long running (i.e. blocking) tests. This is >> particularly useful when running the tests on a CI host. When a test >> is interrupted the stacks of all threads are dumped to stderr, which >> helps you to locate the reason for the blocking. If the system >> supports SIGALRM then the test itself is interrupted using >> pytest.fail() and other tests continue to run, otherwise the stack(s) >> are dumped to stderr and the process exists immediately. >> >> The plugin is available from the cheeseshop: >> http://pypi.python.org/pypi/pytest-timeout and the code lives at >> bitbucket: https://bitbucket.org/flub/pytest-timeout/. Feel free to >> provide any feedback or report issues. >> >> Regards, >> Floris >> >> -- >> Debian GNU/Linux -- The Power of Freedom >> www.debian.org | www.gnu.org | www.kernel.org >> _______________________________________________ From flub at devork.be Thu Mar 8 23:49:46 2012 From: flub at devork.be (Floris Bruynooghe) Date: Thu, 8 Mar 2012 22:49:46 +0000 Subject: [py-dev] New plugin: pytest-timeout In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE42A39E98@sfo-exch-01.dolby.net> References: <05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net> <05BB196AB3DA6C4BBE11AB6C957581FE42A39E96@sfo-exch-01.dolby.net> <05BB196AB3DA6C4BBE11AB6C957581FE42A39E98@sfo-exch-01.dolby.net> Message-ID: Hello Laurent, On 7 March 2012 20:56, Brack, Laurent P. wrote: > Somehow (and I don't know why), registering the plugin as 'timeout' instead > of 'pytest_timeout' was the cause of the problem, > so for it to work on my side, I had to do: > > config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout') Ah, I probably got something wrong with the setuptool entrypoints or something. I'll look into it, thanks for finding this! > I also played with the ability to use markers to override the default > timeout at the test level (here is an excerpt of > your code modified) [...] Thanks for this, I'll incorporate the timeout marker if you don't mind. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From flub at devork.be Fri Mar 9 00:08:20 2012 From: flub at devork.be (Floris Bruynooghe) Date: Thu, 8 Mar 2012 23:08:20 +0000 Subject: [py-dev] New plugin: pytest-timeout In-Reply-To: <20120307215549.GH26028@merlinux.eu> References: <20120307215549.GH26028@merlinux.eu> Message-ID: On 7 March 2012 21:55, holger krekel wrote: > looking at version 0.1 i wonder why you didn't go with always activating > the plugin so not requiring any pytest_plugins setting. ?The default > could be "no timeout" or None which could be modified with: > > ? ?[pytest] > ? ?timeout = 2.3 # secs This simply didn't occur to me, probably because personally I don't install it but just have the module included in the root of our project. Thanks for the idea. > or with a marker. ?Not using some implicit magic number is anyway a good > idea i think. Not sure what you mean. Do you mean using 0 as saying "no timeout" is a magic number but e.g. None is fine? Essentially anything else other then a positive number is "disable timeout" to me. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From flub at devork.be Fri Mar 9 00:17:19 2012 From: flub at devork.be (Floris Bruynooghe) Date: Thu, 8 Mar 2012 23:17:19 +0000 Subject: [py-dev] New plugin: pytest-timeout In-Reply-To: <4F57F0AC.7000603@gmx.de> References: <20120307215549.GH26028@merlinux.eu> <4F57F0AC.7000603@gmx.de> Message-ID: On 7 March 2012 23:35, Ronny Pfannschmidt wrote: > in addition something for eventlet/gevent based timeout objects > as well as a ini option to choose the timeout method between signal/thread, > gevent, eventlet could be nice eventlet/gevent timeouts are also a good idea, I'll look into adding these as well. One issue with this however is that I hate calling import from inside a function (partially because I've looked too close at eventlet's patching code) but unconditionally importing it upfront, in a try-except block, woud be bad for py.test's startup time. Would importing in a function be reasonable in this case? Or am I missing another solution? Thanks for all the feedback everyone! Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From holger at merlinux.eu Fri Mar 9 00:32:11 2012 From: holger at merlinux.eu (holger krekel) Date: Thu, 8 Mar 2012 23:32:11 +0000 Subject: [py-dev] New plugin: pytest-timeout In-Reply-To: References: <20120307215549.GH26028@merlinux.eu> Message-ID: <20120308233211.GI26028@merlinux.eu> On Thu, Mar 08, 2012 at 23:08 +0000, Floris Bruynooghe wrote: > On 7 March 2012 21:55, holger krekel wrote: > > or with a marker. ?Not using some implicit magic number is anyway a good > > idea i think. > > Not sure what you mean. Do you mean using 0 as saying "no timeout" is > a magic number but e.g. None is fine? Essentially anything else other > then a positive number is "disable timeout" to me. just meant "5 seconds" as a default. 0 or None is fine to mean "no timeout" IMHO. best, holger > Regards, > Floris > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > From Ronny.Pfannschmidt at gmx.de Fri Mar 9 00:42:31 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 09 Mar 2012 00:42:31 +0100 Subject: [py-dev] New plugin: pytest-timeout In-Reply-To: References: <20120307215549.GH26028@merlinux.eu> <4F57F0AC.7000603@gmx.de> Message-ID: <4F5943E7.5080204@gmx.de> On 03/09/2012 12:17 AM, Floris Bruynooghe wrote: > On 7 March 2012 23:35, Ronny Pfannschmidt wrote: >> in addition something for eventlet/gevent based timeout objects >> as well as a ini option to choose the timeout method between signal/thread, >> gevent, eventlet could be nice > > eventlet/gevent timeouts are also a good idea, I'll look into adding > these as well. > > One issue with this however is that I hate calling import from inside > a function (partially because I've looked too close at eventlet's > patching code) but unconditionally importing it upfront, in a > try-except block, woud be bad for py.test's startup time. Would > importing in a function be reasonable in this case? Or am I missing > another solution? > Lets go with the assumption that whoever is requesting a gevent/eventlet timeout knows enough of what hes doing (since it would need a config entry) then its reasonable enough to just grab the timeouts in the function wrt monkeypatching, i think its strictly opt-in, and should also be left to the user best, Ronny > > Thanks for all the feedback everyone! > > Floris > > From lpbrac at dolby.com Fri Mar 9 01:28:14 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Thu, 8 Mar 2012 16:28:14 -0800 Subject: [py-dev] New plugin: pytest-timeout References: <05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net><05BB196AB3DA6C4BBE11AB6C957581FE42A39E96@sfo-exch-01.dolby.net><05BB196AB3DA6C4BBE11AB6C957581FE42A39E98@sfo-exch-01.dolby.net> Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A39EDF@sfo-exch-01.dolby.net> Please be my guest and use it as we are using your code. Thanks a lot for that plugin by the way. -----Original Message----- From: floris.bruynooghe at gmail.com on behalf of Floris Bruynooghe Sent: Thu 3/8/2012 2:49 PM To: Brack, Laurent P. Cc: Brard, Valentin; py-dev at codespeak.net Subject: Re: [py-dev] New plugin: pytest-timeout Hello Laurent, On 7 March 2012 20:56, Brack, Laurent P. wrote: > Somehow (and I don't know why), registering the plugin as 'timeout' instead > of 'pytest_timeout' was the cause of the problem, > so for it to work on my side, I had to do: > > config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout') Ah, I probably got something wrong with the setuptool entrypoints or something. I'll look into it, thanks for finding this! > I also played with the ability to use markers to override the default > timeout at the test level (here is an excerpt of > your code modified) [...] Thanks for this, I'll incorporate the timeout marker if you don't mind. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20120308/564dce4f/attachment.htm From flub at devork.be Sat Mar 17 18:31:34 2012 From: flub at devork.be (Floris Bruynooghe) Date: Sat, 17 Mar 2012 18:31:34 +0100 Subject: [py-dev] pytest-timeout 0.2 Message-ID: Hi all, I've made a second release of the pytest-timeout plugin for py.test which can time out long running tests. This release includes a number of suggestions made on this list, major changes include: * Fixed the activation problem * Set timeout using configuration file * Add a timeout marker to modify timeout of one item * The marker can also choose the method (signal/thread) * Renamed --nosigalrm to --timeout_method to future proof adding of eventlet and gevent timeout methods * Works on python 3, tested on 2.6, 2.7 and 3.2 Not yet done: * Automatic enabling of the plugin, you still need to enable it on the command line or configuration file before you can use the marker. This was probably a bad idea but I felt bad about stealing a marker by default. * eventlet and gevent timeouts As before the release is on pypi: http://pypi.python.org/pypi/pytest-timeout and the development repository and issue tracker on bitbucket: https://bitbucket.org/flub/pytest-timeout/ I'd be pleased to receive any further feedback you may have. Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From holger at merlinux.eu Sun Mar 18 00:16:27 2012 From: holger at merlinux.eu (holger krekel) Date: Sat, 17 Mar 2012 23:16:27 +0000 Subject: [py-dev] pytest-timeout 0.2 In-Reply-To: References: Message-ID: <20120317231627.GT26028@merlinux.eu> Hi Floris, On Sat, Mar 17, 2012 at 18:31 +0100, Floris Bruynooghe wrote: > I've made a second release of the pytest-timeout plugin for py.test > which can time out long running tests. This release includes a number > of suggestions made on this list, major changes include: > > * Fixed the activation problem > * Set timeout using configuration file > * Add a timeout marker to modify timeout of one item > * The marker can also choose the method (signal/thread) > * Renamed --nosigalrm to --timeout_method to future proof adding of > eventlet and gevent timeout methods > * Works on python 3, tested on 2.6, 2.7 and 3.2 > > Not yet done: > > * Automatic enabling of the plugin, you still need to enable it on the > command line or configuration file before you can use the marker. > This was probably a bad idea but I felt bad about stealing a marker by > default. I went ahead and created a test function with @pytest.mark.timeout(1) def test_hello(): ... but the timeout was not honoured. Then i skimmed the docs :) added "timeout_method = signal" to my ini-file and ran, still not honoured. Then i figured i need to set some dummy "timeout = 10" in the ini - and now i get the proper timeout of 1 second. I understand the hesitance to grab a general name like "timeout" but then again installing "pytest-timeout" is a deliberate act and it grabbing the "timeout" marker is not surprising IMO. So i'd kindly encourage you to go for it. I wonder btw. if the output of "--markers" should be merged with "--help". The latter would get yet longer but then again it's nice to have all the info at a fingertip. Another feedback item: "@pytest.mark.timeout(5, 'signal')" ought to work. It's slightly awkward because of the marker args/kwargs API but it's expected from a pure user perspective i think. Moreover i'd eventually like to include the timeout plugin in pytest core. It's an important feature for functional testing. > * eventlet and gevent timeouts Here is what i did for eventlet (only accessing the decorator here): https://bitbucket.org/hpk42/detox/src/f9f8c0107cc1/tests/conftest.py#cl-108 cheers & thanks, holger > > As before the release is on pypi: > http://pypi.python.org/pypi/pytest-timeout and the development > repository and issue tracker on bitbucket: > https://bitbucket.org/flub/pytest-timeout/ > > I'd be pleased to receive any further feedback you may have. > > Floris > > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From flub at devork.be Mon Mar 19 00:24:07 2012 From: flub at devork.be (Floris Bruynooghe) Date: Mon, 19 Mar 2012 00:24:07 +0100 Subject: [py-dev] pytest-timeout 0.2 In-Reply-To: <20120317231627.GT26028@merlinux.eu> References: <20120317231627.GT26028@merlinux.eu> Message-ID: Hello Holger, On 18 March 2012 00:16, holger krekel wrote: > I went ahead and created a test function with > > ? ?@pytest.mark.timeout(1) > ? ?def test_hello(): > ? ? ? ?... > > but the timeout was not honoured. ?Then i skimmed the docs :) > added "timeout_method = signal" to my ini-file and ran, still not honoured. > Then i figured i need to set some dummy "timeout = 10" in the ini - and now > i get the proper timeout of 1 second. Actually timeout=0 would be fine too. > I understand the hesitance to grab a general name like "timeout" but then > again installing "pytest-timeout" is a deliberate act and it grabbing the > "timeout" marker is not surprising IMO. ?So i'd kindly encourage you to > go for it. I wonder btw. if the output of "--markers" should be merged with > "--help". ?The latter would get yet longer but then again it's nice to > have all the info at a fingertip. In fact I had no idea --markers existed, I was considering brining this up but I just hadn't looked around enough! This does make me feel a lot better about using the timeout marker and you've convinced me it's outweighs the stumbling blocks of requiring --timeout=0. > Another feedback item: "@pytest.mark.timeout(5, 'signal')" ought to work. > It's slightly awkward because of the marker args/kwargs API but it's expected > from a pure user perspective i think. Yes, I did consider that but then python3 allows keyword only arguments and this seemed a reasonable candidate while the api to make it lends itself to it naturally. But don't see any problem with making it positional as well and am happy to add this. > Moreover i'd eventually like to include the timeout plugin > in pytest core. ?It's an important feature for functional testing. That would be nice! But as you implied it probably needs to mature a bit more outside of pytest core first. >> * eventlet and gevent timeouts > > Here is what i did for eventlet (only accessing the decorator here): > > https://bitbucket.org/hpk42/detox/src/f9f8c0107cc1/tests/conftest.py#cl-108 Thanks for that hint, hadn't started looking yet but this will save me some experimenting. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From tetsuya.morimoto at gmail.com Mon Mar 19 04:23:33 2012 From: tetsuya.morimoto at gmail.com (Tetsuya Morimoto) Date: Sun, 18 Mar 2012 20:23:33 -0700 Subject: [py-dev] pytest-quickcheck plugin Message-ID: Hi, I've made my first pytest plugin named pytest-quickcheck for generating random data. http://pypi.python.org/pypi/pytest-quickcheck/ Originally, this idea is inspired by Haskell QuickCheck (and paycheck, its Python implementation). http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck http://pypi.python.org/pypi/paycheck/ I think it's useful to make sure of the robustness by generating lots of test-data for data-driven test. However, this plugin is still experimental beta, so I'll welcome any comments or suggestions on. thanks, Tetsuya From lpbrac at dolby.com Mon Mar 19 17:38:47 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Mon, 19 Mar 2012 09:38:47 -0700 Subject: [py-dev] pytest-timeout 0.2 In-Reply-To: References: <20120317231627.GT26028@merlinux.eu> Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net> To have this part of the core pytest would be great (even unit tests can hang). It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one. Cheers /Laurent -----Original Message----- From: py-dev-bounces at codespeak.net [mailto:py-dev-bounces at codespeak.net] On Behalf Of Floris Bruynooghe Sent: Sunday, March 18, 2012 4:24 PM To: holger krekel; Floris Bruynooghe; py-dev at codespeak.net Subject: Re: [py-dev] pytest-timeout 0.2 Hello Holger, On 18 March 2012 00:16, holger krekel wrote: > I went ahead and created a test function with > > ? ?@pytest.mark.timeout(1) > ? ?def test_hello(): > ? ? ? ?... > > but the timeout was not honoured. ?Then i skimmed the docs :) added > "timeout_method = signal" to my ini-file and ran, still not honoured. > Then i figured i need to set some dummy "timeout = 10" in the ini - > and now i get the proper timeout of 1 second. Actually timeout=0 would be fine too. > I understand the hesitance to grab a general name like "timeout" but > then again installing "pytest-timeout" is a deliberate act and it > grabbing the "timeout" marker is not surprising IMO. ?So i'd kindly > encourage you to go for it. I wonder btw. if the output of "--markers" > should be merged with "--help". ?The latter would get yet longer but > then again it's nice to have all the info at a fingertip. In fact I had no idea --markers existed, I was considering brining this up but I just hadn't looked around enough! This does make me feel a lot better about using the timeout marker and you've convinced me it's outweighs the stumbling blocks of requiring --timeout=0. > Another feedback item: "@pytest.mark.timeout(5, 'signal')" ought to work. > It's slightly awkward because of the marker args/kwargs API but it's > expected from a pure user perspective i think. Yes, I did consider that but then python3 allows keyword only arguments and this seemed a reasonable candidate while the api to make it lends itself to it naturally. But don't see any problem with making it positional as well and am happy to add this. > Moreover i'd eventually like to include the timeout plugin in pytest > core. ?It's an important feature for functional testing. That would be nice! But as you implied it probably needs to mature a bit more outside of pytest core first. >> * eventlet and gevent timeouts > > Here is what i did for eventlet (only accessing the decorator here): > > https://bitbucket.org/hpk42/detox/src/f9f8c0107cc1/tests/conftest.py#c > l-108 Thanks for that hint, hadn't started looking yet but this will save me some experimenting. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org _______________________________________________ py-dev mailing list py-dev at codespeak.net http://codespeak.net/mailman/listinfo/py-dev From flub at devork.be Mon Mar 19 23:20:56 2012 From: flub at devork.be (Floris Bruynooghe) Date: Mon, 19 Mar 2012 22:20:56 +0000 Subject: [py-dev] pytest-timeout 0.2 In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net> References: <20120317231627.GT26028@merlinux.eu> <05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net> Message-ID: On 19 March 2012 16:38, Brack, Laurent P. wrote: > It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one. I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal. I might investigate the feasibility of such an approach. I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option. If anyone has any other ideas about this please let me know. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From lpbrac at dolby.com Tue Mar 20 00:47:36 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Mon, 19 Mar 2012 16:47:36 -0700 Subject: [py-dev] pytest-timeout 0.2 In-Reply-To: References: <20120317231627.GT26028@merlinux.eu><05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net> Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE445B4106@sfo-exch-01.dolby.net> I don't want to make this issue overly complicated but here are some scenarios (that we have). Our tests, in some instances, interact with python extensions written in C or C++ (or in some case COM inproc server - yuk) which can be multi-threaded. If the hang occurs in a critical section in the C/C++ domain I am not sure what the timeout effect would be but Assuming that the test is pre-empted in a critical section (of the extension), we would most likely be doomed anyway (mutexes would not be released). Next test would hang, be aborted and so on. In our case, the safest thing to do is to run with xdist -n 1 so then the OS takes care of doing most if not all the cleanup. I am not so intimate with xdist but I was hoping that upon detecting that the "remote process" had died, xdist would restart a new one. In this case pytest ended with: [gw0] win32 -- Python 2.7.2 C:\Python27\python.exe Slave 'gw0' crashed while running 'src/timeout/test_sample.py::TestTimeOut::()::test_hang' Which signaled the end of everything (other tests were not run). This is a something we will have to solve internally (and when we have the solution will be more than happy to contribute it back). We might be able to restart a remote Process on pytest_testnodedown(). We would be however too late to run any restore() and chances are that those might hang as well. More food for thoughts than anything at this point. Floris, thanks for your work. We are not yet in a position where we can help much but we will get to it. /Laurent P.S.: Speaking of xdist, further down the road we are interested in extending xdist (via its hooks) to interface it with OpenStack (dynamic provisioning of VM). Anyone has any use for this or thinking about the same thing? -----Original Message----- From: floris.bruynooghe at gmail.com [mailto:floris.bruynooghe at gmail.com] On Behalf Of Floris Bruynooghe Sent: Monday, March 19, 2012 3:21 PM To: Brack, Laurent P. Cc: holger krekel; py-dev at codespeak.net; Brard, Valentin Subject: Re: [py-dev] pytest-timeout 0.2 On 19 March 2012 16:38, Brack, Laurent P. wrote: > It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one. I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal. I might investigate the feasibility of such an approach. I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option. If anyone has any other ideas about this please let me know. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From holger at merlinux.eu Tue Mar 20 07:10:12 2012 From: holger at merlinux.eu (holger krekel) Date: Tue, 20 Mar 2012 06:10:12 +0000 Subject: [py-dev] pytest-timeout 0.2 In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE445B4106@sfo-exch-01.dolby.net> References: <20120317231627.GT26028@merlinux.eu> <05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net> <05BB196AB3DA6C4BBE11AB6C957581FE445B4106@sfo-exch-01.dolby.net> Message-ID: <20120320061012.GC26028@merlinux.eu> Hi Laurent, Floris, On Mon, Mar 19, 2012 at 16:47 -0700, Brack, Laurent P. wrote: > I don't want to make this issue overly complicated but here are some scenarios (that we have). proper termination and handling of hanging tests is involved. So discussion and hearing use cases makes sense to me. > Our tests, in some instances, interact with python extensions written in C or C++ (or in some case COM inproc server - yuk) which can be multi-threaded. > > If the hang occurs in a critical section in the C/C++ domain I am not sure what the timeout effect would be but > Assuming that the test is pre-empted in a critical section (of the extension), we would most likely be doomed anyway (mutexes would not be released). Next test would hang, be aborted and so on. > > In our case, the safest thing to do is to run with xdist -n 1 so then the OS takes care of doing most if not all the cleanup. > I am not so intimate with xdist but I was hoping that upon detecting that the "remote process" had died, > xdist would restart a new one. It's reasonable to expect this from the xdist plugin, even when you run with "-n N" with N>1. > In this case pytest ended with: > > [gw0] win32 -- Python 2.7.2 C:\Python27\python.exe > Slave 'gw0' crashed while running 'src/timeout/test_sample.py::TestTimeOut::()::test_hang' > > Which signaled the end of everything (other tests were not run). This is a something we will have to solve internally > (and when we have the solution will be more than happy to contribute it back). We might be able to restart a remote > Process on pytest_testnodedown(). We would be however too late to run any restore() and chances are that those might hang as well. Indeed, no attempt is made to restart a node properly. It remains offline and when run with "-n1" everything dies because no node is left to continue. Not sure how hard it is to try to fix it. > More food for thoughts than anything at this point. > > Floris, thanks for your work. We are not yet in a position where we can help much but we will get to it. Coming to think of it, the xdist plugin might also do "timeout" handling and send a keyboard interrupt from the outside to a running node. > /Laurent > > P.S.: Speaking of xdist, further down the road we are interested in extending xdist (via its hooks) to interface it with OpenStack (dynamic provisioning of VM). Anyone has any use for this or thinking about the same thing? I am interested in OpenStack but you can you detail a bit more of what you want to achieve? best, holger > > -----Original Message----- > From: floris.bruynooghe at gmail.com [mailto:floris.bruynooghe at gmail.com] On Behalf Of Floris Bruynooghe > Sent: Monday, March 19, 2012 3:21 PM > To: Brack, Laurent P. > Cc: holger krekel; py-dev at codespeak.net; Brard, Valentin > Subject: Re: [py-dev] pytest-timeout 0.2 > > On 19 March 2012 16:38, Brack, Laurent P. wrote: > > It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one. > > I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal. I might investigate the feasibility of such an approach. I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option. > > If anyone has any other ideas about this please let me know. > > Regards, > Floris > > > -- > Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From lpbrac at dolby.com Tue Mar 20 17:43:46 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Tue, 20 Mar 2012 09:43:46 -0700 Subject: [py-dev] pytest-timeout 0.2 References: <20120317231627.GT26028@merlinux.eu> <05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net> <05BB196AB3DA6C4BBE11AB6C957581FE445B4106@sfo-exch-01.dolby.net> <20120320061012.GC26028@merlinux.eu> Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A081@sfo-exch-01.dolby.net> Hi Holger, I am not saying there is anything wrong with the way xdist is handling a node going down at this point. the -n 1 was just my attempt to see if the timeout plugin would behave differently when running the test out of process. I think it did, as pytest did not exit the way it does without xdist ... it just ran out of nodes this time and could not carry forward. > Coming to think of it, the xdist plugin might also do "timeout" handling > and send a keyboard interrupt from the outside to a running node. I think this is a good way of handling this. As Floris pointed out, there is always a danger of using signal (right now his marker allows to switch modes on a per test basis which is a good thing). > I am interested in OpenStack but you can you detail a bit more of what > you want to achieve? I will start a separate thread - OpenStack (as it is somewhat unrelated to the timeout subject). /Laurent -----Original Message----- From: holger krekel [mailto:holger at merlinux.eu] Sent: Mon 3/19/2012 11:10 PM To: Brack, Laurent P. Cc: Floris Bruynooghe; holger krekel; py-dev at codespeak.net; Brard, Valentin Subject: Re: [py-dev] pytest-timeout 0.2 Hi Laurent, Floris, On Mon, Mar 19, 2012 at 16:47 -0700, Brack, Laurent P. wrote: > I don't want to make this issue overly complicated but here are some scenarios (that we have). proper termination and handling of hanging tests is involved. So discussion and hearing use cases makes sense to me. > Our tests, in some instances, interact with python extensions written in C or C++ (or in some case COM inproc server - yuk) which can be multi-threaded. > > If the hang occurs in a critical section in the C/C++ domain I am not sure what the timeout effect would be but > Assuming that the test is pre-empted in a critical section (of the extension), we would most likely be doomed anyway (mutexes would not be released). Next test would hang, be aborted and so on. > > In our case, the safest thing to do is to run with xdist -n 1 so then the OS takes care of doing most if not all the cleanup. > I am not so intimate with xdist but I was hoping that upon detecting that the "remote process" had died, > xdist would restart a new one. It's reasonable to expect this from the xdist plugin, even when you run with "-n N" with N>1. > In this case pytest ended with: > > [gw0] win32 -- Python 2.7.2 C:\Python27\python.exe > Slave 'gw0' crashed while running 'src/timeout/test_sample.py::TestTimeOut::()::test_hang' > > Which signaled the end of everything (other tests were not run). This is a something we will have to solve internally > (and when we have the solution will be more than happy to contribute it back). We might be able to restart a remote > Process on pytest_testnodedown(). We would be however too late to run any restore() and chances are that those might hang as well. Indeed, no attempt is made to restart a node properly. It remains offline and when run with "-n1" everything dies because no node is left to continue. Not sure how hard it is to try to fix it. > More food for thoughts than anything at this point. > > Floris, thanks for your work. We are not yet in a position where we can help much but we will get to it. Coming to think of it, the xdist plugin might also do "timeout" handling and send a keyboard interrupt from the outside to a running node. > /Laurent > > P.S.: Speaking of xdist, further down the road we are interested in extending xdist (via its hooks) to interface it with OpenStack (dynamic provisioning of VM). Anyone has any use for this or thinking about the same thing? I am interested in OpenStack but you can you detail a bit more of what you want to achieve? best, holger > > -----Original Message----- > From: floris.bruynooghe at gmail.com [mailto:floris.bruynooghe at gmail.com] On Behalf Of Floris Bruynooghe > Sent: Monday, March 19, 2012 3:21 PM > To: Brack, Laurent P. > Cc: holger krekel; py-dev at codespeak.net; Brard, Valentin > Subject: Re: [py-dev] pytest-timeout 0.2 > > On 19 March 2012 16:38, Brack, Laurent P. wrote: > > It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one. > > I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal. I might investigate the feasibility of such an approach. I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option. > > If anyone has any other ideas about this please let me know. > > Regards, > Floris > > > -- > Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20120320/37a501a7/attachment-0001.htm From lpbrac at dolby.com Tue Mar 20 18:27:00 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Tue, 20 Mar 2012 10:27:00 -0700 Subject: [py-dev] OpenStack Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A082@sfo-exch-01.dolby.net> Forking from the [py-dev] pytest-timeout 0.2 e-mail thread. > I am interested in OpenStack but you can you detail a bit more of what > you want to achieve? I have attached a rough diagram of what we are building internally (hopefully it will not be filtered out). About a year ago, we attended a presentation on OpenStack (when it was still driven by Anso Labs before they got acquired by Rackspace). We were (and are) currently using a private cloud (VMWare) but contemplated the idea of scaling up to public clouds. Problem we had was that we had to develop custom code for each cloud vendor whether Amazon EC2, Penguin, VMWare, etc. so OpenStack was really a viable path to not lock ourselves in with a given vendor. While a bulk of our tests require embedded devices (in which case virtualization makes little sense) a good chunk can be run on standard workstations (all OS flavors) and in this case it only makes sense to move to a virtual environment. PyTest combined with xdist was the dream come true as we could focus on developing tests meant to run on a single machine and later seamlessly parallelize their execution. There is nothing new here. We were thinking of writing a plugin to xdist (cxdist - c for cloud) that would interface with OpenStack, using the pytest_xdist_setupnodes and pytest_configure_node hooks. In those hooks, the plugin would provision the machine (via OpenStack) and then make xdist believe that it is dealing with physical slaves. Finally at the end of the run, we would teardown the slaves, leaving resources for other tests to run. We have not started yet on this as scalability is not an issue but this is our plan. As the diagram shows, the read boxes are plugins we intend to release to the open source community. Our teams write a lot of "data driven tests" (testing various AV codecs with different configuration parameters using very similar verification procedure) and as a result we make heavy use of funcargs. The pytest_testlink plugin is very similar to the pytest Case Conductor plugin from the Mozilla folks (http://blargon7.com/2012/01/case-conductor-pytest-plugin-proposal/) but interacts with TestLink (teamst.org in which we are currently involved in - improving performance of web services etc.) as opposed to Case Conductor. However in addition to linking a test case ID to a given python test, we have implemented a simple mechanism to generate test cases from what we called Test Description Records. Those are currently stored in excel spreadsheets but will be store in TestLink as test case properties. The object injection is done automatically by the plugin using a custom marker indicating which function argument shall be used for TDR injection as well as correlating TDR with parametrized test functions. The point is that by using the funcarg feature offered by pytest, we can generate tremendous amount of test cases with little effort and this will require scalability and use of virtualization over time. Sorry for the long description, but I wanted to give a complete overview to help comprehend where we are going. Our plan is to eventually release all this to the open source community (when we think the plugins are mature and robust enough). If other people have a need/interest for those plugins (some of them are not illustrated), I would be more than happy to accelerate this process (getting approval from our Open Source Board). Since pytest_cxdist doesn't exist, this could be open source from the get go. Best Laurent -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20120320/b99c4f2a/attachment-0001.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: GTI Plugin.png Type: image/png Size: 59280 bytes Desc: GTI Plugin.png Url : http://codespeak.net/pipermail/py-dev/attachments/20120320/b99c4f2a/attachment-0001.png From holger at merlinux.eu Sat Mar 24 14:41:58 2012 From: holger at merlinux.eu (holger krekel) Date: Sat, 24 Mar 2012 13:41:58 +0000 Subject: [py-dev] OpenStack In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A082@sfo-exch-01.dolby.net> References: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A082@sfo-exch-01.dolby.net> Message-ID: <20120324134157.GM26028@merlinux.eu> Hi Brack, sorry for taking a while. Am still on travel, currently in the Sonoran deserts. My answers up until first half May are thus likely to lag. (They do have suprisingly good internet connection ... better than in the three-times more expensive hotel in the San Francisco valley last week). On Tue, Mar 20, 2012 at 10:27 -0700, Brack, Laurent P. wrote: > Forking from the [py-dev] pytest-timeout 0.2 e-mail thread. > > > I am interested in OpenStack but you can you detail a bit more of what > > you want to achieve? > > I have attached a rough diagram of what we are building internally (hopefully it will not be filtered out). thanks for sharing. > About a year ago, we attended a presentation on OpenStack (when it was still driven by Anso Labs before they got acquired by Rackspace). > We were (and are) currently using a private cloud (VMWare) but contemplated the idea of scaling up to public clouds. Problem we had was that > we had to develop custom code for each cloud vendor whether Amazon EC2, Penguin, VMWare, etc. so OpenStack was really a viable path to not lock > ourselves in with a given vendor. sounds sensible. > While a bulk of our tests require embedded devices (in which case virtualization makes little sense) a good chunk can be run > on standard workstations (all OS flavors) and in this case it only makes sense to move to a virtual environment. > > PyTest combined with xdist was the dream come true as we could focus on developing tests meant to run on > a single machine and later seamlessly parallelize their execution. There is nothing new here. > > We were thinking of writing a plugin to xdist (cxdist - c for cloud) that would interface with OpenStack, using the pytest_xdist_setupnodes > and pytest_configure_node hooks. ok. > In those hooks, the plugin would provision the machine (via OpenStack) and then make xdist believe that it is dealing with > physical slaves. > Finally at the end of the run, we would teardown the slaves, leaving resources for other tests to run. > > We have not started yet on this as scalability is not an issue but this is our plan. As the diagram shows, the read boxes > are plugins we intend to release to the open source community. Is your general idea to use the open stack integration to get parallelizable test runs with totally controllable environments? > Our teams write a lot of "data driven tests" (testing various AV codecs with different configuration parameters using very similar verification procedure) > and as a result we make heavy use of funcargs. Curious, are you using 2.2.3 for this? There are some plans to further improve parametrization where i would be interested in your feedback. > The pytest_testlink plugin is very similar to the pytest Case Conductor plugin from the Mozilla folks (http://blargon7.com/2012/01/case-conductor-pytest-plugin-proposal/) but interacts with TestLink (teamst.org in which we are currently involved in - improving performance of web services etc.) as opposed to Case Conductor. Is pytest-testlink a plugin that you plan to work on? > However in addition to linking a test case ID to a given python test, we have implemented a simple mechanism to generate test cases > from what we called Test Description Records. Those are currently stored in excel spreadsheets but will be store in TestLink as > test case properties. The object injection is done automatically by the plugin using a custom marker indicating which function argument shall > be used for TDR injection as well as correlating TDR with parametrized test functions. right. Driving things from a spreadsheet, maybe even in Google docs, sounds good to me, btw :) > The point is that by using the funcarg feature offered by pytest, we can generate tremendous amount of test cases with little effort > and this will require scalability and use of virtualization over time. > > Sorry for the long description, but I wanted to give a complete overview to help comprehend where we are going. > Our plan is to eventually release all this to the open source community (when we think the plugins are mature and robust enough). > > If other people have a need/interest for those plugins (some of them are not illustrated), I would be more than happy to accelerate > this process (getting approval from our Open Source Board). Since pytest_cxdist doesn't exist, this could be open source from the > get go. I wonder if the provisioning of VMs wouldn't better fit at the level of "tox", see http://tox.testrun.org or even yet a level higher. tox creates virtual environments on the Python level. FYI there are plans to introduce a plugin system to tox and to make pytest-xdist make use of tox configuration directly. Btw, from May on i (and others) should be available through my company merlinux for some consulting in case you need some more substantial support than occassional comments from far away deserts :) best, holger From lpbrac at dolby.com Tue Mar 27 18:23:08 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Tue, 27 Mar 2012 09:23:08 -0700 Subject: [py-dev] OpenStack In-Reply-To: <20120324134157.GM26028@merlinux.eu> References: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A082@sfo-exch-01.dolby.net> <20120324134157.GM26028@merlinux.eu> Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE449EF695@sfo-exch-01.dolby.net> Hi Holger, No problem for the delayed response. I know you were in the bay area a while back and may have felt everyone is always in a rush but this is not our case :) -----Original Message----- From: holger krekel [mailto:holger at merlinux.eu] Sent: Saturday, March 24, 2012 6:42 AM To: Brack, Laurent P. Cc: py-dev at codespeak.net Subject: Re: [py-dev] OpenStack Hi Brack, [Laurent] >> This is my last name :) sorry for taking a while. Am still on travel, currently in the Sonoran deserts. My answers up until first half May are thus likely to lag. (They do have suprisingly good internet connection ... better than in the three-times more expensive hotel in the San Francisco valley last week). On Tue, Mar 20, 2012 at 10:27 -0700, Brack, Laurent P. wrote: > Forking from the [py-dev] pytest-timeout 0.2 e-mail thread. > > > I am interested in OpenStack but you can you detail a bit more of > > what you want to achieve? > > I have attached a rough diagram of what we are building internally (hopefully it will not be filtered out). thanks for sharing. > About a year ago, we attended a presentation on OpenStack (when it was still driven by Anso Labs before they got acquired by Rackspace). > We were (and are) currently using a private cloud (VMWare) but > contemplated the idea of scaling up to public clouds. Problem we had > was that we had to develop custom code for each cloud vendor whether Amazon EC2, Penguin, VMWare, etc. so OpenStack was really a viable path to not lock ourselves in with a given vendor. sounds sensible. > While a bulk of our tests require embedded devices (in which case > virtualization makes little sense) a good chunk can be run on standard workstations (all OS flavors) and in this case it only makes sense to move to a virtual environment. > > PyTest combined with xdist was the dream come true as we could focus > on developing tests meant to run on a single machine and later seamlessly parallelize their execution. There is nothing new here. > > We were thinking of writing a plugin to xdist (cxdist - c for cloud) > that would interface with OpenStack, using the pytest_xdist_setupnodes and pytest_configure_node hooks. ok. > In those hooks, the plugin would provision the machine (via OpenStack) > and then make xdist believe that it is dealing with physical slaves. > Finally at the end of the run, we would teardown the slaves, leaving resources for other tests to run. > > We have not started yet on this as scalability is not an issue but > this is our plan. As the diagram shows, the read boxes are plugins we intend to release to the open source community. Is your general idea to use the open stack integration to get parallelizable test runs with totally controllable environments? [Laurent] >> It is. We have a fairly large VMWare infrastructure but I feel we are locked in. Also there is a point where scaling up this private cloud will simply make no sense from an economic standpoint. Finally, infrastructure like VMWare are of little use for the open source community. > Our teams write a lot of "data driven tests" (testing various AV > codecs with different configuration parameters using very similar verification procedure) and as a result we make heavy use of funcargs. Curious, are you using 2.2.3 for this? There are some plans to further improve parametrization where i would be interested in your feedback. [Laurent] >> Yes. As a matter of fact we had to make changes to our plugin to work properly with the new metafunc.parametrize method. Maybe here is the time to open a small parenthesis. We have simplified the generation process For our common users, although this doesn't prevent them from using custom hook implementations or factories. One thing that the plugin does is attach "meta data" to items (which is carried over from hook to hook). This meta data contains information about test cases on the testlink Server. One of the challenge with metafunc was to find a way to attach this information in a way it would not be lost during the test generation done by pytest. In 2.1.3, with "addcall" we had done this by hacking the "param" formal parameter intended use (while preserving the functionality). In 2.2.3 (the addcall hack was broken - meta data got lost), we found another way. Again a hack and therefore no guaranty it will work with future versions. It would be nice to have a supported way to "attach" data as part of the parametrize process which is then carried over to the "item" being generated. The idea is to carry over information that may be generated by a hook to another hook (whenever it makes sense). > The pytest_testlink plugin is very similar to the pytest Case Conductor plugin from the Mozilla folks (http://blargon7.com/2012/01/case-conductor-pytest-plugin-proposal/) but interacts with TestLink (teamst.org in which we are currently involved in - improving performance of web services etc.) as opposed to Case Conductor. Is pytest-testlink a plugin that you plan to work on? [Laurent] >> Actually we have it working. We have been using it for about 6 months now and I just made a new internal release (to fix the issue with 2.1.3+) which also supports "platforms" (a feature of TestLink) and filtering based on TestLink constructs (outcome, test case ID, keywords). The plugin relies on another package (called pytl) which provides an OO model of the testlink server built on top of their existing web services. Those web services are quite buggy and incomplete but we have built-in workarounds. In parallel we are working with the TestLink team to provide a more complete set of web services. We are also working on restructuring the code base allowing better integration via plugins (requirement managements, reporting - we have a solution using BIRT) as well performance improvements, test case libraries (for re-use between projects), project groups, etc. While the pytest testlink plugin is extensively documented and so far proven to be pretty robust, I want to clean it up even more before releasing it (making it available as an open source package was the intent from the start). > However in addition to linking a test case ID to a given python test, > we have implemented a simple mechanism to generate test cases from > what we called Test Description Records. Those are currently stored in > excel spreadsheets but will be store in TestLink as test case properties. The object injection is done automatically by the plugin using a custom marker indicating which function argument shall be used for TDR injection as well as correlating TDR with parametrized test functions. right. Driving things from a spreadsheet, maybe even in Google docs, sounds good to me, btw :) [Laurent] >> We considered this but decided otherwise just because of the limited functionality offered compared to M$ Office or LibreOffice (we are actually using xlrd/xlwr). As a matter of fact I wanted to find a way to abstract the "data source" for the TDR. Google doc would be a perfect exercise. > The point is that by using the funcarg feature offered by pytest, we > can generate tremendous amount of test cases with little effort and this will require scalability and use of virtualization over time. > > Sorry for the long description, but I wanted to give a complete overview to help comprehend where we are going. > Our plan is to eventually release all this to the open source community (when we think the plugins are mature and robust enough). > > If other people have a need/interest for those plugins (some of them > are not illustrated), I would be more than happy to accelerate this > process (getting approval from our Open Source Board). Since pytest_cxdist doesn't exist, this could be open source from the get go. I wonder if the provisioning of VMs wouldn't better fit at the level of "tox", see http://tox.testrun.org or even yet a level higher. tox creates virtual environments on the Python level. FYI there are plans to introduce a plugin system to tox and to make pytest-xdist make use of tox configuration directly. [Laurent] >> I need to educate myself more on Tox. We are using tox for our CI system (Jenkins) but environment setup is actually done via buildout (tox invoking buildout - maybe we would have worked with tox from the start if we had known about it). But the tox idea is pretty good as it this is really where the testbed should be setup rather than runtime (pytest) when things can go wrong. Btw, from May on i (and others) should be available through my company merlinux for some consulting in case you need some more substantial support than occassional comments from far away deserts :) [Laurent] >> Actually this sounds like interesting. Let me think about it. We are a rather small team so we can use all the help we can get. Enjoy and drink a lot of water :) Cheers/Laurent best, holger From mantihor at gmail.com Sat Mar 31 17:16:55 2012 From: mantihor at gmail.com (Bogdan Opanchuk) Date: Sun, 1 Apr 2012 02:16:55 +1100 Subject: [py-dev] Performance tests with py.test Message-ID: Hello, I am trying to wrap performance tests for my module into py.test. Each testcase is, basically, a data preparation stage (can take some time) and the test itself. Is there some way (hooks, plugins) to make py.test print the value returned from the testcase (and make testcases time whatever they want)? What I want is just a table like the one after "py.test -v", but with return values instead of "PASSED". I googled it and haven't found any solutions. At the moment I am doing it as following: 1. Overload "pytest_pyfunc_call()" and save returned result into "pyfuncitem" parameter. 2. Overload "pytest_runtest_makereport()", look for "perf" mark and somehow output "item.result". 3. I haven't decided about the way to output it yet, because I am currently trying to find the place where "PASSED" is printed. But this seems quite clunky and overcomplicated. Is there any better way to do this? If not for the data preparation, the currently existing time measuring system would probably be enough for me (although I'd still miss the ability to print GFLOPS value instead of seconds), but the data preparation is testcase-specific and I do not want to move it to "pytest_generate_tests()" hook. Thank you in advance.