From nicoe at no-log.org Sun Apr 3 23:51:26 2011 From: nicoe at no-log.org (Nicolas =?utf-8?Q?=C3=89vrard?=) Date: Sun, 3 Apr 2011 23:51:26 +0200 Subject: [py-dev] How can I use py.test in trytond Message-ID: <20110403215126.GA28374@nutellux.dyndns.org> Hello, I develop on tryton[1] and I'd like to use py.test to test the new modules I am writing for it. This framework has a concept of modularity that allows some module to override the definition of objects. An example will probably help understanding the situation. So we have the base trytond module which contains a `modules` directory with the list of available modules (they are all imported but not necessarily installed). project/ bin/ trytond/ modules/ ... sale/ sale_cost/ ... So when you install the sale_cost module the object defined on the sale module are extended with more fields and the methods might be extended to fit the business case. Currently I use a conftest.py file in the top directory in order to define funcargs for my tests. But it looks like a suboptimal solution. I think I can use plugins in order to do the following: 1. Define a funcarg `sale` in the sale module 2. Define a funcarg `sale` in the sale_cost module. This funcarg should ideally use the funcarg from `sale` and extend it with various information (if allowed to do so by the business rule). 3. Calling py.test trytond/modules/sale_cost from the project/ directory should work This setup would allow me to provide funcargs for use to other developers on which they can base their own tests. Right now I noticed that the funcargs are loaded recursively. Do you think it is possible to write a plugin that would read the dependency information in the sale_cost module and use it to setup the conftests file found in those modules to be imported *before* the conftest.py from sale_cost ? If it is, do you have any pointers on how I can do this ? BTW, py.test is a really nice piece of software and I really enjoy using it. [1]: http://www.tryton.org/ -- (?> Nicolas ?vrard ( ) Li?ge `? From holger at merlinux.eu Mon Apr 4 09:37:36 2011 From: holger at merlinux.eu (holger krekel) Date: Mon, 4 Apr 2011 07:37:36 +0000 Subject: [py-dev] How can I use py.test in trytond In-Reply-To: <20110403215126.GA28374@nutellux.dyndns.org> References: <20110403215126.GA28374@nutellux.dyndns.org> Message-ID: <20110404073736.GS16231@merlinux.eu> Hi Nicolas, On Sun, Apr 03, 2011 at 23:51 +0200, Nicolas ?vrard wrote: > Hello, > > I develop on tryton[1] and I'd like to use py.test to test the new > modules I am writing for it. > > This framework has a concept of modularity that allows some module to > override the definition of objects. An example will probably help > understanding the situation. > > So we have the base trytond module which contains a `modules` > directory with the list of available modules (they are all imported > but not necessarily installed). > > project/ > bin/ > trytond/ > modules/ > ... > sale/ > sale_cost/ > ... > > So when you install the sale_cost module the object defined on the > sale module are extended with more fields and the methods might be > extended to fit the business case. > > Currently I use a conftest.py file in the top directory in order to > define funcargs for my tests. But it looks like a suboptimal solution. > > I think I can use plugins in order to do the following: > > 1. Define a funcarg `sale` in the sale module > 2. Define a funcarg `sale` in the sale_cost module. This funcarg > should ideally use the funcarg from `sale` and extend it with > various information (if allowed to do so by the business rule). > 3. Calling py.test trytond/modules/sale_cost from the project/ > directory should work > > This setup would allow me to provide funcargs for use to other > developers on which they can base their own tests. > > Right now I noticed that the funcargs are loaded recursively. Do you > think it is possible to write a plugin that would read the dependency > information in the sale_cost module and use it to setup the conftests > file found in those modules to be imported *before* the conftest.py > from sale_cost ? > > If it is, do you have any pointers on how I can do this ? request.getfuncargvalue() can be called to decorate an existing funcarg, see http://bit.ly/eiByLF However, i guess you could just have a single conftest.py which provides different values for your funcargs depending on which directory they are requested from. (``request.module.fspath`` gives you the file system path of the python module.) This way you would encode all the logic in one place. HTH, holger > BTW, py.test is a really nice piece of software and I really enjoy > using it. BTW, nice to hear :) > > [1]: http://www.tryton.org/ > > -- > (?> Nicolas ?vrard > ( ) Li?ge > `? > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From nicoe at no-log.org Mon Apr 4 13:03:24 2011 From: nicoe at no-log.org (Nicolas =?utf-8?Q?=C3=89vrard?=) Date: Mon, 4 Apr 2011 13:03:24 +0200 Subject: [py-dev] How can I use py.test in trytond In-Reply-To: <20110404073736.GS16231@merlinux.eu> References: <20110403215126.GA28374@nutellux.dyndns.org> <20110404073736.GS16231@merlinux.eu> Message-ID: <20110404110324.GA24950@nutellux.dyndns.org> * holger krekel [2011-04-04 09:37 +0200]: >Hi Nicolas, Hello, >> Currently I use a conftest.py file in the top directory in order to >> define funcargs for my tests. But it looks like a suboptimal solution. >> >> I think I can use plugins in order to do the following: >> >> 1. Define a funcarg `sale` in the sale module >> 2. Define a funcarg `sale` in the sale_cost module. This funcarg >> should ideally use the funcarg from `sale` and extend it with >> various information (if allowed to do so by the business rule). >> 3. Calling py.test trytond/modules/sale_cost from the project/ >> directory should work >> >> This setup would allow me to provide funcargs for use to other >> developers on which they can base their own tests. >> >> Right now I noticed that the funcargs are loaded recursively. Do you >> think it is possible to write a plugin that would read the dependency >> information in the sale_cost module and use it to setup the conftests >> file found in those modules to be imported *before* the conftest.py >> from sale_cost ? >> >> If it is, do you have any pointers on how I can do this ? > >request.getfuncargvalue() can be called to decorate an existing funcarg, >see http://bit.ly/eiByLF Thank you for the link I missed it while going through the doc (although I knew that you could call getfuncargvalue to get the 'previous' value and play with it). >However, i guess you could just have a single conftest.py which provides different >values for your funcargs depending on which directory they are requested from. That's precisely what I don't want to do. Because modules are distributed through pypi and thus you never know which modules are installed. Moreover, I think that such a setup would have many difficulties to address all the possible cases (modules incompatibilities, bad dependency specified in the __tryton__ file, and so on). My idea is the following: - During test conftest collection, locate the nearest __tryton__.py file (by going in the direction of the root of the file system). - Once you found it, collect also conftest information from the modules specified in this file (the modules are usually in a sibling directory from the directory where you found the __tryton__.py file but they may also be found thanks to their entry point). - The collection should of course work recursively It is important not to load all conftest information (because one might have conflicting configuration for the same object) but only the one you depend on. Is this kind of setup doable in a plugin ? Am I completely wrong about how this problematic should be addressed ? >> BTW, py.test is a really nice piece of software and I really enjoy >> using it. > >BTW, nice to hear :) It is well deserved :). -- (?> Nicolas ?vrard ( ) Li?ge `? From prologic at shortcircuit.net.au Tue Apr 5 01:00:58 2011 From: prologic at shortcircuit.net.au (James Mills) Date: Tue, 5 Apr 2011 09:00:58 +1000 Subject: [py-dev] py.test + CI Message-ID: Hey all, Currently what options are available for Continuous Integration with py.test (or any other test suite / runner for that matter) ? cheers James -- -- James Mills -- -- "Problems are solved by method" From flub at devork.be Tue Apr 5 14:24:36 2011 From: flub at devork.be (Floris Bruynooghe) Date: Tue, 5 Apr 2011 13:24:36 +0100 Subject: [py-dev] py.test + CI In-Reply-To: References: Message-ID: On 5 April 2011 00:00, James Mills wrote: > Currently what options are available for > Continuous Integration with py.test (or any other test suite / runner > for that matter) ? py.test has a --junitxml option which writes reports in the de-facto JUnit XML format. E.g. Jenkins/Hudson can read this XML to do detailed reporting. I assume there's more tools that understand that format though never researched this. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From rafalskiy at gmail.com Tue Apr 5 17:42:15 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Tue, 5 Apr 2011 11:42:15 -0400 Subject: [py-dev] migration from 1.3 to 2.0 Message-ID: Hi Holger and list members, I have been using py.test for some time after switching from nose and quite happy with it. However, when migrating from 1.3.1 to 2.0.2 I hit a couple of snags, which makes me hesitate. Perhaps you can help me. 1. Just to clear the point. I understand from another post that option_xxxx variables in conftest.py are gone and recommended replacement is to use pytest_cmdline_preparse(). Is that so? 2. The new behavior is that even if you run py.test --help, the pytest_configure() is still run. In my case pytest_configure() does quite a bit of heavy lifting and raises exceptions at wrong parameters. The result is that if the parameter is wrong I cannot use help to find out what is the right parameter. Of course, the workaround could be: def pytest_configure(config): if config.getvalue('help'): return ... but it seems ugly. Am I in a wrong hook? 3. In my pytest_configure() there are numerous conditions when it can fail. For this conditions I have exceptions with specially crafted messages, intended for different people. Now they are gone, replaced by a long and scary listing with every line prepended with INTERNALERROR. What is internal about it? Can I continue managing my configuration errors? Thanks, Vyacheslav From holger at merlinux.eu Tue Apr 5 22:16:12 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 5 Apr 2011 20:16:12 +0000 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: References: Message-ID: <20110405201612.GW16231@merlinux.eu> Hi Vyachselav, On Tue, Apr 05, 2011 at 11:42 -0400, Vyacheslav Rafalskiy wrote: > Hi Holger and list members, > > I have been using py.test for some time after switching from nose and > quite happy with it. > However, when migrating from 1.3.1 to 2.0.2 I hit a couple of snags, > which makes me hesitate. > Perhaps you can help me. can try but am only online intermittently currently. > 1. Just to clear the point. I understand from another post that > option_xxxx variables > in conftest.py are gone and recommended replacement is to use > pytest_cmdline_preparse(). > Is that so? Not exactly. You can use a .ini file to add command line options: http://pytest.org/customize.html#adding-default-options Or do you need to do something more dynamic? In that case you could play with pytest_cmdline_preparse, indeed. > 2. The new behavior is that even if you run py.test --help, the > pytest_configure() is still run. > In my case pytest_configure() does quite a bit of heavy lifting and > raises exceptions at > wrong parameters. The result is that if the parameter is wrong I > cannot use help to find > out what is the right parameter. Of course, the workaround could be: > > def pytest_configure(config): > if config.getvalue('help'): > return > ... > > but it seems ugly. Am I in a wrong hook? You could try pytest_sessionstart(session) and access session.config for the config object. > 3. In my pytest_configure() there are numerous conditions when it can > fail. For this > conditions I have exceptions with specially crafted messages, intended > for different > people. Now they are gone, replaced by a long and scary listing with every line > prepended with INTERNALERROR. What is internal about it? Can I continue managing > my configuration errors? Sure, you should be able to. I guess it was a bit of an incident that failures in pytest_configure were shown nicely. I am not quite sure immediately what the best way to relay "global" configuration messages to the user is. If you'd use "funcargs" and the "session" scope for setting up resources then it would show nicely. Feel free to open a feature/enhancement request to have py.test show failures in sessionstart or configure hooks in a way that helps you. This way we can write a specific test and make sure it works also in the future. best, holger From prologic at shortcircuit.net.au Wed Apr 6 01:16:54 2011 From: prologic at shortcircuit.net.au (James Mills) Date: Wed, 6 Apr 2011 09:16:54 +1000 Subject: [py-dev] py.test + CI In-Reply-To: References: Message-ID: On Tue, Apr 5, 2011 at 10:24 PM, Floris Bruynooghe wrote: > py.test has a --junitxml option which writes reports in the de-facto > JUnit XML format. ?E.g. Jenkins/Hudson can read this XML to do > detailed reporting. ?I assume there's more tools that understand that > format though never researched this. Yeah I've seen this looking through what py.test can output... I don't know about anyone else though... But I kind of feel like we need a new set of CI tools for Python. Hudson would not work out of the box for me and I get frustrated easily be complexity and things that don't work as "advertised" :) (Even buildbot didn't work) cheers James -- -- James Mills -- -- "Problems are solved by method" From Ronny.Pfannschmidt at gmx.de Wed Apr 6 01:51:55 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Wed, 06 Apr 2011 01:51:55 +0200 Subject: [py-dev] py.test + CI In-Reply-To: References: Message-ID: <1302047515.10381.32.camel@Klappe2> On Wed, 2011-04-06 at 09:16 +1000, James Mills wrote: > On Tue, Apr 5, 2011 at 10:24 PM, Floris Bruynooghe wrote: > > py.test has a --junitxml option which writes reports in the de-facto > > JUnit XML format. E.g. Jenkins/Hudson can read this XML to do > > detailed reporting. I assume there's more tools that understand that > > format though never researched this. > > Yeah I've seen this looking through what py.test can output... > > I don't know about anyone else though... But I kind of feel like > we need a new set of CI tools for Python. Hudson would not work > out of the box for me and I get frustrated easily be complexity > and things that don't work as "advertised" :) (Even buildbot didn't work) there are various people that feel the need for more pythonic CI tools, but currently seems like the time is not right to just gather them for hacking it up its on my wishlist as well, but until anyvc is ready as base lib i wont take a look at the problem again (last time i failed bad) -- Ronny -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110406/9dbc8448/attachment.pgp From holger at merlinux.eu Wed Apr 6 09:01:36 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 6 Apr 2011 07:01:36 +0000 Subject: [py-dev] py.test + CI In-Reply-To: References: Message-ID: <20110406070136.GY16231@merlinux.eu> On Wed, Apr 06, 2011 at 09:16 +1000, James Mills wrote: > On Tue, Apr 5, 2011 at 10:24 PM, Floris Bruynooghe wrote: > > py.test has a --junitxml option which writes reports in the de-facto > > JUnit XML format. ?E.g. Jenkins/Hudson can read this XML to do > > detailed reporting. ?I assume there's more tools that understand that > > format though never researched this. > > Yeah I've seen this looking through what py.test can output... > > I don't know about anyone else though... But I kind of feel like > we need a new set of CI tools for Python. Hudson would not work > out of the box for me and I get frustrated easily be complexity > and things that don't work as "advertised" :) (Even buildbot didn't work) I much understand the sentiment and hope to work myself towards a better solution some day :) Meanwhile you might want to look into tox (http://codespeak.net/tox) which helps to automate environment setup. Given you have a "tox.ini" configuration working for your project you can then relatively easily integrate it with Jenkins/Hudson. holger > cheers > James > > -- > -- James Mills > -- > -- "Problems are solved by method" > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From holger at merlinux.eu Wed Apr 6 10:04:49 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 6 Apr 2011 08:04:49 +0000 Subject: [py-dev] How can I use py.test in trytond In-Reply-To: <20110404110324.GA24950@nutellux.dyndns.org> References: <20110403215126.GA28374@nutellux.dyndns.org> <20110404073736.GS16231@merlinux.eu> <20110404110324.GA24950@nutellux.dyndns.org> Message-ID: <20110406080449.GZ16231@merlinux.eu> On Mon, Apr 04, 2011 at 13:03 +0200, Nicolas ?vrard wrote: > * holger krekel [2011-04-04 09:37 +0200]: > >Hi Nicolas, > > Hello, > > >>Currently I use a conftest.py file in the top directory in order to > >>define funcargs for my tests. But it looks like a suboptimal solution. > >> > >>I think I can use plugins in order to do the following: > >> > >> 1. Define a funcarg `sale` in the sale module > >> 2. Define a funcarg `sale` in the sale_cost module. This funcarg > >> should ideally use the funcarg from `sale` and extend it with > >> various information (if allowed to do so by the business rule). > >> 3. Calling py.test trytond/modules/sale_cost from the project/ > >> directory should work > >> > >>This setup would allow me to provide funcargs for use to other > >>developers on which they can base their own tests. > >> > >>Right now I noticed that the funcargs are loaded recursively. Do you > >>think it is possible to write a plugin that would read the dependency > >>information in the sale_cost module and use it to setup the conftests > >>file found in those modules to be imported *before* the conftest.py > >>from sale_cost ? > >> > >>If it is, do you have any pointers on how I can do this ? > > > >request.getfuncargvalue() can be called to decorate an existing funcarg, > >see http://bit.ly/eiByLF > > Thank you for the link I missed it while going through the doc > (although I knew that you could call getfuncargvalue to get the > 'previous' value and play with it). > > >However, i guess you could just have a single conftest.py which provides different > >values for your funcargs depending on which directory they are requested from. > > That's precisely what I don't want to do. OK, i didn't read your first post correctly. > Because modules are distributed through pypi and thus you never know > which modules are installed. Moreover, I think that such a setup would > have many difficulties to address all the possible cases (modules > incompatibilities, bad dependency specified in the __tryton__ file, > and so on). > > My idea is the following: > > - During test conftest collection, locate the nearest > __tryton__.py file (by going in the direction of the root of the > file system). > - Once you found it, collect also conftest information from > the modules specified in this file (the modules are usually in a > sibling directory from the directory where you found the > __tryton__.py file but they may also be found thanks to their > entry point). > - The collection should of course work recursively > > It is important not to load all conftest information (because one > might have conflicting configuration for the same object) but only the > one you depend on. > > Is this kind of setup doable in a plugin ? Hum, might be tricky. To begin with, conftest.py detection/looading is mostly done via internal APIs. Second, conftest.py definitions are only consulted for the particularly subdirectory so that for a/conftest.py b/ conftest.py test_whatever.py for the path "b/test_whatever.py" funcargs defined in a/conftest.py file will not be consulted. Maybe it's best if you experiment with your own convention. An idea could be to define a pytest_cmdline_main() hook in your top-level conftest.py file (which you can morph to become a pypi-installable installable pytest-tryton plugin once it works nicely). In this hook you would look for your __tryton__.py file, load it and look for which modules need to be loaded. In the module dirs you could have a "conftestresource.py" module which defines funcargs as if you would do it in a conftest.py file. You could import this module and register it as a plugin to py.test like this:: config.pluginmanager.register(pluginmodule_or_object, name, prepend=True) # or False This way your modules grow the ability to interact with testing and you could even use "request.getfuncargvalue()" given that your project definitions and thus the registrations of funcarg resources are accordingly ordered. If not you might be able to use the "@pytest.mark.tryfirst" marker on any base object which modules want to enhance. You might want to read the _pytest/*.py source code a bit, grepping for the bits and pieces i posted. > Am I completely wrong about how this problematic should be addressed ? Depends on if i am wrong about what i am suggesting for your situation :) best, holger > >>BTW, py.test is a really nice piece of software and I really enjoy > >>using it. > > > >BTW, nice to hear :) > > It is well deserved :). > > -- > (?> Nicolas ?vrard > ( ) Li?ge > `? > From holger at merlinux.eu Wed Apr 6 13:48:05 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 6 Apr 2011 11:48:05 +0000 Subject: [py-dev] latest pytest questions for pytest-cov In-Reply-To: References: <20110320112120.GL16231@merlinux.eu> Message-ID: <20110406114805.GB16231@merlinux.eu> Hey Meme, On Sun, Mar 27, 2011 at 22:28 +1100, meme dough wrote: > Hi, > > Now up and working with the latest pytest (think only distributed > testing was an issue). It is still backward compatible with older > versions of pytest (before the split). > > Funcarg is at module level so visible with py.test --funcargs. > > pth file more robust since it doesn't complain if cov-core got deleted > and pth file left. > > More doc for coverage config file since couple of people have asked questions. > > I moved the doc to the docstring in the module. I see that the > website just points to the external plugins at pypi, so that means the > doc can't get out of sync right? right, i'd think so :) nice changes. holger > :) > > On 20 March 2011 22:21, holger krekel wrote: > > Hi Meme, > > > > On Sun, Mar 20, 2011 at 22:00 +1100, meme dough wrote: > >> Hi, > >> > >> Currently looking at fixing pytest-cov to work with latest pytest. ?I > >> see two issues so far. > >> > >> 1. How to determine if non distributed, or distributed master, or > >> distributed slave. > >> > >> This used to look at the session class Session = Central (non > >> distributed), DSession = distributed master, and SlaveSession = > >> distributed slave. > >> > >> So now I'm thinking look at > >> session.config.pluginmanager.hasplugin('dsession') which means > >> distributed master. ?For distributed slave check if config.slaveinput > >> is there which means distributed slave. ?If none of those then must be > >> non distributed. > > > > I guess that's ok. ?There is no "official" way at the moment. > > > >> Is there a better way to determine at pytest_sessionstart if the > >> process is central (non distributed), distributed master or > >> distributed slave? > >> > >> 2. Need to know topdir. > >> > >> This used to use config.topdir to know what the root dir is. ?This is > >> needed since distributed testing may be co-located or not. ?If not > >> then have to rewrite all source paths back to what the master is set > >> to so that all coverage files combine correctly even if have many > >> separate tx envs on diff machines on diff paths. > >> > >> Has this moved somewhere? ?I see config.fspath, is that it? ?I look > >> more I just got to this bit. > > > > No, there now is a "looponfailroots" (see --help) and distributed > > testing uses "rsyncroots". The problem is that "topdir" was a bit > > of an artifical concept. > > > > Maybe it helps, that item.nodeid contains a "::" separated test id where > > the first part is a path relative to the current directory and > > normalized to use "/" as separator. > > > > best, > > holger > > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From rafalskiy at gmail.com Wed Apr 6 18:37:19 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Wed, 6 Apr 2011 12:37:19 -0400 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: <20110405201612.GW16231@merlinux.eu> References: <20110405201612.GW16231@merlinux.eu> Message-ID: Thanks Holger, On Tue, Apr 5, 2011 at 4:16 PM, holger krekel wrote: > Hi Vyachselav, > > On Tue, Apr 05, 2011 at 11:42 -0400, Vyacheslav Rafalskiy wrote: >> Hi Holger and list members, >> >> I have been using py.test for some time after switching from nose and >> quite happy with it. >> However, when migrating from 1.3.1 to 2.0.2 I hit a couple of snags, >> which makes me hesitate. >> Perhaps you can help me. > > can try but am only online intermittently currently. > >> 1. Just to clear the point. I understand from another post that >> option_xxxx variables >> in conftest.py are gone and recommended replacement is to use >> pytest_cmdline_preparse(). >> Is that so? > > Not exactly. ?You can use a .ini file to add command line options: > > ? ?http://pytest.org/customize.html#adding-default-options I wouldn't want to maintain an extra config file. I already have conftest.py with a lot of hardcoded stuff, a bit more does not hurt. Moreover, if you put it like def pytest_cmdline_preparse(args): args[0:0] = [ '--verbose', '--capture=no', ] it looks even cleaner than using option_xxxx variables in spite of a bit of boilerplate. >> 3. In my pytest_configure() there are numerous conditions when it can >> fail. For this >> conditions I have exceptions with specially crafted messages, intended >> for different >> people. Now they are gone, replaced by a long and scary listing with every line >> prepended with INTERNALERROR. What is internal about it? Can I continue managing >> my configuration errors? > > Sure, you should be able to. I guess it was a bit of an incident that > failures in pytest_configure were shown nicely. ?I am not quite sure > immediately what the best way to relay "global" configuration messages > to the user is. ?If you'd use "funcargs" and the "session" scope for > setting up resources then it would show nicely. ?Feel free to open > a feature/enhancement request to have py.test show failures > in sessionstart or configure hooks in a way that helps you. > This way we can write a specific test and make sure it works > also in the future. If I understand correctly, using funcargs means that every test function of hundreds I have in several suites should include a reference to the global setup funcarg. It seems a non-starter. I guess I will stick to the old version and try to think of something to enter in a feature request. Vyacheslav From rafalskiy at gmail.com Wed Apr 6 23:00:59 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Wed, 6 Apr 2011 17:00:59 -0400 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: References: <20110405201612.GW16231@merlinux.eu> Message-ID: >>> 3. In my pytest_configure() there are numerous conditions when it can >>> fail. For this >>> conditions I have exceptions with specially crafted messages, intended >>> for different >>> people. Now they are gone, replaced by a long and scary listing with every line >>> prepended with INTERNALERROR. What is internal about it? Can I continue managing >>> my configuration errors? >> >> Sure, you should be able to. I guess it was a bit of an incident that >> failures in pytest_configure were shown nicely. ?I am not quite sure >> immediately what the best way to relay "global" configuration messages >> to the user is. ?If you'd use "funcargs" and the "session" scope for >> setting up resources then it would show nicely. ?Feel free to open >> a feature/enhancement request to have py.test show failures >> in sessionstart or configure hooks in a way that helps you. >> This way we can write a specific test and make sure it works >> also in the future. > > If I understand correctly, using funcargs means that every test function > of hundreds I have in several suites should include a reference to the > global setup funcarg. It seems a non-starter. > > I guess I will stick to the old version and try to think of something > ?to enter in a feature request. > > Vyacheslav > Looks like I solved my main problem with a monkey patch, admittedly a brutal one import sys import _pytest.core def notify_exception(self, excinfo): excrepr = excinfo.getrepr(style='native') sys.stderr.write(excrepr) _pytest.core.PluginManager.notify_exception = notify_exception From holger at merlinux.eu Thu Apr 7 08:17:15 2011 From: holger at merlinux.eu (holger krekel) Date: Thu, 7 Apr 2011 06:17:15 +0000 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: References: <20110405201612.GW16231@merlinux.eu> Message-ID: <20110407061715.GE16231@merlinux.eu> On Wed, Apr 06, 2011 at 12:37 -0400, Vyacheslav Rafalskiy wrote: > > Not exactly. ?You can use a .ini file to add command line options: > > > > ? ?http://pytest.org/customize.html#adding-default-options > > I wouldn't want to maintain an extra config file. I already have > conftest.py with > a lot of hardcoded stuff, a bit more does not hurt. Moreover, if you put it like > > def pytest_cmdline_preparse(args): > args[0:0] = [ > '--verbose', > '--capture=no', > ] > > it looks even cleaner than using option_xxxx variables in spite of a > bit of boilerplate. Sure, if you have a root conftest.py anyway that works. Note that you can put the .ini config into a setup.cfg as well which is used by the upcoming python package distribution as well (and setuptools). > >> 3. In my pytest_configure() there are numerous conditions when it can > >> fail. For this > >> conditions I have exceptions with specially crafted messages, intended > >> for different > >> people. Now they are gone, replaced by a long and scary listing with every line > >> prepended with INTERNALERROR. What is internal about it? Can I continue managing > >> my configuration errors? > > > > Sure, you should be able to. I guess it was a bit of an incident that > > failures in pytest_configure were shown nicely. ?I am not quite sure > > immediately what the best way to relay "global" configuration messages > > to the user is. ?If you'd use "funcargs" and the "session" scope for > > setting up resources then it would show nicely. ?Feel free to open > > a feature/enhancement request to have py.test show failures > > in sessionstart or configure hooks in a way that helps you. > > This way we can write a specific test and make sure it works > > also in the future. > > If I understand correctly, using funcargs means that every test function > of hundreds I have in several suites should include a reference to the > global setup funcarg. It seems a non-starter. > > I guess I will stick to the old version and try to think of something > to enter in a feature request. sure, makes sense. isn't your main issue that upon actual test start (after collection) you want to do some configuration steps which might result in error conditions which you'd like to see nicely reported to the user? (sidenote: configure and even sessionstart hooks are both a bit not 100% right because they happen even on the master side of a distributed test run and the master side does not collect or run tests at all) best, holger > Vyacheslav > From rafalskiy at gmail.com Thu Apr 7 18:29:09 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Thu, 7 Apr 2011 12:29:09 -0400 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: <20110407061715.GE16231@merlinux.eu> References: <20110405201612.GW16231@merlinux.eu> <20110407061715.GE16231@merlinux.eu> Message-ID: >> >> 3. In my pytest_configure() there are numerous conditions when it can >> >> fail. For this >> >> conditions I have exceptions with specially crafted messages, intended >> >> for different >> >> people. Now they are gone, replaced by a long and scary listing with every line >> >> prepended with INTERNALERROR. What is internal about it? Can I continue managing >> >> my configuration errors? >> > >> > Sure, you should be able to. I guess it was a bit of an incident that >> > failures in pytest_configure were shown nicely. ?I am not quite sure >> > immediately what the best way to relay "global" configuration messages >> > to the user is. ?If you'd use "funcargs" and the "session" scope for >> > setting up resources then it would show nicely. ?Feel free to open >> > a feature/enhancement request to have py.test show failures >> > in sessionstart or configure hooks in a way that helps you. >> > This way we can write a specific test and make sure it works >> > also in the future. >> >> If I understand correctly, using funcargs means that every test function >> of hundreds I have in several suites should include a reference to the >> global setup funcarg. It seems a non-starter. >> >> I guess I will stick to the old version and try to think of something >> ?to enter in a feature request. > > sure, makes sense. ?isn't your main issue that upon actual test start > (after collection) you want to do some configuration steps which might > result in error conditions which you'd like to see nicely reported to > the user? That's exactly my point. >(sidenote: configure and even sessionstart hooks are both a bit > not 100% right because they happen even on the master side of a distributed > test run and the master side does not collect or run tests at all) I see. Perhaps something like setup_package() in the top-level __init__.py could be a solution? Vyacheslav From flub at devork.be Tue Apr 12 00:55:40 2011 From: flub at devork.be (Floris Bruynooghe) Date: Mon, 11 Apr 2011 23:55:40 +0100 Subject: [py-dev] Creating unicode strings in pytest testsuit Message-ID: Hi While working on the patch for the invalid unicode codepoints in the junitxml output I discovered it's fairly hard to create a unicode strings in the the test suite. Currently I decided to use py.builtins._totext() to replace unicode string literals. And to get access to the unichr() builtin I've done this little hack: def foo(): global unichr try: unichr(65) except NameError: unichr = chr I think both are fairly fine, though am surprised by the leading underscore in _totext(), I'd have rather expected a py.builtin.unicode() name or so. And if the unichr hack is acceptable would it be worth to create py.builtins.unichr()? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From Ronny.Pfannschmidt at gmx.de Wed Apr 13 00:38:34 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Wed, 13 Apr 2011 00:38:34 +0200 Subject: [py-dev] the concept of a per test "test dir" and its uses Message-ID: <1302647914.5537.439.camel@Klappe2> hi, after some thought i would like to discuss adding a generic testdir, so for each test different plugins could store data/metadata currently only the tempdir plugin creates a session dir on first use of the tmpdir funcarg/pytest.mktemp i'd rather have to possibility to store for each test * full test id * stdout/err * logfile for stdlib logging * tmpdir for plugin * some kind of serialized full taceback, optionally with some locals/globals * coverage reports * profiler reports since all of those are a misfit for the tmpdir functionality, i'd like to propose a extension of the mechanism the basic tmpdir would move to the "tmpdir" subdir of the testdir, stdout/err, log, would in turn be files in the testdir it would also be easily possible to keep those in sync with xdist+rsync should the need arise, just make a new testdir for the remote id, and sync the directory content. further possible uses include using coverage data and test results of previous runs to decide what tests can likely be skipped without causing too much of a problem or providing relative progress information on the collection status (based on the number of test id's in the last run). Please discuss additional uses and possible problems. Regards Ronny -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110413/0ac25a44/attachment.pgp From holger at merlinux.eu Wed Apr 13 13:53:14 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 13 Apr 2011 11:53:14 +0000 Subject: [py-dev] Creating unicode strings in pytest testsuit In-Reply-To: References: Message-ID: <20110413115314.GO16231@merlinux.eu> Hey Floris, On Mon, Apr 11, 2011 at 23:55 +0100, Floris Bruynooghe wrote: > Hi > > While working on the patch for the invalid unicode codepoints in the > junitxml output I discovered it's fairly hard to create a unicode > strings in the the test suite. Currently I decided to use > py.builtins._totext() to replace unicode string literals. And to get > access to the unichr() builtin I've done this little hack: > > def foo(): > global unichr > try: > unichr(65) > except NameError: > unichr = chr > > I think both are fairly fine, though am surprised by the leading > underscore in _totext(), I'd have rather expected a > py.builtin.unicode() name or so. And if the unichr hack is acceptable > would it be worth to create py.builtins.unichr()? i guess this makes sense so introducing a unicode and unicode (as a duplicate for _totext for now) would make sense. Btw, i'd like to do a py/pytest release around the coming weekend - would be cool to have your fix in, even if the test isn't written in the optimal way yet. best & thanks, holger > Regards > Floris > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From holger at merlinux.eu Wed Apr 13 14:02:52 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 13 Apr 2011 12:02:52 +0000 Subject: [py-dev] the concept of a per test "test dir" and its uses In-Reply-To: <1302647914.5537.439.camel@Klappe2> References: <1302647914.5537.439.camel@Klappe2> Message-ID: <20110413120252.GP16231@merlinux.eu> Hey Ronny, On Wed, Apr 13, 2011 at 00:38 +0200, Ronny Pfannschmidt wrote: > hi, > > after some thought i would like to discuss adding a generic testdir, > so for each test different plugins could store data/metadata > > currently only the tempdir plugin creates a session dir on first use of > the tmpdir funcarg/pytest.mktemp > > i'd rather have to possibility to store for each test > * full test id > * stdout/err > * logfile for stdlib logging > * tmpdir for plugin > * some kind of serialized full taceback, > optionally with some locals/globals > * coverage reports > * profiler reports > > since all of those are a misfit for the tmpdir functionality, > i'd like to propose a extension of the mechanism > > the basic tmpdir would move to the "tmpdir" subdir of the testdir, > stdout/err, log, would in turn be files in the testdir > > it would also be easily possible to keep those in sync with xdist+rsync > should the need arise, just make a new testdir for the remote id, and > sync the directory content. > > further possible uses include using coverage data and test results of > previous runs to decide what tests can likely be skipped without causing > too much of a problem or providing relative progress information on the > collection status (based on the number of test id's in the last run). > > Please discuss additional uses and possible problems. Currently, i'd think going for a "testdir" that easily allows to create dirs, files, run (sub) processes and collect results is one thing. And the other thing, collecting and accessing all kinds of historic information from a test run does not need to be accessible via a funcarg, does it? IMO, a pytest-history plugin would introduce recording all this information and adding cmdline opts for re-run/re-present info to access it as well as an API (pytest.gethistory() or so) best, holger > Regards Ronny > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From schmir at gmail.com Wed Apr 13 16:15:26 2011 From: schmir at gmail.com (schmir at gmail.com) Date: Wed, 13 Apr 2011 16:15:26 +0200 Subject: [py-dev] KeyError in xdist/looponfail.py Message-ID: <87vcyinsox.fsf@muni.brainbot.com> py.test --looponfail did fail with the following traceback: ####################################### waiting for changes ######################################## ### Watching: /home/ralf/bbot/py-nextbot/nextbot/core Traceback (most recent call last): File "/home/ralf/local/bin/py.test", line 9, in load_entry_point('pytest==2.0.2', 'console_scripts', 'py.test')() File "/home/ralf/local/lib/python2.7/site-packages/_pytest/core.py", line 448, in main exitstatus = hook.pytest_cmdline_main(config=config) File "/home/ralf/local/lib/python2.7/site-packages/_pytest/core.py", line 402, in __call__ return self._docall(methods, kwargs) File "/home/ralf/local/lib/python2.7/site-packages/_pytest/core.py", line 413, in _docall res = mc.execute() File "/home/ralf/local/lib/python2.7/site-packages/_pytest/core.py", line 331, in execute res = method(**kwargs) File "/home/ralf/local/lib/python2.7/site-packages/xdist/plugin.py", line 56, in pytest_cmdline_main looponfail_main(config) File "/home/ralf/local/lib/python2.7/site-packages/xdist/looponfail.py", line 26, in looponfail_main statrecorder.waitonchange(checkinterval=2.0) File "/home/ralf/local/lib/python2.7/site-packages/xdist/looponfail.py", line 195, in waitonchange changed = self.check() File "/home/ralf/local/lib/python2.7/site-packages/xdist/looponfail.py", line 213, in check del statcache[path] KeyError: local('/home/ralf/bbot/py-nextbot/nextbot/core/searchproc_flymake.py') I'm using flymake in emacs, and searchproc_flymake.py is a temporary file generated by flymake. cheers, -- ralf From holger at merlinux.eu Thu Apr 14 18:54:38 2011 From: holger at merlinux.eu (holger krekel) Date: Thu, 14 Apr 2011 16:54:38 +0000 Subject: [py-dev] KeyError in xdist/looponfail.py In-Reply-To: <87vcyinsox.fsf@muni.brainbot.com> References: <87vcyinsox.fsf@muni.brainbot.com> Message-ID: <20110414165438.GW16231@merlinux.eu> Hey Ralf, can you give me the output of "py.test --version"? And when exactly is searchproc_flymake.py created and removed? (the looponfail-detection code should be robust against file removals but maybe there is a race condition) holger On Wed, Apr 13, 2011 at 16:15 +0200, schmir at gmail.com wrote: > py.test --looponfail did fail with the following traceback: > > ####################################### waiting for changes ######################################## > ### Watching: /home/ralf/bbot/py-nextbot/nextbot/core > Traceback (most recent call last): > File "/home/ralf/local/bin/py.test", line 9, in > load_entry_point('pytest==2.0.2', 'console_scripts', 'py.test')() > File "/home/ralf/local/lib/python2.7/site-packages/_pytest/core.py", line 448, in main > exitstatus = hook.pytest_cmdline_main(config=config) > File "/home/ralf/local/lib/python2.7/site-packages/_pytest/core.py", line 402, in __call__ > return self._docall(methods, kwargs) > File "/home/ralf/local/lib/python2.7/site-packages/_pytest/core.py", line 413, in _docall > res = mc.execute() > File "/home/ralf/local/lib/python2.7/site-packages/_pytest/core.py", line 331, in execute > res = method(**kwargs) > File "/home/ralf/local/lib/python2.7/site-packages/xdist/plugin.py", line 56, in pytest_cmdline_main > looponfail_main(config) > File "/home/ralf/local/lib/python2.7/site-packages/xdist/looponfail.py", line 26, in looponfail_main > statrecorder.waitonchange(checkinterval=2.0) > File "/home/ralf/local/lib/python2.7/site-packages/xdist/looponfail.py", line 195, in waitonchange > changed = self.check() > File "/home/ralf/local/lib/python2.7/site-packages/xdist/looponfail.py", line 213, in check > del statcache[path] > KeyError: local('/home/ralf/bbot/py-nextbot/nextbot/core/searchproc_flymake.py') > > > I'm using flymake in emacs, and searchproc_flymake.py is a temporary > file generated by flymake. > > cheers, > -- ralf > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From schmir at gmail.com Thu Apr 14 20:51:08 2011 From: schmir at gmail.com (Ralf Schmitt) Date: Thu, 14 Apr 2011 20:51:08 +0200 Subject: [py-dev] KeyError in xdist/looponfail.py In-Reply-To: <20110414165438.GW16231@merlinux.eu> (holger krekel's message of "Thu, 14 Apr 2011 16:54:38 +0000") References: <87vcyinsox.fsf@muni.brainbot.com> <20110414165438.GW16231@merlinux.eu> Message-ID: <874o60zmxv.fsf@brainbot.com> holger krekel writes: > Hey Ralf, > > can you give me the output of "py.test --version"? yes, sorry. should have mentioned that, but I was in a hurry and didn't want to let the issue get lost. ,---- | This is py.test version 2.0.2, imported from /home/ralf/local/lib/python2.7/site-packages/pytest.pyc | setuptools registered plugins: | pytest-xdist-1.5 at /home/ralf/local/lib/python2.7/site-packages/xdist/plugin.pyc `---- > And when exactly is searchproc_flymake.py created and removed? after saving searchproc.py, emacs creates searchproc_flymake.py, runs pyflakes on it and then deletes it. > (the looponfail-detection code should be robust against file > removals but maybe there is a race condition) I'm pretty sure there is a race condition :) cheers, - ralf From holger at merlinux.eu Thu Apr 14 21:37:15 2011 From: holger at merlinux.eu (holger krekel) Date: Thu, 14 Apr 2011 19:37:15 +0000 Subject: [py-dev] KeyError in xdist/looponfail.py In-Reply-To: <874o60zmxv.fsf@brainbot.com> References: <87vcyinsox.fsf@muni.brainbot.com> <20110414165438.GW16231@merlinux.eu> <874o60zmxv.fsf@brainbot.com> Message-ID: <20110414193715.GY16231@merlinux.eu> On Thu, Apr 14, 2011 at 20:51 +0200, Ralf Schmitt wrote: > holger krekel writes: > > > Hey Ralf, > > > > can you give me the output of "py.test --version"? > > yes, sorry. should have mentioned that, but I was in a hurry and didn't > want to let the issue get lost. > > ,---- > | This is py.test version 2.0.2, imported from /home/ralf/local/lib/python2.7/site-packages/pytest.pyc > | setuptools registered plugins: > | pytest-xdist-1.5 at /home/ralf/local/lib/python2.7/site-packages/xdist/plugin.pyc > `---- > > > And when exactly is searchproc_flymake.py created and removed? > > after saving searchproc.py, emacs creates searchproc_flymake.py, runs pyflakes > on it and then deletes it. > > > (the looponfail-detection code should be robust against file > > removals but maybe there is a race condition) > > I'm pretty sure there is a race condition :) i now understand which one, i think :) and fixed it in 1.6.dev4. could you try out "pip install -U -i http://pypi.testrun.org pytest-xdist"? best, holger From flub at devork.be Sat Apr 16 01:51:22 2011 From: flub at devork.be (Floris Bruynooghe) Date: Sat, 16 Apr 2011 00:51:22 +0100 Subject: [py-dev] Creating unicode strings in pytest testsuit In-Reply-To: <20110413115314.GO16231@merlinux.eu> References: <20110413115314.GO16231@merlinux.eu> Message-ID: Hello On 13 April 2011 12:53, holger krekel wrote: > Btw, i'd like to do a py/pytest release around the coming weekend - > would be cool to have your fix in, even if the test isn't written > in the optimal way yet. I've finally managed to stop confusing myself and seem to have succeeded in a more comprehensive fix. Though I did take some shortcuts in the tests as you suggested, there where some more issues with testing unicode characters so the tests just skip things outside of the ascii range. testrun.org seems happy, though two of the builds (pypy, py31) seem to be hanging on unrelated issues. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From holger at merlinux.eu Sun Apr 17 12:30:51 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 17 Apr 2011 10:30:51 +0000 Subject: [py-dev] Creating unicode strings in pytest testsuit In-Reply-To: References: <20110413115314.GO16231@merlinux.eu> Message-ID: <20110417103051.GA16231@merlinux.eu> Hi Floris, thanks, looks good. testrun.org is not too reliable these days, indeed. (I'd like to have a much simpler and more robust service than Hudson/Jenkins some day). best, holger On Sat, Apr 16, 2011 at 00:51 +0100, Floris Bruynooghe wrote: > Hello > > On 13 April 2011 12:53, holger krekel wrote: > > Btw, i'd like to do a py/pytest release around the coming weekend - > > would be cool to have your fix in, even if the test isn't written > > in the optimal way yet. > > I've finally managed to stop confusing myself and seem to have > succeeded in a more comprehensive fix. Though I did take some > shortcuts in the tests as you suggested, there where some more issues > with testing unicode characters so the tests just skip things outside > of the ascii range. > > testrun.org seems happy, though two of the builds (pypy, py31) seem to > be hanging on unrelated issues. > > Regards > Floris > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > From holger at merlinux.eu Sun Apr 17 23:18:38 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 17 Apr 2011 21:18:38 +0000 Subject: [py-dev] pytest-2.0.3: bug fixes and speed ups Message-ID: <20110417211838.GC16231@merlinux.eu> py.test 2.0.3: bug fixes and speed ups =========================================================================== Welcome to pytest-2.0.3, a maintenance and bug fix release of pytest, a mature testing tool for Python, supporting CPython 2.4-3.2, Jython and latest PyPy interpreters. See the extensive docs with tested examples here: http://pytest.org/ If you want to install or upgrade pytest, just type one of:: pip install -U pytest # or easy_install -U pytest There also is a bugfix release 1.6 of pytest-xdist, the plugin that enables seemless distributed and "looponfail" testing for Python. best, holger krekel Changes between 2.0.2 and 2.0.3 ---------------------------------------------- - fix issue38: nicer tracebacks on calls to hooks, particularly early configure/sessionstart ones - fix missing skip reason/meta information in junitxml files, reported via http://lists.idyll.org/pipermail/testing-in-python/2011-March/003928.html - fix issue34: avoid collection failure with "test" prefixed classes deriving from object. - don't require zlib (and other libs) for genscript plugin without --genscript actually being used. - speed up skips (by not doing a full traceback represenation internally) - fix issue37: avoid invalid characters in junitxml's output From Ronny.Pfannschmidt at gmx.de Wed Apr 20 22:47:48 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Wed, 20 Apr 2011 22:47:48 +0200 Subject: [py-dev] initial take at a generic funcags hook Message-ID: <1303332468.8027.7.camel@Klappe2> hi, i just tried a initial implementation of generic funcargs to solve the issue i introduced the hook pytest_reconfigure_funcargs(request) the current patch is at https://bitbucket.org/RonnyPfannschmidt/pytest-patches/qseries?apply=t&qs_apply=reconfigure-funcargs regards, Ronny -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110420/87744669/attachment.pgp From schettino72 at gmail.com Mon Apr 25 17:40:18 2011 From: schettino72 at gmail.com (Eduardo Schettino) Date: Mon, 25 Apr 2011 23:40:18 +0800 Subject: [py-dev] [ANN] plugin: pytest-incremental Message-ID: Hi all, I have just pushed a new pytest plugin to pypi (http://pypi.python.org/pypi/pytest-incremental). The idea is to execute your tests faster by executing not all of them but only the "required" ones. This is very new (alpha), feedback is welcome :) Regards, Eduardo +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ pytest-incremental ==================== an incremental test runner (pytest plugin) What is an "incremental test runner" ? ======================================= When talking about build-tools it is common to refer to the terms: * initial (full) build - all files are compiled * incremental build (or partial rebuild) - just changed files are compiled * no-op build - no files are compiled (none changed since last execution) So an "incremental test runner" will only re-execute tests that were affected by changes in the source code since last test execution. How it works ? ================ `pytest-incremental` is a `pytest `_ plugin. So if you can run your test suite with pytest you can use `pytest-incremental`. The plugin will analyse your python source files and through its imports define the dependencies of the modules. `doit `_ is used to keep track of the dependencies and save results. The plugin will modify how pytest collect your tests. pytest do the rest of the job of actually running the tests and reporting the results. Install ========= pytest-incremental is tested on python 2.6, 2.7. ``pip install pytest-incremental``` ``python setup.py install`` local installation -------------------- You can also just grab the plugin `module `_ file and put in your project path. Then enable it (check `pytest docs `_). Usage ====== Just pass the parameter ``--incremental`` when calling from the command line:: $ py.test --incremental You can also enable it by default adding the following line to your ``pytest.ini``:: [pytest] addopts = --incremental watched packages ------------------ By default all modules collected by pytest will used as dependencies if imported. In order to limit or extend the watched folders you must use the parameter ``--watch-pkg`` Limitations ============== ``pytest-incremental`` looks for imports recursively to find dependencies (using AST). But given the very dynamic nature of python there are still some cases that a module can be affected by a module that are not detected. * `from package import *` modules imported from __all__ in a package are not counted as a dependency * modules imported not using the *import* statement * modules not explictitly imported but used at runtime (i.e. conftest.py when running your tests with pytest) * monkey-patching. (i.e. A imports X. B monkey-patches X. In this case A might depend on B) * others ? Project Details =============== - Project code + issue track on `bitbucket `_ - `Discussion group `_ From prologic at shortcircuit.net.au Wed Apr 27 01:25:49 2011 From: prologic at shortcircuit.net.au (James Mills) Date: Wed, 27 Apr 2011 09:25:49 +1000 Subject: [py-dev] [ANN] plugin: pytest-incremental In-Reply-To: References: Message-ID: On Tue, Apr 26, 2011 at 1:40 AM, Eduardo Schettino wrote: > I have just pushed a new pytest plugin to pypi > (http://pypi.python.org/pypi/pytest-incremental). > > The idea is to execute your tests faster by executing not all of them > but only the "required" ones. Nice work. I might find this useful in my own project :) cheers James -- -- James Mills -- -- "Problems are solved by method" From baptiste.lepilleur at gmail.com Wed Apr 27 22:29:20 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Wed, 27 Apr 2011 22:29:20 +0200 Subject: [py-dev] Empty traceback reported after running tests Message-ID: Hi, I'm running into the following odd output when running some tests: --- py.test --nomagic mphelpertest.py ================================================= test session starts ================================================= platform win32 -- Python 3.2.0 -- pytest-1.3.4 test path 1: mphelpertest.py mphelpertest.py ... ============================================== 3 passed in 1.16 seconds =============================================== Traceback (most recent call last): --- Notes that the traceback is empty. Any idea of what could explain that? The test is installing and (hopefully) uninstalling some logging handler and use the multiprocessing module, so it may be possible that some exceptions are thrown on shutdown when logging handler are closed (but I have not yet managed to reproduce the issue outside py.test). Sources are available here: http://gaiacrtn.free.fr/temp/pytestmp/ Baptiste. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110427/3b1ae9b0/attachment.htm From Ronny.Pfannschmidt at gmx.de Thu Apr 28 08:30:34 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 28 Apr 2011 08:30:34 +0200 Subject: [py-dev] Empty traceback reported after running tests In-Reply-To: References: Message-ID: <1303972234.30044.3.camel@Klappe2> On Wed, 2011-04-27 at 22:29 +0200, Baptiste Lepilleur wrote: > Hi, > > > I'm running into the following odd output when running some tests: > > > --- > py.test --nomagic mphelpertest.py > ================================================= test session starts > ================================================= > platform win32 -- Python 3.2.0 -- pytest-1.3.4 > test path 1: mphelpertest.py > > > mphelpertest.py ... > > > ============================================== 3 passed in 1.16 > seconds =============================================== > Traceback (most recent call last): > --- > > > Notes that the traceback is empty. Any idea of what could explain > that? > > > The test is installing and (hopefully) uninstalling some logging > handler and use the multiprocessing module, so it may be possible that > some exceptions are thrown on shutdown when logging handler are closed > (but I have not yet managed to reproduce the issue outside py.test). > can you retry with a more recent py.test afair this problem has been fixed after 1.3.4 also i would suggest to follow the py.test convention and name the test files test_{foo} instead {foo}test -- Ronny > > Sources are available here: > http://gaiacrtn.free.fr/temp/pytestmp/ > > > Baptiste. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110428/e6100ce9/attachment.pgp From baptiste.lepilleur at gmail.com Thu Apr 28 09:54:45 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Thu, 28 Apr 2011 09:54:45 +0200 Subject: [py-dev] Empty traceback reported after running tests In-Reply-To: <1303972234.30044.3.camel@Klappe2> References: <1303972234.30044.3.camel@Klappe2> Message-ID: 2011/4/28 Ronny Pfannschmidt > On Wed, 2011-04-27 at 22:29 +0200, Baptiste Lepilleur wrote: > > Hi, > [...] > > > > Notes that the traceback is empty. Any idea of what could explain > > that? > > can you retry with a more recent py.test > afair this problem has been fixed after 1.3.4 > > also i would suggest to follow the py.test convention and name the test > files test_{foo} instead {foo}test > > What is the procedure for upgrading an existing install done easy_install? Do I just reinstall using the latest tarball? Or do I need to remove previous module manually (If so is it just py.test module)? Thanks, Baptiste. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110428/bb17a31c/attachment.htm From Ronny.Pfannschmidt at gmx.de Thu Apr 28 09:59:11 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 28 Apr 2011 09:59:11 +0200 Subject: [py-dev] Empty traceback reported after running tests In-Reply-To: References: <1303972234.30044.3.camel@Klappe2> Message-ID: <1303977551.30044.4.camel@Klappe2> On Thu, 2011-04-28 at 09:54 +0200, Baptiste Lepilleur wrote: > > > 2011/4/28 Ronny Pfannschmidt > > On Wed, 2011-04-27 at 22:29 +0200, Baptiste Lepilleur wrote: > > Hi, > [...] > > > > Notes that the traceback is empty. Any idea of what could > explain > > that? > > > can you retry with a more recent py.test > afair this problem has been fixed after 1.3.4 > > also i would suggest to follow the py.test convention and name > the test > files test_{foo} instead {foo}test > > > > What is the procedure for upgrading an existing install done > easy_install? Do I just reinstall using the latest tarball? Or do I > need to remove previous module manually (If so is it just py.test > module)? > easy_install -U pytest since easy_install wont upgrade already installed packages unless asked to -- Ronny > > Thanks, > Baptiste. > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110428/146adc8b/attachment.pgp From baptiste.lepilleur at gmail.com Thu Apr 28 19:59:01 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Thu, 28 Apr 2011 19:59:01 +0200 Subject: [py-dev] Empty traceback reported after running tests In-Reply-To: <1303977551.30044.4.camel@Klappe2> References: <1303972234.30044.3.camel@Klappe2> <1303977551.30044.4.camel@Klappe2> Message-ID: Upgrading to 2.0.3 has indeed fixed the issue (the empty traceback no longer appear). Though I don't really see an issue corresponding to that in the changelog... Thanks, Baptiste. 2011/4/28 Ronny Pfannschmidt > On Thu, 2011-04-28 at 09:54 +0200, Baptiste Lepilleur wrote: > > > > > > 2011/4/28 Ronny Pfannschmidt > > > > On Wed, 2011-04-27 at 22:29 +0200, Baptiste Lepilleur wrote: > > > Hi, > > [...] > > > > > > Notes that the traceback is empty. Any idea of what could > > explain > > > that? > > > > > > can you retry with a more recent py.test > > afair this problem has been fixed after 1.3.4 > > > > also i would suggest to follow the py.test convention and name > > the test > > files test_{foo} instead {foo}test > > > > > > > > What is the procedure for upgrading an existing install done > > easy_install? Do I just reinstall using the latest tarball? Or do I > > need to remove previous module manually (If so is it just py.test > > module)? > > > easy_install -U pytest > since easy_install wont upgrade already installed packages unless asked > to > > -- Ronny > > > > > Thanks, > > Baptiste. > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110428/8bec8e4c/attachment.htm From baptiste.lepilleur at gmail.com Thu Apr 28 20:00:17 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Thu, 28 Apr 2011 20:00:17 +0200 Subject: [py-dev] Empty traceback reported after running tests In-Reply-To: <1303977551.30044.4.camel@Klappe2> References: <1303972234.30044.3.camel@Klappe2> <1303977551.30044.4.camel@Klappe2> Message-ID: Upgrading to 2.0.3 has indeed fixed the issue (the empty traceback no longer appear). Though I don't really see an issue corresponding to that in the changelog... Thanks, Baptiste. 2011/4/28 Ronny Pfannschmidt > On Thu, 2011-04-28 at 09:54 +0200, Baptiste Lepilleur wrote: > > > > > > 2011/4/28 Ronny Pfannschmidt > > > > On Wed, 2011-04-27 at 22:29 +0200, Baptiste Lepilleur wrote: > > > Hi, > > [...] > > > > > > Notes that the traceback is empty. Any idea of what could > > explain > > > that? > > > > > > can you retry with a more recent py.test > > afair this problem has been fixed after 1.3.4 > > > > also i would suggest to follow the py.test convention and name > > the test > > files test_{foo} instead {foo}test > > > > > > > > What is the procedure for upgrading an existing install done > > easy_install? Do I just reinstall using the latest tarball? Or do I > > need to remove previous module manually (If so is it just py.test > > module)? > > > easy_install -U pytest > since easy_install wont upgrade already installed packages unless asked > to > > -- Ronny > > > > > Thanks, > > Baptiste. > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110428/772d74e1/attachment.htm From Ronny.Pfannschmidt at gmx.de Thu Apr 28 20:44:47 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 28 Apr 2011 20:44:47 +0200 Subject: [py-dev] Empty traceback reported after running tests In-Reply-To: References: <1303972234.30044.3.camel@Klappe2> <1303977551.30044.4.camel@Klappe2> Message-ID: <1304016287.30044.10.camel@Klappe2> On Thu, 2011-04-28 at 19:59 +0200, Baptiste Lepilleur wrote: > Upgrading to 2.0.3 has indeed fixed the issue (the empty traceback no > longer appear). > > > Though I don't really see an issue corresponding to that in the > changelog... [snip] Changes between 2.0.0 and 2.0.1 ---------------------------------------------- - refine and unify initial capturing so that it works nicely even if the logging module is used on an early-loaded conftest.py file or plugin. [snip] it's not obvious unless you know what it means. -- Ronny > Thanks, > Baptiste. > > 2011/4/28 Ronny Pfannschmidt > > On Thu, 2011-04-28 at 09:54 +0200, Baptiste Lepilleur wrote: > > > > > > 2011/4/28 Ronny Pfannschmidt > > > > On Wed, 2011-04-27 at 22:29 +0200, Baptiste > Lepilleur wrote: > > > Hi, > > [...] > > > > > > Notes that the traceback is empty. Any idea of > what could > > explain > > > that? > > > > > > can you retry with a more recent py.test > > afair this problem has been fixed after 1.3.4 > > > > also i would suggest to follow the py.test > convention and name > > the test > > files test_{foo} instead {foo}test > > > > > > > > What is the procedure for upgrading an existing install done > > easy_install? Do I just reinstall using the latest tarball? > Or do I > > need to remove previous module manually (If so is it just > py.test > > module)? > > > > easy_install -U pytest > since easy_install wont upgrade already installed packages > unless asked > to > > -- Ronny > > > > > Thanks, > > Baptiste. > > > > > > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110428/7ddf84f7/attachment-0001.pgp From baptiste.lepilleur at gmail.com Fri Apr 29 10:12:26 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Fri, 29 Apr 2011 10:12:26 +0200 Subject: [py-dev] Help understanding pytest_generate_tests related error message Message-ID: Hi, I just upgraded from py.test 1.3.4 to 2.0.3, and I'm running into a new failure related to pytest_generate_tests that I can not make sense of: --- py.test src\nitroz\test_dbo2.py ================================================= test session starts ================================================= platform win32 -- Python 3.2.0 -- pytest-2.0.3 collected 0 items / 1 errors ======================================================= ERRORS ======================================================== ______________________________________ ERROR collecting src/nitroz/test_dbo2.py _______________________________________ src\nitroz\test_dbo2.py:151: in pytest_generate_tests > good_value=good_value) ) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\python.py:550: in addcall > pytest.fail("funcarg %r not used in this function." % name) E Failed: funcarg 'property' not used in this function. =============================================== 1 error in 0.12 seconds =============================================== --- What does this error message means? What 'this function' refer to? Source of the test module can be found there: http://pastebin.com/1D2VArGt Baptiste. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110429/b48ce527/attachment.htm From Ronny.Pfannschmidt at gmx.de Fri Apr 29 12:09:24 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 29 Apr 2011 12:09:24 +0200 Subject: [py-dev] Help understanding pytest_generate_tests related error message In-Reply-To: References: Message-ID: <1304071764.30044.18.camel@Klappe2> On Fri, 2011-04-29 at 10:12 +0200, Baptiste Lepilleur wrote: > Hi, > > > I just upgraded from py.test 1.3.4 to 2.0.3, and I'm running into a > new failure related to pytest_generate_tests that I can not make sense > of: > > > --- > > > py.test src\nitroz\test_dbo2.py > ================================================= test session starts > ================================================= > platform win32 -- Python 3.2.0 -- pytest-2.0.3 > collected 0 items / 1 errors > > > ======================================================= ERRORS > ======================================================== > ______________________________________ ERROR collecting > src/nitroz/test_dbo2.py _______________________________________ > src\nitroz\test_dbo2.py:151: in pytest_generate_tests > > good_value=good_value) ) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest > \python.py:550: in addcall > > pytest.fail("funcarg %r not used in this > function." % name) > E Failed: funcarg 'property' not used in this > function. > =============================================== 1 error in 0.12 > seconds =============================================== > > > --- > > > What does this error message means? What 'this function' refer to? it seems the function is test_property_validation the current check for the metafunc funcarg dict does not consider arguments with default values funcargs thus it complains since you pass in something that has a default at the function level > > Source of the test module can be found > there: http://pastebin.com/1D2VArGt > i would like to suggest splitting the scenario types into multiple test functions, (good/bad values/types) also giving names/id's to the scenarios > > Baptiste. > > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110429/4af90420/attachment.pgp From holger at merlinux.eu Fri Apr 29 13:45:22 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 29 Apr 2011 11:45:22 +0000 Subject: [py-dev] initial take at a generic funcags hook In-Reply-To: <1303332468.8027.7.camel@Klappe2> References: <1303332468.8027.7.camel@Klappe2> Message-ID: <20110429114522.GP4071@merlinux.eu> Hi Ronny, On Wed, Apr 20, 2011 at 22:47 +0200, Ronny Pfannschmidt wrote: > hi, > > i just tried a initial implementation of generic funcargs > to solve the issue i introduced the hook > pytest_reconfigure_funcargs(request) > > the current patch is at > https://bitbucket.org/RonnyPfannschmidt/pytest-patches/qseries?apply=t&qs_apply=reconfigure-funcargs Can you detail your use case? If anything i'd rather go for a hook that would implement the whole funcarg filling protocol. This way one could use generic mechanisms (*) for forcing to execute first/last or wrapping the default protocol. I'd also like to have such a hook get called even if there are no function arguments present. It would allow to make use of request.cached_setup() with session scopes to setup global state for use by Python code. best, holger (*) you can decorate any hook impl with @pytest.mark.trylast, @pytest.mark.tryfirst and you may accept the special __multicall__ arg. these are not yet documented features but they are used in internal plugins - see there for examples. However, i guess the API is ripe to be documented and published sometime. > regards, > Ronny > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From holger at merlinux.eu Fri Apr 29 14:00:38 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 29 Apr 2011 12:00:38 +0000 Subject: [py-dev] [ANN] plugin: pytest-incremental In-Reply-To: References: Message-ID: <20110429120038.GQ4071@merlinux.eu> Hi Eduardo, i like the idea. A few notes: * it's not compatible with pytest-xdist, is it? * i got BSDDB database corruption (i CTRL-Ced the run before) * can you add an example of a project layout and what one would call wrt to watch_pkg? I guess things don't work for me on pytest itself because it has a plugin-based dynamic namespace construction/imports so your AST scanning method does not really see the deps. A different method would be to try to record imports during the import and running of a test. Myself, i also experimented with specifying dependencies manually at some point which also solves the issue when invoking shell commands provided a project - i guess those would not naturally be found by your scanner. best, holger On Mon, Apr 25, 2011 at 23:40 +0800, Eduardo Schettino wrote: > Hi all, > > I have just pushed a new pytest plugin to pypi > (http://pypi.python.org/pypi/pytest-incremental). > > The idea is to execute your tests faster by executing not all of them > but only the "required" ones. > > This is very new (alpha), feedback is welcome :) > > Regards, > Eduardo > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > pytest-incremental > ==================== > > an incremental test runner (pytest plugin) > > > What is an "incremental test runner" ? > ======================================= > > When talking about build-tools it is common to refer to the terms: > > * initial (full) build - all files are compiled > * incremental build (or partial rebuild) - just changed files are compiled > * no-op build - no files are compiled (none changed since last execution) > > So an "incremental test runner" will only re-execute tests that were affected > by changes in the source code since last test execution. > > > How it works ? > ================ > > `pytest-incremental` is a `pytest `_ plugin. So if > you can run your test suite with pytest you can use > `pytest-incremental`. > > The plugin will analyse your python source files and through its > imports define the dependencies of the modules. `doit > `_ is used to keep track of the > dependencies and save results. The plugin will modify how pytest > collect your tests. pytest do the rest of the job of actually running > the tests and reporting the results. > > > Install > ========= > > pytest-incremental is tested on python 2.6, 2.7. > > ``pip install pytest-incremental``` > > ``python setup.py install`` > > local installation > -------------------- > > You can also just grab the plugin `module > `_ > file and put in your project path. Then enable it (check `pytest docs > `_). > > > Usage > ====== > > Just pass the parameter ``--incremental`` when calling from the command line:: > > $ py.test --incremental > > > You can also enable it by default adding the following line to your > ``pytest.ini``:: > > [pytest] > addopts = --incremental > > > watched packages > ------------------ > > By default all modules collected by pytest will used as dependencies > if imported. In order to limit or extend the watched folders you must > use the parameter ``--watch-pkg`` > > > Limitations > ============== > > ``pytest-incremental`` looks for imports recursively to find > dependencies (using AST). But given the very dynamic nature of python > there are still some cases that a module can be affected by a module > that are not detected. > > * `from package import *` modules imported from __all__ in a package > are not counted as a dependency > * modules imported not using the *import* statement > * modules not explictitly imported but used at runtime (i.e. > conftest.py when running your tests with pytest) > * monkey-patching. (i.e. A imports X. B monkey-patches X. In this > case A might depend on B) > * others ? > > > Project Details > =============== > > - Project code + issue track on `bitbucket > `_ > - `Discussion group `_ > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From Ronny.Pfannschmidt at gmx.de Fri Apr 29 15:05:04 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 29 Apr 2011 15:05:04 +0200 Subject: [py-dev] initial take at a generic funcags hook In-Reply-To: <20110429114522.GP4071@merlinux.eu> References: <1303332468.8027.7.camel@Klappe2> <20110429114522.GP4071@merlinux.eu> Message-ID: <1304082304.30044.79.camel@Klappe2> On Fri, 2011-04-29 at 11:45 +0000, holger krekel wrote: > Hi Ronny, > > On Wed, Apr 20, 2011 at 22:47 +0200, Ronny Pfannschmidt wrote: > > hi, > > > > i just tried a initial implementation of generic funcargs > > to solve the issue i introduced the hook > > pytest_reconfigure_funcargs(request) > > > > the current patch is at > > https://bitbucket.org/RonnyPfannschmidt/pytest-patches/qseries?apply=t&qs_apply=reconfigure-funcargs > > Can you detail your use case? > in the setup for the bitbucket hook tests of pypy i have a little mess of interacting funcargs (im intercepting various calls by default) that also needs some additional funcargs to functions that shouldn't need them so having a generic hook to set the interaction between up would be helpful > If anything i'd rather go for a hook that would implement the whole > funcarg filling protocol. This way one could use generic mechanisms (*) > for forcing to execute first/last or wrapping the default protocol. > I'd also like to have such a hook get called even if there are no > function arguments present. It would allow to make use of > request.cached_setup() with session scopes to setup global state > for use by Python code. > i suppose a pytest_configure_funcargs(request) hook should do the builtin implementation would be a tryfirst, my use of "reconfigure" would be the same -- Ronny > best, > holger > > (*) you can decorate any hook impl with @pytest.mark.trylast, > @pytest.mark.tryfirst and you may accept the special __multicall__ arg. > these are not yet documented features but they are used in internal > plugins - see there for examples. However, i guess the API is ripe > to be documented and published sometime. > > > > regards, > > Ronny > > > > > _______________________________________________ > > py-dev mailing list > > py-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/py-dev > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110429/00977b7a/attachment.pgp From schettino72 at gmail.com Fri Apr 29 15:54:27 2011 From: schettino72 at gmail.com (Eduardo Schettino) Date: Fri, 29 Apr 2011 21:54:27 +0800 Subject: [py-dev] [ANN] plugin: pytest-incremental In-Reply-To: <20110429120038.GQ4071@merlinux.eu> References: <20110429120038.GQ4071@merlinux.eu> Message-ID: On Fri, Apr 29, 2011 at 8:00 PM, holger krekel wrote: > * it's not compatible with pytest-xdist, is it? I actually had never tried pytest-xdist... is there anything that I could do to make them compatible? > * i got BSDDB database corruption (i CTRL-Ced the run before) I tried hitting CTRL-C at several points and never got a corruption. At which point you hit Ctrl-C? before the test execution starts? Although I got a bug that it does not detect that not all tests were executed and mark them as successful. > * can you add an example of a project layout The plugin is supposed to work with any project layout... > and what one would call wrt to watch_pkg? By default it will look for changes in all python modules that pass through py.test collection. This way doesnt work well when you try to run tests from a single file like: $ py.test tests/test_foo.py If you try to use the plugin like this it will give an error message saying that you must specify watch_pkg. lets say you have the folders: /tests /my_lib you should call $ py.test --incremental --watch-pkg my_lib tests/test_foo.py (no need to pass the package of the test file itself) It can also be used in case you want to watch for changes in modules that are in another project. for example if you are testing pytest and want to check for changes in dependencies from your "py" package. $ py.test --incremental --watch-pkg my_lib --watch-pkg ../py-trunk/py > > I guess things don't work for me on pytest itself because > it has a plugin-based dynamic namespace construction/imports > so your AST scanning method does not really see the deps. > A different method would be to try to record imports > during the import and running of a test. ?Myself, i also > experimented with specifying dependencies manually at > some point which also solves the issue when invoking > shell commands provided a project - i guess those > would not naturally be found by your scanner. > Yes. There is also the problem of dependencies on text files (or any other non-python files). I think dependencies should really be defined by the user, this AST scanner should be just one way of doing it that works out of the box for most projects. Regards, Eduardo From holger at merlinux.eu Fri Apr 29 20:05:55 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 29 Apr 2011 18:05:55 +0000 Subject: [py-dev] [ANN] plugin: pytest-incremental In-Reply-To: References: <20110429120038.GQ4071@merlinux.eu> Message-ID: <20110429180555.GU4071@merlinux.eu> On Fri, Apr 29, 2011 at 21:54 +0800, Eduardo Schettino wrote: > On Fri, Apr 29, 2011 at 8:00 PM, holger krekel wrote: > > * it's not compatible with pytest-xdist, is it? > I actually had never tried pytest-xdist... is there anything that I > could do to make them compatible? Many people use it for distributing tests to multiple CPUs (or hosts). If you just consider the multi-CPU case the main issue is to make sure the slave processes don't step onto each other when writing or determining your state information. > > * i got BSDDB database corruption (i CTRL-Ced the run before) > I tried hitting CTRL-C at several points and never got a corruption. > At which point you hit Ctrl-C? before the test execution starts? > Although I got a bug that it does not detect that not all tests were > executed and mark them as successful. Not sure i can help with reproducing it. > > * can you add an example of a project layout > The plugin is supposed to work with any project layout... ok ... > > and what one would call wrt to watch_pkg? > By default it will look for changes in all python modules that pass > through py.test collection. This way doesnt work well when you try to > run tests from a single file like: > $ py.test tests/test_foo.py > > If you try to use the plugin like this it will give an error message > saying that you must specify watch_pkg. lets say you have the folders: > /tests > /my_lib > > you should call > $ py.test --incremental --watch-pkg my_lib tests/test_foo.py > (no need to pass the package of the test file itself) > > It can also be used in case you want to watch for changes in modules > that are in another project. for example if you are testing pytest and > want to check for changes in dependencies from your "py" package. > $ py.test --incremental --watch-pkg my_lib --watch-pkg ../py-trunk/py ok, now it's clear that one need to specify file system paths. Throughout the python world "--XYZ-pkg" might mean a module import path like "os.path" or a filesystem path. Maybe good to mention this in the help string for the option - i'd probably rather call it "--watch-path" to disambiguate. > > I guess things don't work for me on pytest itself because > > it has a plugin-based dynamic namespace construction/imports > > so your AST scanning method does not really see the deps. > > A different method would be to try to record imports > > during the import and running of a test. ?Myself, i also > > experimented with specifying dependencies manually at > > some point which also solves the issue when invoking > > shell commands provided a project - i guess those > > would not naturally be found by your scanner. > > > Yes. There is also the problem of dependencies on text files (or any > other non-python files). > I think dependencies should really be defined by the user, this AST > scanner should be just one way of doing it that works out of the box > for most projects. Right. Do you plan to implement a manual way to specify deps for your plugin? sidenote: you may want to announce the next release of the plugin also to the TIP (testing in python) list - http://lists.idyll.org/listinfo/testing-in-python - a number of people are rather following there rather than here. best, holger > Regards, > Eduardo > From baptiste.lepilleur at gmail.com Fri Apr 29 22:18:20 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Fri, 29 Apr 2011 22:18:20 +0200 Subject: [py-dev] Help understanding pytest_generate_tests related error message In-Reply-To: <1304071764.30044.18.camel@Klappe2> References: <1304071764.30044.18.camel@Klappe2> Message-ID: 2011/4/29 Ronny Pfannschmidt > [...] > > E Failed: funcarg 'property' not used in this > > function. > [...] > it seems the function is test_property_validation > the current check for the metafunc funcarg dict does not consider > arguments with default values funcargs > thus it complains since you pass in something that has a default at the > function level > It was indeed the cause. Changing: def test_property_validation( validation_scenario_iter, property=None, value=None, good_value=None ): to: def test_property_validation( validation_scenario_iter, property, value, good_value ): fixed the issue. This is weird as it used to work in 1.3.4... > > > > Source of the test module can be found > > there: http://pastebin.com/1D2VArGt > > > > i would like to suggest splitting the scenario types into multiple test > functions, (good/bad values/types) > > I'm not sure how I would go about that: it would means going through the VALIDATION_SCENARIOS test data for each test function, and duplicating some setup code of the test function... also giving names/id's to the scenarios I did not know it was possible to give name. I added this and it greatly improve the output! Thanks a lot for helping figuring this out, Baptiste. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110429/30c5d37f/attachment.htm From Ronny.Pfannschmidt at gmx.de Fri Apr 29 22:25:04 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 29 Apr 2011 22:25:04 +0200 Subject: [py-dev] Help understanding pytest_generate_tests related error message In-Reply-To: References: <1304071764.30044.18.camel@Klappe2> Message-ID: <1304108704.30044.81.camel@Klappe2> On Fri, 2011-04-29 at 22:18 +0200, Baptiste Lepilleur wrote: > 2011/4/29 Ronny Pfannschmidt > > [...] > > E Failed: funcarg 'property' not used in > this > > function. > [...] > it seems the function is test_property_validation > the current check for the metafunc funcarg dict does not > consider > arguments with default values funcargs > thus it complains since you pass in something that has a > default at the > function level > > > It was indeed the cause. Changing: > def test_property_validation( validation_scenario_iter, property=None, > value=None, good_value=None ): > > > to: > > > def test_property_validation( validation_scenario_iter, property, > value, good_value ): > > > fixed the issue. This is weird as it used to work in 1.3.4... > > > > > Source of the test module can be found > > there: http://pastebin.com/1D2VArGt > > > > > i would like to suggest splitting the scenario types into > multiple test > functions, (good/bad values/types) > > > > I'm not sure how I would go about that: it would means going through > the VALIDATION_SCENARIOS test data for each test function, and > duplicating some setup code of the test function... unless i'm misstaken, only the creation of the class is shared which means that could actually be something like a funcarg or a param to a funcarg -- Ronny > > > also giving names/id's to the scenarios > I did not know it was possible to give name. I added this and it > greatly improve the output! > > > Thanks a lot for helping figuring this out, > Baptiste. > > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110429/1b93bac9/attachment.pgp From holger at merlinux.eu Sat Apr 30 16:22:25 2011 From: holger at merlinux.eu (holger krekel) Date: Sat, 30 Apr 2011 14:22:25 +0000 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: References: <20110405201612.GW16231@merlinux.eu> <20110407061715.GE16231@merlinux.eu> Message-ID: <20110430142225.GY4071@merlinux.eu> On Thu, Apr 07, 2011 at 12:29 -0400, Vyacheslav Rafalskiy wrote: > >> >> 3. In my pytest_configure() there are numerous conditions when it can > >> >> fail. For this > >> >> conditions I have exceptions with specially crafted messages, intended > >> >> for different > >> >> people. Now they are gone, replaced by a long and scary listing with every line > >> >> prepended with INTERNALERROR. What is internal about it? Can I continue managing > >> >> my configuration errors? > >> > > >> > Sure, you should be able to. I guess it was a bit of an incident that > >> > failures in pytest_configure were shown nicely. ?I am not quite sure > >> > immediately what the best way to relay "global" configuration messages > >> > to the user is. ?If you'd use "funcargs" and the "session" scope for > >> > setting up resources then it would show nicely. ?Feel free to open > >> > a feature/enhancement request to have py.test show failures > >> > in sessionstart or configure hooks in a way that helps you. > >> > This way we can write a specific test and make sure it works > >> > also in the future. > >> > >> If I understand correctly, using funcargs means that every test function > >> of hundreds I have in several suites should include a reference to the > >> global setup funcarg. It seems a non-starter. > >> > >> I guess I will stick to the old version and try to think of something > >> ?to enter in a feature request. > > > > sure, makes sense. ?isn't your main issue that upon actual test start > > (after collection) you want to do some configuration steps which might > > result in error conditions which you'd like to see nicely reported to > > the user? > > That's exactly my point. > > >(sidenote: configure and even sessionstart hooks are both a bit > > not 100% right because they happen even on the master side of a distributed > > test run and the master side does not collect or run tests at all) > > I see. Perhaps something like setup_package() in the top-level __init__.py > could be a solution? I guess you mean an __init__.py file of a test directory. With a layout of test dirs within an application this might mean one has to put this setup into the main package __init__.py and mixing test code and application code is often not a good idea. So i'd rather put it into a conftest.py file as a normal hook. Maybe "pytest_pyfunc_setup(request)" would be good where request is the same object as for the funcarg factories. You could then write: # content of conftest.py def pytest_pyfunc_setup(request): val = request.cached_setup(setup=makeval, scope="session") # use val for some global setting of the package Alternatively we could see to call something like: def pytest_setup_testloop(config): val = makeval() # use val for some global setting of the package def pytest_teardown_testloop(config): ... which would be called once for a test process. best, holger > Vyacheslav > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From memedough at gmail.com Sun May 1 11:15:16 2011 From: memedough at gmail.com (meme dough) Date: Sun, 1 May 2011 19:15:16 +1000 Subject: [py-dev] conftest global scope Message-ID: Hi, pytest-cov has an issue ticket that conftest global scope has no coverage reported. I don't think it will be possible to cover though. My understanding is that conftest file is local plugin which pytest will load quite early. It then can have hooks in conftest like to add args or change args. Coverage starts at session start hook which I think most appropriate. The hooks in conftest (like configure) could change things to turn coverage on / off or set coverage paths. So starting coverage any earlier than session start may not be good. Is this correct with conftest? It is not reloaded is it? It gets loaded early and hooks in it called to modify pytest operations. So no way we could get coverage at conftest global scope anyway? :) From baptiste.lepilleur at gmail.com Sun May 1 11:59:15 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Sun, 1 May 2011 11:59:15 +0200 Subject: [py-dev] Help understanding pytest_generate_tests related error message In-Reply-To: <1304108704.30044.81.camel@Klappe2> References: <1304071764.30044.18.camel@Klappe2> <1304108704.30044.81.camel@Klappe2> Message-ID: 2011/4/29 Ronny Pfannschmidt > On Fri, 2011-04-29 at 22:18 +0200, Baptiste Lepilleur wrote: > > [...] > > I'm not sure how I would go about that: it would means going through > > the VALIDATION_SCENARIOS test data for each test function, and > > duplicating some setup code of the test function... > > unless i'm misstaken, only the creation of the class is shared > which means that could actually be something like a funcarg or a param > to a funcarg > Indeed. I did the split but I feel that the scenario types/test function association is very clumsy: http://pastebin.com/gTsVJLWK Is there a simpler way to do this? Ideally, I'd rather have a simple way to associate a test function with a test parameter generator. I'm thinking of something like this for example: @parametrized( _generate_property_validation_tests, ['good_values'] ) def test_property_validation_good_values( scenario_type, dbo_class, value, good_value ): ... The parametrized decorator would call _generate_property_validation_tests( metafunc, ['good_values'] ). Is there a way to do that? Thanks, Baptiste. > > -- Ronny > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110501/8424f174/attachment.htm From baptiste.lepilleur at gmail.com Sun May 1 12:08:16 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Sun, 1 May 2011 12:08:16 +0200 Subject: [py-dev] How to run a given test by name? Message-ID: Hi, typically when troubleshooting multiple test failures, I want to be able to run a single parametrized test case. How can you tell py.test to run only tests matching a specific name. For example, I'd like to be able to run: py.test src --filter-by-name "test_mandatory_property[ZText]" This would run all tests with a name matching "test_mandatory_property[ZText]". Is there a way to do that? Baptiste. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110501/06d643bf/attachment.htm From holger at merlinux.eu Sun May 1 12:22:41 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 1 May 2011 10:22:41 +0000 Subject: [py-dev] How to run a given test by name? In-Reply-To: References: Message-ID: <20110501102241.GB4071@merlinux.eu> On Sun, May 01, 2011 at 12:08 +0200, Baptiste Lepilleur wrote: > typically when troubleshooting multiple test failures, I want to be able to > run a single parametrized test case. > > How can you tell py.test to run only tests matching a specific name. > > For example, I'd like to be able to run: > > py.test src --filter-by-name "test_mandatory_property[ZText]" > > This would run all tests with a name > matching "test_mandatory_property[ZText]". > > Is there a way to do that? two possibilites. You can do a run with "py.test -rf" which will report test IDs for all failures. You can then pass one or more of those IDs to a py.test invocation. Secondly you can use the keyword option try something like "-k ZTEXT", see option help. holger From Ronny.Pfannschmidt at gmx.de Sun May 1 12:29:09 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Sun, 01 May 2011 12:29:09 +0200 Subject: [py-dev] Help understanding pytest_generate_tests related error message In-Reply-To: References: <1304071764.30044.18.camel@Klappe2> <1304108704.30044.81.camel@Klappe2> Message-ID: <1304245750.30044.89.camel@Klappe2> On Sun, 2011-05-01 at 11:59 +0200, Baptiste Lepilleur wrote: > 2011/4/29 Ronny Pfannschmidt > > On Fri, 2011-04-29 at 22:18 +0200, Baptiste Lepilleur wrote: > > [...] > > I'm not sure how I would go about that: it would means going > through > > the VALIDATION_SCENARIOS test data for each test function, > and > > duplicating some setup code of the test function... > > > unless i'm misstaken, only the creation of the class is shared > which means that could actually be something like a funcarg or > a param > to a funcarg > > > Indeed. I did the split but I feel that the scenario types/test > function association is very clumsy: > http://pastebin.com/gTsVJLWK > > > Is there a simpler way to do this? Ideally, I'd rather have a simple > way to associate a test function with a test parameter generator. I'm > thinking of something like this for example: > > > @parametrized( _generate_property_validation_tests, ['good_values'] ) > def test_property_validation_good_values( scenario_type, dbo_class, > value, good_value ): > ... > > > The parametrized decorator would > call _generate_property_validation_tests( metafunc, ['good_values'] ). > > > Is there a way to do that? you can use the pytest.mark tool to make a decorator which annotates the value types nicely -- Ronny > > > Thanks, > Baptiste. > > > -- Ronny > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110501/5edd0e49/attachment-0001.pgp From baptiste.lepilleur at gmail.com Sun May 1 12:41:06 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Sun, 1 May 2011 12:41:06 +0200 Subject: [py-dev] How to run a given test by name? In-Reply-To: <20110501102241.GB4071@merlinux.eu> References: <20110501102241.GB4071@merlinux.eu> Message-ID: options -k was what I was looking for. I somehow managed to miss it. Thanks, Baptiste. 2011/5/1 holger krekel > On Sun, May 01, 2011 at 12:08 +0200, Baptiste Lepilleur wrote: > > typically when troubleshooting multiple test failures, I want to be able > to > > run a single parametrized test case. > > > > How can you tell py.test to run only tests matching a specific name. > > > > For example, I'd like to be able to run: > > > > py.test src --filter-by-name "test_mandatory_property[ZText]" > > > > This would run all tests with a name > > matching "test_mandatory_property[ZText]". > > > > Is there a way to do that? > > two possibilites. You can do a run with "py.test -rf" which will report > test IDs > for all failures. You can then pass one or more of those IDs to a py.test > invocation. > Secondly you can use the keyword option try something like "-k ZTEXT", see > option help. > > holger > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110501/35fe9a99/attachment.htm From baptiste.lepilleur at gmail.com Sun May 1 13:47:19 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Sun, 1 May 2011 13:47:19 +0200 Subject: [py-dev] Help understanding pytest_generate_tests related error message In-Reply-To: <1304245750.30044.89.camel@Klappe2> References: <1304071764.30044.18.camel@Klappe2> <1304108704.30044.81.camel@Klappe2> <1304245750.30044.89.camel@Klappe2> Message-ID: 2011/5/1 Ronny Pfannschmidt > On Sun, 2011-05-01 at 11:59 +0200, Baptiste Lepilleur wrote: > ... > > @parametrized( _generate_property_validation_tests, ['good_values'] ) > > def test_property_validation_good_values( scenario_type, dbo_class, > > value, good_value ): > > ... > > > > > > The parametrized decorator would > > call _generate_property_validation_tests( metafunc, ['good_values'] ). > > > > > > Is there a way to do that? > you can use the pytest.mark tool to make a decorator which annotates > the value types nicely > Great, this is exactly what I wanted! The pytest_generate_tests function is now generic: def pytest_generate_tests(metafunc): if hasattr( metafunc.function, 'parametrized' ): args = metafunc.function.parametrized.args kwargs = metafunc.function.parametrized.kwargs test_generator, args = args[0], args[1:] test_generator( metafunc, *args, **kwargs ) And the parameter generator function can be specified as I wanted: @py.test.mark.parametrized( _generate_property_validation_tests, ['good_values'] ) def test_property_validation_good_values( dbo_class, value, good_value ): ... Here is the source: http://pastebin.com/kfgvM7L9 The only pitfall to be aware of is that you must always provide two parameters to @py.test.mark.parametrized otherwise MarkDecorator will think that the first parameter, which is callable, must be decorated. Thanks a lot for your help, Baptiste. > > -- Ronny > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110501/e2261310/attachment.htm From schettino72 at gmail.com Mon May 2 05:33:39 2011 From: schettino72 at gmail.com (Eduardo Schettino) Date: Mon, 2 May 2011 11:33:39 +0800 Subject: [py-dev] [ANN] plugin: pytest-incremental In-Reply-To: <20110429180555.GU4071@merlinux.eu> References: <20110429120038.GQ4071@merlinux.eu> <20110429180555.GU4071@merlinux.eu> Message-ID: On Sat, Apr 30, 2011 at 2:05 AM, holger krekel wrote: > On Fri, Apr 29, 2011 at 21:54 +0800, Eduardo Schettino wrote: >> On Fri, Apr 29, 2011 at 8:00 PM, holger krekel wrote: >> > * it's not compatible with pytest-xdist, is it? I gave it a try... Right now it is definitely not compatible. In the plugin test results are saved internally on pytest_runtest_makereport and processed at pytest_sessionfinish. I could see 2 issues: 1) pytest_sessionfinish is called once for every subprocess and once for the main process. So on sessionfinish I need to detect if it is a subprocess and send the result to the runner in the main process. In the main process it would work as it is today. Any clues how to that? I looked into junitxml plugin but it seems it does not work together with xdist also. 2) pytest_runtest_makereport when using xdist "call.result" is None (or not defined) when the test fails. I guess this is a bug on xdist. > > ok, now it's clear that one need to specify file system paths. > Throughout the python world "--XYZ-pkg" might mean a module > import path like "os.path" or a filesystem path. ?Maybe good > to mention this in the help string for the option - i'd probably > rather call it "--watch-path" to disambiguate. thanks. fixed. > > Right. ?Do you plan to implement a manual way to specify deps > for your plugin? I could do that, but I don't know how :) I will join the py.test IRC so we can talk about it. I've been thinking in going the other way. doit calling pytest instead of pytest calling doit in a plugin. I've done that before.. the problem is that I am not sure it can be done without losing all the features from py.test runner. > > sidenote: you may want to announce the next release of the plugin > also to the TIP (testing in python) list - > http://lists.idyll.org/listinfo/testing-in-python - > a number of people are rather following there rather than here. > I announced there. this thread was my motivation to write the plugin (http://lists.idyll.org/pipermail/testing-in-python/2011-April/004083.html) > best, > holger > cheers, eduardo From memedough at gmail.com Mon May 2 12:01:13 2011 From: memedough at gmail.com (meme dough) Date: Mon, 2 May 2011 20:01:13 +1000 Subject: [py-dev] [ANN] plugin: pytest-incremental In-Reply-To: References: <20110429120038.GQ4071@merlinux.eu> <20110429180555.GU4071@merlinux.eu> Message-ID: Hi, pytest-cov has pytest-xdist support. It was a big reason for writing it (pytest-cov that is) and holger kindly added to the protocol between master / slave to allow info to travel that pytest-cov needed. It is generic so you can use it too. Have a look at: https://bitbucket.org/memedough/pytest-cov/src/a7d65cc2458a/pytest_cov.py At pytest_sessionstart you can check if you are a master process or a slave process. The most important thing to look for is if config has slaveinput. If the config has that slaveinput then the process is a slave. My one work with older pytest and 2.x too. So that give you answer to non distributed testing process, master process or slave process. On master pytest_configure_node you get node. On that put info in the node.slaveinput and pytest send it to the slave. Look at DistMaster: https://bitbucket.org/memedough/cov-core/src/b9a3c9756366/cov_core.py On slave at pytest_sessionstart you can get the slaveinput. Look at DistSlave. On slave at pytest_sessionfinish you put info in that config.slaveoutput dict to go back to the master. Then on the master at pytest_testnodedown you get the slaveoutput from the node that went down. So then at pytest_sessionfinish on master you know all slaves have finished and sent back info that been collected by master so can do stuff with it. Feel free to use anything from pytest-cov if useful for you. :) On 2 May 2011 13:33, Eduardo Schettino wrote: > On Sat, Apr 30, 2011 at 2:05 AM, holger krekel wrote: >> On Fri, Apr 29, 2011 at 21:54 +0800, Eduardo Schettino wrote: >>> On Fri, Apr 29, 2011 at 8:00 PM, holger krekel wrote: >>> > * it's not compatible with pytest-xdist, is it? > I gave it a try... Right now it is definitely not compatible. > > In the plugin test results are saved internally on > pytest_runtest_makereport and processed at pytest_sessionfinish. I > could see 2 issues: > > 1) pytest_sessionfinish is called once for every subprocess and once > for the main process. So on sessionfinish I need to detect if it is a > subprocess and send the result to the runner in the main process. In > the main process it would work as it is today. Any clues how to that? > I looked into junitxml plugin but it seems it does not work together > with xdist also. > > 2) pytest_runtest_makereport when using xdist "call.result" is None > (or not defined) when the test fails. I guess this is a bug on xdist. > > >> >> ok, now it's clear that one need to specify file system paths. >> Throughout the python world "--XYZ-pkg" might mean a module >> import path like "os.path" or a filesystem path. ?Maybe good >> to mention this in the help string for the option - i'd probably >> rather call it "--watch-path" to disambiguate. > thanks. fixed. > >> >> Right. ?Do you plan to implement a manual way to specify deps >> for your plugin? > I could do that, but I don't know how :) ? I will join the py.test IRC > so we can talk about it. > > I've been thinking in going the other way. doit calling pytest instead > of pytest calling doit in a plugin. I've done that before.. the > problem is that I am not sure it can be done without losing all the > features from py.test runner. > >> >> sidenote: you may want to announce the next release of the plugin >> also to the TIP (testing in python) list - >> http://lists.idyll.org/listinfo/testing-in-python - >> a number of people are rather following there rather than here. >> > I announced there. this thread was my motivation to write the plugin > (http://lists.idyll.org/pipermail/testing-in-python/2011-April/004083.html) > >> best, >> holger >> > cheers, > ?eduardo > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From rafalskiy at gmail.com Mon May 2 23:40:33 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Mon, 2 May 2011 17:40:33 -0400 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: <20110430142225.GY4071@merlinux.eu> References: <20110405201612.GW16231@merlinux.eu> <20110407061715.GE16231@merlinux.eu> <20110430142225.GY4071@merlinux.eu> Message-ID: On Sat, Apr 30, 2011 at 10:22 AM, holger krekel wrote: > On Thu, Apr 07, 2011 at 12:29 -0400, Vyacheslav Rafalskiy wrote: >> >> >(sidenote: configure and even sessionstart hooks are both a bit >> > not 100% right because they happen even on the master side of a distributed >> > test run and the master side does not collect or run tests at all) >> >> I see. Perhaps something like setup_package() in the top-level __init__.py >> could be a solution? > > I guess you mean an __init__.py file of a test directory. > With a layout of test dirs within an application this might mean > one has to put this setup into the main package __init__.py > and mixing test code and application code is often not a good idea. Yes, exactly. In my case of functional testing I don't even have application code here. When I start the tests I tell the runner where in the network the system under test is. > > So i'd rather put it into a conftest.py file as a normal hook. > Maybe "pytest_pyfunc_setup(request)" would be good where request > is the same object as for the funcarg factories. > > You could then write: > > ? ?# content of conftest.py > ? ?def pytest_pyfunc_setup(request): > ? ? ? ?val = request.cached_setup(setup=makeval, scope="session") > ? ? ? ?# use val for some global setting of the package > > Alternatively we could see to call something like: > > ? ?def pytest_setup_testloop(config): > ? ? ? ?val = makeval() > ? ? ? ?# use val for some global setting of the package > > ? ?def pytest_teardown_testloop(config): > ? ? ? ?... > > which would be called once for a test process. The reason why I suggested setup_package() is that you already have setup_function, setup_method, setup_class and setup_module so the former would just complete the line-up. And the natural place for it would be __init__.py of that package. On the other hand, you can put conftest.py in every folder, which can do precisely the same thing as well as many others. The question is which way is more intuitive and results in cleaner code. The answer is perhaps "It depends". I like setup_module(module) because it lets me dump the configuration straight into the namespace where I use it and setup_package(package) could do the same. But all in all, whatever works is fine with me. Thanks, Vyacheslav > > best, > holger > >> Vyacheslav >> _______________________________________________ >> py-dev mailing list >> py-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/py-dev >> > From holger at merlinux.eu Tue May 3 11:48:14 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 3 May 2011 09:48:14 +0000 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: References: <20110405201612.GW16231@merlinux.eu> <20110407061715.GE16231@merlinux.eu> <20110430142225.GY4071@merlinux.eu> Message-ID: <20110503094814.GU4071@merlinux.eu> On Mon, May 02, 2011 at 17:40 -0400, Vyacheslav Rafalskiy wrote: > On Sat, Apr 30, 2011 at 10:22 AM, holger krekel wrote: > > On Thu, Apr 07, 2011 at 12:29 -0400, Vyacheslav Rafalskiy wrote: > > >> > >> >(sidenote: configure and even sessionstart hooks are both a bit > >> > not 100% right because they happen even on the master side of a distributed > >> > test run and the master side does not collect or run tests at all) > >> > >> I see. Perhaps something like setup_package() in the top-level __init__.py > >> could be a solution? > > > > I guess you mean an __init__.py file of a test directory. > > With a layout of test dirs within an application this might mean > > one has to put this setup into the main package __init__.py > > and mixing test code and application code is often not a good idea. > > Yes, exactly. In my case of functional testing I don't even have > application code here. > When I start the tests I tell the runner where in the network the > system under test is. > > > > > So i'd rather put it into a conftest.py file as a normal hook. > > Maybe "pytest_pyfunc_setup(request)" would be good where request > > is the same object as for the funcarg factories. > > > > You could then write: > > > > ? ?# content of conftest.py > > ? ?def pytest_pyfunc_setup(request): > > ? ? ? ?val = request.cached_setup(setup=makeval, scope="session") > > ? ? ? ?# use val for some global setting of the package > > > > Alternatively we could see to call something like: > > > > ? ?def pytest_setup_testloop(config): > > ? ? ? ?val = makeval() > > ? ? ? ?# use val for some global setting of the package > > > > ? ?def pytest_teardown_testloop(config): > > ? ? ? ?... > > > > which would be called once for a test process. > > The reason why I suggested setup_package() is that you already have > setup_function, setup_method, setup_class and setup_module so > the former would just complete the line-up. And the natural place > for it would be __init__.py of that package. > > On the other hand, you can put conftest.py in every folder, which > can do precisely the same thing as well as many others. The > question is which way is more intuitive and results in cleaner code. > The answer is perhaps "It depends". > > I like setup_module(module) because it lets me dump the configuration > straight into the namespace where I use it and setup_package(package) > could do the same. good line of reasoning. It's mostly my intuition making me hesitant to add setup_package like you suggest. And i wonder what it is about :) Maybe it's that the root namespace of a test directory is often called "testing" or "tests" (the "test" one is taken by Python stdlib already). And therefore you would end up having to do "import testing" and then use global state with something like "testing.STATE". But i guess this doesn't look so bad to you, does it? :) (The plugin/conftest system is designed such that it hardly requires any imports to manage test state.) Any more opinions on setup_package(package)? If others find it useful as well, i will consider introducing it with pytest-2.1. best, holger From rafalskiy at gmail.com Wed May 4 16:53:12 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Wed, 4 May 2011 10:53:12 -0400 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: <20110503094814.GU4071@merlinux.eu> References: <20110405201612.GW16231@merlinux.eu> <20110407061715.GE16231@merlinux.eu> <20110430142225.GY4071@merlinux.eu> <20110503094814.GU4071@merlinux.eu> Message-ID: On Tue, May 3, 2011 at 5:48 AM, holger krekel wrote: > On Mon, May 02, 2011 at 17:40 -0400, Vyacheslav Rafalskiy wrote: >> On Sat, Apr 30, 2011 at 10:22 AM, holger krekel wrote: >> > On Thu, Apr 07, 2011 at 12:29 -0400, Vyacheslav Rafalskiy wrote: >> >> >> >> >> >(sidenote: configure and even sessionstart hooks are both a bit >> >> > not 100% right because they happen even on the master side of a distributed >> >> > test run and the master side does not collect or run tests at all) >> >> >> >> I see. Perhaps something like setup_package() in the top-level __init__.py >> >> could be a solution? >> > >> > I guess you mean an __init__.py file of a test directory. >> > With a layout of test dirs within an application this might mean >> > one has to put this setup into the main package __init__.py >> > and mixing test code and application code is often not a good idea. >> >> Yes, exactly. In my case of functional testing I don't even have >> application code here. >> When I start the tests I tell the runner where in the network the >> system under test is. >> >> > >> > So i'd rather put it into a conftest.py file as a normal hook. >> > Maybe "pytest_pyfunc_setup(request)" would be good where request >> > is the same object as for the funcarg factories. >> > >> > You could then write: >> > >> > ? ?# content of conftest.py >> > ? ?def pytest_pyfunc_setup(request): >> > ? ? ? ?val = request.cached_setup(setup=makeval, scope="session") >> > ? ? ? ?# use val for some global setting of the package >> > >> > Alternatively we could see to call something like: >> > >> > ? ?def pytest_setup_testloop(config): >> > ? ? ? ?val = makeval() >> > ? ? ? ?# use val for some global setting of the package >> > >> > ? ?def pytest_teardown_testloop(config): >> > ? ? ? ?... >> > >> > which would be called once for a test process. >> >> The reason why I suggested setup_package() is that you already have >> setup_function, setup_method, setup_class and setup_module so >> the former would just complete the line-up. And the natural place >> for it would be __init__.py of that package. >> >> On the other hand, you can put conftest.py in every folder, which >> can do precisely the same thing as well as many others. The >> question is which way is more intuitive and results in cleaner code. >> The answer is perhaps "It depends". >> >> I like setup_module(module) because it lets me dump the configuration >> straight into the namespace where I use it and setup_package(package) >> could do the same. > > good line of reasoning. ?It's mostly my intuition making me hesitant > to add setup_package like you suggest. ?And i wonder what it is about :) > Maybe it's that the root namespace of a test directory is often called > "testing" or "tests" ?(the "test" one is taken by Python stdlib already). > And therefore you would end up having to do "import testing" and > then use global state with something like "testing.STATE". > But i guess this doesn't look so bad to you, does it? :) > (The plugin/conftest system is designed such that it hardly > requires any imports to manage test state.) > > Any more opinions on setup_package(package)? If others find it useful > as well, i will consider introducing it with pytest-2.1. I guess I will have to withdraw the idea. Having to explicitly import the test package does not look nice at all. conftest.py rules! As to the two alternatives above I'd rather use pytest_setup_testloop(config) with direct access to config. Regards, Vyacheslav > > best, > holger > From schettino72 at gmail.com Fri May 6 19:29:28 2011 From: schettino72 at gmail.com (Eduardo Schettino) Date: Sat, 7 May 2011 01:29:28 +0800 Subject: [py-dev] [ANN] plugin: pytest-incremental In-Reply-To: References: <20110429120038.GQ4071@merlinux.eu> <20110429180555.GU4071@merlinux.eu> Message-ID: On Mon, May 2, 2011 at 6:01 PM, meme dough wrote: > Hi, > > pytest-cov has pytest-xdist support. It was a big reason for writing > it (pytest-cov that is) and holger kindly added to the protocol > between master / slave to allow info to travel that pytest-cov needed. > It is generic so you can use it too. > > Have a look at: > > https://bitbucket.org/memedough/pytest-cov/src/a7d65cc2458a/pytest_cov.py > > At pytest_sessionstart you can check if you are a master process or a > slave process. The most important thing to look for is if config has > slaveinput. If the config has that slaveinput then the process is a > slave. My one work with older pytest and 2.x too. So that give you > answer to non distributed testing process, master process or slave > process. > > On master pytest_configure_node you get node. On that put info in the > node.slaveinput and pytest send it to the slave. Look at DistMaster: > > https://bitbucket.org/memedough/cov-core/src/b9a3c9756366/cov_core.py > > On slave at pytest_sessionstart you can get the slaveinput. Look at > DistSlave. > > On slave at pytest_sessionfinish you put info in that > config.slaveoutput dict to go back to the master. > > Then on the master at pytest_testnodedown you get the slaveoutput from > the node that went down. > > So then at pytest_sessionfinish on master you know all slaves have > finished and sent back info that been collected by master so can do > stuff with it. > > Feel free to use anything from pytest-cov if useful for you. > > :) > > Thanks Dough, it was very useful... now pytest-incremental works for me with xdist -n. i could also run --load=each but not sure it has the expected behaviour... the code is here https://bitbucket.org/schettino72/pytest-incremental cheers, eduardo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110507/5debb504/attachment.htm From baptiste.lepilleur at gmail.com Wed May 18 17:42:30 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Wed, 18 May 2011 17:42:30 +0200 Subject: [py-dev] What is the recommended way to run test with a coverage report? Message-ID: The pytest documentation page indicates that it is supported, but provides no pointer on how do to this... Doing a search seem to reveal multiple plug-ins to do that. What is the recommended one? I'm working on Windows XP / Python 2.6 & 3.2. By the way, is there a centralized list of useful plug-ins for pytest ? Baptiste. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110518/17f0bfa0/attachment.htm From holger at merlinux.eu Wed May 18 22:04:43 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 18 May 2011 20:04:43 +0000 Subject: [py-dev] What is the recommended way to run test with a coverage report? In-Reply-To: References: Message-ID: <20110518200443.GA20287@merlinux.eu> Hey Baptiste, On Wed, May 18, 2011 at 17:42 +0200, Baptiste Lepilleur wrote: > The pytest documentation page indicates that it is supported, but provides > no pointer on how do to this... > Doing a search seem to reveal multiple plug-ins to do that. What is the > recommended one? I'm working on Windows XP / Python 2.6 & 3.2. sorry about that. The plugin is named pytest-cov, see here http://pypi.python.org/pypi/pytest-cov If it doesn't work for you i am sure Meme (also here on the list) can answer questions or to issues. > By the way, is there a centralized list of useful plug-ins for pytest ? not really. However, i recommend to install "pip" and then type: pip search pytest to get a good list. best, holger > > Baptiste. > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From baptiste.lepilleur at gmail.com Thu May 19 23:53:39 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Thu, 19 May 2011 23:53:39 +0200 Subject: [py-dev] What is the recommended way to run test with a coverage report? In-Reply-To: <20110518200443.GA20287@merlinux.eu> References: <20110518200443.GA20287@merlinux.eu> Message-ID: Thanks for the pointer concerning pip, I did not know you could search using it. Now, I running into a strange missing import error when trying to the test with coverage: E ImportError: No module named restdoc Any ideas what could be the cause and what this module is? Detail below: py.test --cov src\nitroz\test_collections.py ================================================= test session starts ================================================= platform win32 -- Python 3.2.0 -- pytest-2.0.3 collected 0 items / 1 errors Coverage.py warning: No data was collected. ======================================================= ERRORS ======================================================== _________________________________________________ ERROR collecting . __________________________________________________ C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:312: in visit > for x in Visitor(fil, rec, ignore, bf, sort).gen(self): C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: in gen > for p in self.gen(subdir): C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: in gen > for p in self.gen(subdir): C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: in gen > for p in self.gen(subdir): C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:347: in gen > dirs = self.optsort([p for p in entries C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:348: in > if p.check(dir=1) and (rec is None or rec(p))]) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\main.py:450: in _recurse > ihook.pytest_collect_directory(path=path, parent=self) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\main.py:118: in call_matching_hooks > plugins = self.config._getmatchingplugins(self.fspath) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:283: in _getmatchingplugins > plugins += self._conftest.getconftestmodules(fspath) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:185: in getconftestmodules > clist.append(self.importconftest(conftestpath)) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:221: in importconftest > self._postimport(mod) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:226: in _postimport > self._onimport(mod) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:272: in _onimportconftest > self.pluginmanager.consider_conftest(conftestmodule) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:181: in consider_conftest > self.consider_module(conftestmodule) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:189: in consider_module > self.import_plugin(spec) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:202: in import_plugin > return self.import_plugin(modname[7:]) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:197: in import_plugin > mod = importplugin(modname) C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:323: in importplugin > return __import__(importspec, None, None, '__doc__') E ImportError: No module named restdoc ----------------------------------- coverage: platform win32, python 3.2.0-final-0 ------------------------------------ Name Stmts Miss Cover --------------------------- ============================================== 1 error in 18.02 seconds =============================================== 2011/5/18 holger krekel > Hey Baptiste, > > On Wed, May 18, 2011 at 17:42 +0200, Baptiste Lepilleur wrote: > > The pytest documentation page indicates that it is supported, but > provides > > no pointer on how do to this... > > > Doing a search seem to reveal multiple plug-ins to do that. What is the > > recommended one? I'm working on Windows XP / Python 2.6 & 3.2. > > sorry about that. The plugin is named pytest-cov, see here > > http://pypi.python.org/pypi/pytest-cov > > If it doesn't work for you i am sure Meme (also here on the list) > can answer questions or to issues. > > > By the way, is there a centralized list of useful plug-ins for pytest ? > > not really. However, i recommend to install "pip" and then type: > > pip search pytest > > to get a good list. > > best, > holger > > > > > > > Baptiste. > > > _______________________________________________ > > py-dev mailing list > > py-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/py-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110519/d3c7a934/attachment.htm From memedough at gmail.com Sun May 22 06:14:43 2011 From: memedough at gmail.com (meme dough) Date: Sun, 22 May 2011 14:14:43 +1000 Subject: [py-dev] What is the recommended way to run test with a coverage report? In-Reply-To: References: <20110518200443.GA20287@merlinux.eu> Message-ID: You need an arg to --cov which is the source root you want covered. py.test -h coverage reporting with distributed testing support: --cov=path measure coverage for filesystem path (multi-allowed) Try something like: py.test --cov . src\nitroz\test_collections.py py.test --cov src src\nitroz\test_collections.py On 20 May 2011 07:53, Baptiste Lepilleur wrote: > Thanks for the pointer concerning pip, I did not know you could search using > it. > Now, I running into a strange missing import error when trying to the test > with coverage: > E ? ImportError: No module named restdoc > Any ideas what could be the cause and what this module is? > Detail below: > py.test --cov src\nitroz\test_collections.py > ================================================= test session starts > ================================================= > platform win32 -- Python 3.2.0 -- pytest-2.0.3 > collected 0 items / 1 errors > Coverage.py warning: No data was collected. > ======================================================= ERRORS > ======================================================== > _________________________________________________ ERROR collecting . > __________________________________________________ > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:312: in > visit >> ? ? ? for x in Visitor(fil, rec, ignore, bf, sort).gen(self): > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: in > gen >> ? ? ? ? ? ? ? for p in self.gen(subdir): > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: in > gen >> ? ? ? ? ? ? ? for p in self.gen(subdir): > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: in > gen >> ? ? ? ? ? ? ? for p in self.gen(subdir): > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:347: in > gen >> ? ? ? dirs = self.optsort([p for p in entries > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:348: in > >> ? ? ? ? ? ? ? if p.check(dir=1) and (rec is None or rec(p))]) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\main.py:450: in > _recurse >> ? ? ? ihook.pytest_collect_directory(path=path, parent=self) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\main.py:118: in > call_matching_hooks >> ? ? ? plugins = self.config._getmatchingplugins(self.fspath) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:283: > in _getmatchingplugins >> ? ? ? plugins += self._conftest.getconftestmodules(fspath) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:185: > in getconftestmodules >> ? ? ? ? ? ? ? ? ? ? ? clist.append(self.importconftest(conftestpath)) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:221: > in importconftest >> ? ? ? ? ? self._postimport(mod) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:226: > in _postimport >> ? ? ? ? ? self._onimport(mod) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:272: > in _onimportconftest >> ? ? ? self.pluginmanager.consider_conftest(conftestmodule) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:181: in > consider_conftest >> ? ? ? ? ? self.consider_module(conftestmodule) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:189: in > consider_module >> ? ? ? ? ? ? ? self.import_plugin(spec) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:202: in > import_plugin >> ? ? ? ? ? ? ? return self.import_plugin(modname[7:]) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:197: in > import_plugin >> ? ? ? ? ? mod = importplugin(modname) > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:323: in > importplugin >> ? ? ? return __import__(importspec, None, None, '__doc__') > E ? ImportError: No module named restdoc > ----------------------------------- coverage: platform win32, python > 3.2.0-final-0 ------------------------------------ > Name ? ?Stmts ? Miss ?Cover > --------------------------- > ============================================== 1 error in 18.02 seconds > =============================================== > > > 2011/5/18 holger krekel >> >> Hey Baptiste, >> >> On Wed, May 18, 2011 at 17:42 +0200, Baptiste Lepilleur wrote: >> > The pytest documentation page indicates that it is supported, but >> > provides >> > no pointer on how do to this... >> >> > Doing a search seem to reveal multiple plug-ins to do that. What is the >> > recommended one? I'm working on Windows XP / Python 2.6 & 3.2. >> >> sorry about that. ?The plugin is named pytest-cov, see here >> >> ? ?http://pypi.python.org/pypi/pytest-cov >> >> If it doesn't work for you i am sure Meme (also here on the list) >> can answer questions or to issues. >> >> > By the way, is there a centralized list of useful plug-ins for pytest ? >> >> not really. ?However, i recommend to install "pip" and then type: >> >> ? ?pip search pytest >> >> to get a good list. >> >> best, >> holger >> >> >> >> > >> > Baptiste. >> >> > _______________________________________________ >> > py-dev mailing list >> > py-dev at codespeak.net >> > http://codespeak.net/mailman/listinfo/py-dev >> > > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > > From baptiste.lepilleur at gmail.com Sun May 22 09:56:55 2011 From: baptiste.lepilleur at gmail.com (Baptiste Lepilleur) Date: Sun, 22 May 2011 09:56:55 +0200 Subject: [py-dev] What is the recommended way to run test with a coverage report? In-Reply-To: References: <20110518200443.GA20287@merlinux.eu> Message-ID: That was it! The restdoc mess was coming from an old version of the sources of py-1.3.4 my third-parties sub-directory that py.test was attempting to execute. The pytest-cov plug-in is exactly what I was looking for. By the way, the documentation indicates that it is possible to use standard coverage tool for result data analysis. Do those tool provided more info than the html report? If so could you provide some relevant pointers (I've never use python coverage tool before)? Thanks a lot, Baptiste. 2011/5/22 meme dough > You need an arg to --cov which is the source root you want covered. > > py.test -h > > coverage reporting with distributed testing support: > --cov=path measure coverage for filesystem path (multi-allowed) > > Try something like: > > py.test --cov . src\nitroz\test_collections.py > > py.test --cov src src\nitroz\test_collections.py > > On 20 May 2011 07:53, Baptiste Lepilleur > wrote: > > Thanks for the pointer concerning pip, I did not know you could search > using > > it. > > Now, I running into a strange missing import error when trying to the > test > > with coverage: > > E ImportError: No module named restdoc > > Any ideas what could be the cause and what this module is? > > Detail below: > > py.test --cov src\nitroz\test_collections.py > > ================================================= test session starts > > ================================================= > > platform win32 -- Python 3.2.0 -- pytest-2.0.3 > > collected 0 items / 1 errors > > Coverage.py warning: No data was collected. > > ======================================================= ERRORS > > ======================================================== > > _________________________________________________ ERROR collecting . > > __________________________________________________ > > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:312: > in > > visit > >> for x in Visitor(fil, rec, ignore, bf, sort).gen(self): > > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: > in > > gen > >> for p in self.gen(subdir): > > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: > in > > gen > >> for p in self.gen(subdir): > > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: > in > > gen > >> for p in self.gen(subdir): > > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:347: > in > > gen > >> dirs = self.optsort([p for p in entries > > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:348: > in > > > >> if p.check(dir=1) and (rec is None or rec(p))]) > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\main.py:450: > in > > _recurse > >> ihook.pytest_collect_directory(path=path, parent=self) > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\main.py:118: > in > > call_matching_hooks > >> plugins = self.config._getmatchingplugins(self.fspath) > > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:283: > > in _getmatchingplugins > >> plugins += self._conftest.getconftestmodules(fspath) > > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:185: > > in getconftestmodules > >> clist.append(self.importconftest(conftestpath)) > > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:221: > > in importconftest > >> self._postimport(mod) > > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:226: > > in _postimport > >> self._onimport(mod) > > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:272: > > in _onimportconftest > >> self.pluginmanager.consider_conftest(conftestmodule) > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:181: > in > > consider_conftest > >> self.consider_module(conftestmodule) > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:189: > in > > consider_module > >> self.import_plugin(spec) > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:202: > in > > import_plugin > >> return self.import_plugin(modname[7:]) > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:197: > in > > import_plugin > >> mod = importplugin(modname) > > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:323: > in > > importplugin > >> return __import__(importspec, None, None, '__doc__') > > E ImportError: No module named restdoc > > ----------------------------------- coverage: platform win32, python > > 3.2.0-final-0 ------------------------------------ > > Name Stmts Miss Cover > > --------------------------- > > ============================================== 1 error in 18.02 seconds > > =============================================== > > > > > > 2011/5/18 holger krekel > >> > >> Hey Baptiste, > >> > >> On Wed, May 18, 2011 at 17:42 +0200, Baptiste Lepilleur wrote: > >> > The pytest documentation page indicates that it is supported, but > >> > provides > >> > no pointer on how do to this... > >> > >> > Doing a search seem to reveal multiple plug-ins to do that. What is > the > >> > recommended one? I'm working on Windows XP / Python 2.6 & 3.2. > >> > >> sorry about that. The plugin is named pytest-cov, see here > >> > >> http://pypi.python.org/pypi/pytest-cov > >> > >> If it doesn't work for you i am sure Meme (also here on the list) > >> can answer questions or to issues. > >> > >> > By the way, is there a centralized list of useful plug-ins for pytest > ? > >> > >> not really. However, i recommend to install "pip" and then type: > >> > >> pip search pytest > >> > >> to get a good list. > >> > >> best, > >> holger > >> > >> > >> > >> > > >> > Baptiste. > >> > >> > _______________________________________________ > >> > py-dev mailing list > >> > py-dev at codespeak.net > >> > http://codespeak.net/mailman/listinfo/py-dev > >> > > > > > > _______________________________________________ > > py-dev mailing list > > py-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/py-dev > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/py-dev/attachments/20110522/08d33af5/attachment.htm From memedough at gmail.com Sun May 22 12:57:45 2011 From: memedough at gmail.com (meme dough) Date: Sun, 22 May 2011 20:57:45 +1000 Subject: [py-dev] What is the recommended way to run test with a coverage report? In-Reply-To: References: <20110518200443.GA20287@merlinux.eu> Message-ID: >From py.test you can get report on stdout with or without the missing line numbers displayed, a html report, an xml report for feeding into something like jenkins or annotated source code. These are the options for reporting that coverage has and pytest-cov allows any combinations of those that you want. Further to that, the coverage data file is called .coverage and will be in the dir that you ran py.test from. You can then use coverage itself to operate on that if you wish. Such as coverage --help coverage report coverage html Ned has docs at http://nedbatchelder.com/code/coverage pytest-cov provides access to all the report types, but having a .coveragerc can be good to get more control over how coverage operates such as turning branch coverage on. On 22 May 2011 17:56, Baptiste Lepilleur wrote: > That was it! > The restdoc mess was coming from an old version of the sources of py-1.3.4 > my third-parties sub-directory that py.test was attempting to execute. > The pytest-cov plug-in is exactly what I was looking for. > By the way, the documentation indicates that it is possible to use standard > coverage tool for result data analysis. Do those tool provided more info > than the html report? If so could you provide some relevant pointers (I've > never use python coverage tool before)? > Thanks a lot, > Baptiste. > 2011/5/22 meme dough >> >> You need an arg to --cov which is the source root you want covered. >> >> py.test -h >> >> ?coverage reporting with distributed testing support: >> ? ?--cov=path ? ? ? ? ?measure coverage for filesystem path >> (multi-allowed) >> >> Try something like: >> >> py.test --cov . src\nitroz\test_collections.py >> >> py.test --cov src src\nitroz\test_collections.py >> >> On 20 May 2011 07:53, Baptiste Lepilleur >> wrote: >> > Thanks for the pointer concerning pip, I did not know you could search >> > using >> > it. >> > Now, I running into a strange missing import error when trying to the >> > test >> > with coverage: >> > E ? ImportError: No module named restdoc >> > Any ideas what could be the cause and what this module is? >> > Detail below: >> > py.test --cov src\nitroz\test_collections.py >> > ================================================= test session starts >> > ================================================= >> > platform win32 -- Python 3.2.0 -- pytest-2.0.3 >> > collected 0 items / 1 errors >> > Coverage.py warning: No data was collected. >> > ======================================================= ERRORS >> > ======================================================== >> > _________________________________________________ ERROR collecting . >> > __________________________________________________ >> > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:312: >> > in >> > visit >> >> ? ? ? for x in Visitor(fil, rec, ignore, bf, sort).gen(self): >> > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: >> > in >> > gen >> >> ? ? ? ? ? ? ? for p in self.gen(subdir): >> > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: >> > in >> > gen >> >> ? ? ? ? ? ? ? for p in self.gen(subdir): >> > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:358: >> > in >> > gen >> >> ? ? ? ? ? ? ? for p in self.gen(subdir): >> > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:347: >> > in >> > gen >> >> ? ? ? dirs = self.optsort([p for p in entries >> > C:\Python32\lib\site-packages\py-1.4.3-py3.2.egg\py\_path\common.py:348: >> > in >> > >> >> ? ? ? ? ? ? ? if p.check(dir=1) and (rec is None or rec(p))]) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\main.py:450: in >> > _recurse >> >> ? ? ? ihook.pytest_collect_directory(path=path, parent=self) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\main.py:118: in >> > call_matching_hooks >> >> ? ? ? plugins = self.config._getmatchingplugins(self.fspath) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:283: >> > in _getmatchingplugins >> >> ? ? ? plugins += self._conftest.getconftestmodules(fspath) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:185: >> > in getconftestmodules >> >> ? ? ? ? ? ? ? ? ? ? ? clist.append(self.importconftest(conftestpath)) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:221: >> > in importconftest >> >> ? ? ? ? ? self._postimport(mod) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:226: >> > in _postimport >> >> ? ? ? ? ? self._onimport(mod) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\config.py:272: >> > in _onimportconftest >> >> ? ? ? self.pluginmanager.consider_conftest(conftestmodule) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:181: in >> > consider_conftest >> >> ? ? ? ? ? self.consider_module(conftestmodule) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:189: in >> > consider_module >> >> ? ? ? ? ? ? ? self.import_plugin(spec) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:202: in >> > import_plugin >> >> ? ? ? ? ? ? ? return self.import_plugin(modname[7:]) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:197: in >> > import_plugin >> >> ? ? ? ? ? mod = importplugin(modname) >> > >> > C:\Python32\lib\site-packages\pytest-2.0.3-py3.2.egg\_pytest\core.py:323: in >> > importplugin >> >> ? ? ? return __import__(importspec, None, None, '__doc__') >> > E ? ImportError: No module named restdoc >> > ----------------------------------- coverage: platform win32, python >> > 3.2.0-final-0 ------------------------------------ >> > Name ? ?Stmts ? Miss ?Cover >> > --------------------------- >> > ============================================== 1 error in 18.02 seconds >> > =============================================== >> > >> > >> > 2011/5/18 holger krekel >> >> >> >> Hey Baptiste, >> >> >> >> On Wed, May 18, 2011 at 17:42 +0200, Baptiste Lepilleur wrote: >> >> > The pytest documentation page indicates that it is supported, but >> >> > provides >> >> > no pointer on how do to this... >> >> >> >> > Doing a search seem to reveal multiple plug-ins to do that. What is >> >> > the >> >> > recommended one? I'm working on Windows XP / Python 2.6 & 3.2. >> >> >> >> sorry about that. ?The plugin is named pytest-cov, see here >> >> >> >> ? ?http://pypi.python.org/pypi/pytest-cov >> >> >> >> If it doesn't work for you i am sure Meme (also here on the list) >> >> can answer questions or to issues. >> >> >> >> > By the way, is there a centralized list of useful plug-ins for pytest >> >> > ? >> >> >> >> not really. ?However, i recommend to install "pip" and then type: >> >> >> >> ? ?pip search pytest >> >> >> >> to get a good list. >> >> >> >> best, >> >> holger >> >> >> >> >> >> >> >> > >> >> > Baptiste. >> >> >> >> > _______________________________________________ >> >> > py-dev mailing list >> >> > py-dev at codespeak.net >> >> > http://codespeak.net/mailman/listinfo/py-dev >> >> >> > >> > >> > _______________________________________________ >> > py-dev mailing list >> > py-dev at codespeak.net >> > http://codespeak.net/mailman/listinfo/py-dev >> > >> > > > From benjamin at python.org Thu May 26 18:42:36 2011 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 26 May 2011 11:42:36 -0500 Subject: [py-dev] assertion code moving out of pylib Message-ID: Hello everyone, I've been working on refactoring the assertion debugging code for py.test. Part of this has involved moving the assertion code from pylib to py.test proper. I don't think many people are using the assertion related code from pylib, but I wanted to give a heads up I plan to remove it soon from pylib. If you do use pylib assertion code and removing it would be a problem for you, I'd like to hear from you. -- Regards, Benjamin From holger at merlinux.eu Sat May 28 16:56:42 2011 From: holger at merlinux.eu (holger krekel) Date: Sat, 28 May 2011 14:56:42 +0000 Subject: [py-dev] tox 1.0 - rapid multi-python test automation Message-ID: <20110528145642.GU20287@merlinux.eu> tox 1.0: the rapid multi-python test automation =========================================================================== I am happy to announce tox 1.0, a stabilization and maintenance release with some small improvements. tox automates tedious test activities driven from a simple ``tox.ini`` file, including: * creation and management of different virtualenv environments with different Python interpreters * packaging and installing your package into each of them * running your test tool of choice, be it nose, py.test or unittest2 or other tools such as "sphinx" doc checks * testing dev packages against each other without needing to upload to PyPI Docs and examples are now hosted at: http://tox.readthedocs.org Installation or upgrade with: pip install -U tox Note that code hosting and issue tracking has moved from Google to Bitbucket: http://bitbucket.org/hpk42/tox The 1.0 release includes contributions and is based on feedback and work from Chris Rose, Ronny Pfannschmidt, Jannis Leidel, Jakob Kaplan-Moss, Sridhar Ratnakumar, Carl Meyer and others. Many thanks! best, Holger Krekel CHANGES --------------------- - fix issue24: introduce a way to set environment variables for for test commands (thanks Chris Rose) - fix issue22: require virtualenv-1.6.1, obsoleting virtualenv5 (thanks Jannis Leidel) and making things work with pypy-1.5 and python3 more seemlessly - toxbootstrap.py (used by jenkins build slaves) now follows the latest release of virtualenv - fix issue20: document format of URLs for specifying dependencies - fix issue19: substitute Hudson for Jenkins everywhere following the renaming of the project. NOTE: if you used the special [tox:hudson] section it will now need to be named [tox:jenkins]. - fix issue 23 / apply some ReST fixes - change the positional argument specifier to use {posargs:} syntax and fix issues #15 and #10 by refining the argument parsing method (Chris Rose) - remove use of inipkg lazy importing logic - the namespace/imports are anyway very small with tox. - fix a fspath related assertion to work with debian installs which uses symlinks - show path of the underlying virtualenv invocation and bootstrap virtualenv.py into a working subdir - added a CONTRIBUTORS file From holger at merlinux.eu Sun May 29 09:43:27 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 29 May 2011 07:43:27 +0000 Subject: [py-dev] migration from 1.3 to 2.0 In-Reply-To: References: <20110405201612.GW16231@merlinux.eu> <20110407061715.GE16231@merlinux.eu> <20110430142225.GY4071@merlinux.eu> <20110503094814.GU4071@merlinux.eu> Message-ID: <20110529074327.GX20287@merlinux.eu> Hey Vyacheslav, On Wed, May 04, 2011 at 10:53 -0400, Vyacheslav Rafalskiy wrote: > On Tue, May 3, 2011 at 5:48 AM, holger krekel wrote: > > On Mon, May 02, 2011 at 17:40 -0400, Vyacheslav Rafalskiy wrote: > >> On Sat, Apr 30, 2011 at 10:22 AM, holger krekel wrote: > >> > On Thu, Apr 07, 2011 at 12:29 -0400, Vyacheslav Rafalskiy wrote: > >> > >> >> > >> >> >(sidenote: configure and even sessionstart hooks are both a bit > >> >> > not 100% right because they happen even on the master side of a distributed > >> >> > test run and the master side does not collect or run tests at all) > >> >> > >> >> I see. Perhaps something like setup_package() in the top-level __init__.py > >> >> could be a solution? > >> > > >> > I guess you mean an __init__.py file of a test directory. > >> > With a layout of test dirs within an application this might mean > >> > one has to put this setup into the main package __init__.py > >> > and mixing test code and application code is often not a good idea. > >> > >> Yes, exactly. In my case of functional testing I don't even have > >> application code here. > >> When I start the tests I tell the runner where in the network the > >> system under test is. > >> > >> > > >> > So i'd rather put it into a conftest.py file as a normal hook. > >> > Maybe "pytest_pyfunc_setup(request)" would be good where request > >> > is the same object as for the funcarg factories. > >> > > >> > You could then write: > >> > > >> > ? ?# content of conftest.py > >> > ? ?def pytest_pyfunc_setup(request): > >> > ? ? ? ?val = request.cached_setup(setup=makeval, scope="session") > >> > ? ? ? ?# use val for some global setting of the package > >> > > >> > Alternatively we could see to call something like: > >> > > >> > ? ?def pytest_setup_testloop(config): > >> > ? ? ? ?val = makeval() > >> > ? ? ? ?# use val for some global setting of the package > >> > > >> > ? ?def pytest_teardown_testloop(config): > >> > ? ? ? ?... > >> > > >> > which would be called once for a test process. > >> > >> The reason why I suggested setup_package() is that you already have > >> setup_function, setup_method, setup_class and setup_module so > >> the former would just complete the line-up. And the natural place > >> for it would be __init__.py of that package. > >> > >> On the other hand, you can put conftest.py in every folder, which > >> can do precisely the same thing as well as many others. The > >> question is which way is more intuitive and results in cleaner code. > >> The answer is perhaps "It depends". > >> > >> I like setup_module(module) because it lets me dump the configuration > >> straight into the namespace where I use it and setup_package(package) > >> could do the same. > > > > good line of reasoning. ?It's mostly my intuition making me hesitant > > to add setup_package like you suggest. ?And i wonder what it is about :) > > Maybe it's that the root namespace of a test directory is often called > > "testing" or "tests" ?(the "test" one is taken by Python stdlib already). > > And therefore you would end up having to do "import testing" and > > then use global state with something like "testing.STATE". > > But i guess this doesn't look so bad to you, does it? :) > > (The plugin/conftest system is designed such that it hardly > > requires any imports to manage test state.) > > > > Any more opinions on setup_package(package)? If others find it useful > > as well, i will consider introducing it with pytest-2.1. > > I guess I will have to withdraw the idea. Having to explicitly import > the test package does not look nice at all. > conftest.py rules! > > As to the two alternatives above I'd rather use > pytest_setup_testloop(config) with direct access to config. I am now pondering following your original intention and introduce a "setup_directory" to be put into conftest.py files. You could then access the config object via pytest.config. Would that make sense to you as well? best, holger > Regards, > Vyacheslav > > > > > best, > > holger > > > From holger at merlinux.eu Sun May 29 09:59:38 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 29 May 2011 07:59:38 +0000 Subject: [py-dev] setup_directory / xUnit extension? Re: migration from 1.3 to 2.0 In-Reply-To: <20110529074327.GX20287@merlinux.eu> References: <20110405201612.GW16231@merlinux.eu> <20110407061715.GE16231@merlinux.eu> <20110430142225.GY4071@merlinux.eu> <20110503094814.GU4071@merlinux.eu> <20110529074327.GX20287@merlinux.eu> Message-ID: <20110529075938.GY20287@merlinux.eu> Hey again, On Sun, May 29, 2011 at 07:43 +0000, holger krekel wrote: > > > good line of reasoning. ?It's mostly my intuition making me hesitant > > > to add setup_package like you suggest. ?And i wonder what it is about :) > > > Maybe it's that the root namespace of a test directory is often called > > > "testing" or "tests" ?(the "test" one is taken by Python stdlib already). > > > And therefore you would end up having to do "import testing" and > > > then use global state with something like "testing.STATE". > > > But i guess this doesn't look so bad to you, does it? :) > > > (The plugin/conftest system is designed such that it hardly > > > requires any imports to manage test state.) > > > > > > Any more opinions on setup_package(package)? If others find it useful > > > as well, i will consider introducing it with pytest-2.1. > > > > I guess I will have to withdraw the idea. Having to explicitly import > > the test package does not look nice at all. > > conftest.py rules! > > > > As to the two alternatives above I'd rather use > > pytest_setup_testloop(config) with direct access to config. > > I am now pondering following your original intention and introduce a > "setup_directory" to be put into conftest.py files. You could then > access the config object via pytest.config. Would that make sense > to you as well? To elaborate a wee bit: * setup_directory would be guranteed to be called for any test (both python, doctest or other test) within the directory hierarchy of the conftest.py dir and before any setup_module/class etc. is called. * teardown_directory would be guranteed to be called when a test is run that is not in the directory hierarchy. * if neccessary one can have setup_directory push test related state to some global module (from which tests could import for example) i am not yet sure about the idea but i guess it would be somewhat natural and complete the setup_*/teardown_* xUnit style fixture methods. holger > best, > holger > > > Regards, > > Vyacheslav > > > > > > > > best, > > > holger > > > > > > From rafalskiy at gmail.com Mon May 30 20:24:57 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Mon, 30 May 2011 14:24:57 -0400 Subject: [py-dev] setup_directory / xUnit extension? Re: migration from 1.3 to 2.0 In-Reply-To: <20110529075938.GY20287@merlinux.eu> References: <20110405201612.GW16231@merlinux.eu> <20110407061715.GE16231@merlinux.eu> <20110430142225.GY4071@merlinux.eu> <20110503094814.GU4071@merlinux.eu> <20110529074327.GX20287@merlinux.eu> <20110529075938.GY20287@merlinux.eu> Message-ID: On Sun, May 29, 2011 at 3:59 AM, holger krekel wrote: > Hey again, > > On Sun, May 29, 2011 at 07:43 +0000, holger krekel wrote: >> > > good line of reasoning. ?It's mostly my intuition making me hesitant >> > > to add setup_package like you suggest. ?And i wonder what it is about :) >> > > Maybe it's that the root namespace of a test directory is often called >> > > "testing" or "tests" ?(the "test" one is taken by Python stdlib already). >> > > And therefore you would end up having to do "import testing" and >> > > then use global state with something like "testing.STATE". >> > > But i guess this doesn't look so bad to you, does it? :) >> > > (The plugin/conftest system is designed such that it hardly >> > > requires any imports to manage test state.) >> > > >> > > Any more opinions on setup_package(package)? If others find it useful >> > > as well, i will consider introducing it with pytest-2.1. >> > >> > I guess I will have to withdraw the idea. Having to explicitly import >> > the test package does not look nice at all. >> > conftest.py rules! >> > >> > As to the two alternatives above I'd rather use >> > pytest_setup_testloop(config) with direct access to config. >> >> I am now pondering following your original intention and introduce a >> "setup_directory" to be put into conftest.py files. ?You could then >> access the config object via pytest.config. Would that make sense >> to you as well? > > To elaborate a wee bit: > > * setup_directory would be guranteed to be called for any test > ?(both python, doctest or other test) within the directory hierarchy > ?of the conftest.py dir and before any setup_module/class etc. is called. > > * teardown_directory would be guranteed to be called when a test > ?is run that is not in the directory hierarchy. > > * if neccessary one can have setup_directory push test related > ?state to some global module (from which tests could import for example) > > i am not yet sure about the idea but i guess it would be somewhat natural > and complete the setup_*/teardown_* xUnit style fixture methods. > > holger Sounds great to me. Not much to add. Perhaps a parameter, say directory_config, which can then be imported by a test module like for example from pytest import directory_config This object could have different meaning depending on the directory where the test module resides. Vyacheslav From rafalskiy at gmail.com Mon May 30 21:17:32 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Mon, 30 May 2011 15:17:32 -0400 Subject: [py-dev] Decorators and funcargs in py.test Message-ID: Hi Holger, I am trying to make decorators work with test functions, which depend on funcargs. As it stands, they don't. Decorated functions lose funcargs. A workaround would be to decorate an internal function like this: def test_it(funcarg_it): @decorate_it def _test_it(): # test it _test_it() This works, but it is not nice. I'd rather wrote a decorator like def decorate_it(f): def _wrap_it(*args, **kwargs): # wrap f() here _wrap_it._varnames = _pytest.core.varnames(f) return _wrap_it and apply it straight to the test function. After examining the source code I even expected it to just work (magically of course) but it didn't. Do you think it is worthwhile? If so I can enter a feature request. Thanks, Vyacheslav From benjamin at python.org Mon May 30 21:23:12 2011 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 30 May 2011 14:23:12 -0500 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: References: Message-ID: 2011/5/30 Vyacheslav Rafalskiy : > Hi Holger, > > I am trying to make decorators work with test functions, which depend > on funcargs. As it stands, they don't. > Decorated functions lose funcargs. A workaround would be to decorate > an internal function like this: > > def test_it(funcarg_it): > ? ?@decorate_it > ? ?def _test_it(): > ? ? ? ?# test it > > ? ?_test_it() > > This works, but it is not nice. I'd rather wrote a decorator like > > def decorate_it(f): > ? ?def _wrap_it(*args, **kwargs): > ? ? ? ?# wrap f() here > > ? ?_wrap_it._varnames = _pytest.core.varnames(f) > ? ?return _wrap_it > > and apply it straight to the test function. > > After examining the source code I even expected it to just work > (magically of course) but it didn't. > Do you think it is worthwhile? If so I can enter a feature request. It's a difficult problem because there's no way to know if a function is being decorated. I suppose py.test could look for a _decorated_ attribute but decorators would have to conform and set that attribute. -- Regards, Benjamin From Ronny.Pfannschmidt at gmx.de Mon May 30 21:55:15 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Mon, 30 May 2011 21:55:15 +0200 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: References: Message-ID: <1306785315.3360.1.camel@Klappe2> Hi, can you try to explain the usecase those decorators are fulfilling, there may be a better integrated way using pytest.mark + setup/teardown hooks -- Ronny On Mon, 2011-05-30 at 15:17 -0400, Vyacheslav Rafalskiy wrote: > Hi Holger, > > I am trying to make decorators work with test functions, which depend > on funcargs. As it stands, they don't. > Decorated functions lose funcargs. A workaround would be to decorate > an internal function like this: > > def test_it(funcarg_it): > @decorate_it > def _test_it(): > # test it > > _test_it() > > This works, but it is not nice. I'd rather wrote a decorator like > > def decorate_it(f): > def _wrap_it(*args, **kwargs): > # wrap f() here > > _wrap_it._varnames = _pytest.core.varnames(f) > return _wrap_it > > and apply it straight to the test function. > > After examining the source code I even expected it to just work > (magically of course) but it didn't. > Do you think it is worthwhile? If so I can enter a feature request. > > Thanks, > Vyacheslav > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110530/7d8dd393/attachment.pgp From Ronny.Pfannschmidt at gmx.de Mon May 30 22:38:30 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Mon, 30 May 2011 22:38:30 +0200 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: References: <1306785315.3360.1.camel@Klappe2> Message-ID: <1306787910.3360.15.camel@Klappe2> On Mon, 2011-05-30 at 16:23 -0400, Vyacheslav Rafalskiy wrote: > No problem. Here is my (real life) example. > > My functional test functions may or may not return for different > reasons (like a faulty web application or middleware). I want to > declare a fail if it takes more than so many seconds to complete. So I > write a decorator > run_with_timeout(), which will start the function in a new thread and > abandon it after timeout. This can easily be solved by combining something like pytest.mark('timeout') and a override to pytest_pyfunc_call using it i suppose this could also take a look at extending the "--boxed" mode (which forks for each test and uses subprocesses but doesn?t handle timeouts atm. > > A I stated in OP, it is not that I cannot do it. I can and I do. My > point is that it makes sense to allow decorators and should not be > very difficult (see the example in OP). personally i am opposed to creating new data conventions for problems that can be solved with plain marks + a hook usually there are 2 reasons people use to decorate tests my opinions for implementing those are a) add arguments + their cleanups -> funcargs please, they are made for that b) use more sophisticated call's -> hooks please, maybe add a bug ticket for empowering one to return a exception-info, so stuff like thread-wrappers can pass that more nicely for failures *i* really think it is wrong to decorate test functions that way and expect stuff to work there are already plenty of mechanisms to change the behavior of pytest test function calling in the desired way, none of those require hacks to pass around argspecs -- Ronny > > Thanks, > Vyacheslav > > On Mon, May 30, 2011 at 3:55 PM, Ronny Pfannschmidt > wrote: > > Hi, > > > > can you try to explain the usecase those decorators are fulfilling, > > > > there may be a better integrated way using pytest.mark + setup/teardown > > hooks > > > > -- Ronny > > > > On Mon, 2011-05-30 at 15:17 -0400, Vyacheslav Rafalskiy wrote: > >> Hi Holger, > >> > >> I am trying to make decorators work with test functions, which depend > >> on funcargs. As it stands, they don't. > >> Decorated functions lose funcargs. A workaround would be to decorate > >> an internal function like this: > >> > >> def test_it(funcarg_it): > >> @decorate_it > >> def _test_it(): > >> # test it > >> > >> _test_it() > >> > >> This works, but it is not nice. I'd rather wrote a decorator like > >> > >> def decorate_it(f): > >> def _wrap_it(*args, **kwargs): > >> # wrap f() here > >> > >> _wrap_it._varnames = _pytest.core.varnames(f) > >> return _wrap_it > >> > >> and apply it straight to the test function. > >> > >> After examining the source code I even expected it to just work > >> (magically of course) but it didn't. > >> Do you think it is worthwhile? If so I can enter a feature request. > >> > >> Thanks, > >> Vyacheslav > >> _______________________________________________ > >> py-dev mailing list > >> py-dev at codespeak.net > >> http://codespeak.net/mailman/listinfo/py-dev > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/py-dev/attachments/20110530/4e1284a0/attachment-0001.pgp From holger at merlinux.eu Wed Jun 1 09:49:13 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 1 Jun 2011 07:49:13 +0000 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: <1306787910.3360.15.camel@Klappe2> References: <1306785315.3360.1.camel@Klappe2> <1306787910.3360.15.camel@Klappe2> Message-ID: <20110601074913.GE20287@merlinux.eu> Hi Vyacheslav, hi Ronny, On Mon, May 30, 2011 at 22:38 +0200, Ronny Pfannschmidt wrote: > On Mon, 2011-05-30 at 16:23 -0400, Vyacheslav Rafalskiy wrote: > > No problem. Here is my (real life) example. > > > > My functional test functions may or may not return for different > > reasons (like a faulty web application or middleware). I want to > > declare a fail if it takes more than so many seconds to complete. So I > > write a decorator > > run_with_timeout(), which will start the function in a new thread and > > abandon it after timeout. > > This can easily be solved by combining something like > pytest.mark('timeout') and a override to pytest_pyfunc_call using it I agree. Vyacheslav, if you can't make sense of the "use pytest.mark()" and provide-a-hook suggestion, one of us will be certainly be happy to provide a more complete example and add it to the docs. > i suppose this could also take a look at extending the "--boxed" mode > (which forks for each test and uses subprocesses but doesn?t handle > timeouts atm. this is part of the pytest-xdist plugin, however. Question is, if we shouldn't eventually grow timeout-support in core pytest through "alarm" or so. best, holger > > > > A I stated in OP, it is not that I cannot do it. I can and I do. My > > point is that it makes sense to allow decorators and should not be > > very difficult (see the example in OP). > > personally i am opposed to creating new data conventions for problems > that can be solved with plain marks + a hook > > usually there are 2 reasons people use to decorate tests > > my opinions for implementing those are > a) add arguments + their cleanups -> funcargs please, they are made for > that > b) use more sophisticated call's -> hooks please, maybe add a bug ticket > for empowering one to return a exception-info, so stuff like > thread-wrappers can pass that more nicely for failures > > *i* really think it is wrong to decorate test functions that way and > expect stuff to work > > there are already plenty of mechanisms to change the behavior of pytest > test function calling in the desired way, none of those require hacks to > pass around argspecs > > -- Ronny > > > > Thanks, > > Vyacheslav > > > > On Mon, May 30, 2011 at 3:55 PM, Ronny Pfannschmidt > > wrote: > > > Hi, > > > > > > can you try to explain the usecase those decorators are fulfilling, > > > > > > there may be a better integrated way using pytest.mark + setup/teardown > > > hooks > > > > > > -- Ronny > > > > > > On Mon, 2011-05-30 at 15:17 -0400, Vyacheslav Rafalskiy wrote: > > >> Hi Holger, > > >> > > >> I am trying to make decorators work with test functions, which depend > > >> on funcargs. As it stands, they don't. > > >> Decorated functions lose funcargs. A workaround would be to decorate > > >> an internal function like this: > > >> > > >> def test_it(funcarg_it): > > >> @decorate_it > > >> def _test_it(): > > >> # test it > > >> > > >> _test_it() > > >> > > >> This works, but it is not nice. I'd rather wrote a decorator like > > >> > > >> def decorate_it(f): > > >> def _wrap_it(*args, **kwargs): > > >> # wrap f() here > > >> > > >> _wrap_it._varnames = _pytest.core.varnames(f) > > >> return _wrap_it > > >> > > >> and apply it straight to the test function. > > >> > > >> After examining the source code I even expected it to just work > > >> (magically of course) but it didn't. > > >> Do you think it is worthwhile? If so I can enter a feature request. > > >> > > >> Thanks, > > >> Vyacheslav > > >> _______________________________________________ > > >> py-dev mailing list > > >> py-dev at codespeak.net > > >> http://codespeak.net/mailman/listinfo/py-dev > > > > > > > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From holger at merlinux.eu Wed Jun 1 09:57:43 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 1 Jun 2011 07:57:43 +0000 Subject: [py-dev] setup_directory / xUnit extension? Re: migration from 1.3 to 2.0 In-Reply-To: References: <20110407061715.GE16231@merlinux.eu> <20110430142225.GY4071@merlinux.eu> <20110503094814.GU4071@merlinux.eu> <20110529074327.GX20287@merlinux.eu> <20110529075938.GY20287@merlinux.eu> Message-ID: <20110601075743.GF20287@merlinux.eu> On Mon, May 30, 2011 at 14:24 -0400, Vyacheslav Rafalskiy wrote: > On Sun, May 29, 2011 at 3:59 AM, holger krekel wrote: > > On Sun, May 29, 2011 at 07:43 +0000, holger krekel wrote: > >> > > good line of reasoning. ?It's mostly my intuition making me hesitant > >> > > to add setup_package like you suggest. ?And i wonder what it is about :) > >> > > Maybe it's that the root namespace of a test directory is often called > >> > > "testing" or "tests" ?(the "test" one is taken by Python stdlib already). > >> > > And therefore you would end up having to do "import testing" and > >> > > then use global state with something like "testing.STATE". > >> > > But i guess this doesn't look so bad to you, does it? :) > >> > > (The plugin/conftest system is designed such that it hardly > >> > > requires any imports to manage test state.) > >> > > > >> > > Any more opinions on setup_package(package)? If others find it useful > >> > > as well, i will consider introducing it with pytest-2.1. > >> > > >> > I guess I will have to withdraw the idea. Having to explicitly import > >> > the test package does not look nice at all. > >> > conftest.py rules! > >> > > >> > As to the two alternatives above I'd rather use > >> > pytest_setup_testloop(config) with direct access to config. > >> > >> I am now pondering following your original intention and introduce a > >> "setup_directory" to be put into conftest.py files. ?You could then > >> access the config object via pytest.config. Would that make sense > >> to you as well? > > > > To elaborate a wee bit: > > > > * setup_directory would be guranteed to be called for any test > > ?(both python, doctest or other test) within the directory hierarchy > > ?of the conftest.py dir and before any setup_module/class etc. is called. > > > > * teardown_directory would be guranteed to be called when a test > > ?is run that is not in the directory hierarchy. > > > > * if neccessary one can have setup_directory push test related > > ?state to some global module (from which tests could import for example) > > > > i am not yet sure about the idea but i guess it would be somewhat natural > > and complete the setup_*/teardown_* xUnit style fixture methods. > > > > holger > > Sounds great to me. Not much to add. Perhaps a parameter, say > directory_config, which > can then be imported by a test module like for example > > from pytest import directory_config > > This object could have different meaning depending on the directory > where the test module resides. This suggestion would replicate funcarg-functionality which allows to provide different/adapted resources to test modules from conftest.py files. I'd rather like to think about generally giving access to funcarg resources from xUnit setup functions which would enlarge what you can do from xUnit fixtures. Meanwhile, we could maybe introduce a global "pytest.appdata" object which can be used for passing state between various places in test like setup_directory or hooks and test methods or other setup methods. Several python packages (particularly web stuff) have global data anyway and maybe it's best to provide one managed storage for it. best, holger > > Vyacheslav > From rafalskiy at gmail.com Wed Jun 1 18:31:06 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Wed, 1 Jun 2011 12:31:06 -0400 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: <20110601074913.GE20287@merlinux.eu> References: <1306785315.3360.1.camel@Klappe2> <1306787910.3360.15.camel@Klappe2> <20110601074913.GE20287@merlinux.eu> Message-ID: Gentlemen! Thanks for your input. I really appreciate it. @Ronny 1. I do agree that creating a new data convention is a big deal. What I tried to suggest is using something that already exists in _pytest.core.varnames(). A comment in _pytest.python.getfuncargnames() indicates that the two functions may merge. If they do, and towards the former, my original suggestion will *just work*. 2. Using decorators is a great deal for me. They are powerful and expressive. I understand however the reluctance to open a can of worms here. The replacement using pytest.mark below seems acceptable to me. It still uses the same decorator function, just at a different place. @Holger, Ronny Here is what I came up with: #---------->> in test_1.py @pytest.mark.timeout(10) def test_f1(): # test here #---------->> in conftest.py def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] item.obj = run_with_timeout(timeout)(item.obj) Your comments are welcome. Regards, Vyacheslav On Wed, Jun 1, 2011 at 3:49 AM, holger krekel wrote: > Hi Vyacheslav, hi Ronny, > > On Mon, May 30, 2011 at 22:38 +0200, Ronny Pfannschmidt wrote: >> On Mon, 2011-05-30 at 16:23 -0400, Vyacheslav Rafalskiy wrote: >> > No problem. Here is my (real life) example. >> > >> > My functional test functions may or may not return for different >> > reasons (like a faulty web application or middleware). I want to >> > declare a fail if it takes more than so many seconds to complete. So I >> > write a decorator >> > run_with_timeout(), which will start the function in a new thread and >> > abandon it after timeout. >> >> This can easily be solved by combining something like >> pytest.mark('timeout') and a override to pytest_pyfunc_call using it > > I agree. > > Vyacheslav, if you can't make sense of the "use pytest.mark()" and > provide-a-hook suggestion, one of us will be certainly be happy to > provide a more complete example and add it to the docs. > >> i suppose this could also take a look at extending the "--boxed" mode >> (which forks for each test and uses subprocesses but doesn?t handle >> timeouts atm. > > this is part of the pytest-xdist plugin, however. ?Question is, > if we shouldn't eventually grow timeout-support in core pytest > through "alarm" or so. > > best, > holger > >> > >> > A I stated in OP, it is not that I cannot do it. I can and I do. My >> > point is that it makes sense to allow decorators and should not be >> > very difficult (see the example in OP). >> >> personally i am opposed to creating new data conventions for problems >> that can be solved with plain marks + a hook >> >> usually there are 2 reasons people use to decorate tests >> >> my opinions for implementing those are >> a) add arguments + their cleanups -> funcargs please, they are made for >> that >> b) use more sophisticated call's -> hooks please, maybe add a bug ticket >> for empowering one to return a exception-info, so stuff like >> thread-wrappers can pass that more nicely for failures >> >> *i* really think it is wrong to decorate test functions that way and >> expect stuff to work >> >> there are already plenty of mechanisms to change the behavior of pytest >> test function calling in the desired way, none of those require hacks to >> pass around argspecs >> >> -- Ronny >> > >> > Thanks, >> > Vyacheslav >> > >> > On Mon, May 30, 2011 at 3:55 PM, Ronny Pfannschmidt >> > wrote: >> > > Hi, >> > > >> > > can you try to explain the usecase those decorators are fulfilling, >> > > >> > > there may be a better integrated way using pytest.mark + setup/teardown >> > > hooks >> > > >> > > -- Ronny >> > > >> > > On Mon, 2011-05-30 at 15:17 -0400, Vyacheslav Rafalskiy wrote: >> > >> Hi Holger, >> > >> >> > >> I am trying to make decorators work with test functions, which depend >> > >> on funcargs. As it stands, they don't. >> > >> Decorated functions lose funcargs. A workaround would be to decorate >> > >> an internal function like this: >> > >> >> > >> def test_it(funcarg_it): >> > >> ? ? @decorate_it >> > >> ? ? def _test_it(): >> > >> ? ? ? ? # test it >> > >> >> > >> ? ? _test_it() >> > >> >> > >> This works, but it is not nice. I'd rather wrote a decorator like >> > >> >> > >> def decorate_it(f): >> > >> ? ? def _wrap_it(*args, **kwargs): >> > >> ? ? ? ? # wrap f() here >> > >> >> > >> ? ? _wrap_it._varnames = _pytest.core.varnames(f) >> > >> ? ? return _wrap_it >> > >> >> > >> and apply it straight to the test function. >> > >> >> > >> After examining the source code I even expected it to just work >> > >> (magically of course) but it didn't. >> > >> Do you think it is worthwhile? If so I can enter a feature request. >> > >> >> > >> Thanks, >> > >> Vyacheslav >> > >> _______________________________________________ >> > >> py-dev mailing list >> > >> py-dev at codespeak.net >> > >> http://codespeak.net/mailman/listinfo/py-dev >> > > >> > > >> > > > >> _______________________________________________ >> py-dev mailing list >> py-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/py-dev > > From holger at merlinux.eu Wed Jun 1 22:40:01 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 1 Jun 2011 20:40:01 +0000 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: References: <1306785315.3360.1.camel@Klappe2> <1306787910.3360.15.camel@Klappe2> <20110601074913.GE20287@merlinux.eu> Message-ID: <20110601204001.GI20287@merlinux.eu> On Wed, Jun 01, 2011 at 12:31 -0400, Vyacheslav Rafalskiy wrote: > Gentlemen! > > Thanks for your input. I really appreciate it. > > @Ronny > 1. I do agree that creating a new data convention is a big deal. > What I tried to suggest is using something that already exists in > _pytest.core.varnames(). A comment in _pytest.python.getfuncargnames() > indicates that the two functions may merge. If they do, and towards > the former, my original suggestion will *just work*. > > 2. Using decorators is a great deal for me. They are powerful and > expressive. I understand however the reluctance to open a can of worms > here. The replacement using pytest.mark below seems acceptable to me. > It still uses the same decorator function, just at a different place. > > @Holger, Ronny > Here is what I came up with: > > #---------->> in test_1.py > @pytest.mark.timeout(10) > def test_f1(): > # test here > > #---------->> in conftest.py > def pytest_runtest_call(item): > if hasattr(item.obj, 'timeout'): > timeout = item.obj.timeout.args[0] > item.obj = run_with_timeout(timeout)(item.obj) > > Your comments are welcome. it's basically ok but there are some bits that could be improved. You are actually implementing the general test item call. Also I am not sure why you assign "item.obj = ...". I'd probably write something like this: @pytest.mark.tryfirst def pytest_pyfunc_call(pyfuncitem, __multicall__): if 'timeout' in pyfuncitem.keywords: timeout = pyfuncitem.keywords['timeout'].args[0] run_with_timeout(timeout)(__multicall__.execute) main differences: * only applies to python test function calls * hook invocation will be "tried first" before other pytest_pyfunc_call hook impls and it will call those other hooks through the "__multicall__" bit which actually represents the ongoing hook call * will call other hook implementations How do you actually implement run_with_timeout, btw? best, holger From rafalskiy at gmail.com Wed Jun 1 23:16:43 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Wed, 1 Jun 2011 17:16:43 -0400 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: <20110601204001.GI20287@merlinux.eu> References: <1306785315.3360.1.camel@Klappe2> <1306787910.3360.15.camel@Klappe2> <20110601074913.GE20287@merlinux.eu> <20110601204001.GI20287@merlinux.eu> Message-ID: >> #---------->> ?in test_1.py >> @pytest.mark.timeout(10) >> def test_f1(): >> ? ? # test here >> >> #---------->> ?in conftest.py >> def pytest_runtest_call(item): >> ? ? if hasattr(item.obj, 'timeout'): >> ? ? ? ? timeout = item.obj.timeout.args[0] >> ? ? ? ? item.obj = run_with_timeout(timeout)(item.obj) >> >> Your comments are welcome. > > it's basically ok but there are some bits that could > be improved. ?You are actually implementing the general > test item call. ?Also I am not sure why you assign > "item.obj = ...". I replace (or so I think) the original test function by the decorated one. It gets called elsewhere. > > I'd probably write something like this: > > ? ?@pytest.mark.tryfirst > ? ?def pytest_pyfunc_call(pyfuncitem, __multicall__): > ? ? ? ?if 'timeout' in pyfuncitem.keywords: > ? ? ? ? ? ?timeout = pyfuncitem.keywords['timeout'].args[0] > ? ? ? ? ? ?run_with_timeout(timeout)(__multicall__.execute) this will take a while to digest > main differences: > > * only applies to python test function calls > * hook invocation will be "tried first" before other > ?pytest_pyfunc_call hook impls and it will call those > ?other hooks through the "__multicall__" bit > ?which actually represents the ongoing hook call > * will call other hook implementations > > How do you actually implement run_with_timeout, btw? def run_with_timeout(timeout=None): def _decorator(f): def _wrapper(*args, **kwargs): def _f(*args, **kwargs): try: _result = f(*args, **kwargs) except Exception as e: thread._exception = e else: thread._result = _result thread = threading.Thread(target=_f, args=args, kwargs=kwargs) thread.daemon = True thread._exception = None thread.start() thread.join(timeout=timeout) if thread.isAlive(): raise RuntimeError('function *%s* exceeded configured timeout of %ss' % (f.__name__, timeout)) if thread._exception is None: return thread._result else: raise thread._exception _wrapper.__name__ = f.__name__ return _wrapper return _decorator > > best, > holger > Vyacheslav From holger at merlinux.eu Thu Jun 2 06:46:06 2011 From: holger at merlinux.eu (holger krekel) Date: Thu, 2 Jun 2011 04:46:06 +0000 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: References: <1306785315.3360.1.camel@Klappe2> <1306787910.3360.15.camel@Klappe2> <20110601074913.GE20287@merlinux.eu> <20110601204001.GI20287@merlinux.eu> Message-ID: <20110602044606.GJ20287@merlinux.eu> On Wed, Jun 01, 2011 at 17:16 -0400, Vyacheslav Rafalskiy wrote: > >> #---------->> ?in test_1.py > >> @pytest.mark.timeout(10) > >> def test_f1(): > >> ? ? # test here > >> > >> #---------->> ?in conftest.py > >> def pytest_runtest_call(item): > >> ? ? if hasattr(item.obj, 'timeout'): > >> ? ? ? ? timeout = item.obj.timeout.args[0] > >> ? ? ? ? item.obj = run_with_timeout(timeout)(item.obj) > >> > >> Your comments are welcome. > > > > it's basically ok but there are some bits that could > > be improved. ?You are actually implementing the general > > test item call. ?Also I am not sure why you assign > > "item.obj = ...". > > I replace (or so I think) the original test function by the decorated > one. It gets called elsewhere. ah, of course :) > > I'd probably write something like this: > > > > ? ?@pytest.mark.tryfirst > > ? ?def pytest_pyfunc_call(pyfuncitem, __multicall__): > > ? ? ? ?if 'timeout' in pyfuncitem.keywords: > > ? ? ? ? ? ?timeout = pyfuncitem.keywords['timeout'].args[0] > > ? ? ? ? ? ?run_with_timeout(timeout)(__multicall__.execute) > > this will take a while to digest it's actually wrong if run_with_timeout is only decorating but not running the function. I think it makes sense to rather directly call a helper which calls the function (note that __multicall__.execute() will execute the remainder of the hook chain one of which will actually execute the function). Such a helper would look like call_with_timeout(timeout, func) i guess. best, holger > > main differences: > > > > * only applies to python test function calls > > * hook invocation will be "tried first" before other > > ?pytest_pyfunc_call hook impls and it will call those > > ?other hooks through the "__multicall__" bit > > ?which actually represents the ongoing hook call > > * will call other hook implementations > > > > How do you actually implement run_with_timeout, btw? > > def run_with_timeout(timeout=None): > def _decorator(f): > def _wrapper(*args, **kwargs): > def _f(*args, **kwargs): > try: > _result = f(*args, **kwargs) > except Exception as e: > thread._exception = e > else: > thread._result = _result > > thread = threading.Thread(target=_f, args=args, kwargs=kwargs) > thread.daemon = True > thread._exception = None > > thread.start() > thread.join(timeout=timeout) > if thread.isAlive(): > raise RuntimeError('function *%s* exceeded configured > timeout of %ss' % (f.__name__, timeout)) > if thread._exception is None: > return thread._result > else: > raise thread._exception > _wrapper.__name__ = f.__name__ > return _wrapper > return _decorator > > > > > > > best, > > holger > > > > Vyacheslav > From rafalskiy at gmail.com Fri Jun 3 18:06:06 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Fri, 3 Jun 2011 12:06:06 -0400 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: <20110602044606.GJ20287@merlinux.eu> References: <1306785315.3360.1.camel@Klappe2> <1306787910.3360.15.camel@Klappe2> <20110601074913.GE20287@merlinux.eu> <20110601204001.GI20287@merlinux.eu> <20110602044606.GJ20287@merlinux.eu> Message-ID: On Thu, Jun 2, 2011 at 12:46 AM, holger krekel wrote: > On Wed, Jun 01, 2011 at 17:16 -0400, Vyacheslav Rafalskiy wrote: >> >> #---------->> ?in test_1.py >> >> @pytest.mark.timeout(10) >> >> def test_f1(): >> >> ? ? # test here >> >> >> >> #---------->> ?in conftest.py >> >> def pytest_runtest_call(item): >> >> ? ? if hasattr(item.obj, 'timeout'): >> >> ? ? ? ? timeout = item.obj.timeout.args[0] >> >> ? ? ? ? item.obj = run_with_timeout(timeout)(item.obj) >> >> >> >> Your comments are welcome. >> > >> > it's basically ok but there are some bits that could >> > be improved. ?You are actually implementing the general >> > test item call. ?Also I am not sure why you assign >> > "item.obj = ...". >> >> I replace (or so I think) the original test function by the decorated >> one. It gets called elsewhere. > > ah, of course :) > >> > I'd probably write something like this: >> > >> > ? ?@pytest.mark.tryfirst >> > ? ?def pytest_pyfunc_call(pyfuncitem, __multicall__): >> > ? ? ? ?if 'timeout' in pyfuncitem.keywords: >> > ? ? ? ? ? ?timeout = pyfuncitem.keywords['timeout'].args[0] >> > ? ? ? ? ? ?run_with_timeout(timeout)(__multicall__.execute) >> >> this will take a while to digest > > it's actually wrong if run_with_timeout is only decorating > but not running the function. ?I think it makes sense to > rather directly call a helper which calls the function > (note that __multicall__.execute() will execute the remainder > of the hook chain one of which will actually execute the > function). ?Such a helper would look like > call_with_timeout(timeout, func) i guess. To call the function I need to know what arguments to give it to. The following seems to work, but this is just a guess: def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] run_with_timeout(timeout)(item.obj)(**item.funcargs) I still don't quite get your example, specifically the __multicall__.execute part. Btw, the pytest_pyfunc_call() parameters seem to be in the wrong order based on _pytest.python.py and the prototype in _pytest.hookspec.py only lists one parameter. Vyacheslav From holger at merlinux.eu Fri Jun 3 19:38:44 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 3 Jun 2011 17:38:44 +0000 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: References: <1306785315.3360.1.camel@Klappe2> <1306787910.3360.15.camel@Klappe2> <20110601074913.GE20287@merlinux.eu> <20110601204001.GI20287@merlinux.eu> <20110602044606.GJ20287@merlinux.eu> Message-ID: <20110603173844.GM20287@merlinux.eu> On Fri, Jun 03, 2011 at 12:06 -0400, Vyacheslav Rafalskiy wrote: > On Thu, Jun 2, 2011 at 12:46 AM, holger krekel wrote: > > On Wed, Jun 01, 2011 at 17:16 -0400, Vyacheslav Rafalskiy wrote: > >> >> #---------->> ?in test_1.py > >> >> @pytest.mark.timeout(10) > >> >> def test_f1(): > >> >> ? ? # test here > >> >> > >> >> #---------->> ?in conftest.py > >> >> def pytest_runtest_call(item): > >> >> ? ? if hasattr(item.obj, 'timeout'): > >> >> ? ? ? ? timeout = item.obj.timeout.args[0] > >> >> ? ? ? ? item.obj = run_with_timeout(timeout)(item.obj) > >> >> > >> >> Your comments are welcome. > >> > > >> > it's basically ok but there are some bits that could > >> > be improved. ?You are actually implementing the general > >> > test item call. ?Also I am not sure why you assign > >> > "item.obj = ...". > >> > >> I replace (or so I think) the original test function by the decorated > >> one. It gets called elsewhere. > > > > ah, of course :) > > > >> > I'd probably write something like this: > >> > > >> > ? ?@pytest.mark.tryfirst > >> > ? ?def pytest_pyfunc_call(pyfuncitem, __multicall__): > >> > ? ? ? ?if 'timeout' in pyfuncitem.keywords: > >> > ? ? ? ? ? ?timeout = pyfuncitem.keywords['timeout'].args[0] > >> > ? ? ? ? ? ?run_with_timeout(timeout)(__multicall__.execute) > >> > >> this will take a while to digest > > > > it's actually wrong if run_with_timeout is only decorating > > but not running the function. ?I think it makes sense to > > rather directly call a helper which calls the function > > (note that __multicall__.execute() will execute the remainder > > of the hook chain one of which will actually execute the > > function). ?Such a helper would look like > > call_with_timeout(timeout, func) i guess. > > To call the function I need to know what arguments to give it to. > The following seems to work, but this is just a guess: > > def pytest_runtest_call(item): > if hasattr(item.obj, 'timeout'): > timeout = item.obj.timeout.args[0] > run_with_timeout(timeout)(item.obj)(**item.funcargs) > > I still don't quite get your example, specifically the > __multicall__.execute part. A few things that might help you to understand: * There can be multiple hook functions implementing the same hook. * MultiCall instances manage the calling into a list of hook implementation functions. * A hook impl function can use zero or any number of available arguments. i.e. if a hookspec is "pyest_myfunc(x,y,z)" then a hook impl function can just receive only one of them "pytest_myfunc(x)" This slicing is done from the MultiCall class. * the actual call to a hook impl function is always done by passing keyword arguments (i.e. **kwargs) and thus ordering of parameters is irrelevant. * a hook impl function can receive a "__multicall__" parameter which is its managing class and which it can use to call __multicall__.execute() to execute the rest of the hook implementations. * with all this in mind the few lines of code in _pytest.core.MultiCall hopefully make more sense :) best, holger > Btw, the pytest_pyfunc_call() parameters seem to be in the wrong order > based on _pytest.python.py and the prototype in _pytest.hookspec.py > only lists one parameter. > > > Vyacheslav > From rafalskiy at gmail.com Mon Jun 6 15:24:55 2011 From: rafalskiy at gmail.com (Vyacheslav Rafalskiy) Date: Mon, 6 Jun 2011 09:24:55 -0400 Subject: [py-dev] Decorators and funcargs in py.test In-Reply-To: <20110603173844.GM20287@merlinux.eu> References: <1306785315.3360.1.camel@Klappe2> <1306787910.3360.15.camel@Klappe2> <20110601074913.GE20287@merlinux.eu> <20110601204001.GI20287@merlinux.eu> <20110602044606.GJ20287@merlinux.eu> <20110603173844.GM20287@merlinux.eu> Message-ID: Thanks Holger, It is clearer now. Vyacheslav On Fri, Jun 3, 2011 at 1:38 PM, holger krekel wrote: > On Fri, Jun 03, 2011 at 12:06 -0400, Vyacheslav Rafalskiy wrote: >> On Thu, Jun 2, 2011 at 12:46 AM, holger krekel wrote: >> > On Wed, Jun 01, 2011 at 17:16 -0400, Vyacheslav Rafalskiy wrote: >> >> >> #---------->> ?in test_1.py >> >> >> @pytest.mark.timeout(10) >> >> >> def test_f1(): >> >> >> ? ? # test here >> >> >> >> >> >> #---------->> ?in conftest.py >> >> >> def pytest_runtest_call(item): >> >> >> ? ? if hasattr(item.obj, 'timeout'): >> >> >> ? ? ? ? timeout = item.obj.timeout.args[0] >> >> >> ? ? ? ? item.obj = run_with_timeout(timeout)(item.obj) >> >> >> >> >> >> Your comments are welcome. >> >> > >> >> > it's basically ok but there are some bits that could >> >> > be improved. ?You are actually implementing the general >> >> > test item call. ?Also I am not sure why you assign >> >> > "item.obj = ...". >> >> >> >> I replace (or so I think) the original test function by the decorated >> >> one. It gets called elsewhere. >> > >> > ah, of course :) >> > >> >> > I'd probably write something like this: >> >> > >> >> > ? ?@pytest.mark.tryfirst >> >> > ? ?def pytest_pyfunc_call(pyfuncitem, __multicall__): >> >> > ? ? ? ?if 'timeout' in pyfuncitem.keywords: >> >> > ? ? ? ? ? ?timeout = pyfuncitem.keywords['timeout'].args[0] >> >> > ? ? ? ? ? ?run_with_timeout(timeout)(__multicall__.execute) >> >> >> >> this will take a while to digest >> > >> > it's actually wrong if run_with_timeout is only decorating >> > but not running the function. ?I think it makes sense to >> > rather directly call a helper which calls the function >> > (note that __multicall__.execute() will execute the remainder >> > of the hook chain one of which will actually execute the >> > function). ?Such a helper would look like >> > call_with_timeout(timeout, func) i guess. >> >> To call the function I need to know what arguments to give it to. >> The following seems to work, but this is just a guess: >> >> def pytest_runtest_call(item): >> ? ? if hasattr(item.obj, 'timeout'): >> ? ? ? ? timeout = item.obj.timeout.args[0] >> ? ? ? ? run_with_timeout(timeout)(item.obj)(**item.funcargs) >> >> I still don't quite get your example, specifically the >> __multicall__.execute part. > > A few things that might help you to understand: > > * There can be multiple hook functions implementing the same hook. > * MultiCall instances manage the calling into a list of hook > ?implementation functions. > * A hook impl function can use zero or any number of available > ?arguments. ?i.e. if a hookspec is "pyest_myfunc(x,y,z)" then > ?a hook impl function can just receive only one of them "pytest_myfunc(x)" > ?This slicing is done from the MultiCall class. > * the actual call to a hook impl function is always done by > ?passing keyword arguments (i.e. **kwargs) and thus ordering > ?of parameters is irrelevant. > * a hook impl function can receive a "__multicall__" parameter > ?which is its managing class and which it can use to call > ?__multicall__.execute() to execute the rest of the hook implementations. > * with all this in mind the few lines of code in _pytest.core.MultiCall > ?hopefully make more sense :) > > best, > holger > >> Btw, the pytest_pyfunc_call() parameters seem to be in the wrong order >> based on _pytest.python.py and the prototype in _pytest.hookspec.py >> only lists one parameter. >> >> >> ?Vyacheslav >> > From ralf at systemexit.de Fri Jun 10 17:09:25 2011 From: ralf at systemexit.de (Ralf Schmitt) Date: Fri, 10 Jun 2011 17:09:25 +0200 Subject: [py-dev] error in teardown wrongly reported as error in setup Message-ID: <8739jh910a.fsf@muni.brainbot.com> I have a directory with two test files with the following content: ,----[ test_1.py ] | #! /usr/bin/env py.test | | def pytest_funcarg__hello(request): | def setup(): | return "hello world" | | def teardown(idx): | raise RuntimeError("fail") | | return request.cached_setup( | setup=setup, | teardown=teardown, | scope="module") | | | def test_val(hello): | assert hello == "hello world" `---- ,----[ test_2.py ] | def test_foo(): | pass `---- py.test reports an error for test_foo: ,---- | ========================== test session starts ========================== | platform linux2 -- Python 2.7.2 -- pytest-2.0.3 | collected 2 items | | test_1.py . | test_2.py E | | ================================ ERRORS ================================= | ______________________ ERROR at setup of test_foo _______________________ | | idx = 'hello world' | | def teardown(idx): | > raise RuntimeError("fail") | E RuntimeError: fail | | test_1.py:8: RuntimeError | =================== 1 passed, 1 error in 0.10 seconds =================== `---- Cheers, - Ralf