[pypy-dev] Re: about to check in refactored test-framework

[Michael Hudson]

holger krekel <hpk at trillke.net> writes:

> hello pypy,
>
> since yesterday evening i cleaned up the all of our test stuff.

Cool!

> I haven't checked it in, yet. I'd like to know if anybody 
> has any strong feelings about the way it currently is...

I have strong feelings about the way it currently is: it sucks.

And I say that as the person most responsible for the current
situation.

> And here is how the refactored test stuff works:
>
> - all the testsupport, testcts, testwice and all the code duplications
>   that came with it are gone. 

Is the functionality of testcts.py still present?

> - not need to set any PYTHONPATH anymore to start a test or
>   interpreter/interactive.py or interpreter/main.py 

It wasn't necessary for main.py already.  But yay!

> - you can start any test by simply calling it directly via
>
>     python test_XXX.py 

This was true already, no?

> - you can run all tests by 
>
>     python test_all.py  # in the root directory

Do you mean python test/all.py?  (From what you said in your checkin
message).

> - you can run all tests below your current working dir
>   say e.g. in interpreter/
>
>     python ../test_all.py  # will start from the current dir

I like this less, I'd prefer it if running the test_all.py script ran
all the tests in the tree independently of where you start it from.

> - you can run all tests in one object space via
>
>     python test_all.py -S  # standard objectspace
>     python test_all.py -T  # trivial objectspace
>     python test_all.py -ST # first standard, then trivial objectspace

It would be nice if the last option only reran the tests that would
actually run differently under the different object spaces.

Have you cleaned up how you switch main.py/interactive.py select their
object space too?

> - you can pass some regular expressions 
>
>     python test_all.py pypy.interpreter # will only run tests whose path
>                                           matches pypy.interpreter
>
>     python test_all.py -S "%pypy.(obj|app)space"
>     # will avoid objspace and appspace tests and otherwise run with
>     # stdobjspace, the '%' inverts the match, btw 

Well, OK.  This is not a feature I'd have thought of, or asked for,
but we'll see if it turns out to be sueful.

> - autopath configuration. 
>
>   every startable script (the tests among them, but also
>   interpreter/interactive.py interpreter/main.py) first does a
>
>     import autopath 
>
>   which configures the python path to include the directory
>   which contains 'pypy'. moreover autopath provides some
>   attributes 
>
>     pypydir   the 'pypy' directory path itself 
>     this_dir  the directory where your "autopath" lives
>               useful for finding test data
>
>   if you run pypy/tool/autopath.py then this will copy itself
>   over all other 'autopath.py' instances and report "sync" status. 
>   No ugly shell scripting anymore. OTOH autopath.py will hardly 
>   need to be modified anymore because it just does path manipulation.

Cool.  Something autopath should perhaps do as well: remove
sys.path[0].  At the moment if you try to run (say)
pypy/interpreter/main.py under Python 2.3 the presence of the
pypy/interpreter directory on sys.path causes problems, as in 2.3
dis.py wants to be able to find a standard library module called
'opcode'...

But then autopath.py needs to act differently iff it is the first
autopath to be imported[*].  Maybe autopath can just ensure that no
directory downtree of pypydir is on sys.path.

[*] Actually, maybe not.  Let me think about this one for a bit.

> - every test_*.py usually does
>
>     import autopath
>     from pypy.tool import test
>
>   and derives from test.TestCase or test.AppTestCase and 
>   on __name__=='__main__' invokes
>     
>     test.main()
>
>   to get to the "cmdline switched" objectspace you call
>
>     test.objspace()
>
>   to get to a specific objectspace you call
>     
>     test.objspace('std') or test.objspace('trivial')
>
> Some stuff regarding running tests successively in StdObjSpace and
> TrivialObjSpace (switch "-ST") is not completly right at the moment. But this
> is not so much a thing of the test framework, i think. 

Ah :-)

> As this checkin involves a lot of files i'd like
> to do the checkin ASAP to avoid conflicts. 
>
> Any oppinions? 

Yes: check it in already.

Cheers,
M.

-- 
  Any form of evilness that can be detected without *too* much effort
  is worth it...  I have no idea what kind of evil we're looking for
  here or how to detect is, so I can't answer yes or no.
                                       -- Guido Van Rossum, python-dev