Created attachment 197313 [details] patch Many packages define 'pytest' tests. USE_PYTHON=pytest adds the test dependency and the do-test target.
Definitely +1 on this idea. Nice one. Could you allow for adding additional environment variables and commandline arguments? Something like: ``` @cd ${WRKSRC} && ${SETENV} ${MAKE_ENV} ${PYTEST_ENV} \ ${PYTHON_CMD} -m pytest ${PYTEST_ARGS} ``` where ${PYTEST_ENV} and ${PYTEST_ARGS} are empty by default.
Created attachment 197333 [details] patch Thanks Matthew, Good improvement idea. I updated the patch. Yuri
I'd like to see something like this leverage/extend the test framework bits if necessary to achieve what we want, to make it usable for cases other than just pytest (nosetests, unittest, whatever). python.mk could then setup/wrap python specific test scenarios, using generic TEST_* (TEST_COMMAND/TEST_CMD_ARGS, etc) variables wherever possible. I see most of the value created in leveraging/extending the test framework, than is obtained from removing the need for TEST_DEPENDS and not needing to specify a test target. Test invocations are *notoriously* non-standard in the Python world (granted, python -m pytest 'is' a common one), with various permutations of a env/commands/args combination necessary to get it just right. Further, its very commonly the case that pytest along is not a standalone/sufficient dependency by itself, and having them split implicitly in the framework and explicitly in the port is not particularly desirable. This idea has been an open task [1] for the Python team for a while, looking for at least a *minimally* considered/designed schema to support a plurality of test execution needs, rather than just added on an adhoc basic, and hopefully one that could be leveraged to the benefit of other softwares/language stacks in the ports tree. Perhaps a good place to start would be to take stock of the existing test methods for existing python ports and see what scheme we can come up with to support the 'major classes' of test execution methods.
Created attachment 197357 [details] patch (In reply to Kubilay Kocak from comment #3) Kubilay, you are right, thanks for your suggestion. I plugged this test into the existing test framework. This made it simpler. nosetests, unittest, whatever can be added later in a likewise fashion.
Please don't invent new variables, there is already TEST_ENV
Created attachment 197361 [details] patch Use TEST_ENV/TEST_ARGS.
Does this actually work?
I'd say it lacks a: --- Mk/bsd.port.mk +++ Mk/bsd.port.mk @@ -3363,3 +3363,3 @@ do-install: -.if !target(do-test) && defined(TEST_TARGET) +.if !target(do-test) && ( defined(TEST_TARGET) || defined(DO_MAKE_TEST) ) DO_MAKE_TEST?= ${SETENV} ${TEST_ENV} ${MAKE_CMD} ${MAKE_FLAGS} ${MAKEFILE} ${TEST_ARGS:C,^${DESTDIRNAME}=.*,,g}
Remark: actually two kinds of tests exist, the post-build ones and the post-install cases. ATM the target do-test of bsd.port.mk handles the post-build cases. In all the Python ports that I know, the test (or check) target requires to load their module, and can only be satisfied on post-install.
That's definitely good idea to use devel/py-pytest for wide testing. Only one thing I'd like to mention here - devel/py-pytest is outdated, 6.2.2 is the recent release and the ports tree contains 4.5.0.
I wrote an update to pytest port here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256624
(In reply to Goran Mekić from comment #11) Thank you so much, Goran!
I think this is implemented now? Can we close it?
Yes, closing.