Bug 231555 - Mk/Uses/python.mk: Add USE_PYTHON=pytest
Summary: Mk/Uses/python.mk: Add USE_PYTHON=pytest
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Port Management Team
Keywords: feature
Depends on:
Reported: 2018-09-21 17:14 UTC by Yuri Victorovich
Modified: 2021-06-15 18:34 UTC (History)
5 users (show)

See Also:
yuri: maintainer-feedback? (python)

patch (918 bytes, patch)
2018-09-21 17:14 UTC, Yuri Victorovich
no flags Details | Diff
patch (1.51 KB, patch)
2018-09-22 06:17 UTC, Yuri Victorovich
no flags Details | Diff
patch (1.47 KB, patch)
2018-09-22 17:30 UTC, Yuri Victorovich
no flags Details | Diff
patch (947 bytes, patch)
2018-09-22 18:32 UTC, Yuri Victorovich
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Yuri Victorovich freebsd_committer 2018-09-21 17:14:53 UTC
Created attachment 197313 [details]

Many packages define 'pytest' tests.

USE_PYTHON=pytest adds the test dependency and the do-test target.
Comment 1 Matthew Seaman freebsd_committer 2018-09-22 05:57:16 UTC
Definitely +1 on this idea.  Nice one.

Could you allow for adding additional environment variables and commandline arguments?  Something like:

     @cd ${WRKSRC} && ${SETENV} ${MAKE_ENV} ${PYTEST_ENV} \
         ${PYTHON_CMD} -m pytest ${PYTEST_ARGS}

where ${PYTEST_ENV} and ${PYTEST_ARGS} are empty by default.
Comment 2 Yuri Victorovich freebsd_committer 2018-09-22 06:17:56 UTC
Created attachment 197333 [details]

Thanks Matthew,

Good improvement idea. I updated the patch.

Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2018-09-22 09:54:07 UTC
I'd like to see something like this leverage/extend the test framework bits if necessary to achieve what we want, to make it usable for cases other than just pytest (nosetests, unittest, whatever). python.mk could then setup/wrap python specific test scenarios, using generic TEST_* (TEST_COMMAND/TEST_CMD_ARGS, etc) variables wherever possible.

I see most of the value created in leveraging/extending the test framework, than is obtained from removing the need for TEST_DEPENDS and not needing to specify a test target.

Test invocations are *notoriously* non-standard in the Python world (granted, python -m pytest 'is' a common one), with various permutations of a env/commands/args combination necessary to get it just right.

Further, its very commonly the case that pytest along is not a standalone/sufficient dependency by itself, and having them split implicitly in the framework and explicitly in the port is not particularly desirable.

This idea has been an open task [1] for the Python team for a while, looking for at least a *minimally* considered/designed schema to support a plurality of test execution needs, rather than just added on an adhoc basic, and hopefully one that could be leveraged to the benefit of other softwares/language stacks in the ports tree.

Perhaps a good place to start would be to take stock of the existing test methods for existing python ports and see what scheme we can come up with to support the 'major classes' of test execution methods.
Comment 4 Yuri Victorovich freebsd_committer 2018-09-22 17:30:33 UTC
Created attachment 197357 [details]

(In reply to Kubilay Kocak from comment #3)

Kubilay, you are right, thanks for your suggestion.
I plugged this test into the existing test framework. This made it simpler.

nosetests, unittest, whatever can be added later in a likewise fashion.
Comment 5 Antoine Brodin freebsd_committer 2018-09-22 18:27:11 UTC
Please don't invent new variables, there is already TEST_ENV
Comment 6 Yuri Victorovich freebsd_committer 2018-09-22 18:32:16 UTC
Created attachment 197361 [details]

Comment 7 Mathieu Arnold freebsd_committer 2018-09-24 12:58:02 UTC
Does this actually work?
Comment 8 Mathieu Arnold freebsd_committer 2018-09-24 12:59:34 UTC
I'd say it lacks a:

--- Mk/bsd.port.mk
+++ Mk/bsd.port.mk
@@ -3363,3 +3363,3 @@ do-install:

-.if !target(do-test) && defined(TEST_TARGET)
+.if !target(do-test) && ( defined(TEST_TARGET) || defined(DO_MAKE_TEST) )
Comment 9 Thierry Thomas freebsd_committer 2021-01-31 21:20:17 UTC
Remark: actually two kinds of tests exist, the post-build ones and the post-install cases. ATM the target do-test of bsd.port.mk handles the post-build cases.

In all the Python ports that I know, the test (or check) target requires to load their module, and can only be satisfied on post-install.
Comment 10 Sergey A. Osokin freebsd_committer 2021-02-08 18:52:04 UTC
That's definitely good idea to use devel/py-pytest for wide testing.

Only one thing I'd like to mention here - devel/py-pytest is outdated, 6.2.2 is the recent release and the ports tree contains 4.5.0.
Comment 11 Goran Mekić 2021-06-15 18:14:14 UTC
I wrote an update to pytest port here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256624
Comment 12 Sergey A. Osokin freebsd_committer 2021-06-15 18:34:58 UTC
(In reply to Goran Mekić from comment #11)

Thank you so much, Goran!