Created attachment 225833 [details] pytest.diff QA: * portlint: OK (looks fine.) * testport: OK (poudriere: 13.0, amd64 tested) * maketest: OK (3035 passed, 10 skipped, 10 xfailed)
Needs reverse dependents test and QA confirmation, as dependent consumers are known to regularly not support pytest major version updates for a while after release. May require a pytest4 port
As number of ports using pytest is huge, how do we continue? I don't have a server that can run all those tests.
Thought not completely useful, the consumer (*_depends) set is only <50 ports, almost all very lightweight. A grep of *_DEPENDS looking for '<' lines is also a decent first pass. This wont include ports that don't correctly or completely declare their dependency <version-specs> however.
If you refer to {RUN,BUILD}_DEPENDS, yes, you're probably right, but TEST_DEPENDS will probably yield a much larger port collection dependent on pytest. I mean, https://www.freshports.org/devel/py-pytest/ reveals which ports depend on pytest, but I think there's no TEST_DEPENDS in it. If freshports list is only I should test, no problem!
Such a command line will list all ports using devel/py-pytest in TEST_DEPENDS: $ find /usr/ports -type d ! -name files -print -exec make -V TEST_DEPENDS -C {} \; 2>/dev/null | grep -B 1 devel/py-pytest Remark: there are several ways to achieve the same result, do not hesitate to find a better one!
Tested with "make -DBATCH test" and following ports reported no problems: cad/netgen devel/py-pytest-factoryboy www/py-textile devel/py-pytest-aiohttp devel/py-pytest-asyncio devel/py-pytest-azurepipelines devel/py-pytest-benchmark devel/py-pytest-black devel/py-pytest-cov devel/py-pytest-datadir devel/py-pytest-django devel/py-pytest-env devel/py-pytest-factoryboy devel/py-pytest-fixture-config devel/py-pytest-flake8 devel/py-pytest-flakes devel/py-pytest-freezegun devel/py-pytest-lazy-fixture devel/py-pytest-localserver devel/py-pytest-mypy devel/py-pytest-pycodestyle devel/py-pytest-randomly devel/py-pytest-regressions devel/py-pytest-rerunfailures devel/py-pytest-sugar devel/py-pytest-tornado devel/py-pytest-translations devel/py-pytest-trio devel/py-pytest-twisted devel/py-pytest-virtualenv devel/py-testinfra devel/py-ttictoc finance/fava science/py-MDAnalysisTests The ports that fail even on main branch: devel/py-pytest-cache devel/py-pytest-flask devel/py-pytest-forked devel/py-pytest-mock devel/py-pytest-mutagen devel/py-pytest-relaxed devel/py-pytest-shutil devel/py-pytest-timeout devel/py-pytest-verbose-parametrize Ports that fail with upgraded pytest, but after bumping their version to match PyPi succeed: devel/py-pytest-xdist devel/py-pytest-xprocess Koobs, xdist is yours, so can you please update that one? I will contact maintainer of xprocess port. Thank you!
(In reply to Goran Mekić from comment #6) Thank you for running this QAS Goran Can you detail the failure modes experienced? Such as the "The ports that fail even on main branch:" group?
For any of the failure cases that are ports bugs (incorrectly incompletely specified version specs of *_DEPENDS), any changes are Approved by: portmgr (blanket: bugfix) And wont require issues to be created or maintainer approval
(In reply to Kubilay Kocak from comment #7) I have no idea what the errors were, I didn't write them down.
Updating pytest-flask also makes it pass the tests, but all other ports are either on latest version or fail for some reason (I'm suspecting some deeper dependency needs update). Failing ports: devel/py-pytest-cache devel/py-pytest-forked devel/py-pytest-mock devel/py-pytest-mutagen devel/py-pytest-relaxed devel/py-pytest-shutil devel/py-pytest-timeout devel/py-pytest-verbose-parametrize
What's our next move?
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=0c6ce3c9c26d878acf943d53fb2a01ba602b71c2 commit 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2 Author: Goran Meki? <meka@tilda.center> AuthorDate: 2021-06-23 20:30:35 +0000 Commit: Kevin Bowling <kbowling@FreeBSD.org> CommitDate: 2021-06-23 20:34:47 +0000 devel/py-pytest: Update to 6.2.4 PR: 256624 devel/py-pytest/Makefile | 12 ++++-------- devel/py-pytest/distinfo | 6 +++--- devel/py-pytest/files/patch-testing-test_parseopt.py (gone) | 11 ----------- 3 files changed, 7 insertions(+), 22 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=d07f92b9bf00821e372f003d698525cd1ae19dc2 commit d07f92b9bf00821e372f003d698525cd1ae19dc2 Author: Kevin Bowling <kbowling@FreeBSD.org> AuthorDate: 2021-06-23 20:37:23 +0000 Commit: Kevin Bowling <kbowling@FreeBSD.org> CommitDate: 2021-06-23 20:37:23 +0000 devel/py-pytest-xdist: Update to 2.3.0 PR: 256624 Approved by: maintainer (koobs) devel/py-pytest-xdist/Makefile | 2 +- devel/py-pytest-xdist/distinfo | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
The failures are now a subset of existing failures, closing. If you make any progress on cleaning them up please send in new PRs. Thanks for your contributions!
@Kevin This was committed without approval or reference, and has broken at least one port (at build time), and likely more at runtime. "failures are now a subset of existing failures, closing." is an insufficient rationale for the committing of a change that has not completed or confirmed QA. [130amd64-default-job-14] Extracting py38-pytest-6.2.4: .......... done ===> py38-pytest-relaxed-1.1.5_1 depends on package: py38-pytest>=3<5 - not found *** Error code 1 Can you please resolve the fallout, referencing this issue on commit, thanks
For the moment let's just imagine py-pytest-relaxed is fixed. I asked already and I have to ask again "What's our next move?" I'm not trying to rush anything, but nothing will happen if person who's "lost" doesn't get guidance.
(In reply to Kubilay Kocak from comment #15) Sure, it's somewhat intricate so I want to confirm with you to revert the following git hashes: * 5989f395a01e1afc9b1b3a0c668d3cf48226f0c3 * d07f92b9bf00821e372f003d698525cd1ae19dc2 * b15351babf6846ec3cdc902203874e55413ee08a * 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2 Note that if I do this I wont work on it again because I disagree, loosely and respectfully. (In reply to Goran Mekić from comment #16) koobs I'd appreciate if you can provide guidance on Goran's question. In my humble opinion the better move here is to mark your port as broken because the upstream isn't lively and nothing depends on it. I admit I didn't see that in my testing and should have been more thorough so I'm willing to revert it regardless.
(In reply to Kevin Bowling from comment #17) Please do whatever you believe is now necessary to correctly and completely resolve the issue. On the question of responding to comment 11, I believe it was clear: - comment 1 (complete consumer QA against <version-spec>'s they support, which involves fixing any that are incorrect or not declared) - comment 1 "May require a pytest4 port" (dependent on outcome of the first qa) - comment 7 (understanding of failure modes precluding update) On the question of marking pytest-relaxed broken, it is/was not broken with the version of pytest prior to the update from this issue. this leads to and is related to comment 1 (may require a pytest4) and the reverse dependents QA request. I don't understand the context of "I wont work on it again because I disagree". If that's with respect to "unapproved and unreferenced" (comment 15), the commit message contained: devel/py-pytest: Update to 6.2.4 PR: 256624 Ignoring the request for QA, the request to understand failure modes for consumer ports, and the knowledge that there were remaining and unknown issues/errors in consumers (comment 15), on what basis was the commit made? Not ignoring the above, on what basis was the commit made? Ultimately these are now rhetorical questions, and we can discuss them offline if you're available and interested in doing so. Let's just get this resolved properly for our users. Over to you Kevin.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=46a663d44cb038c4b5fcca21f1c0151db0e8302e commit 46a663d44cb038c4b5fcca21f1c0151db0e8302e Author: Kevin Bowling <kbowling@FreeBSD.org> AuthorDate: 2021-06-27 05:07:31 +0000 Commit: Kevin Bowling <kbowling@FreeBSD.org> CommitDate: 2021-06-27 05:08:40 +0000 devel/py-pytest-flask: revert 6.2.4 update Revert "devel/py-pytest: Update to 6.2.4" This reverts commit 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2. Revert "devel/py-pytest-translations: Update to 3.1.0" This reverts commit 5989f395a01e1afc9b1b3a0c668d3cf48226f0c3. Revert "devel/py-pytest-xdist: Update to 2.3.0" This reverts commit d07f92b9bf00821e372f003d698525cd1ae19dc2. Revert "devel/py-pytest-flask: Update to 1.2.0" This reverts commit b15351babf6846ec3cdc902203874e55413ee08a. PR: 256624 Requested by: koobs devel/py-pytest-flask/Makefile | 2 +- devel/py-pytest-flask/distinfo | 6 +++--- devel/py-pytest-translations/Makefile | 15 +++++++++++---- devel/py-pytest-translations/distinfo | 6 +++--- devel/py-pytest-xdist/Makefile | 2 +- devel/py-pytest-xdist/distinfo | 6 +++--- devel/py-pytest/Makefile | 12 ++++++++---- devel/py-pytest/distinfo | 6 +++--- .../py-pytest/files/patch-testing-test_parseopt.py (new) | 11 +++++++++++ 9 files changed, 44 insertions(+), 22 deletions(-)
(In reply to Kubilay Kocak from comment #18) Thanks, since you haven't expressed technical issue with my revert plan above I will implement the revert of those commits. All I said is I don't want to work on this that way, it's ok for me to choose how I volunteer :) I have a political platform: I want main to be a little more Fedora a little less Enterprise Linux and this looked ready to go: Goran has methodically been testing and fixing the issues leading to one leaf port that is known to be newly broken while fixing many other issues. I'm unconvinced it's not an improvement, but you're right I made a procedural error as a newer ports committer and although I executed many tests I did not do a thorough enough job to identify this failure ahead of time to discuss it. I also misunderstood where you were leading and what approvals remained. My experience thus far has been that it's easier to work on src over ports because of all these latches which is unexpected. It's open to another contributor because I would like to focus on things with less gatekeepers and put my time into making faster improvements and attracting new talent and integrating their work. I'm not saying anyone's wrong, just that I don't want to do it the way you want. My position is based in part by dialog in https://github.com/bitprophet/pytest-relaxed/issues/12, where you can see the methodologies of other maintainers who are dealing with tails wagging dogs. I hope you will respect me as I do you and not agreeing and getting out of the way is part of that. And for Goran, sorry for the false start keep working on the bugs and hopefully someone else can get it done.
Created attachment 226076 [details] pytest6and4.diff Pytest4 added, pytest-relaxed changed. Anything else?
(In reply to Kevin Bowling from comment #20) Let's have a chat offline, because (a) i want the same things you do (b) i do my best to be exactly not a gatekeeper, by making clear exactly what expectations are up front, which saves everyone the overhead of repeating the same things over and over, which unfortunately the project spends the vast majority of its time doing, which wastes everyone's time. I have even gone so far as to document a maintainer policy, which explicitly enables and allows people to just get stuff done, with no "all these latches" https://wiki.freebsd.org/KubilayKocak#MAINTAINER_POLICY I wont apologise to anyone however for high QA expectations. This is the second major source of waste of our limited resources. Again, let's have a pow-wow offline so we can get better things done together.
(In reply to Kubilay Kocak from comment #22) I've apologized privately for trying to get involved here and tried to undo it. I apologize again.
Created attachment 226168 [details] pytest-check.sh This is the script I quickly assembled to help find the ports that depend on pytest and run "make test" on them. The way to use it: mkdir pytest-check cd pytest-check # place script here chmod +x pytest-check.sh ./pytest-check.sh list >list.txt ./pytest-check.sh test . It will generate {success,failed}.log files and any subsequent call of the last command will remove failed.log and check success.log for ports to skip. I'm running it currently on 68268248585a32a80cb0e381cfd19a288e13d5f2 to check which ports are failing already. I know this is no way to check for failed/success port tests reliably, but it's a start.
Created attachment 226194 [details] failed-main.log
Created attachment 226195 [details] success-main.log
Created attachment 226196 [details] failed-pytest.log
Created attachment 226197 [details] success-pytest.log
The {failed,success}-main.log are ports that failed/succeeded on main git branch and analogous is for pytest branch which is at https://github.com/mekanix/freebsd-ports/tree/port/pytest. I would say success-pytest.log is a definite list of ports that should not be considered any more while other logs are here just for comparison. I am sure there are a lot of ports failing because same port with other name is already there (one example is py-sqlalchemy). I will start testing individual ports from failed-pytest.log, but it's a tedious job so I would really appreciate some joint effort.
As I don't have the time to fix all the ports, may I suggest alternative solution? If I go through the list of ports that use pytest using pytest-check.sh and replace pytest with pytest4 that way nobody has to break as that is exact version like pytest in main branch, yet nobody has to wait as new ports will probably use newer pytest. I need a feedback on this before I continue, so please advise.
Created attachment 226639 [details] pytest4and6.diff This patch moves py-pytest to py-pytest4, creates py-pytest version 6 and replaces devel/py-pytest with devel/py-pytest4 in ports Makefile.
Created attachment 226640 [details] pytest4and6.diff Edit made to MOVED
Created attachment 226643 [details] pytest-check.sh Based on https://github.com/mekanix/freebsd-ports/commit/c6437e2421811d27d6205f9ac0863b8bc115e97f pytest-check.sh list >pytest-ports.txt pytest-check.sh pin pytest-ports.txt
Created attachment 226644 [details] pytest.diff Bump PORTREVISION
So? What are we waiting now?
Is there anything else to be done here? If there is, could anyone tell me and I'll do it. If there's not, can I get an explanation what are we waiting?
(In reply to Goran Mekić from comment #36) I need time/cycles to land it or to coordinate with someone else who can. It is at the top of my list and hasn't been forgotten.
*** Bug 259018 has been marked as a duplicate of this bug. ***
(In reply to Kubilay Kocak from comment #37) Hi Goran, Does the patch switching pytest consumers to a new pytest4 port only switch those that don't support anything above 4 (explicitly don't work with, or pin to < 5), or were all just changed to pytest4 ?
(In reply to Kubilay Kocak from comment #39) All were changed.
Reviewing rev deps for > 6|5 pytest dependency switch
Moin moin Quick reminder, that the pkg-version went backwards from 6.x to 4.x -- so either please commit the update again soon, or add a PORTEPOCH. mfg Tobias
Or can new pytest be commited as a separate port? Then other ports can be reviewed switched one by one.
Ping koobs: any progress on this PR?
Maintainer reset.
I'm picking this up. Let's start with preserving pytest 4 and confirming that it works for some port which is not compatible with pytest 6.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=8802adcf7cd9e7a53d6995a7fea998908d3f81a2 commit 8802adcf7cd9e7a53d6995a7fea998908d3f81a2 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-18 16:24:20 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-03-18 17:11:38 +0000 ftp/py-aioftp: switch to pytest4 - With 3 preceeding commits, the tests are now able to run when py-pytest is at 6.x PR: 256624 ftp/py-aioftp/Makefile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=7db055349957ee7f79e5c2452430c71a3e86b6eb commit 7db055349957ee7f79e5c2452430c71a3e86b6eb Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-18 16:24:00 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-03-18 17:11:37 +0000 devel/py-pytest4: preserve 4.x version of devel/py-pytest - Preserve 4.x version of pytest for the ports which are not compatible with pytest 6.x. This is a prerequisite for updating pytest. PR: 256624 Approved by: maintainer timeout (python@, many months) devel/Makefile | 1 + devel/py-pytest/Makefile | 2 + devel/py-pytest4/Makefile (new) | 61 ++++++++++++++++++++++ devel/py-pytest4/distinfo (new) | 3 ++ .../files/patch-testing-test_parseopt.py (new) | 11 ++++ devel/py-pytest4/pkg-descr (new) | 12 +++++ 6 files changed, 90 insertions(+)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=dc826ba1280fc889e95b8c12fc1aba6279c3d67d commit dc826ba1280fc889e95b8c12fc1aba6279c3d67d Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-18 16:37:40 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-03-18 17:11:37 +0000 devel/py-pytest4-{asyncio,cov}: preserve pytest4 counterparts of corresponding ports PR: 256624 devel/Makefile | 2 ++ devel/py-pytest-asyncio/Makefile | 2 ++ devel/py-pytest-cov/Makefile | 2 ++ devel/py-pytest4-asyncio/Makefile (new) | 22 ++++++++++++++++++++++ devel/py-pytest4-asyncio/distinfo (new) | 3 +++ devel/py-pytest4-asyncio/pkg-descr (new) | 5 +++++ devel/py-pytest4-cov/Makefile (new) | 25 +++++++++++++++++++++++++ devel/py-pytest4-cov/distinfo (new) | 3 +++ devel/py-pytest4-cov/pkg-descr (new) | 8 ++++++++ 9 files changed, 72 insertions(+)
I've built all ports which run/build depend on pytest, and there are no failures. We'll do an exp-run when the change is ready just to be sure. Currently I'm running `make test` for all ports which test-depend on pytest with pytest4. Next I'll do the same with pytest7 (I'm testing 7.11 right now) and look at the new failures more closely. The ongoing work is at https://github.com/AMDmi3/freebsd-ports/tree/pytest
So it turned out that most failures after pytest update are caused by outdated pytest-asyncio and are fixed by updating it as well. 4 remaining cases are fixed by switching them to pytest4, which I'm going to commit right away. There's also databases/py-fakeredis which fails regardless of the update: cd /work/usr/ports/databases/py-fakeredis/work-py38/fakeredis-1.7.1 && /usr/local/bin/python3.8 test_fakeredis.py /usr/local/bin/python3.8: can't open file 'test_fakeredis.py': [Errno 2] No such file or directory *** Error code 2 I'm switching it to pytest4 as well, as it limits the dependency versions from above. Note that py-fakeredis and py-parso where broken by the revert of the previous pytest update attempt, as these limit pytest dependency version from above and are not taking EPOCH bump into account.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=32ec2042864d3a2de26c774739b133dd3a0a2022 commit 32ec2042864d3a2de26c774739b133dd3a0a2022 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-25 19:23:34 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-03-28 12:29:52 +0000 math/py-levmar: switch to pytest4 - Switch to pytest4 to avoid test failures after updating pytest to 7.x PR: 256624 math/py-levmar/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=1d244cdd602dc64184d79d9f9687f918d1fdfe35 commit 1d244cdd602dc64184d79d9f9687f918d1fdfe35 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-28 12:35:35 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-03-28 12:38:39 +0000 textproc/py-parso: switch to pytest4 - Switch to pytest4 to avoid test failures after updating pytest to 7.x - Don't forget epoch when limiting pytest version PR: 256624 textproc/py-parso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=429facb51b6f90477d81e2c9b118a765eafb96a7 commit 429facb51b6f90477d81e2c9b118a765eafb96a7 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-25 19:24:07 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-03-28 12:31:10 +0000 textproc/py-sphinx: switch to pytest4 - Switch to pytest4 to avoid test failures after updating pytest to 7.x PR: 256624 textproc/py-sphinx/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=f08db191ff75ca34e8295bb60899b6cfbbbedeb2 commit f08db191ff75ca34e8295bb60899b6cfbbbedeb2 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-25 19:23:50 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-03-28 12:31:03 +0000 net/py-dugong: switch to pytest4 - Switch to pytest4 to avoid test failures after updating pytest to 7.x PR: 256624 net/py-dugong/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=aec4847321880fd6c01576ca577050a9d114b282 commit aec4847321880fd6c01576ca577050a9d114b282 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-28 12:34:27 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-03-28 12:38:27 +0000 databases/py-fakeredis: switch to pytest4 - Switch to pytest4 to avoid test failures after updating pytest to 7.x - Don't forget epoch when limiting pytest version PR: 256624 databases/py-fakeredis/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Created attachment 232778 [details] pytest+pytest-asyncio patch Here's an updated patch for pytest + pytest-asyncio, ready for review and testing. Note that previous pytest patch was missing some dependency changes (and some more were added in 7.1.1): - atomicwwites: windows-only - more-itertools: removed in 6.1.0 - six: no longer needed with dropped support for python 2 - wcwidth: removed in 6.0.0
Requesting an exp-run for the latest patch.
Comment on attachment 232778 [details] pytest+pytest-asyncio patch > + ${PYTHON_PKGNAMEPREFIX}tomli>=0:textproc/py-toml@${PY_FLAVOR} \ Found a typo: $ poudriere bulk -j 130amd64 devel/py-pytest-asyncio [...] [00:00:03] Warning: devel/py-pytest dependency on textproc/py-toml@py38 has wrong PKGNAME of 'py38-tomli' but should be 'py38-toml' [...]
Created attachment 232944 [details] pytest+pytest-asyncio patch > Found a typo: > > [00:00:03] Warning: devel/py-pytest dependency on textproc/py-toml@py38 has wrong Thanks, fixed!
Created attachment 232958 [details] pytest+pytest-asyncio patch Updated patch with pytest-asyncio fixes
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=f5a4de53e143baa4e22486d37125ac272c1a0032 commit f5a4de53e143baa4e22486d37125ac272c1a0032 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-04-06 12:26:08 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-04-06 22:34:34 +0000 devel/py-joblib: switch to pytest4 The port is not compatible with recent pytest PR: 256624 devel/py-joblib/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=d01e186749d9e2bd3be15a4038fe62e751dfcf70 commit d01e186749d9e2bd3be15a4038fe62e751dfcf70 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-04-06 12:51:29 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-04-06 22:34:34 +0000 devel/py-pytest-relaxed: switch to pytest4 This plugin is not compatible with recent pytest PR: 256624 devel/py-pytest-relaxed/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=35a9ac41d04e7bee62e88a2fc13eb3028b9476c7 commit 35a9ac41d04e7bee62e88a2fc13eb3028b9476c7 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-04-06 22:04:55 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-04-06 22:38:34 +0000 devel/py-pytest-{pep8,translations}: switch to pytest4 These plugins are not compatible with recent pytest. Consumers (sysutils/iocage and www/py-flask-security) were switched to pytest4 as well. Note that py-flask-security tests were and still are broken due to unrelated problem - py-sqlalchemy12 and py-sqlalchemy14 conflict. PR: 256624 devel/py-pytest-pep8/Makefile | 4 ++-- devel/py-pytest-translations/Makefile | 2 +- sysutils/iocage/Makefile | 4 ++-- www/py-flask-security/Makefile | 8 ++++---- 4 files changed, 9 insertions(+), 9 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=e38e0e6063c33fc7cea17ca466902ebde17f772f commit e38e0e6063c33fc7cea17ca466902ebde17f772f Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-04-06 17:20:10 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-04-06 22:38:33 +0000 devel/py-pytest4-cache: preserve pytest4 counterpart of corresponding port PR: 256624 devel/Makefile | 1 + devel/py-pytest-cache/Makefile | 2 ++ devel/py-pytest4-cache/Makefile (new) | 29 ++++++++++++++++++ devel/py-pytest4-cache/distinfo (new) | 3 ++ devel/py-pytest4-cache/files/patch-setup.py (new) | 36 +++++++++++++++++++++++ devel/py-pytest4-cache/pkg-descr (new) | 3 ++ 6 files changed, 74 insertions(+)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=21f418e9aa447a296d36f17aee6837d899fa91c8 commit 21f418e9aa447a296d36f17aee6837d899fa91c8 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-04-06 22:01:06 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-04-06 22:38:33 +0000 devel/py-pytest4-flakes: preserve pytest4 counterpart of corresponding port PR: 256624 devel/Makefile | 1 + devel/py-pytest-flakes/Makefile | 2 ++ devel/py-pytest4-flakes/Makefile (new) | 23 +++++++++++++++++++++++ devel/py-pytest4-flakes/distinfo (new) | 3 +++ devel/py-pytest4-flakes/pkg-descr (new) | 3 +++ 5 files changed, 32 insertions(+)
The current status: - Well, I've had to write my own poudriere which is capable of bulk `make test` runs. The last bunch of fixes was possible solely because of it. Hopefully the tool will be useful on its own. - Right now it's running another test of all *pytest* consumers. - Notably, there are a lot of failures even with pytest4, which shows that tests are not getting enough attentions. I hope to fix that by releasing the mentioned utility and, for python ports specifically, by introducing pytest.mk https://reviews.freebsd.org/D34693?vs=on&id=104613#toc which makes it easier to silence failing tests and thus make passing tests more useful. - Also note that the pytest update will fix about a dozen of ports on its own. Still, I'm striving not to break anything. I'll fix some more ports by hand, as failures are trivial in many cases, such as missing depends.
Another run has finished, it has revealed 3 failures which I've already fixed. I've also committed a bunch of fixes for other test problems not directly related to pytest7 update (missing TEST_DEPENDS in most cases), however not all of these fix testing for corresponding ports completely. It was also revealed that I use overly restrictive jail networking settings (ip4=disable ipv6=disable) in my porttester, which has led to a lot of failures in tests which try to use sockets (and it may happen that some pytest7 related failures were masked by these). Next run will allow networking limited with local addresses.
The last run shows no failures caused by pytest update and no apparent problems which may mask test results. Here's the list of ports which are FIXED by pytest update: devel/py-fastnumbers failed → OK devel/py-flaky failed → OK devel/py-pytest failed → OK devel/py-pytest-xdist failed → OK math/py-cvxopt failed → OK net-mgmt/py-napalm failed → OK security/py-pynacl failed → OK sysutils/py-ansible-base failed → OK sysutils/py-ansible-core211 failed → OK textproc/py-ocrmypdf failed → OK textproc/py-pygments failed → OK So I'm committing the pytest update now.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=c11512de05705ac6b2099551144d1b4f3d0b6e0e commit c11512de05705ac6b2099551144d1b4f3d0b6e0e Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-03-25 19:14:57 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-04-08 15:27:02 +0000 devel/py-pytest-asyncio: update to 0.18.2 PR: 256624, 262884 Approved by: jbeich (maintainer) devel/py-pytest-asyncio/Makefile | 20 ++++++++++++++++---- devel/py-pytest-asyncio/distinfo | 6 +++--- 2 files changed, 19 insertions(+), 7 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=2ada39de05b12da1d1083c40ad676c061b6e8a70 commit 2ada39de05b12da1d1083c40ad676c061b6e8a70 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2021-07-08 12:31:51 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-04-08 15:26:07 +0000 devel/py-pytest: update to 7.1.1 PR: 256624 Approved by: maintainer timeout (python@, 9 months) devel/py-pytest/Makefile | 27 +++++++++------------- devel/py-pytest/distinfo | 6 ++--- .../files/patch-testing-test_parseopt.py (gone) | 11 --------- 3 files changed, 14 insertions(+), 30 deletions(-)
Thanks portmgr for no reaction in 2 weeks, thankfully I've managed on my own.
Thanks for taking care of this Dmitry. Would love to see/use your poudriere make test bits.
(In reply to Kubilay Kocak from comment #73) > Would love to see/use your poudriere make test bits. See ports-mgmt/reprise (needs FLAVOR=py310 specified to build, because of another poudriere problem), also announced on ports-developers@