Bug 256624 - devel/py-pytest: Update to 6.2.5
Summary: devel/py-pytest: Update to 6.2.5
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Dmitry Marakasov
URL: https://docs.pytest.org/en/stable/cha...
Keywords:
: 259018 (view as bug list)
Depends on: 262884
Blocks: 259648 256745
  Show dependency treegraph
 
Reported: 2021-06-15 18:13 UTC by Goran Mekić
Modified: 2022-04-13 18:48 UTC (History)
9 users (show)

See Also:
amdmi3: exp-run?


Attachments
pytest.diff (3.46 KB, patch)
2021-06-15 18:13 UTC, Goran Mekić
no flags Details | Diff
pytest6and4.diff (7.53 KB, patch)
2021-06-27 10:50 UTC, Goran Mekić
no flags Details | Diff
pytest-check.sh (1.06 KB, application/x-shellscript)
2021-07-02 11:51 UTC, Goran Mekić
no flags Details
failed-main.log (8.13 KB, text/plain)
2021-07-03 14:20 UTC, Goran Mekić
no flags Details
success-main.log (5.01 KB, text/plain)
2021-07-03 14:20 UTC, Goran Mekić
no flags Details
failed-pytest.log (8.28 KB, text/plain)
2021-07-03 14:21 UTC, Goran Mekić
no flags Details
success-pytest.log (4.87 KB, text/plain)
2021-07-03 14:21 UTC, Goran Mekić
no flags Details
pytest4and6.diff (252.96 KB, patch)
2021-07-23 19:07 UTC, Goran Mekić
no flags Details | Diff
pytest4and6.diff (253.52 KB, patch)
2021-07-23 19:08 UTC, Goran Mekić
no flags Details | Diff
pytest-check.sh (1.28 KB, application/x-shellscript)
2021-07-23 22:16 UTC, Goran Mekić
no flags Details
pytest.diff (320.68 KB, patch)
2021-07-23 22:17 UTC, Goran Mekić
no flags Details | Diff
pytest+pytest-asyncio patch (6.05 KB, patch)
2022-03-28 12:59 UTC, Dmitry Marakasov
no flags Details | Diff
pytest+pytest-asyncio patch (6.05 KB, patch)
2022-04-04 19:56 UTC, Dmitry Marakasov
no flags Details | Diff
pytest+pytest-asyncio patch (6.24 KB, patch)
2022-04-05 11:42 UTC, Dmitry Marakasov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Goran Mekić 2021-06-15 18:13:07 UTC
Created attachment 225833 [details]
pytest.diff

QA:

  * portlint: OK (looks fine.)
  * testport: OK (poudriere: 13.0, amd64 tested)
  * maketest: OK (3035 passed, 10 skipped, 10 xfailed)
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-16 00:32:27 UTC
Needs reverse dependents test and QA confirmation, as dependent consumers are known to regularly not support pytest major version updates for a while after release. May require a pytest4 port
Comment 2 Goran Mekić 2021-06-16 08:01:18 UTC
As number of ports using pytest is huge, how do we continue? I don't have a server that can run all those tests.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-17 00:11:53 UTC
Thought not completely useful, the consumer (*_depends) set is only <50 ports, almost all very lightweight.

A grep of *_DEPENDS looking for '<' lines is also a decent first pass. This wont include ports that don't correctly or completely declare their dependency <version-specs> however.
Comment 4 Goran Mekić 2021-06-17 11:02:18 UTC
If you refer to {RUN,BUILD}_DEPENDS, yes, you're probably right, but TEST_DEPENDS will probably yield a much larger port collection dependent on pytest. I mean, https://www.freshports.org/devel/py-pytest/ reveals which ports depend on pytest, but I think there's no TEST_DEPENDS in it. If freshports list is only I should test, no problem!
Comment 5 Thierry Thomas freebsd_committer freebsd_triage 2021-06-17 20:10:25 UTC
Such a command line will list all ports using devel/py-pytest in TEST_DEPENDS:

$ find /usr/ports -type d ! -name files -print -exec make -V TEST_DEPENDS -C {} \; 2>/dev/null | grep -B 1 devel/py-pytest

Remark: there are several ways to achieve the same result, do not hesitate to find a better one!
Comment 6 Goran Mekić 2021-06-20 19:01:22 UTC
Tested with "make -DBATCH test" and following ports reported no problems:
cad/netgen
devel/py-pytest-factoryboy
www/py-textile
devel/py-pytest-aiohttp
devel/py-pytest-asyncio
devel/py-pytest-azurepipelines
devel/py-pytest-benchmark
devel/py-pytest-black
devel/py-pytest-cov
devel/py-pytest-datadir
devel/py-pytest-django
devel/py-pytest-env
devel/py-pytest-factoryboy
devel/py-pytest-fixture-config
devel/py-pytest-flake8
devel/py-pytest-flakes
devel/py-pytest-freezegun
devel/py-pytest-lazy-fixture
devel/py-pytest-localserver
devel/py-pytest-mypy
devel/py-pytest-pycodestyle
devel/py-pytest-randomly
devel/py-pytest-regressions
devel/py-pytest-rerunfailures
devel/py-pytest-sugar
devel/py-pytest-tornado
devel/py-pytest-translations
devel/py-pytest-trio
devel/py-pytest-twisted
devel/py-pytest-virtualenv
devel/py-testinfra
devel/py-ttictoc
finance/fava
science/py-MDAnalysisTests

The ports that fail even on main branch:
devel/py-pytest-cache
devel/py-pytest-flask
devel/py-pytest-forked
devel/py-pytest-mock
devel/py-pytest-mutagen
devel/py-pytest-relaxed
devel/py-pytest-shutil
devel/py-pytest-timeout
devel/py-pytest-verbose-parametrize

Ports that fail with upgraded pytest, but after bumping their version to match PyPi succeed:
devel/py-pytest-xdist
devel/py-pytest-xprocess

Koobs, xdist is yours, so can you please update that one? I will contact maintainer of xprocess port. Thank you!
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-21 01:54:13 UTC
(In reply to Goran Mekić from comment #6)

Thank you for running this QAS Goran

Can you detail the failure modes experienced? Such as the "The ports that fail even on main branch:" group?
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-21 01:55:01 UTC
For any of the failure cases that are ports bugs (incorrectly incompletely specified version specs of *_DEPENDS), any changes are 

Approved by: portmgr (blanket: bugfix)

And wont require issues to be created or maintainer approval
Comment 9 Goran Mekić 2021-06-21 10:02:10 UTC
(In reply to Kubilay Kocak from comment #7)
I have no idea what the errors were, I didn't write them down.
Comment 10 Goran Mekić 2021-06-21 11:28:25 UTC
Updating pytest-flask also makes it pass the tests, but all other ports are either on latest version or fail for some reason (I'm suspecting some deeper dependency needs update). Failing ports:

devel/py-pytest-cache
devel/py-pytest-forked
devel/py-pytest-mock
devel/py-pytest-mutagen
devel/py-pytest-relaxed
devel/py-pytest-shutil
devel/py-pytest-timeout
devel/py-pytest-verbose-parametrize
Comment 11 Goran Mekić 2021-06-22 10:52:14 UTC
What's our next move?
Comment 12 commit-hook freebsd_committer freebsd_triage 2021-06-23 20:35:49 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=0c6ce3c9c26d878acf943d53fb2a01ba602b71c2

commit 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2
Author:     Goran Meki? <meka@tilda.center>
AuthorDate: 2021-06-23 20:30:35 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-06-23 20:34:47 +0000

    devel/py-pytest: Update to 6.2.4

    PR:             256624

 devel/py-pytest/Makefile                                    | 12 ++++--------
 devel/py-pytest/distinfo                                    |  6 +++---
 devel/py-pytest/files/patch-testing-test_parseopt.py (gone) | 11 -----------
 3 files changed, 7 insertions(+), 22 deletions(-)
Comment 13 commit-hook freebsd_committer freebsd_triage 2021-06-23 20:38:51 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=d07f92b9bf00821e372f003d698525cd1ae19dc2

commit d07f92b9bf00821e372f003d698525cd1ae19dc2
Author:     Kevin Bowling <kbowling@FreeBSD.org>
AuthorDate: 2021-06-23 20:37:23 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-06-23 20:37:23 +0000

    devel/py-pytest-xdist: Update to 2.3.0

    PR:             256624
    Approved by:    maintainer (koobs)

 devel/py-pytest-xdist/Makefile | 2 +-
 devel/py-pytest-xdist/distinfo | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)
Comment 14 Kevin Bowling freebsd_committer freebsd_triage 2021-06-23 20:39:56 UTC
The failures are now a subset of existing failures, closing.  If you make any progress on cleaning them up please send in new PRs.  Thanks for your contributions!
Comment 15 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-26 01:59:27 UTC
@Kevin This was committed without approval or reference, and has broken at least one port (at build time), and likely more at runtime. "failures are now a subset of existing failures, closing." is an insufficient rationale for the committing of a change that has not completed or confirmed QA.

[130amd64-default-job-14] Extracting py38-pytest-6.2.4: .......... done
===>   py38-pytest-relaxed-1.1.5_1 depends on package: py38-pytest>=3<5 - not found
*** Error code 1

Can you please resolve the fallout, referencing this issue on commit, thanks
Comment 16 Goran Mekić 2021-06-26 09:46:44 UTC
For the moment let's just imagine py-pytest-relaxed is fixed. I asked already and I have to ask again "What's our next move?" I'm not trying to rush anything, but nothing will happen if person who's "lost" doesn't get guidance.
Comment 17 Kevin Bowling freebsd_committer freebsd_triage 2021-06-26 21:54:10 UTC
(In reply to Kubilay Kocak from comment #15)
Sure, it's somewhat intricate so I want to confirm with you to revert the following git hashes: 
* 5989f395a01e1afc9b1b3a0c668d3cf48226f0c3
* d07f92b9bf00821e372f003d698525cd1ae19dc2
* b15351babf6846ec3cdc902203874e55413ee08a
* 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2

Note that if I do this I wont work on it again because I disagree, loosely and respectfully.

(In reply to Goran Mekić from comment #16)
koobs I'd appreciate if you can provide guidance on Goran's question.  In my humble opinion the better move here is to mark your port as broken because the upstream isn't lively and nothing depends on it.  I admit I didn't see that in my testing and should have been more thorough so I'm willing to revert it regardless.
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-27 01:38:46 UTC
(In reply to Kevin Bowling from comment #17)

Please do whatever you believe is now necessary to correctly and completely resolve the issue.

On the question of responding to comment 11, I believe it was clear:

 - comment 1 (complete consumer QA against <version-spec>'s they support, which involves fixing any that are incorrect or not declared)

 - comment 1 "May require a pytest4 port" (dependent on outcome of the first qa)

 - comment 7 (understanding of failure modes precluding update)

On the question of marking pytest-relaxed broken, it is/was not broken with the version of pytest prior to the update from this issue. this leads to and is related to comment 1 (may require a pytest4) and the reverse dependents QA request.

I don't understand the context of "I wont work on it again because I disagree". If that's with respect to "unapproved and unreferenced" (comment 15), the commit message contained:

    devel/py-pytest: Update to 6.2.4

    PR:             256624

Ignoring the request for QA, the request to understand failure modes for consumer ports, and the knowledge that there were remaining and unknown issues/errors in consumers (comment 15), on what basis was the commit made?

Not ignoring the above, on what basis was the commit made?

Ultimately these are now rhetorical questions, and we can discuss them offline if you're available and interested in doing so.

Let's just get this resolved properly for our users. Over to you Kevin.
Comment 19 commit-hook freebsd_committer freebsd_triage 2021-06-27 05:12:31 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=46a663d44cb038c4b5fcca21f1c0151db0e8302e

commit 46a663d44cb038c4b5fcca21f1c0151db0e8302e
Author:     Kevin Bowling <kbowling@FreeBSD.org>
AuthorDate: 2021-06-27 05:07:31 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-06-27 05:08:40 +0000

    devel/py-pytest-flask: revert 6.2.4 update

    Revert "devel/py-pytest: Update to 6.2.4"

    This reverts commit 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2.

    Revert "devel/py-pytest-translations: Update to 3.1.0"

    This reverts commit 5989f395a01e1afc9b1b3a0c668d3cf48226f0c3.

    Revert "devel/py-pytest-xdist: Update to 2.3.0"

    This reverts commit d07f92b9bf00821e372f003d698525cd1ae19dc2.

    Revert "devel/py-pytest-flask: Update to 1.2.0"

    This reverts commit b15351babf6846ec3cdc902203874e55413ee08a.

    PR:             256624
    Requested by:   koobs

 devel/py-pytest-flask/Makefile                            |  2 +-
 devel/py-pytest-flask/distinfo                            |  6 +++---
 devel/py-pytest-translations/Makefile                     | 15 +++++++++++----
 devel/py-pytest-translations/distinfo                     |  6 +++---
 devel/py-pytest-xdist/Makefile                            |  2 +-
 devel/py-pytest-xdist/distinfo                            |  6 +++---
 devel/py-pytest/Makefile                                  | 12 ++++++++----
 devel/py-pytest/distinfo                                  |  6 +++---
 .../py-pytest/files/patch-testing-test_parseopt.py (new)  | 11 +++++++++++
 9 files changed, 44 insertions(+), 22 deletions(-)
Comment 20 Kevin Bowling freebsd_committer freebsd_triage 2021-06-27 05:29:25 UTC
(In reply to Kubilay Kocak from comment #18)
Thanks, since you haven't expressed technical issue with my revert plan above I will implement the revert of those commits.  All I said is I don't want to work on this that way, it's ok for me to choose how I volunteer :)

I have a political platform:  I want main to be a little more Fedora a little less Enterprise Linux and this looked ready to go:  Goran has methodically been testing and fixing the issues leading to one leaf port that is known to be newly broken while fixing many other issues.  I'm unconvinced it's not an improvement, but you're right I made a procedural error as a newer ports committer and although I executed many tests I did not do a thorough enough job to identify this failure ahead of time to discuss it. I also misunderstood where you were leading and what approvals remained.  My experience thus far has been that it's easier to work on src over ports because of all these latches which is unexpected.  It's open to another contributor because I would like to focus on things with less gatekeepers and put my time into making faster improvements and attracting new talent and integrating their work.  I'm not saying anyone's wrong, just that I don't want to do it the way you want.  My position is based in part by dialog in https://github.com/bitprophet/pytest-relaxed/issues/12, where you can see the methodologies of other maintainers who are dealing with tails wagging dogs.

I hope you will respect me as I do you and not agreeing and getting out of the way is part of that.  And for Goran, sorry for the false start keep working on the bugs and hopefully someone else can get it done.
Comment 21 Goran Mekić 2021-06-27 10:50:16 UTC
Created attachment 226076 [details]
pytest6and4.diff

Pytest4 added, pytest-relaxed changed. Anything else?
Comment 22 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-28 02:21:36 UTC
(In reply to Kevin Bowling from comment #20)

Let's have a chat offline, because (a) i want the same things you do (b) i do my best to be exactly not a gatekeeper, by making clear exactly what expectations are up front, which saves everyone the overhead of repeating the same things over and over, which unfortunately the project spends the vast majority of its time doing, which wastes everyone's time.

I have even gone so far as to document a maintainer policy, which explicitly enables and allows people to just get stuff done, with no "all these latches"

https://wiki.freebsd.org/KubilayKocak#MAINTAINER_POLICY

I wont apologise to anyone however for high QA expectations. This is the second major source of waste of our limited resources.

Again, let's have a pow-wow offline so we can get better things done together.
Comment 23 Kevin Bowling freebsd_committer freebsd_triage 2021-06-28 05:05:54 UTC
(In reply to Kubilay Kocak from comment #22)
I've apologized privately for trying to get involved here and tried to undo it.  I apologize again.
Comment 24 Goran Mekić 2021-07-02 11:51:39 UTC
Created attachment 226168 [details]
pytest-check.sh

This is the script I quickly assembled to help find the ports that depend on pytest and run "make test" on them. The way to use it:

mkdir pytest-check
cd pytest-check
# place script here
chmod +x pytest-check.sh
./pytest-check.sh list >list.txt
./pytest-check.sh test .

It will generate {success,failed}.log files and any subsequent call of the last command will remove failed.log and check success.log for ports to skip. I'm running it currently on 68268248585a32a80cb0e381cfd19a288e13d5f2 to check which ports are failing already.

I know this is no way to check for failed/success port tests reliably, but it's a start.
Comment 25 Goran Mekić 2021-07-03 14:20:23 UTC
Created attachment 226194 [details]
failed-main.log
Comment 26 Goran Mekić 2021-07-03 14:20:41 UTC
Created attachment 226195 [details]
success-main.log
Comment 27 Goran Mekić 2021-07-03 14:21:00 UTC
Created attachment 226196 [details]
failed-pytest.log
Comment 28 Goran Mekić 2021-07-03 14:21:21 UTC
Created attachment 226197 [details]
success-pytest.log
Comment 29 Goran Mekić 2021-07-03 14:29:54 UTC
The {failed,success}-main.log are ports that failed/succeeded on main git branch and analogous is for pytest branch which is at https://github.com/mekanix/freebsd-ports/tree/port/pytest. I would say success-pytest.log is a definite list of ports that should not be considered any more while other logs are here just for comparison. I am sure there are a lot of ports failing because same port with other name is already there (one example is py-sqlalchemy). I will start testing individual ports from failed-pytest.log, but it's a tedious job so I would really appreciate some joint effort.
Comment 30 Goran Mekić 2021-07-22 15:02:35 UTC
As I don't have the time to fix all the ports, may I suggest alternative solution? If I go through the list of ports that use pytest using pytest-check.sh and replace pytest with pytest4 that way nobody has to break as that is exact version like pytest in main branch, yet nobody has to wait as new ports will probably use newer pytest. I need a feedback on this before I continue, so please advise.
Comment 31 Goran Mekić 2021-07-23 19:07:25 UTC
Created attachment 226639 [details]
pytest4and6.diff

This patch moves py-pytest to py-pytest4, creates py-pytest version 6 and replaces devel/py-pytest with devel/py-pytest4 in ports Makefile.
Comment 32 Goran Mekić 2021-07-23 19:08:31 UTC
Created attachment 226640 [details]
pytest4and6.diff

Edit made to MOVED
Comment 33 Goran Mekić 2021-07-23 22:16:54 UTC
Created attachment 226643 [details]
pytest-check.sh

Based on https://github.com/mekanix/freebsd-ports/commit/c6437e2421811d27d6205f9ac0863b8bc115e97f

pytest-check.sh list >pytest-ports.txt
pytest-check.sh pin pytest-ports.txt
Comment 34 Goran Mekić 2021-07-23 22:17:44 UTC
Created attachment 226644 [details]
pytest.diff

Bump PORTREVISION
Comment 35 Goran Mekić 2021-07-28 09:43:16 UTC
So? What are we waiting now?
Comment 36 Goran Mekić 2021-09-26 08:41:48 UTC
Is there anything else to be done here? If there is, could anyone tell me and I'll do it. If there's not, can I get an explanation what are we waiting?
Comment 37 Kubilay Kocak freebsd_committer freebsd_triage 2021-09-26 23:15:11 UTC
(In reply to Goran Mekić from comment #36)

I need time/cycles to land it or to coordinate with someone else who can. It is at the top of my list and hasn't been forgotten.
Comment 38 Kubilay Kocak freebsd_committer freebsd_triage 2021-10-17 00:45:18 UTC
*** Bug 259018 has been marked as a duplicate of this bug. ***
Comment 39 Kubilay Kocak freebsd_committer freebsd_triage 2021-10-17 00:53:07 UTC
(In reply to Kubilay Kocak from comment #37)

Hi Goran, 

Does the patch switching pytest consumers to a new pytest4 port only switch those that don't support anything above 4 (explicitly don't work with, or pin to < 5), or were all just changed to pytest4 ?
Comment 40 Goran Mekić 2021-10-17 09:32:13 UTC
(In reply to Kubilay Kocak from comment #39)
All were changed.
Comment 41 Kubilay Kocak freebsd_committer freebsd_triage 2021-10-27 22:40:45 UTC
Reviewing rev deps for > 6|5 pytest dependency switch
Comment 42 Tobias C. Berner freebsd_committer freebsd_triage 2021-11-27 13:44:58 UTC
Moin moin 

Quick reminder, that the pkg-version went backwards from 6.x to 4.x -- so either please commit the update again soon, or add a PORTEPOCH.


mfg Tobias
Comment 43 Oleg Sidorkin 2021-11-27 14:35:47 UTC
Or can new pytest be commited as a separate port? Then other ports can be reviewed switched one by one.
Comment 44 Thierry Thomas freebsd_committer freebsd_triage 2022-02-26 13:51:48 UTC
Ping koobs: any progress on this PR?
Comment 45 Rene Ladan freebsd_committer freebsd_triage 2022-03-07 19:55:20 UTC
Maintainer reset.
Comment 46 Dmitry Marakasov freebsd_committer freebsd_triage 2022-03-18 17:11:26 UTC
I'm picking this up. Let's start with preserving pytest 4 and confirming that it works for some port which is not compatible with pytest 6.
Comment 47 commit-hook freebsd_committer freebsd_triage 2022-03-18 17:13:31 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=8802adcf7cd9e7a53d6995a7fea998908d3f81a2

commit 8802adcf7cd9e7a53d6995a7fea998908d3f81a2
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-18 16:24:20 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-03-18 17:11:38 +0000

    ftp/py-aioftp: switch to pytest4

    - With 3 preceeding commits, the tests are now able to run when
      py-pytest is at 6.x

    PR:             256624

 ftp/py-aioftp/Makefile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
Comment 48 commit-hook freebsd_committer freebsd_triage 2022-03-18 17:13:33 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=7db055349957ee7f79e5c2452430c71a3e86b6eb

commit 7db055349957ee7f79e5c2452430c71a3e86b6eb
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-18 16:24:00 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-03-18 17:11:37 +0000

    devel/py-pytest4: preserve 4.x version of devel/py-pytest

    - Preserve 4.x version of pytest for the ports which are not
      compatible with pytest 6.x. This is a prerequisite for updating
      pytest.

    PR:             256624
    Approved by:    maintainer timeout (python@, many months)

 devel/Makefile                                     |  1 +
 devel/py-pytest/Makefile                           |  2 +
 devel/py-pytest4/Makefile (new)                    | 61 ++++++++++++++++++++++
 devel/py-pytest4/distinfo (new)                    |  3 ++
 .../files/patch-testing-test_parseopt.py (new)     | 11 ++++
 devel/py-pytest4/pkg-descr (new)                   | 12 +++++
 6 files changed, 90 insertions(+)
Comment 49 commit-hook freebsd_committer freebsd_triage 2022-03-18 17:13:34 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=dc826ba1280fc889e95b8c12fc1aba6279c3d67d

commit dc826ba1280fc889e95b8c12fc1aba6279c3d67d
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-18 16:37:40 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-03-18 17:11:37 +0000

    devel/py-pytest4-{asyncio,cov}: preserve pytest4 counterparts of corresponding ports

    PR:             256624

 devel/Makefile                           |  2 ++
 devel/py-pytest-asyncio/Makefile         |  2 ++
 devel/py-pytest-cov/Makefile             |  2 ++
 devel/py-pytest4-asyncio/Makefile (new)  | 22 ++++++++++++++++++++++
 devel/py-pytest4-asyncio/distinfo (new)  |  3 +++
 devel/py-pytest4-asyncio/pkg-descr (new) |  5 +++++
 devel/py-pytest4-cov/Makefile (new)      | 25 +++++++++++++++++++++++++
 devel/py-pytest4-cov/distinfo (new)      |  3 +++
 devel/py-pytest4-cov/pkg-descr (new)     |  8 ++++++++
 9 files changed, 72 insertions(+)
Comment 50 Dmitry Marakasov freebsd_committer freebsd_triage 2022-03-23 14:44:55 UTC
I've built all ports which run/build depend on pytest, and there are no failures. We'll do an exp-run when the change is ready just to be sure.

Currently I'm running `make test` for all ports which test-depend on pytest with pytest4. Next I'll do the same with pytest7 (I'm testing 7.11 right now) and look at the new failures more closely.

The ongoing work is at https://github.com/AMDmi3/freebsd-ports/tree/pytest
Comment 51 Dmitry Marakasov freebsd_committer freebsd_triage 2022-03-28 12:50:18 UTC
So it turned out that most failures after pytest update are caused by outdated pytest-asyncio and are fixed by updating it as well.

4 remaining cases are fixed by switching them to pytest4, which I'm going to commit right away.

There's also databases/py-fakeredis which fails regardless of the update:
cd /work/usr/ports/databases/py-fakeredis/work-py38/fakeredis-1.7.1 && /usr/local/bin/python3.8 test_fakeredis.py
/usr/local/bin/python3.8: can't open file 'test_fakeredis.py': [Errno 2] No such file or directory
*** Error code 2

I'm switching it to pytest4 as well, as it limits the dependency versions from above.

Note that py-fakeredis and py-parso where broken by the revert of the previous pytest update attempt, as these limit pytest dependency version from above and are not taking EPOCH bump into account.
Comment 52 commit-hook freebsd_committer freebsd_triage 2022-03-28 12:51:51 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=32ec2042864d3a2de26c774739b133dd3a0a2022

commit 32ec2042864d3a2de26c774739b133dd3a0a2022
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-25 19:23:34 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-03-28 12:29:52 +0000

    math/py-levmar: switch to pytest4

    - Switch to pytest4 to avoid test failures after updating pytest to 7.x

    PR:             256624

 math/py-levmar/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 53 commit-hook freebsd_committer freebsd_triage 2022-03-28 12:51:52 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=1d244cdd602dc64184d79d9f9687f918d1fdfe35

commit 1d244cdd602dc64184d79d9f9687f918d1fdfe35
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-28 12:35:35 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-03-28 12:38:39 +0000

    textproc/py-parso: switch to pytest4

    - Switch to pytest4 to avoid test failures after updating pytest to 7.x
    - Don't forget epoch when limiting pytest version

    PR:             256624

 textproc/py-parso/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 54 commit-hook freebsd_committer freebsd_triage 2022-03-28 12:51:54 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=429facb51b6f90477d81e2c9b118a765eafb96a7

commit 429facb51b6f90477d81e2c9b118a765eafb96a7
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-25 19:24:07 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-03-28 12:31:10 +0000

    textproc/py-sphinx: switch to pytest4

    - Switch to pytest4 to avoid test failures after updating pytest to 7.x

    PR:             256624

 textproc/py-sphinx/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Comment 55 commit-hook freebsd_committer freebsd_triage 2022-03-28 12:51:55 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=f08db191ff75ca34e8295bb60899b6cfbbbedeb2

commit f08db191ff75ca34e8295bb60899b6cfbbbedeb2
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-25 19:23:50 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-03-28 12:31:03 +0000

    net/py-dugong: switch to pytest4

    - Switch to pytest4 to avoid test failures after updating pytest to 7.x

    PR:             256624

 net/py-dugong/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 56 commit-hook freebsd_committer freebsd_triage 2022-03-28 12:51:56 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=aec4847321880fd6c01576ca577050a9d114b282

commit aec4847321880fd6c01576ca577050a9d114b282
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-28 12:34:27 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-03-28 12:38:27 +0000

    databases/py-fakeredis: switch to pytest4

    - Switch to pytest4 to avoid test failures after updating pytest to 7.x
    - Don't forget epoch when limiting pytest version

    PR:             256624

 databases/py-fakeredis/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Comment 57 Dmitry Marakasov freebsd_committer freebsd_triage 2022-03-28 12:59:16 UTC
Created attachment 232778 [details]
pytest+pytest-asyncio patch

Here's an updated patch for pytest + pytest-asyncio, ready for review and testing. Note that previous pytest patch was missing some dependency changes (and some more were added in 7.1.1):

- atomicwwites: windows-only
- more-itertools: removed in 6.1.0
- six: no longer needed with dropped support for python 2
- wcwidth: removed in 6.0.0
Comment 58 Dmitry Marakasov freebsd_committer freebsd_triage 2022-03-28 13:43:38 UTC
Requesting an exp-run for the latest patch.
Comment 59 Jan Beich freebsd_committer freebsd_triage 2022-03-30 13:22:03 UTC
Comment on attachment 232778 [details]
pytest+pytest-asyncio patch

> +		${PYTHON_PKGNAMEPREFIX}tomli>=0:textproc/py-toml@${PY_FLAVOR} \

Found a typo:

$ poudriere bulk -j 130amd64 devel/py-pytest-asyncio
[...]
[00:00:03] Warning: devel/py-pytest dependency on textproc/py-toml@py38 has wrong PKGNAME of 'py38-tomli' but should be 'py38-toml'
[...]
Comment 60 Dmitry Marakasov freebsd_committer freebsd_triage 2022-04-04 19:56:42 UTC
Created attachment 232944 [details]
pytest+pytest-asyncio patch

> Found a typo:
>
> [00:00:03] Warning: devel/py-pytest dependency on textproc/py-toml@py38 has wrong 

Thanks, fixed!
Comment 61 Dmitry Marakasov freebsd_committer freebsd_triage 2022-04-05 11:42:05 UTC
Created attachment 232958 [details]
pytest+pytest-asyncio patch

Updated patch with pytest-asyncio fixes
Comment 62 commit-hook freebsd_committer freebsd_triage 2022-04-06 22:38:45 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=f5a4de53e143baa4e22486d37125ac272c1a0032

commit f5a4de53e143baa4e22486d37125ac272c1a0032
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-04-06 12:26:08 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-04-06 22:34:34 +0000

    devel/py-joblib: switch to pytest4

    The port is not compatible with recent pytest

    PR:             256624

 devel/py-joblib/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 63 commit-hook freebsd_committer freebsd_triage 2022-04-06 22:38:47 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=d01e186749d9e2bd3be15a4038fe62e751dfcf70

commit d01e186749d9e2bd3be15a4038fe62e751dfcf70
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-04-06 12:51:29 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-04-06 22:34:34 +0000

    devel/py-pytest-relaxed: switch to pytest4

    This plugin is not compatible with recent pytest

    PR:             256624

 devel/py-pytest-relaxed/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 64 commit-hook freebsd_committer freebsd_triage 2022-04-06 22:39:48 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=35a9ac41d04e7bee62e88a2fc13eb3028b9476c7

commit 35a9ac41d04e7bee62e88a2fc13eb3028b9476c7
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-04-06 22:04:55 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-04-06 22:38:34 +0000

    devel/py-pytest-{pep8,translations}: switch to pytest4

    These plugins are not compatible with recent pytest.

    Consumers (sysutils/iocage and www/py-flask-security) were switched
    to pytest4 as well. Note that py-flask-security tests were and still
    are broken due to unrelated problem - py-sqlalchemy12 and
    py-sqlalchemy14 conflict.

    PR:             256624

 devel/py-pytest-pep8/Makefile         | 4 ++--
 devel/py-pytest-translations/Makefile | 2 +-
 sysutils/iocage/Makefile              | 4 ++--
 www/py-flask-security/Makefile        | 8 ++++----
 4 files changed, 9 insertions(+), 9 deletions(-)
Comment 65 commit-hook freebsd_committer freebsd_triage 2022-04-06 22:39:50 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=e38e0e6063c33fc7cea17ca466902ebde17f772f

commit e38e0e6063c33fc7cea17ca466902ebde17f772f
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-04-06 17:20:10 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-04-06 22:38:33 +0000

    devel/py-pytest4-cache: preserve pytest4 counterpart of corresponding port

    PR:             256624

 devel/Makefile                                    |  1 +
 devel/py-pytest-cache/Makefile                    |  2 ++
 devel/py-pytest4-cache/Makefile (new)             | 29 ++++++++++++++++++
 devel/py-pytest4-cache/distinfo (new)             |  3 ++
 devel/py-pytest4-cache/files/patch-setup.py (new) | 36 +++++++++++++++++++++++
 devel/py-pytest4-cache/pkg-descr (new)            |  3 ++
 6 files changed, 74 insertions(+)
Comment 66 commit-hook freebsd_committer freebsd_triage 2022-04-06 22:39:51 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=21f418e9aa447a296d36f17aee6837d899fa91c8

commit 21f418e9aa447a296d36f17aee6837d899fa91c8
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-04-06 22:01:06 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-04-06 22:38:33 +0000

    devel/py-pytest4-flakes: preserve pytest4 counterpart of corresponding port

    PR:             256624

 devel/Makefile                          |  1 +
 devel/py-pytest-flakes/Makefile         |  2 ++
 devel/py-pytest4-flakes/Makefile (new)  | 23 +++++++++++++++++++++++
 devel/py-pytest4-flakes/distinfo (new)  |  3 +++
 devel/py-pytest4-flakes/pkg-descr (new) |  3 +++
 5 files changed, 32 insertions(+)
Comment 67 Dmitry Marakasov freebsd_committer freebsd_triage 2022-04-06 23:19:42 UTC
The current status:

- Well, I've had to write my own poudriere which is capable of bulk `make test` runs. The last bunch of fixes was possible solely because of it. Hopefully the tool will be useful on its own.

- Right now it's running another test of all *pytest* consumers.

- Notably, there are a lot of failures even with pytest4, which shows that tests are not getting enough attentions. I hope to fix that by releasing the mentioned utility and, for python ports specifically, by introducing pytest.mk https://reviews.freebsd.org/D34693?vs=on&id=104613#toc which makes it easier to silence failing tests and thus make passing tests more useful.

- Also note that the pytest update will fix about a dozen of ports on its own. Still, I'm striving not to break anything. I'll fix some more ports by hand, as failures are trivial in many cases, such as missing depends.
Comment 68 Dmitry Marakasov freebsd_committer freebsd_triage 2022-04-07 19:18:06 UTC
Another run has finished, it has revealed 3 failures which I've already fixed.

I've also committed a bunch of fixes for other test problems not directly related to pytest7 update (missing TEST_DEPENDS in most cases), however not all of these fix testing for corresponding ports completely.

It was also revealed that I use overly restrictive jail networking settings (ip4=disable ipv6=disable) in my porttester, which has led to a lot of failures in tests which try to use sockets (and it may happen that some pytest7 related failures were masked by these). Next run will allow networking limited with local addresses.
Comment 69 Dmitry Marakasov freebsd_committer freebsd_triage 2022-04-08 15:26:38 UTC
The last run shows no failures caused by pytest update and no apparent problems which may mask test results. Here's the list of ports which are FIXED by pytest update:

devel/py-fastnumbers failed → OK
devel/py-flaky failed → OK
devel/py-pytest failed → OK
devel/py-pytest-xdist failed → OK
math/py-cvxopt failed → OK
net-mgmt/py-napalm failed → OK
security/py-pynacl failed → OK
sysutils/py-ansible-base failed → OK
sysutils/py-ansible-core211 failed → OK
textproc/py-ocrmypdf failed → OK
textproc/py-pygments failed → OK

So I'm committing the pytest update now.
Comment 70 commit-hook freebsd_committer freebsd_triage 2022-04-08 15:29:36 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=c11512de05705ac6b2099551144d1b4f3d0b6e0e

commit c11512de05705ac6b2099551144d1b4f3d0b6e0e
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-03-25 19:14:57 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-04-08 15:27:02 +0000

    devel/py-pytest-asyncio: update to 0.18.2

    PR:             256624, 262884
    Approved by:    jbeich (maintainer)

 devel/py-pytest-asyncio/Makefile | 20 ++++++++++++++++----
 devel/py-pytest-asyncio/distinfo |  6 +++---
 2 files changed, 19 insertions(+), 7 deletions(-)
Comment 71 commit-hook freebsd_committer freebsd_triage 2022-04-08 15:29:38 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=2ada39de05b12da1d1083c40ad676c061b6e8a70

commit 2ada39de05b12da1d1083c40ad676c061b6e8a70
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2021-07-08 12:31:51 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-04-08 15:26:07 +0000

    devel/py-pytest: update to 7.1.1

    PR:             256624
    Approved by:    maintainer timeout (python@, 9 months)

 devel/py-pytest/Makefile                           | 27 +++++++++-------------
 devel/py-pytest/distinfo                           |  6 ++---
 .../files/patch-testing-test_parseopt.py (gone)    | 11 ---------
 3 files changed, 14 insertions(+), 30 deletions(-)
Comment 72 Dmitry Marakasov freebsd_committer freebsd_triage 2022-04-08 15:32:42 UTC
Thanks portmgr for no reaction in 2 weeks, thankfully I've managed on my own.
Comment 73 Kubilay Kocak freebsd_committer freebsd_triage 2022-04-09 01:04:35 UTC
Thanks for taking care of this Dmitry. Would love to see/use your poudriere make test bits.
Comment 74 Dmitry Marakasov freebsd_committer freebsd_triage 2022-04-13 18:48:09 UTC
(In reply to Kubilay Kocak from comment #73)

> Would love to see/use your poudriere make test bits.

See ports-mgmt/reprise (needs FLAVOR=py310 specified to build, because of another poudriere problem), also announced on ports-developers@