Bug 256624 - devel/py-pytest: Update to 6.2.4
Summary: devel/py-pytest: Update to 6.2.4
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Kubilay Kocak
URL: https://docs.pytest.org/en/stable/cha...
Keywords:
Depends on:
Blocks: 256745
  Show dependency treegraph
 
Reported: 2021-06-15 18:13 UTC by Goran Mekić
Modified: 2021-07-28 09:43 UTC (History)
5 users (show)

See Also:


Attachments
pytest.diff (3.46 KB, patch)
2021-06-15 18:13 UTC, Goran Mekić
no flags Details | Diff
pytest6and4.diff (7.53 KB, patch)
2021-06-27 10:50 UTC, Goran Mekić
no flags Details | Diff
pytest-check.sh (1.06 KB, application/x-shellscript)
2021-07-02 11:51 UTC, Goran Mekić
no flags Details
failed-main.log (8.13 KB, text/plain)
2021-07-03 14:20 UTC, Goran Mekić
no flags Details
success-main.log (5.01 KB, text/plain)
2021-07-03 14:20 UTC, Goran Mekić
no flags Details
failed-pytest.log (8.28 KB, text/plain)
2021-07-03 14:21 UTC, Goran Mekić
no flags Details
success-pytest.log (4.87 KB, text/plain)
2021-07-03 14:21 UTC, Goran Mekić
no flags Details
pytest4and6.diff (252.96 KB, patch)
2021-07-23 19:07 UTC, Goran Mekić
no flags Details | Diff
pytest4and6.diff (253.52 KB, patch)
2021-07-23 19:08 UTC, Goran Mekić
no flags Details | Diff
pytest-check.sh (1.28 KB, application/x-shellscript)
2021-07-23 22:16 UTC, Goran Mekić
no flags Details
pytest.diff (320.68 KB, patch)
2021-07-23 22:17 UTC, Goran Mekić
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Goran Mekić 2021-06-15 18:13:07 UTC
Created attachment 225833 [details]
pytest.diff

QA:

  * portlint: OK (looks fine.)
  * testport: OK (poudriere: 13.0, amd64 tested)
  * maketest: OK (3035 passed, 10 skipped, 10 xfailed)
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-16 00:32:27 UTC
Needs reverse dependents test and QA confirmation, as dependent consumers are known to regularly not support pytest major version updates for a while after release. May require a pytest4 port
Comment 2 Goran Mekić 2021-06-16 08:01:18 UTC
As number of ports using pytest is huge, how do we continue? I don't have a server that can run all those tests.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-17 00:11:53 UTC
Thought not completely useful, the consumer (*_depends) set is only <50 ports, almost all very lightweight.

A grep of *_DEPENDS looking for '<' lines is also a decent first pass. This wont include ports that don't correctly or completely declare their dependency <version-specs> however.
Comment 4 Goran Mekić 2021-06-17 11:02:18 UTC
If you refer to {RUN,BUILD}_DEPENDS, yes, you're probably right, but TEST_DEPENDS will probably yield a much larger port collection dependent on pytest. I mean, https://www.freshports.org/devel/py-pytest/ reveals which ports depend on pytest, but I think there's no TEST_DEPENDS in it. If freshports list is only I should test, no problem!
Comment 5 Thierry Thomas freebsd_committer 2021-06-17 20:10:25 UTC
Such a command line will list all ports using devel/py-pytest in TEST_DEPENDS:

$ find /usr/ports -type d ! -name files -print -exec make -V TEST_DEPENDS -C {} \; 2>/dev/null | grep -B 1 devel/py-pytest

Remark: there are several ways to achieve the same result, do not hesitate to find a better one!
Comment 6 Goran Mekić 2021-06-20 19:01:22 UTC
Tested with "make -DBATCH test" and following ports reported no problems:
cad/netgen
devel/py-pytest-factoryboy
www/py-textile
devel/py-pytest-aiohttp
devel/py-pytest-asyncio
devel/py-pytest-azurepipelines
devel/py-pytest-benchmark
devel/py-pytest-black
devel/py-pytest-cov
devel/py-pytest-datadir
devel/py-pytest-django
devel/py-pytest-env
devel/py-pytest-factoryboy
devel/py-pytest-fixture-config
devel/py-pytest-flake8
devel/py-pytest-flakes
devel/py-pytest-freezegun
devel/py-pytest-lazy-fixture
devel/py-pytest-localserver
devel/py-pytest-mypy
devel/py-pytest-pycodestyle
devel/py-pytest-randomly
devel/py-pytest-regressions
devel/py-pytest-rerunfailures
devel/py-pytest-sugar
devel/py-pytest-tornado
devel/py-pytest-translations
devel/py-pytest-trio
devel/py-pytest-twisted
devel/py-pytest-virtualenv
devel/py-testinfra
devel/py-ttictoc
finance/fava
science/py-MDAnalysisTests

The ports that fail even on main branch:
devel/py-pytest-cache
devel/py-pytest-flask
devel/py-pytest-forked
devel/py-pytest-mock
devel/py-pytest-mutagen
devel/py-pytest-relaxed
devel/py-pytest-shutil
devel/py-pytest-timeout
devel/py-pytest-verbose-parametrize

Ports that fail with upgraded pytest, but after bumping their version to match PyPi succeed:
devel/py-pytest-xdist
devel/py-pytest-xprocess

Koobs, xdist is yours, so can you please update that one? I will contact maintainer of xprocess port. Thank you!
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-21 01:54:13 UTC
(In reply to Goran Mekić from comment #6)

Thank you for running this QAS Goran

Can you detail the failure modes experienced? Such as the "The ports that fail even on main branch:" group?
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-21 01:55:01 UTC
For any of the failure cases that are ports bugs (incorrectly incompletely specified version specs of *_DEPENDS), any changes are 

Approved by: portmgr (blanket: bugfix)

And wont require issues to be created or maintainer approval
Comment 9 Goran Mekić 2021-06-21 10:02:10 UTC
(In reply to Kubilay Kocak from comment #7)
I have no idea what the errors were, I didn't write them down.
Comment 10 Goran Mekić 2021-06-21 11:28:25 UTC
Updating pytest-flask also makes it pass the tests, but all other ports are either on latest version or fail for some reason (I'm suspecting some deeper dependency needs update). Failing ports:

devel/py-pytest-cache
devel/py-pytest-forked
devel/py-pytest-mock
devel/py-pytest-mutagen
devel/py-pytest-relaxed
devel/py-pytest-shutil
devel/py-pytest-timeout
devel/py-pytest-verbose-parametrize
Comment 11 Goran Mekić 2021-06-22 10:52:14 UTC
What's our next move?
Comment 12 commit-hook freebsd_committer 2021-06-23 20:35:49 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=0c6ce3c9c26d878acf943d53fb2a01ba602b71c2

commit 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2
Author:     Goran Meki? <meka@tilda.center>
AuthorDate: 2021-06-23 20:30:35 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-06-23 20:34:47 +0000

    devel/py-pytest: Update to 6.2.4

    PR:             256624

 devel/py-pytest/Makefile                                    | 12 ++++--------
 devel/py-pytest/distinfo                                    |  6 +++---
 devel/py-pytest/files/patch-testing-test_parseopt.py (gone) | 11 -----------
 3 files changed, 7 insertions(+), 22 deletions(-)
Comment 13 commit-hook freebsd_committer 2021-06-23 20:38:51 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=d07f92b9bf00821e372f003d698525cd1ae19dc2

commit d07f92b9bf00821e372f003d698525cd1ae19dc2
Author:     Kevin Bowling <kbowling@FreeBSD.org>
AuthorDate: 2021-06-23 20:37:23 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-06-23 20:37:23 +0000

    devel/py-pytest-xdist: Update to 2.3.0

    PR:             256624
    Approved by:    maintainer (koobs)

 devel/py-pytest-xdist/Makefile | 2 +-
 devel/py-pytest-xdist/distinfo | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)
Comment 14 Kevin Bowling freebsd_committer 2021-06-23 20:39:56 UTC
The failures are now a subset of existing failures, closing.  If you make any progress on cleaning them up please send in new PRs.  Thanks for your contributions!
Comment 15 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-26 01:59:27 UTC
@Kevin This was committed without approval or reference, and has broken at least one port (at build time), and likely more at runtime. "failures are now a subset of existing failures, closing." is an insufficient rationale for the committing of a change that has not completed or confirmed QA.

[130amd64-default-job-14] Extracting py38-pytest-6.2.4: .......... done
===>   py38-pytest-relaxed-1.1.5_1 depends on package: py38-pytest>=3<5 - not found
*** Error code 1

Can you please resolve the fallout, referencing this issue on commit, thanks
Comment 16 Goran Mekić 2021-06-26 09:46:44 UTC
For the moment let's just imagine py-pytest-relaxed is fixed. I asked already and I have to ask again "What's our next move?" I'm not trying to rush anything, but nothing will happen if person who's "lost" doesn't get guidance.
Comment 17 Kevin Bowling freebsd_committer 2021-06-26 21:54:10 UTC
(In reply to Kubilay Kocak from comment #15)
Sure, it's somewhat intricate so I want to confirm with you to revert the following git hashes: 
* 5989f395a01e1afc9b1b3a0c668d3cf48226f0c3
* d07f92b9bf00821e372f003d698525cd1ae19dc2
* b15351babf6846ec3cdc902203874e55413ee08a
* 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2

Note that if I do this I wont work on it again because I disagree, loosely and respectfully.

(In reply to Goran Mekić from comment #16)
koobs I'd appreciate if you can provide guidance on Goran's question.  In my humble opinion the better move here is to mark your port as broken because the upstream isn't lively and nothing depends on it.  I admit I didn't see that in my testing and should have been more thorough so I'm willing to revert it regardless.
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-27 01:38:46 UTC
(In reply to Kevin Bowling from comment #17)

Please do whatever you believe is now necessary to correctly and completely resolve the issue.

On the question of responding to comment 11, I believe it was clear:

 - comment 1 (complete consumer QA against <version-spec>'s they support, which involves fixing any that are incorrect or not declared)

 - comment 1 "May require a pytest4 port" (dependent on outcome of the first qa)

 - comment 7 (understanding of failure modes precluding update)

On the question of marking pytest-relaxed broken, it is/was not broken with the version of pytest prior to the update from this issue. this leads to and is related to comment 1 (may require a pytest4) and the reverse dependents QA request.

I don't understand the context of "I wont work on it again because I disagree". If that's with respect to "unapproved and unreferenced" (comment 15), the commit message contained:

    devel/py-pytest: Update to 6.2.4

    PR:             256624

Ignoring the request for QA, the request to understand failure modes for consumer ports, and the knowledge that there were remaining and unknown issues/errors in consumers (comment 15), on what basis was the commit made?

Not ignoring the above, on what basis was the commit made?

Ultimately these are now rhetorical questions, and we can discuss them offline if you're available and interested in doing so.

Let's just get this resolved properly for our users. Over to you Kevin.
Comment 19 commit-hook freebsd_committer 2021-06-27 05:12:31 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=46a663d44cb038c4b5fcca21f1c0151db0e8302e

commit 46a663d44cb038c4b5fcca21f1c0151db0e8302e
Author:     Kevin Bowling <kbowling@FreeBSD.org>
AuthorDate: 2021-06-27 05:07:31 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-06-27 05:08:40 +0000

    devel/py-pytest-flask: revert 6.2.4 update

    Revert "devel/py-pytest: Update to 6.2.4"

    This reverts commit 0c6ce3c9c26d878acf943d53fb2a01ba602b71c2.

    Revert "devel/py-pytest-translations: Update to 3.1.0"

    This reverts commit 5989f395a01e1afc9b1b3a0c668d3cf48226f0c3.

    Revert "devel/py-pytest-xdist: Update to 2.3.0"

    This reverts commit d07f92b9bf00821e372f003d698525cd1ae19dc2.

    Revert "devel/py-pytest-flask: Update to 1.2.0"

    This reverts commit b15351babf6846ec3cdc902203874e55413ee08a.

    PR:             256624
    Requested by:   koobs

 devel/py-pytest-flask/Makefile                            |  2 +-
 devel/py-pytest-flask/distinfo                            |  6 +++---
 devel/py-pytest-translations/Makefile                     | 15 +++++++++++----
 devel/py-pytest-translations/distinfo                     |  6 +++---
 devel/py-pytest-xdist/Makefile                            |  2 +-
 devel/py-pytest-xdist/distinfo                            |  6 +++---
 devel/py-pytest/Makefile                                  | 12 ++++++++----
 devel/py-pytest/distinfo                                  |  6 +++---
 .../py-pytest/files/patch-testing-test_parseopt.py (new)  | 11 +++++++++++
 9 files changed, 44 insertions(+), 22 deletions(-)
Comment 20 Kevin Bowling freebsd_committer 2021-06-27 05:29:25 UTC
(In reply to Kubilay Kocak from comment #18)
Thanks, since you haven't expressed technical issue with my revert plan above I will implement the revert of those commits.  All I said is I don't want to work on this that way, it's ok for me to choose how I volunteer :)

I have a political platform:  I want main to be a little more Fedora a little less Enterprise Linux and this looked ready to go:  Goran has methodically been testing and fixing the issues leading to one leaf port that is known to be newly broken while fixing many other issues.  I'm unconvinced it's not an improvement, but you're right I made a procedural error as a newer ports committer and although I executed many tests I did not do a thorough enough job to identify this failure ahead of time to discuss it. I also misunderstood where you were leading and what approvals remained.  My experience thus far has been that it's easier to work on src over ports because of all these latches which is unexpected.  It's open to another contributor because I would like to focus on things with less gatekeepers and put my time into making faster improvements and attracting new talent and integrating their work.  I'm not saying anyone's wrong, just that I don't want to do it the way you want.  My position is based in part by dialog in https://github.com/bitprophet/pytest-relaxed/issues/12, where you can see the methodologies of other maintainers who are dealing with tails wagging dogs.

I hope you will respect me as I do you and not agreeing and getting out of the way is part of that.  And for Goran, sorry for the false start keep working on the bugs and hopefully someone else can get it done.
Comment 21 Goran Mekić 2021-06-27 10:50:16 UTC
Created attachment 226076 [details]
pytest6and4.diff

Pytest4 added, pytest-relaxed changed. Anything else?
Comment 22 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-28 02:21:36 UTC
(In reply to Kevin Bowling from comment #20)

Let's have a chat offline, because (a) i want the same things you do (b) i do my best to be exactly not a gatekeeper, by making clear exactly what expectations are up front, which saves everyone the overhead of repeating the same things over and over, which unfortunately the project spends the vast majority of its time doing, which wastes everyone's time.

I have even gone so far as to document a maintainer policy, which explicitly enables and allows people to just get stuff done, with no "all these latches"

https://wiki.freebsd.org/KubilayKocak#MAINTAINER_POLICY

I wont apologise to anyone however for high QA expectations. This is the second major source of waste of our limited resources.

Again, let's have a pow-wow offline so we can get better things done together.
Comment 23 Kevin Bowling freebsd_committer 2021-06-28 05:05:54 UTC
(In reply to Kubilay Kocak from comment #22)
I've apologized privately for trying to get involved here and tried to undo it.  I apologize again.
Comment 24 Goran Mekić 2021-07-02 11:51:39 UTC
Created attachment 226168 [details]
pytest-check.sh

This is the script I quickly assembled to help find the ports that depend on pytest and run "make test" on them. The way to use it:

mkdir pytest-check
cd pytest-check
# place script here
chmod +x pytest-check.sh
./pytest-check.sh list >list.txt
./pytest-check.sh test .

It will generate {success,failed}.log files and any subsequent call of the last command will remove failed.log and check success.log for ports to skip. I'm running it currently on 68268248585a32a80cb0e381cfd19a288e13d5f2 to check which ports are failing already.

I know this is no way to check for failed/success port tests reliably, but it's a start.
Comment 25 Goran Mekić 2021-07-03 14:20:23 UTC
Created attachment 226194 [details]
failed-main.log
Comment 26 Goran Mekić 2021-07-03 14:20:41 UTC
Created attachment 226195 [details]
success-main.log
Comment 27 Goran Mekić 2021-07-03 14:21:00 UTC
Created attachment 226196 [details]
failed-pytest.log
Comment 28 Goran Mekić 2021-07-03 14:21:21 UTC
Created attachment 226197 [details]
success-pytest.log
Comment 29 Goran Mekić 2021-07-03 14:29:54 UTC
The {failed,success}-main.log are ports that failed/succeeded on main git branch and analogous is for pytest branch which is at https://github.com/mekanix/freebsd-ports/tree/port/pytest. I would say success-pytest.log is a definite list of ports that should not be considered any more while other logs are here just for comparison. I am sure there are a lot of ports failing because same port with other name is already there (one example is py-sqlalchemy). I will start testing individual ports from failed-pytest.log, but it's a tedious job so I would really appreciate some joint effort.
Comment 30 Goran Mekić 2021-07-22 15:02:35 UTC
As I don't have the time to fix all the ports, may I suggest alternative solution? If I go through the list of ports that use pytest using pytest-check.sh and replace pytest with pytest4 that way nobody has to break as that is exact version like pytest in main branch, yet nobody has to wait as new ports will probably use newer pytest. I need a feedback on this before I continue, so please advise.
Comment 31 Goran Mekić 2021-07-23 19:07:25 UTC
Created attachment 226639 [details]
pytest4and6.diff

This patch moves py-pytest to py-pytest4, creates py-pytest version 6 and replaces devel/py-pytest with devel/py-pytest4 in ports Makefile.
Comment 32 Goran Mekić 2021-07-23 19:08:31 UTC
Created attachment 226640 [details]
pytest4and6.diff

Edit made to MOVED
Comment 33 Goran Mekić 2021-07-23 22:16:54 UTC
Created attachment 226643 [details]
pytest-check.sh

Based on https://github.com/mekanix/freebsd-ports/commit/c6437e2421811d27d6205f9ac0863b8bc115e97f

pytest-check.sh list >pytest-ports.txt
pytest-check.sh pin pytest-ports.txt
Comment 34 Goran Mekić 2021-07-23 22:17:44 UTC
Created attachment 226644 [details]
pytest.diff

Bump PORTREVISION
Comment 35 Goran Mekić 2021-07-28 09:43:16 UTC
So? What are we waiting now?