Created attachment 225176 [details] updates py-urlwatch to v2.23, updating TEST_DEPENDS along the way. Hi Kai, find attached an update of py-urlwatch to 2.23, with minor tweaking for tests. We may also (this is not included) add a pkg-message that points the user to either py-pip based install instructions or optional dependencies for further filters. The attached patch adds TEST_DEPENDS that suffice for "make test". Have tested this on FreeBSD 13.0-RELEASE amd64 including Tesseract.
test builds succeeded on all Python versions 3.6 3.7 3.8 3.9 on 13.0-RELEASE amd64, and also on default Python 3.8 on 11.4-RELEASE i386. Light run-time testing on 12.2-RELEASE amd64 with Python 3.8 as dogfooding successful, too.
Created attachment 225189 [details] update urlwatch to v2.23, add DOCS, fix TEST_DEPENDS new update to also build & install documentation unless DOCS option disabled
Comment on attachment 225189 [details] update urlwatch to v2.23, add DOCS, fix TEST_DEPENDS Many thanks for the patch, Matthias! It looks good to me and I'd suggest to add > USE_LOCALE=en_US.UTF-8 and change the command of the "do-test" target to: > cd ${WRKSRC} && ${SET_ENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest -v -rs to make the testsuite work with Python 3.6. Feel free to take maintainership (and of databases/py-minidb, which is also required by www/py-urlwatch) if you like. As a side note, the testsuite revealed two issues in ports you maintain: graphics/py-pytesseract: ~~~~~~~~~~~~~~~~~~~~~~~~ - Needs graphics/py-pillow for RUN_DEPENDS and USE_PYTHON=concurrent as well, because a script is installed outside of Python's site-lib directory. textproc/py-jq: ~~~~~~~~~~~~~~~ - Has staging issues with py36 and py37. Using "s/jq.*.so/jq*.so/" should remedy the issue.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=020dff9f9564604959680378f0d1eed19ff9be0a commit 020dff9f9564604959680378f0d1eed19ff9be0a Author: Matthias Andree <mandree@FreeBSD.org> AuthorDate: 2021-05-22 11:01:55 +0000 Commit: Matthias Andree <mandree@FreeBSD.org> CommitDate: 2021-06-04 21:50:58 +0000 www/py-urlwatch: update to v2.23, add DOCS Add DOCS option (uses py-sphinx and GNU make) to build/install html and text documentation. Ensure build works for non-default Python flavors. Update TEST_DEPENDS to list additional self-test requisites. Changelog: https://github.com/thp/urlwatch/blob/2.23/CHANGELOG.md#223----2021-04-10 PR: 256085 Approved by: kai@FreeBSD.org www/py-urlwatch/Makefile | 29 +++++++++++++++++++++++++---- www/py-urlwatch/distinfo | 6 +++--- 2 files changed, 28 insertions(+), 7 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=2b87ac22b9c9400755350a9f05f34a380d73db9d commit 2b87ac22b9c9400755350a9f05f34a380d73db9d Author: Matthias Andree <mandree@FreeBSD.org> AuthorDate: 2021-06-04 21:49:49 +0000 Commit: Matthias Andree <mandree@FreeBSD.org> CommitDate: 2021-06-04 21:50:58 +0000 graphics/py-pytesseract: require py-pillow... ...and USE_PYTHON=concurrent. Reported by: kai@ PR: 256085 graphics/py-pytesseract/Makefile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=d15f0e60444d2e0f5eb527e9a5965b977f0c38d1 commit d15f0e60444d2e0f5eb527e9a5965b977f0c38d1 Author: Matthias Andree <mandree@FreeBSD.org> AuthorDate: 2021-06-04 21:45:15 +0000 Commit: Matthias Andree <mandree@FreeBSD.org> CommitDate: 2021-06-04 21:50:58 +0000 textproc/py-jq: package properly with py <= 3.7 Reported by: kai@ PR: 256085 textproc/py-jq/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Hi Kai, thanks for the review and the offer to take maintainership. I am not too familiar/experienced with porting Python code so far and have not too much experience with version portability, so I took your advice. I am wondering about - what your proposed USE_LOCALE achieves and why it was needed (I took the advice) - how to properly go about testing (for instance, what I missed in my testing so I did not notice the py-pillow dependency for py-pytesseract). I have enabled my poudriere to all Python flavors and could reproduce the py-jq staging issues with that, but the py-pillow requisite somehow eludes me. How did you figure? What did I need to try to see the missing py-pillow requisite myself? How do you test Python ports?
(In reply to Matthias Andree from comment #7) Hi Matthias, > - what your proposed USE_LOCALE achieves and why it was needed (I took the advice) If "USE_LOCALE" is missing, the whole test suite fails (in a non-UTF-8 environment) with Python 3.6: > /usr/local/lib/python3.6/encodings/ascii.py:26: in decode > return codecs.ascii_decode(input, self.errors)[0] > E UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4621: ordinal not in range(128) This is because Python 3.6 doesn't enable by default the UTF-8 mode when the locale is C. This was changed in Python 3.7+. A (IMHO) technically more correct solution would be to add a conditional like ".if ${PYTHON_REL} < 30700" and place then "USE_LOCALE" there or better, fix the issue in the Python package. But using "USE_LOCALE" alone doesn't cause much overhead so I mentioned it instead to keep the Makefile simple. > I have enabled my poudriere to all Python flavors and could reproduce the py-jq staging issues with that, but the py-pillow requisite somehow eludes me. > > How did you figure? What did I need to try to see the missing py-pillow requisite myself? I stumbled across it when I ran the the testsuite of www/py-urlwatch. The test "test_url[https://example.net/ocr-test.png-job27]" failed with following error: > if pytesseract is None: > > raise ImportError('Please install pytesseract') > E ImportError: Please install pytesseract This error then surprised me a little, because graphics/py-pytesseract was already in TEST_DEPENDS and was also installed. So I took a look at graphics/py-pytesseract and tried the script ${PREFIX}/bin/pytesseract which failed at runtime: > File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve > raise DistributionNotFound(req, requirers) > pkg_resources.DistributionNotFound: The 'Pillow' distribution was not found and is required by pytesseract The same error also occurred if one tried to run "${PYTHON_CMD} -m pytesseract". The graphics/py-pytesseract port has following lines in its "setup.py" where graphics/py-pillow is defined as a dependency: > INSTALL_REQUIRES = ['Pillow'] > [...] > install_requires=INSTALL_REQUIRES, The above is also an good example why QA'ing/testing of major Python libraries (e.g. net/py-urllib, security/py-cryptography, etc.) takes so much time. This is because some ports (like graphics/py-pytesseract) check the dependencies only at runtime and not during buildtime (e.g. a bulk-run won't reveal any issues). > How do you test Python ports? For the most ports I do following steps: - Check setup.{py,cfg} and pyproject.toml for required dependencies In a clean poudriere environment, e.g. via "testport -i", for each supported Python version: - Run testsuite - Try to import the Python package, e.g. via "${PYTHON_CMD} -m NewPackage - Runtime test scripts (if supplied) If the Python port is required by other ports, I do following steps as well: - Do a bulk-run against all consumers - Check the setup.{py,cfg} and pyproject.toml of the consumers
Thanks for explaining your port testing workflows!