Bug 256085 - www/py-urlwatch: Update to 2.23
Summary: www/py-urlwatch: Update to 2.23
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Matthias Andree
URL: https://github.com/thp/urlwatch/blob/...
Keywords:
Depends on:
Blocks:
 
Reported: 2021-05-22 14:01 UTC by Matthias Andree
Modified: 2021-06-19 23:16 UTC (History)
2 users (show)

See Also:
kai: maintainer-feedback+


Attachments
updates py-urlwatch to v2.23, updating TEST_DEPENDS along the way. (2.11 KB, patch)
2021-05-22 14:01 UTC, Matthias Andree
no flags Details | Diff
update urlwatch to v2.23, add DOCS, fix TEST_DEPENDS (3.09 KB, patch)
2021-05-23 10:03 UTC, Matthias Andree
kai: maintainer-approval+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Andree freebsd_committer freebsd_triage 2021-05-22 14:01:47 UTC
Created attachment 225176 [details]
updates py-urlwatch to v2.23, updating TEST_DEPENDS along the way.

Hi Kai,

find attached an update of py-urlwatch to 2.23, with minor tweaking for tests.

We may also (this is not included) add a pkg-message that points the user to either py-pip based install instructions or optional dependencies for further filters.

The attached patch adds TEST_DEPENDS that suffice for "make test".

Have tested this on FreeBSD 13.0-RELEASE amd64 including Tesseract.
Comment 1 Matthias Andree freebsd_committer freebsd_triage 2021-05-22 14:50:12 UTC
test builds succeeded on 
all Python versions 3.6 3.7 3.8 3.9 on 13.0-RELEASE amd64, and also on 
default Python 3.8 on 11.4-RELEASE i386. 

Light run-time testing on 12.2-RELEASE amd64 with Python 3.8 as dogfooding successful, too.
Comment 2 Matthias Andree freebsd_committer freebsd_triage 2021-05-23 10:03:22 UTC
Created attachment 225189 [details]
update urlwatch to v2.23, add DOCS, fix TEST_DEPENDS

new update to also build & install documentation unless DOCS option disabled
Comment 3 Kai Knoblich freebsd_committer freebsd_triage 2021-06-04 20:04:39 UTC
Comment on attachment 225189 [details]
update urlwatch to v2.23, add DOCS, fix TEST_DEPENDS

Many thanks for the patch, Matthias! It looks good to me and I'd suggest to add

> USE_LOCALE=en_US.UTF-8

and change the command of the "do-test" target to:

> cd ${WRKSRC} && ${SET_ENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest -v -rs

to make the testsuite work with Python 3.6.

Feel free to take maintainership (and of databases/py-minidb, which is also required by www/py-urlwatch) if you like.

As a side note, the testsuite revealed two issues in ports you maintain:

graphics/py-pytesseract:
~~~~~~~~~~~~~~~~~~~~~~~~
- Needs graphics/py-pillow for RUN_DEPENDS and USE_PYTHON=concurrent as well, because a script is installed outside of Python's site-lib directory.

textproc/py-jq:
~~~~~~~~~~~~~~~
- Has staging issues with py36 and py37. Using "s/jq.*.so/jq*.so/" should remedy the issue.
Comment 4 commit-hook freebsd_committer freebsd_triage 2021-06-04 21:52:59 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=020dff9f9564604959680378f0d1eed19ff9be0a

commit 020dff9f9564604959680378f0d1eed19ff9be0a
Author:     Matthias Andree <mandree@FreeBSD.org>
AuthorDate: 2021-05-22 11:01:55 +0000
Commit:     Matthias Andree <mandree@FreeBSD.org>
CommitDate: 2021-06-04 21:50:58 +0000

    www/py-urlwatch: update to v2.23, add DOCS

    Add DOCS option (uses py-sphinx and GNU make) to build/install
    html and text documentation.

    Ensure build works for non-default Python flavors.

    Update TEST_DEPENDS to list additional self-test requisites.

    Changelog: https://github.com/thp/urlwatch/blob/2.23/CHANGELOG.md#223----2021-04-10

    PR:             256085

    Approved by:    kai@FreeBSD.org

 www/py-urlwatch/Makefile | 29 +++++++++++++++++++++++++----
 www/py-urlwatch/distinfo |  6 +++---
 2 files changed, 28 insertions(+), 7 deletions(-)
Comment 5 commit-hook freebsd_committer freebsd_triage 2021-06-04 21:53:00 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=2b87ac22b9c9400755350a9f05f34a380d73db9d

commit 2b87ac22b9c9400755350a9f05f34a380d73db9d
Author:     Matthias Andree <mandree@FreeBSD.org>
AuthorDate: 2021-06-04 21:49:49 +0000
Commit:     Matthias Andree <mandree@FreeBSD.org>
CommitDate: 2021-06-04 21:50:58 +0000

    graphics/py-pytesseract: require py-pillow...

    ...and USE_PYTHON=concurrent.

    Reported by:    kai@
    PR:             256085

 graphics/py-pytesseract/Makefile | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Comment 6 commit-hook freebsd_committer freebsd_triage 2021-06-04 21:53:01 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=d15f0e60444d2e0f5eb527e9a5965b977f0c38d1

commit d15f0e60444d2e0f5eb527e9a5965b977f0c38d1
Author:     Matthias Andree <mandree@FreeBSD.org>
AuthorDate: 2021-06-04 21:45:15 +0000
Commit:     Matthias Andree <mandree@FreeBSD.org>
CommitDate: 2021-06-04 21:50:58 +0000

    textproc/py-jq: package properly with py <= 3.7

    Reported by:    kai@
    PR:             256085

 textproc/py-jq/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 7 Matthias Andree freebsd_committer freebsd_triage 2021-06-05 07:13:58 UTC
Hi Kai,

thanks for the review and the offer to take maintainership. 

I am not too familiar/experienced with porting Python code so far and have not too much experience with version portability, so I took your advice. I am wondering about 

- what your proposed USE_LOCALE achieves and why it was needed (I took the advice)

- how to properly go about testing (for instance, what I missed in my testing so I did not notice the py-pillow dependency for py-pytesseract).  

I have enabled my poudriere to all Python flavors and could reproduce the py-jq staging issues with that, but the py-pillow requisite somehow eludes me. 

How did you figure?  What did I need to try to see the missing py-pillow requisite myself? How do you test Python ports?
Comment 8 Kai Knoblich freebsd_committer freebsd_triage 2021-06-05 19:38:18 UTC
(In reply to Matthias Andree from comment #7)

Hi Matthias,

> - what your proposed USE_LOCALE achieves and why it was needed (I took the advice)

If "USE_LOCALE" is missing, the whole test suite fails (in a non-UTF-8 environment) with Python 3.6:

> /usr/local/lib/python3.6/encodings/ascii.py:26: in decode
>     return codecs.ascii_decode(input, self.errors)[0]
> E   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4621: ordinal not in range(128)

This is because Python 3.6 doesn't enable by default the UTF-8 mode when the locale is C. This was changed in Python 3.7+.

A (IMHO) technically more correct solution would be to add a conditional like ".if ${PYTHON_REL} < 30700" and place then "USE_LOCALE" there or better, fix the issue in the Python package.

But using "USE_LOCALE" alone doesn't cause much overhead so I mentioned it instead to keep the Makefile simple.

> I have enabled my poudriere to all Python flavors and could reproduce the py-jq staging issues with that, but the py-pillow requisite somehow eludes me. 
>
> How did you figure?  What did I need to try to see the missing py-pillow requisite myself?

I stumbled across it when I ran the the testsuite of www/py-urlwatch. The test "test_url[https://example.net/ocr-test.png-job27]" failed with following error:

>         if pytesseract is None:
> >           raise ImportError('Please install pytesseract')
> E           ImportError: Please install pytesseract

This error then surprised me a little, because graphics/py-pytesseract was already in TEST_DEPENDS and was also installed.  So I took a look at graphics/py-pytesseract and tried the script ${PREFIX}/bin/pytesseract which failed at runtime:

>   File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
>     raise DistributionNotFound(req, requirers)
> pkg_resources.DistributionNotFound: The 'Pillow' distribution was not found and is required by pytesseract

The same error also occurred if one tried to run "${PYTHON_CMD} -m pytesseract".  

The graphics/py-pytesseract port has following lines in its "setup.py" where graphics/py-pillow is defined as a dependency:

> INSTALL_REQUIRES = ['Pillow']
> [...]
> install_requires=INSTALL_REQUIRES,

The above is also an good example why QA'ing/testing of major Python libraries (e.g. net/py-urllib, security/py-cryptography, etc.) takes so much time.  This is because some ports (like graphics/py-pytesseract) check the dependencies only at runtime and not during buildtime (e.g. a bulk-run won't reveal any issues).

> How do you test Python ports?

For the most ports I do following steps:

- Check setup.{py,cfg} and pyproject.toml for required dependencies

In a clean poudriere environment, e.g. via "testport -i", for each supported Python version:

- Run testsuite
- Try to import the Python package, e.g. via "${PYTHON_CMD} -m NewPackage
- Runtime test scripts (if supplied)

If the Python port is required by other ports, I do following steps as well:

- Do a bulk-run against all consumers
- Check the setup.{py,cfg} and pyproject.toml of the consumers
Comment 9 Matthias Andree freebsd_committer freebsd_triage 2021-06-19 23:16:06 UTC
Thanks for explaining your port testing workflows!