Bug 206643

Summary:	mail/py-pyspf always depends on dns/py-py3dns while dns/py-py3dns is python 3 only
Product:	Ports & Packages	Reporter:	Daniel Gerzo <danger>
Component:	Individual Port(s)	Assignee:	Kubilay Kocak <koobs>
Status:	Closed FIXED
Severity:	Affects Some People	CC:	danger, koobs, mayhem30, sunpoet
Priority:	---	Keywords:	patch
Version:	Latest	Flags:	bugzilla: maintainer-feedback? (sunpoet) koobs: merge-quarterly+
Hardware:	Any
OS:	Any
URL:	https://reviews.freebsd.org/D5082
See Also:	https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205789
Bug Depends on:
Bug Blocks:	205789, 206645

Description Daniel Gerzo freebsd_committer

2016-01-26 11:11:28 UTC

mail/py-pyspf has recently been updated to always depend on dns/py-py3dns, however py3dns is python3 dns client and does not work under python2. This breaks python2 users who use mail/py-pyspf.

Comment 1 Kubilay Kocak freebsd_committer

2016-01-26 11:14:14 UTC

Two changes are needed:

* dns/py-py3dns should be locked to USES=python:3.0+ 
* py-pyspf should conditionally RUN_DEPENDS on py-py3dns for >= 3000 and py-pydns for <= 3000

Both need MFH'ing

I have changes in progress now

Comment 2 commit-hook freebsd_committer

2016-01-26 14:02:13 UTC

A commit references this bug:

Author: koobs
Date: Tue Jan 26 14:01:38 UTC 2016
New revision: 407273
URL: https://svnweb.freebsd.org/changeset/ports/407273

Log:
  mail/py-pyspf: Fix dependencies (missing & incorrect)

  Revision 404778 [1] modified py-pyspf to only use dns/py3dns instead
  of conditionally using dns/py-dns and py-py3dns depending on whether
  Python 2.x or 3.x was being used.

  dns/py-py3dns is a Python 3.x *only* package, but the port does not
  currently [2] limit itself to USES=python:3.0+.

  This results in errors for all dependent ports of py-pyspf when
  Python 2.x is used, which was reported for
  mail/postfix-policyd-spf-python. [3]

  pyspf's README notes the following requirements:

    This package requires PyDNS (or Py3DNS for running with Python 3) and
    either the ipaddr or python3.3 and later

    This package requires authres from either pypi or
    http://launchpad.net/authentication-results-python to process
    and generate RFC 5451 Authentication Results headers.

    The spf module in this version has been tested with python3.2 and does
    not require using 2to3.

  Accordingly, this change:

  - Reverts to conditional RUN_DEPENDS on dns/py-dns or py-py3dns
    depending on whether Python 2.x or Python 3.x is being used.
  - Adds authres to RUN_DEPENDS
  - Adds a conditional RUN_DEPENDS on devel/py-ipaddr, depending on
    Python version.

  While I'm here:

  - Sort and group USE{S} entries
  - Match COMMENT to setup.py:description
  - Add test and post-extract targets, TEST_DEPENDS, and patch files
    so that tests can be run properly
  - Pet several portlint warnings

  [1] https://svnweb.freebsd.org/changeset/ports/404778
  [2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206645
  [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206643

  QA:

   * portlint: OK (WARN: Makefile: [47]: possible use of "${CHMOD}")
   * porttest: OK (poudriere: 11amd64, python27, python34)
   * unittest: OK (Ran 425 tests in 0.155s)

  PR:			206643
  Reported by:		danger
  Approved by:		portmgr (blanket)
  MFH:			2016Q1
  Differential Revision:	DXXXX

Changes:
  head/mail/py-pyspf/Makefile
  head/mail/py-pyspf/files/
  head/mail/py-pyspf/files/patch-test_testspf.py
  head/mail/py-pyspf/pkg-descr

Comment 3 Ken 2016-01-26 17:53:23 UTC

This patch did not work for me.

Here is the new error message I am receiving :

policyd-spf[22181]: ERROR: 127.0.0.0/8 in skip_addresses not IP network.  Message: '21x.xx.xx.xx' does not appear to be an IPv4 or IPv6 address. Did you pass in a bytes (str in Python 2) instead of a unicode object?. Aborting whitelist processing.

Comment 4 Daniel Gerzo freebsd_committer

2016-01-26 20:05:22 UTC

(In reply to mayhem30 from comment #3)
make sure you end up having installed just dns/py-dns and not dns/py-py3dns, this fix works for me.

Comment 5 Ken 2016-01-26 20:20:02 UTC

(In reply to Daniel Gerzo from comment #4)

Only py27-dns-2.3.6_1 is installed. I removed dns/py-py3dns and reinstalled py27-postfix-policyd-spf-python before testing.

Comment 6 Daniel Gerzo freebsd_committer

2016-01-26 21:42:24 UTC

(In reply to mayhem30 from comment #5)
that's strange because the error you mention is exactly the error I was getting before this fix.

You did not have a problem before this commit?
what version of py27-pyspf do you have installed?

Comment 7 Ken 2016-01-26 21:46:43 UTC

(In reply to Daniel Gerzo from comment #6)

I had much worse issues before this commit, my maillog was packed with 20+ lines of python errors (per connection).

Now, I only receive that one line error per connection.

I have py27-pyspf-2.0.12_4 installed.

Comment 8 Ken 2016-01-27 06:56:27 UTC

I've tried reinstalling everything and have the same result.

Is there anything else I can try?

$ portmaster -r python27-2.7.11_1

===>>> The following actions were performed:
        Re-installation of python27-2.7.11_1
        Re-installation of py27-setuptools27-19.2
        Re-installation of py27-ipaddr-2.1.11
        Re-installation of python2-2_3
        Re-installation of py27-dns-2.3.6_1
        Re-installation of py27-authres-0.800
        Re-installation of py27-pyspf-2.0.12_4
        Re-installation of py27-six-1.9.0
        Re-installation of scons-2.3.0_1
        Re-installation of py27-alabaster-0.7.6
        Re-installation of py27-pytz-2015.7,1
        Re-installation of py27-Babel-2.2.0_1
        Re-installation of py27-MarkupSafe-0.23
        Re-installation of py27-Jinja2-2.8
        Re-installation of py27-docutils-0.12
        Re-installation of py27-pygments-2.1
        Re-installation of py27-snowballstemmer-1.2.0_1
        Re-installation of py27-sphinx_rtd_theme-0.1.8
        Re-installation of py27-sphinx-1.3.1_2
        Re-installation of py27-ipaddress-1.0.14
        Re-installation of py27-postfix-policyd-spf-python-1.3.2_1

$ /usr/local/etc/rc.d/postfix restart
postfix/postfix-script: stopping the Postfix mail system
postfix/postfix-script: starting the Postfix mail system

$ more /var/log/maillog

...
...

Jan 26 22:52:35 localhost policyd-spf[10266]: ERROR: 127.0.0.0/8 in skip_addresses not IP network.  Message: '211.22.33.44' does not appear to be an IPv4 or IPv6 address. Did you pass in a bytes (str in Python 2) instead of a unicode object?. Aborting whitelist processing.

Comment 9 Kubilay Kocak freebsd_committer

2016-01-27 12:36:04 UTC

Can you check the upstream bug list for possible candidates? While QA'ing I noticed that the Debian package had a patch reverting something that may have had something to do with this.

Let me know if you find anything, and we'll create a new issue for it if its relevant

Comment 10 Ken 2016-01-29 02:25:21 UTC

No one else is having this issue?

I've completely removed the package (and all the dependency's) and reinstalled it - and it's still not working for me.

Comment 11 Ken 2016-01-29 21:02:01 UTC

I fixed the issue. In case anyone else is having troubles, this is what I had to do :

Remove py27-ipaddress-1.0.14 (left over from an older version??) 

portmaster -r python27-2.7.11_1 (rebuild everything that requires python)

Restart Postfix

Comment 12 commit-hook freebsd_committer

2016-02-02 15:07:31 UTC

A commit references this bug:

Author: koobs
Date: Tue Feb  2 15:06:56 UTC 2016
New revision: 407820
URL: https://svnweb.freebsd.org/changeset/ports/407820

Log:
  MFH: r407273 mail/py-pyspf: Fix dependencies (missing & incorrect)

  Revision 404778 [1] modified py-pyspf to only use dns/py3dns instead
  of conditionally using dns/py-dns and py-py3dns depending on whether
  Python 2.x or 3.x was being used.

  dns/py-py3dns is a Python 3.x *only* package, but the port does not
  currently [2] limit itself to USES=python:3.0+.

  This results in errors for all dependent ports of py-pyspf when
  Python 2.x is used, which was reported for
  mail/postfix-policyd-spf-python. [3]

  pyspf's README notes the following requirements:

    This package requires PyDNS (or Py3DNS for running with Python 3) and
    either the ipaddr or python3.3 and later

    This package requires authres from either pypi or
    http://launchpad.net/authentication-results-python to process
    and generate RFC 5451 Authentication Results headers.

    The spf module in this version has been tested with python3.2 and does
    not require using 2to3.

  Accordingly, this change:

  - Reverts to conditional RUN_DEPENDS on dns/py-dns or py-py3dns
    depending on whether Python 2.x or Python 3.x is being used.
  - Adds authres to RUN_DEPENDS
  - Adds a conditional RUN_DEPENDS on devel/py-ipaddr, depending on
    Python version.

  While I'm here:

  - Sort and group USE{S} entries
  - Match COMMENT to setup.py:description
  - Add test and post-extract targets, TEST_DEPENDS, and patch files
    so that tests can be run properly
  - Pet several portlint warnings

  [1] https://svnweb.freebsd.org/changeset/ports/404778
  [2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206645
  [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206643

  QA:

   * portlint: OK (WARN: Makefile: [47]: possible use of "${CHMOD}")
   * porttest: OK (poudriere: 11amd64, python27, python34)
   * unittest: OK (Ran 425 tests in 0.155s)

  PR:			206643
  Reported by:		danger
  Approved by:		portmgr (blanket)
  Differential Revision:	D5082

  Approved by:		ports-secteam (feld)

Changes:
_U  branches/2016Q1/
  branches/2016Q1/mail/py-pyspf/Makefile
  branches/2016Q1/mail/py-pyspf/files/
  branches/2016Q1/mail/py-pyspf/pkg-descr

Comment 13 Kubilay Kocak freebsd_committer

2016-02-02 15:37:29 UTC

The issues a number of people have reported are likely due to slightly corrupt/invalid python installations, containing multiple (version) references to a package.

This can be caused either by issues with incomplete package removal, pkg-plist bugs (mostly a non-issue due to autoplist for most python ports), OR most likely, the use of pip or easy_install to install packages from PyPI into the system site-packages location, which pkg does not manage.

There is an open bug 205881 to add a warning against this to the pip port.

If anyone continues to experience issues related to this change, please create a separate issue, and add this issue (bug 206643) to the See Also: field so they can be investigated on a case by case basis.