Bug 237952 - mail/notmuch: port still has build dependency python27 (transitive, via devel/talloc)
Summary: mail/notmuch: port still has build dependency python27 (transitive, via devel...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Kubilay Kocak
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-05-17 16:07 UTC by p5B2E9A8F
Modified: 2020-01-07 12:12 UTC (History)
4 users (show)

See Also:
koobs: maintainer-feedback? (timur)


Attachments
notmuch-build-dependencies.svg (374.33 KB, image/svg+xml)
2019-05-20 20:01 UTC, Sebastian Schwarz
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description p5B2E9A8F 2019-05-17 16:07:30 UTC
Even with forced DEFAULT_VERSIONS+=python=3.6
the port has build dependency python27.

Please fix for python3.
Comment 1 Sebastian Schwarz 2019-05-20 20:01:16 UTC
Created attachment 204490 [details]
notmuch-build-dependencies.svg

I'm building the notmuch packages using poudriere without any custom make.conf or ports options and I can find no direct direct build dependency on python27.   However there are a few transitive dependencies on python27:

# cd /usr/ports
# poudriere logclean -a
(...)
# poudriere bulk -j FreeBSD:12:amd64 -n mail/*notmuch*
(...)
# awk '$2 ~ /py(thon)?2/ { print $1 }' /usr/local/poudriere/data/logs/bulk/FreeBSD:12:amd64-default/latest/.poudriere.pkg_deps% | sort -u
llvm60-6.0.1_6
mesa-libs-18.3.2
py27-enum34-1.1.6
py27-setuptools-41.0.0
scons-3.0.1
spidermonkey52-52.9.0_3
talloc-2.1.14

Can you share the output of these (or similar) commands on your system so that I can try to reproduce your problem?  Otherwise please file PRs about the python27 build dependencies with above's packages' maintainers.

I've attached the full dependency graph, which was generated with the following command:

awk '
    BEGIN {
        print "digraph {"
    }
    {
        printf "\"%s\" -> \"%s\";\n", $1, $2
    }
    END {
        print "}"
    }
' < /usr/local/poudriere/data/logs/bulk/FreeBSD:12:amd64-default/latest/.poudriere.pkg_deps% | tred | dot -Tsvg -o notmuch-build-dependencies.svg
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-20 23:51:11 UTC
@Sebastian If/when isolated to another ports dependency declarations, we can amend this issues summary to suite, rather than creating another issue.

Alternatively (most likely) we can create sub-issues from this one to resolve 'blockers' to resolving this issue in other ports, since the symptoms are observable in/with this port.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-21 00:11:18 UTC
poudriere build on 12amd64 (-zpy36)

===>   notmuch-0.28.2 depends on shared library: libtalloc.so - not found
===>   Installing existing package /packages/All/talloc-2.1.14.txz
[12amd64-koobs] Installing talloc-2.1.14...
[12amd64-koobs] `-- Installing python27-2.7.16_1...

From the looks of things (I looked at the last 3 months of talloc's mailing list archive) ...

1) Python 3.x is the default
2) Python 2.x support is being removed from 4.11+
3) Python 2/3 is supported now

See also:

https://lists.samba.org/archive/samba-technical/2019-April/133196.html
https://lists.samba.org/archive/samba-technical/2019-May/133450.html
https://lists.samba.org/archive/samba-technical/2019-March/132899.html
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-21 01:05:44 UTC
The talloc port is currently set to limit Python support to 2.7, which was added in ports r454556 ("Fix too broad Python specification, cut down to 2.7 explicitly.) where it previously declared 2.7+, but why is was too broad is not detailed.

Looking for previous issues in Bugzilla, it appears bug 223761 was the originator of limiting it to 2.x.

Building talloc after removing only the 2.7 restriction, results in a build error as follows:

===>  Configuring for talloc-2.1.14
Traceback (most recent call last):
  File "buildtools/bin/waf", line 75, in <module>
    import Scripting
  File "/wrkdirs/usr/ports/devel/talloc/work/talloc-2.1.14/third_party/waf/wafadmin/Scripting.py", line 146
    except Utils.WafError, e:
                         ^
SyntaxError: invalid syntax

See Also: https://trac.macports.org/ticket/30617 (and my other google results)

The port also uses USES=waf (but does *not* specify WAF_CMD), which contains the following:

.if !${USES:Mpython*}
python_ARGS=    2.7,build
.include "${USESDIR}/python.mk"
.endif

Upstream says Python 2 and 3 are supported [1], so I don't know why this is limited to 2.7 either.

The bundled waf script (WRKSRC/buildtools/bin) is "./waf --version waf 1.5.19 (9709M)"

waf moved to Python 3 syntax by defaultin:

NEW IN WAF 1.6.0
----------------

General:
* Python 3 syntax by default (runs unmodified for 2.6, 2.7, 3.0 and 3.1)


There is an additional section after USES is set in the port:

# XXX: This is a gross hack to make port use both Python 2.7+ and 3.3+
# This is not officially supported, use at your own risk
.if defined(WITH_SAMBA4_PYTHON3) && ${WITH_SAMBA4_PYTHON3:M3\.[0-9]}
<snip>

which allows the user to specify those variables and add 'additional' python version support via ./configure:

--extra-python=PYTHON
build selected libraries for the specified additional version of Python (example: --extra-python=/usr/bin/python3)

[1] https://gitlab.com/ita1024/waf/  "Python compatibility: cPython 2.5 to 3.x, Jython 2.5, IronPython, and Pypy"
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-21 01:40:07 UTC
Fixing the exception syntax throughout the bundled waf results in further errors:

os.chmod:

File "./buildtools/wafsamba/wafsamba.py", line 801
    os.chmod(installed_location, 0755)
                                    ^
SyntaxError: invalid token

and after fixing that:

NameError: name 'xrange' is not defined (function removed from Python 3)

I stopped my 'quick fix investigation' at this point.

Suffice it to say:

1) devel/talloc uses :2.7 because the bundled waf bundled is (very) old, pre-Python 3 support, and only supports 2.x.

The options from here are, in no specific order:

a) Port/fix the bundled waf script to work with python 3.x

b) Ask/get talloc upstream to upgrade their bundled waf version

This may already be occurring: 

https://bugzilla.samba.org/show_bug.cgi?id=13504

c) In the meantime, get devel/talloc port to expose a PYTHON option that is disabled by default (might impact other samba ports?). This may be an incomplete solution, and still has a *build* dependency on Python 2.7, but removes the *runtime* requirement for it.

d) Investigate/Get notmuch to not use talloc. I don't know if its compulsory, or an optional/swappable backend

IMO, 

(a) is too much (and duplicate) work, and has a high QA and maintenance overhead.
(b) is worth adding your support to.
(c) is worth exploring with samba port maintainer (not as a bug), as a backup for, and in the meantime to (b)
(d) is worth investigating

Once we have the answers to (c) and (d), we can consider resolution for this issue. If they bear no fruit, this will likely need to be closed "Not A Bug", "Works As Intended" or "Not Accepted" as necessary

Either way I'm happy to be the coordindator on resolution for this
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-21 02:35:44 UTC
Accidentally closed. Re-open.
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-21 02:37:12 UTC
It appears talloc is *not* an optional dependency [1] which precludes option (d) in comment 5:

Notmuch depends on four libraries: Xapian, GMime 3.0,
Talloc, and zlib which are each described below

[1] https://github.com/notmuch/notmuch/blob/master/INSTALL
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-21 02:40:27 UTC
I forgot to note that for option (c), there is an 'at present' workaround in the absence of a new PYTHON option being created.

That is, to pass "NO_PYTHON=1" to the devel/talloc port build, which will remove the python 2.7 runtime requirement, leaving only a build requirement on python 2.7 (can be deleted after build)
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-21 02:48:13 UTC
And I made a mistake in comment4, devel/talloc *does* set WAF_CMD, which, either way, is not relevant to the root cause(s) as they stand currently.

If, on the other hand WAF_CMD wasn't set, and a Python 3.x compatible devel/py-waf port did exist, and talloc's build configuration files were compatible with it without changes, it would have been relevant, which is why I (incorrectly at the time) commented on it
Comment 10 Timur I. Bakeyev freebsd_committer 2019-05-21 21:26:01 UTC
That was interesting reading, thanks :)

So, you found most of the answers yourself, so let me summarize the current state of the Samba development and related ports.

Samba 4.x up to 4.10 uses highly customized, but old version of WAF. That leads to :

1. Hard dependency, at least on a build time, on Python 2.7 exclusively.
2. Inability to use external generic WAF.

There is no clean and easy way to use arbitrary Python version and/or external WAF to build those ports.

On a popular demand from embedded systems users I've added "hidden" NO_PYTHON flag to the related ports that disables building related Python bindings, which allow you to have packages(on the price of losing AD functionality) without Python 2.7 dependency. Still you'll need it for the build stage. But you found that out yourself.

It's also possible to get Python3 bindings in ADDITION to Python2 ones, but that would contaminate your system with the Python 2.7.

There is no clean way out of this legacy until Samba 4.10+, which is coming.

But that be a complicated process AFAIK, as we'd need to get parallel versions of talloc/tevent/tdb/ldb ports with Python3 only dependency until Samba .49 will be phased out.

As for your problem with talloc in particular, there is a new talloc 2.2.1 release, which uses new Python3 dependent WAF(no, it's not compatible with Python2, at least officially).

I have it in my tree and the only problem is how to name the port to keep, at least for this year, parallel versions of the "old" and "new" ports.

Advice is welcome how to implement it gracefully.