Bug 226135 - lang/python27: Fails to package after r462630 (_elementtree / pyexpat modules)
Summary: lang/python27: Fails to package after r462630 (_elementtree / pyexpat modules)
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Kubilay Kocak
URL:
Keywords: regression
: 226256 226321 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-02-23 02:43 UTC by Chip Cuccio
Modified: 2018-05-07 04:34 UTC (History)
10 users (show)

See Also:
koobs: maintainer-feedback+


Attachments
Build logs, env and ver. details, config options. (506.03 KB, text/plain)
2018-02-23 02:43 UTC, Chip Cuccio
no flags Details
Updated build log after make.conf changes. (490.75 KB, text/plain)
2018-02-24 12:52 UTC, Chip Cuccio
no flags Details
Use CPPFLAGS, Make CFLAGS comment more prominent (1.65 KB, patch)
2018-02-25 07:38 UTC, Kubilay Kocak
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Chip Cuccio 2018-02-23 02:43:41 UTC
Created attachment 190911 [details]
Build logs, env and ver. details, config options.

Previous commit builds fine on both amd64 and powerpc. Today's r462630 commit won't build/install packages on both archs: 11.1-RELEASE;

pkg-static: Unable to access file /usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/_elementtree.so:No such file or directory
pkg-static: Unable to access file /usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/pyexpat.so:No such file or directory

Attached is the build log, uname, pkg version -v, etc. in a single file.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2018-02-24 06:51:52 UTC
This will likely be related to the CFLAGS/LIBS change (additions) in ports r462630

For more context see this block in python27/Makefile:

# Use CPPFLAGS over CFLAGS due to -I ordering, causing elementtree and pyexpat
# to break in Python 2.7, or preprocessor complaints in Python >= 3.3
# Upstream Issue: https://bugs.python.org/issue6299

The issue is not scoped only to NLS.

For more context see:

https://bugs.python.org/issue6299#msg210189

There have been countless ports issues reported in the past related to hacking/modifying CFLAGS (and other variables, and these changes should not be attempted until the root underlying issue(s) are fixed (correctly/completely).

I'll see if I can prepare a patch for you to test. If that can't be done, we'll need to revert the change.
Comment 2 Sunpoet Po-Chuan Hsieh freebsd_committer 2018-02-24 09:02:52 UTC
I cannot reproduce the failure in poudriere builds (11.1-RELEASE amd64 with the specific option setting: -DEBUG +IPV6 +LIBFFI +NLS +PYMALLOC +THREADS -UCS2 +UCS4).
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2018-02-24 09:13:11 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #2)

Likewise.

@Chip Can you provide system environment/configuration information (uname -a, sanitized /etc/make.conf) along with an the full build log (as an attachmennt)
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2018-02-24 09:13:59 UTC
@Sunpoet, Can we hold off on MFH'ing ports r462630 until we figure out whats happening?
Comment 5 Chip Cuccio 2018-02-24 12:23:45 UTC
(In reply to Kubilay Kocak from comment #3)

@Kubilay there should be an attachment already with my build log, env/config details etc.

However here's my make.conf

FAVORITE_COMPILER=clang
COMPILER_TYPE=  clang
MAKE_JOBS_NUMBER?=3

CC=clang
CXX=clang++
CPP=clang-cpp
WITH_CCACHE_BUILD=yes
CPUTYPE?=core2

DEFAULT_VERSIONS+=ssl=openssl
OPENSSLBASE=/usr/local  # Use ports version of openssl with TLS 1.2
OPTIONS_UNSET+= GSSAPI_BASE
OPTIONS_SET+=   GSSAPI_MIT

Thanks!
Comment 6 Bernard Spil freebsd_committer 2018-02-24 12:41:12 UTC
(In reply to Chip Cuccio from comment #5)
Hi Chip,

Please remove

> CC=clang
> CXX=clang++
> CPP=clang-cpp
> OPENSSLBASE=/usr/local

from your make.conf, these are redundant and set either by Mk/Uses/ssl.mk or in the ports framework.
Comment 7 Chip Cuccio 2018-02-24 12:43:59 UTC
(In reply to Bernard Spil from comment #6)

@Bernard,

OK I will do that, and report back. Thanks!
- chip
Comment 8 Chip Cuccio 2018-02-24 12:52:17 UTC
Created attachment 190956 [details]
Updated build log after make.conf changes.
Comment 9 Chip Cuccio 2018-02-24 12:53:31 UTC
(In reply to Bernard Spil from comment #6)

@Bernard,
I have made the make.conf changes as you advised: Same build issue.
Updated build log attached.

Thanks!
Comment 10 Sunpoet Po-Chuan Hsieh freebsd_committer 2018-02-24 13:22:47 UTC
(In reply to Kubilay Kocak from comment #4)

Yes, MFH is suspended.
Comment 11 Sunpoet Po-Chuan Hsieh freebsd_committer 2018-02-24 13:27:51 UTC
Here is my poudriere build log of lang/python27-pr226135 (copied from lang/python27 with following change).
https://people.freebsd.org/~sunpoet/logs/python27-pr226135.log

% diff -u lang/python27/Makefile lang/python27-pr226135/Makefile
--- lang/python27/Makefile      2018-02-23 03:51:29.048434000 +0800
+++ lang/python27-pr226135/Makefile     2018-02-24 01:22:44.675219000 +0800
@@ -5,7 +5,8 @@
 PORTREVISION=  1
 CATEGORIES=    lang python ipv6
 MASTER_SITES=  PYTHON/ftp/python/${PORTVERSION}
-PKGNAMESUFFIX= 27
+#PKGNAMESUFFIX=        27
+PKGNAMESUFFIX= 27-pr226135
 DISTNAME=      Python-${PORTVERSION}
 DIST_SUBDIR=   python

@@ -44,7 +45,8 @@
                        OSMAJOR=${OSVERSION:C/([0-9]*)[0-9]{5}/\1/}             # For plat-freebsd* in pkg-plist. https://bugs.python.org/issue19554

 OPTIONS_DEFINE=                DEBUG IPV6 LIBFFI NLS PYMALLOC THREADS
-OPTIONS_DEFAULT=       LIBFFI PYMALLOC THREADS UCS4
+#OPTIONS_DEFAULT=      LIBFFI PYMALLOC THREADS UCS4
+OPTIONS_DEFAULT=       LIBFFI NLS PYMALLOC THREADS UCS4
 OPTIONS_SINGLE=                UNICODE
 OPTIONS_SINGLE_UNICODE=        UCS2 UCS4
 OPTIONS_SUB=           yes
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2018-02-25 05:35:14 UTC
@Chip Could you please:

- Comment out the LIBS entry
- Rebuild, confirm whether issue is still reproducible
- In either case, attach the build log
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2018-02-25 05:41:14 UTC
@Chip

Additionally, could you please run a second test:

Using python27 *as is* (verbatim r462630, *without* requested change in comment 10)

- pkg delete -f expat
- rebuild python27
- confirm whether issue is reproducible

I'm continuing to isolate locally
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2018-02-25 07:16:49 UTC
The issue is reproducible with:

- DEFAULT_VERSIONS=ssl=openssl (more precisely CFLAGS+=-I/usr/local/include)
- expat2 port/package installed

The root cause is that described in comment #1

I'll have a patch shortly
Comment 15 Kubilay Kocak freebsd_committer freebsd_triage 2018-02-25 07:38:45 UTC
Created attachment 190984 [details]
Use CPPFLAGS, Make CFLAGS comment more prominent

@Chip 

Please test/confirm that this patch fixes the issue.
Comment 16 Chip Cuccio 2018-02-26 15:40:10 UTC
(In reply to Kubilay Kocak from comment #15)

@Koobs,

I can confirm that your patch corrects the build issue on both amd64 and powerpc archs!

Thank you.
- Chip
Comment 17 Sunpoet Po-Chuan Hsieh freebsd_committer 2018-02-26 23:16:08 UTC
(In reply to Kubilay Kocak from comment #15)

This patch looks good to me. And it's reasonable for CFLAGS/CPPFLAGS issue you mentioned before.

I'll check if it works with 3 SSL settings (ssl=base, openssl and openssl-devel).

Thanks!
Comment 18 gessel 2018-02-27 11:35:52 UTC
I had the same issue on FreeBSD 10.3-RELEASE-p26 #0 r327588 building python27-2.7.14_1, applied the patch and build completed successfully. 

Thank you for the patch.
Comment 19 Dmitry 2018-02-27 12:44:24 UTC
I've got the same problem with port build (latest 11.1 amd64)
Also can confirm: patch from comment #15 was helpful, thanks.
Hope it will be committed to ports soon.
Comment 20 Sunpoet Po-Chuan Hsieh freebsd_committer 2018-02-27 17:18:10 UTC
@koobs,

I've tested it with different SSL settings. It works.

Please DO NOT BUMP PORTREVISION.
It fixes the build but does not change the package, especially for people who did not encounter build failure.
Comment 21 Walter Schwarzenfeld freebsd_triage 2018-02-28 01:17:58 UTC
*** Bug 226256 has been marked as a duplicate of this bug. ***
Comment 22 Franco Fichtner 2018-03-01 06:58:55 UTC
Why not back out the old commit? It has been failing for over a week now.
Comment 23 Kubilay Kocak freebsd_committer freebsd_triage 2018-03-01 08:45:24 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #20)

I note that other python ports didn't get the original change (ports r462630, as part of bug 222795), but also:

- python34 still has BROKEN_SSL=openssl-devel
- python35 doesn't
- python36 doesn't

Were you going to apply r462630 to the other ports as well?
What is the status with regard to ssl support in other ports? (>2.7)

Barnerd, if you can comment on the change delta required to get all lang/python?? ports to support openssl/openssl-devel/libressl, that would be appreciated.

We (python) want as much consistency across the ports as possible, so I'll want to replicate this change into every port unless there are reasons not to.
Comment 24 Sunpoet Po-Chuan Hsieh freebsd_committer 2018-03-01 12:37:32 UTC
(In reply to Kubilay Kocak from comment #23)

Other python port do not need r462630 as follows:
- python34 is confirmed broken with/without r462630.
- python35 doesn't need r462630.
- python36 doesn't need r462630.
Comment 25 commit-hook freebsd_committer 2018-03-01 13:47:29 UTC
A commit references this bug:

Author: koobs
Date: Thu Mar  1 13:47:11 UTC 2018
New revision: 463321
URL: https://svnweb.freebsd.org/changeset/ports/463321

Log:
  lang/python27: Fix build regression

  r462630 added OpenSSL include/library paths to CFLAGS/LIBS which caused
  the _elementtree and pyexpat modules to fail to build. Adding paths to CFLAGS
  is known to cause issues [1].

  Accordingly:

  - Switch to using CPPFLAGS for OpenSSL include/library paths (like NLS).
  - Move the comment describing why this is necessary to the top of the port
    with reasonable warning about what can happen and why.

  [1] https://svnweb.freebsd.org/ports?view=revision&revision=326729

  PR:		226135, 222795)
  Reported by:	many
  Reviewed by:	sunpoet (python)
  Approved by:	koobs (python)

Changes:
  head/lang/python27/Makefile
Comment 26 Kubilay Kocak freebsd_committer freebsd_triage 2018-03-01 13:56:14 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #24)

Thank you. Anything left to do here?
Comment 27 Sunpoet Po-Chuan Hsieh freebsd_committer 2018-03-01 14:24:18 UTC
(In reply to Kubilay Kocak from comment #26)

Well done. Thank you!
Comment 28 Franco Fichtner 2018-03-01 14:25:13 UTC
yes, thank you!
Comment 29 commit-hook freebsd_committer 2018-03-02 06:54:54 UTC
A commit references this bug:

Author: antoine
Date: Fri Mar  2 06:54:14 UTC 2018
New revision: 463375
URL: https://svnweb.freebsd.org/changeset/ports/463375

Log:
  Revert r462630, it introduced regressions

  PR:		226135
  With hat:	portmgr

Changes:
  head/lang/python27/Makefile
Comment 30 Kubilay Kocak freebsd_committer freebsd_triage 2018-03-02 07:10:07 UTC
Thank you Antoine.

All: ports r463321 (comment 25), fixing the regression was reverted due to a secondary readline build failure on what appears to be 10.x, so I asked for r462630 (the original regression) to be reverted as it was wider in scope.
Comment 31 Bernard Spil freebsd_committer 2018-03-03 20:47:41 UTC
*** Bug 226321 has been marked as a duplicate of this bug. ***
Comment 32 Ivan 2018-03-04 07:45:14 UTC
I'm not sure, that my https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226321 is duplicate of this one (I tried reverted patch from here) as failure occurs only with libressl, but OK, let it be.
Comment 33 Walter Schwarzenfeld freebsd_triage 2018-05-06 13:50:09 UTC
Only for information: There is python 2.7.15 with fixes for at least CVE-2018-1060 and CVE-2018-1061.
Comment 34 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-07 04:34:33 UTC
The original bug report (regression) is resolved.

Re comment 33, see bug 228028