Bug 238821 - lang/python35: Use -std=c99
Summary: lang/python35: Use -std=c99
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: powerpc Any
: --- Affects Some People
Assignee: Piotr Kubaj
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-06-26 10:08 UTC by Piotr Kubaj
Modified: 2019-07-01 07:10 UTC (History)
2 users (show)

See Also:
koobs: maintainer-feedback+
koobs: merge-quarterly+


Attachments
patch (343 bytes, patch)
2019-06-26 10:08 UTC, Piotr Kubaj
koobs: maintainer-approval+
Details | Diff
change to gnu99 (345 bytes, patch)
2019-06-27 10:41 UTC, Piotr Kubaj
no flags Details | Diff
head-powerpc64-default-logs-python35-3.5.7_2.log (800.67 KB, text/plain)
2019-06-27 11:23 UTC, Kubilay Kocak
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Piotr Kubaj freebsd_committer 2019-06-26 10:08:16 UTC
Created attachment 205349 [details]
patch

_pickle.so isn't build with base GCC by default:
/usr/local/poudriere/ports/default/lang/python35/work/Python-3.5.7/Modules/_pickle.c: In function 'PyMemoTable_Copy':
/usr/local/poudriere/ports/default/lang/python35/work/Python-3.5.7/Modules/_pickle.c:677: error: 'for' loop initial declaration used outside C99 mode
/usr/local/poudriere/ports/default/lang/python35/work/Python-3.5.7/Modules/_pickle.c: In function '_pickle_PicklerMemoProxy_copy_impl':
/usr/local/poudriere/ports/default/lang/python35/work/Python-3.5.7/Modules/_pickle.c:4207: error: 'for' loop initial declaration used outside C99 mode
/usr/local/poudriere/ports/default/lang/python35/work/Python-3.5.7/Modules/_pickle.c: In function 'Unpickler_set_memo':
/usr/local/poudriere/ports/default/lang/python35/work/Python-3.5.7/Modules/_pickle.c:6794: error: 'for' loop initial declaration used outside C99 mode
/usr/local/poudriere/ports/default/lang/python35/work/Python-3.5.7/Modules/_pickle.c:6842: error: 'for' loop initial declaration used outside C99 mode

Set USE_CSTD=c99 to fix build.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 09:44:29 UTC
(In reply to Piotr Kubaj from comment #0)

Can you test (if you haven't already) and let us know if any other lang/python* versions are affected.

I don't believe Python has changed its C coding standard for particular branches, so we may want to report this upstream too
Comment 2 Piotr Kubaj freebsd_committer 2019-06-27 10:21:08 UTC
(In reply to Kubilay Kocak from comment #1)
Here are successful build logs for other versions:
https://talos.anongoth.pl/data/powerpc64-default/latest-per-pkg/python27-2.7.16_1.log
https://talos.anongoth.pl/data/powerpc64-default/latest-per-pkg/python36-3.6.8_2.log
https://talos.anongoth.pl/data/powerpc64-default/latest-per-pkg/python37-3.7.3_1.log

Only python35 is broken and it's been that way for at least last 9 months (note that only builds with base GCC are broken, so that's why no one noticed).
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 10:26:06 UTC
(In reply to Piotr Kubaj from comment #2)

Thank you for following up Piotr

In future, when providing logs please provide them as attachments as external URL references tend to go missing/stale over time.

Given 3.5 is the only branch not affected, it may be worth perusing the upstream sources to identify potential commits that fixed the problem, but may not have been merged back to 3.5 (Python only bugfixes the latest branch + 2 previous)

So, look at 3.6 for the failing code in question, compare it to 3.5.

It's preferable to backport upstream changes where they exist, rather than switch the C standard across the board (in only one version)
Comment 4 Piotr Kubaj freebsd_committer 2019-06-27 10:40:37 UTC
(In reply to Kubilay Kocak from comment #3)
The patch that fixes build on 3.6 is here:
https://github.com/python/cpython/commit/7490577f6a7debf35bd13adf0f00fa9b272e8f8c

They later changed to gnu99:
https://github.com/python/cpython/commit/7490577f6a7debf35bd13adf0f00fa9b272e8f8c

But since they esentially add -std=gnu99, we could achieve the same with USE_CSTD=gnu99.
Comment 5 Piotr Kubaj freebsd_committer 2019-06-27 10:41:28 UTC
Created attachment 205372 [details]
change to gnu99
Comment 6 Piotr Kubaj freebsd_committer 2019-06-27 10:42:49 UTC
(In reply to Piotr Kubaj from comment #4)
Comment 7 Piotr Kubaj freebsd_committer 2019-06-27 10:43:09 UTC
Sorry, the commit changing to gnu99 is here:
https://github.com/python/cpython/commit/7f9eb6eda30398ea7a5b630856abce81f91591f6
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 11:04:40 UTC
Issue 28017 [1] shows that that commit was followed up by a subsequent commit:

compile with -std=c99 instead of -std=gnu99; use kiddie-gloves with bluetooth/bluetooh.h
https://hg.python.org/cpython/rev/91017e2202ae

Changing just the cstd doesn't appear to be sufficient, we should backport 91017e22 incl. the bluetooth header check block too.

FWIW, the bluetooth headers were fixed to compile with strict C in 

https://git.kernel.org/pub/scm/bluetooth/bluez.git/commit?id=cf52a40

So theoretically we could fix the root cause, but unless Python went down that track later (maybe they did?), its probably not best course of action to deviate from upstream code and carry the patch locally.

[1] https://bugs.python.org/issue28017
Comment 9 Piotr Kubaj freebsd_committer 2019-06-27 11:10:27 UTC
(In reply to Kubilay Kocak from comment #8)
This bug report says:
PPC Fedora build bot is not able to build the _ssl and _socket module. It looks like bluetooth.h is not compatible with std=c99. It uses some GNU C extensions like __extension__, __attribute__((packed)) and __typeof__. The C99 variant -std=gnu99 should do the trick.

But, i'm able to build on FreeBSD/powerpc64 (that's what I test on). So this bug is probably only present on Linux.
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 11:23:40 UTC
Created attachment 205373 [details]
head-powerpc64-default-logs-python35-3.5.7_2.log

Sorry, I completelty missed that it was an unrelated module.

However, it (python35/powerpc) appears to be failing on our build cluster

We need to identify why Python >= 3.6 is not failing in this manner, as that will be the likely best candidate resolution
Comment 11 Piotr Kubaj freebsd_committer 2019-06-27 11:28:33 UTC
(In reply to Kubilay Kocak from comment #10)
Yes, it's failing because of the error in my 1st comment:
/wrkdirs/usr/ports/lang/python35/work/Python-3.5.7/Modules/_pickle.c: In function 'PyMemoTable_Copy':
/wrkdirs/usr/ports/lang/python35/work/Python-3.5.7/Modules/_pickle.c:677: error: 'for' loop initial declaration used outside C99 mode
/wrkdirs/usr/ports/lang/python35/work/Python-3.5.7/Modules/_pickle.c: In function '_pickle_PicklerMemoProxy_copy_impl':
/wrkdirs/usr/ports/lang/python35/work/Python-3.5.7/Modules/_pickle.c:4207: error: 'for' loop initial declaration used outside C99 mode
/wrkdirs/usr/ports/lang/python35/work/Python-3.5.7/Modules/_pickle.c: In function 'Unpickler_set_memo':
/wrkdirs/usr/ports/lang/python35/work/Python-3.5.7/Modules/_pickle.c:6794: error: 'for' loop initial declaration used outside C99 mode
/wrkdirs/usr/ports/lang/python35/work/Python-3.5.7/Modules/_pickle.c:6842: error: 'for' loop initial declaration used outside C99 mode

This is what my patch fixes.
Comment 12 Piotr Kubaj freebsd_committer 2019-06-27 11:30:18 UTC
(In reply to Kubilay Kocak from comment #10)
Also, Python >= 3.6 doesn't fail, because it uses -std=gnu99.
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 11:39:27 UTC
(In reply to Piotr Kubaj from comment #12)

Sorry, brain tired, no workey :) But let's use std=c99 as originally submitted as that's what python36 uses
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 11:46:58 UTC
lang/python35: Use -std=c99

Python 3.5 is currently failing to build the pickle module on GCC-based
architectures, with the following (several) errors:

  error: 'for' loop initial declaration used outside C99 mode

This causes packaging to fail, as the pickle module filename changes on
failure to build, so the plist ends up incorrect

Python 3.6+ switched to using -std=c99 [1][2], but the changes were not
backported to 3.5

[1] https://bugs.python.org/issue28017
[2] https://hg.python.org/cpython/rev/b5b2bb56d303
[3] https://hg.python.org/cpython/rev/91017e2202ae

PR: 238821
Reviewed by: koobs (python)
Approved by: koobs (python)
MFH: 2019Q2 (blanket: build fix)
Comment 15 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 11:47:20 UTC
Comment on attachment 205349 [details]
patch

Approved by: koobs (python, maintainer)
Comment 16 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 11:47:43 UTC
Over to you Piotr for commit/merge, thank you for the report, patch and putting up with me
Comment 17 Piotr Kubaj freebsd_committer 2019-06-27 11:51:52 UTC
(In reply to Kubilay Kocak from comment #16)
Just to be clear, it doesn't affect only powerpc, but also other GCC-based architectures (mips*, sparc64).
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-27 11:54:16 UTC
That's fine, I set the arch for the initial report case, and we can't select multiple.

I made sure the commit log message (please use it) did not imply it was ppc specific
Comment 19 commit-hook freebsd_committer 2019-06-27 17:13:50 UTC
A commit references this bug:

Author: pkubaj
Date: Thu Jun 27 17:12:59 UTC 2019
New revision: 505210
URL: https://svnweb.freebsd.org/changeset/ports/505210

Log:
  lang/python35: Use -std=c99

  Python 3.5 is currently failing to build the pickle module on GCC-based
  architectures, with the following (several) errors:

    error: 'for' loop initial declaration used outside C99 mode

  This causes packaging to fail, as the pickle module filename changes on
  failure to build, so the plist ends up incorrect

  Python 3.6+ switched to using -std=c99 [1][2], but the changes were not
  backported to 3.5

  [1] https://bugs.python.org/issue28017
  [2] https://hg.python.org/cpython/rev/b5b2bb56d303
  [3] https://hg.python.org/cpython/rev/91017e2202ae

  PR: 238821
  Reviewed by: koobs (python)
  Approved by: koobs (python), mat (mentor)
  MFH: 2019Q2 (blanket: build fix)
  Differential Revision:  https://reviews.freebsd.org/D20778

Changes:
  head/lang/python35/Makefile
Comment 20 commit-hook freebsd_committer 2019-07-01 07:09:29 UTC
A commit references this bug:

Author: koobs
Date: Mon Jul  1 07:08:36 UTC 2019
New revision: 505546
URL: https://svnweb.freebsd.org/changeset/ports/505546

Log:
  MFH: r505210 lang/python35: Use -std=c99

  Python 3.5 is currently failing to build the pickle module on GCC-based
  architectures, with the following (several) errors:

    error: 'for' loop initial declaration used outside C99 mode

  This causes packaging to fail, as the pickle module filename changes on
  failure to build, so the plist ends up incorrect

  Python 3.6+ switched to using -std=c99 [1][2], but the changes were not
  backported to 3.5

  [1] https://bugs.python.org/issue28017
  [2] https://hg.python.org/cpython/rev/b5b2bb56d303
  [3] https://hg.python.org/cpython/rev/91017e2202ae

  PR: 238821
  Reviewed by: koobs (python)
  Approved by: koobs (python), mat (mentor)
  Differential Revision:  https://reviews.freebsd.org/D20778

  Approved by:	ports-secteam (blanket: build fix)

Changes:
_U  branches/2019Q2/
  branches/2019Q2/lang/python35/Makefile
Comment 21 Kubilay Kocak freebsd_committer freebsd_triage 2019-07-01 07:10:04 UTC
Merged to 2019Q2 quarterly