Bug 261974 - lang/python3*: Enable Link Time Optimization (via autoconf)
Summary: lang/python3*: Enable Link Time Optimization (via autoconf)
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Piotr Kubaj
URL:
Keywords:
Depends on:
Blocks: 258536 258822
  Show dependency treegraph
 
Reported: 2022-02-15 17:10 UTC by Piotr Kubaj
Modified: 2022-05-04 01:33 UTC (History)
2 users (show)

See Also:
koobs: maintainer-feedback+


Attachments
patch (511 bytes, patch)
2022-02-15 17:10 UTC, Piotr Kubaj
no flags Details | Diff
v2 (7.69 KB, patch)
2022-02-23 20:34 UTC, Piotr Kubaj
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Piotr Kubaj freebsd_committer freebsd_triage 2022-02-15 17:10:24 UTC
Off for riscv64 and powerpc64.

configure.ac checks $CC, so we need to set it to clang if $CC == cc.
LD takes around 300MB for linking so it should be ok to enable it even for 32 bit architectures.

This patch should also be merged to older Python. At least 3.10 and 3.9 should build with LTO. It would be nice if it's done before 3.9 is made the default.
Comment 1 Piotr Kubaj freebsd_committer freebsd_triage 2022-02-15 17:10:54 UTC
Created attachment 231841 [details]
patch
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2022-02-15 22:14:28 UTC
I have a WIP for python ports to LTO, PGO, et al (PIE, etc), and more broadly prefer to handle this at the framework level than at an individual ports/software build level. Most build systems that provide pie/lto/etc features are relatively flaky and produce inconsistent results
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2022-02-15 22:15:20 UTC
@Piotr Let's touch base and solve this by making some progress at the higher level, I've made significant progress for establishing a baseline configuration for large swathes of the ports tree for LTO
Comment 4 Piotr Kubaj freebsd_committer freebsd_triage 2022-02-15 22:54:56 UTC
I saw your site at https://wiki.freebsd.org/KubilayKocak/Ports/LinkTimeOptimization.

However, until the toolchain issues are resolved, I'd prefer to enable LTO only for individual ports.
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2022-02-23 03:06:18 UTC
(In reply to Piotr Kubaj from comment #4)

Sounds reasonable. Could we:

- Wrap this in an LTO option, enabled by default, excluded on architectures its not relevent or ok for.

- Update the patch to include changes for all lang/python* ports that support it (LTO). I think more than one does.

- Can we get away without the 'CC= environment override', leaving in either:

1) the current check for == cc, or 
2) using appropriate ports framework mechanisms to determine whether the compiler is clang? ie:

if excluding options based on arch is feasible:

   if option and correct-compiler 

If not:

   if option and correct compiler and not bad-arch

Note that while pythons configure checks CC, it also incorrectly detects 'GNU linker' (based on version string), and the build system is not very robust with respect to toolchains, and if we can just do the 'is this the right toolchain for lto' parts in ports, that would be preferable.
Comment 6 Piotr Kubaj freebsd_committer freebsd_triage 2022-02-23 20:34:30 UTC
Created attachment 232057 [details]
v2

Here's the patch for all Python 3 versions. Only Python 3.11 supports Thin LTO.

Python 2.7 supports LTO but using it requires --enable-optimizations. Once this patch is committed, I plan to do the next one for --enable-optimizations and then will enable LTO for 2.7.
Comment 7 commit-hook freebsd_committer freebsd_triage 2022-03-09 17:35:25 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=9a31e1b6d3bff1eef9fcd5e72268c238ec5335fb

commit 9a31e1b6d3bff1eef9fcd5e72268c238ec5335fb
Author:     Piotr Kubaj <pkubaj@FreeBSD.org>
AuthorDate: 2022-03-09 17:30:00 +0000
Commit:     Piotr Kubaj <pkubaj@FreeBSD.org>
CommitDate: 2022-03-09 17:30:00 +0000

    lang/python3*: add LTO option and enable by default everywhere except powerpc64 and riscv64

    PR:     261974
    Approved by:    python (koobs got his commit bit revoked during review process)

 lang/python310/Makefile                    |  9 +++++++--
 lang/python310/files/patch-configure (new) | 11 +++++++++++
 lang/python311/Makefile                    |  9 +++++++--
 lang/python311/files/patch-configure (new) | 11 +++++++++++
 lang/python37/Makefile                     | 10 +++++++---
 lang/python37/files/patch-configure (new)  | 11 +++++++++++
 lang/python38/Makefile                     | 10 +++++++---
 lang/python38/files/patch-configure (new)  | 11 +++++++++++
 lang/python39/Makefile                     |  9 +++++++--
 lang/python39/files/patch-configure (new)  | 11 +++++++++++
 10 files changed, 90 insertions(+), 12 deletions(-)
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2022-03-11 01:31:15 UTC
(In reply to commit-hook from comment #7)

Thanks for that Piotr, next time let's ensure a review and explicit accept prior to commit. Don't hesitate to reach out if you need support on leveling up Python ports

^Triage: Assign to committer that resolved
Comment 9 Piotr Kubaj freebsd_committer freebsd_triage 2022-03-11 02:13:15 UTC
(In reply to Kubilay Kocak from comment #8)
OK, I assumed that if you revoked your commit bit, you abandoned FreeBSD development completely.
Comment 10 Kyle Evans freebsd_committer freebsd_triage 2022-03-11 03:45:50 UTC
(In reply to Piotr Kubaj from comment #9)

That's not how that's supposed to be interpreted, no.