Bug 262495

Summary: lang/python*: add option PGO and enable by default, add LTO option to python27 and enable by default
Product: Ports & Packages Reporter: Piotr Kubaj <pkubaj>
Component: Individual Port(s)Assignee: freebsd-python (Nobody) <python>
Status: Open ---    
Severity: Affects Many People CC: koobs, python
Priority: --- Keywords: needs-qa
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
patch
pkubaj: maintainer-approval? (python)
clang as cc detection for pgo
none
pgo-clang-is-cc-v2
none
build log none

Description Piotr Kubaj freebsd_committer freebsd_triage 2022-03-12 00:34:10 UTC
Created attachment 232396 [details]
patch

All the python ports build fine.

python27 needs PGO to use LTO.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2022-03-12 03:16:32 UTC
Created attachment 232397 [details]
clang as cc detection for pgo

Thanks Piotr

What is the nature of the dependence on PGO for LTO? 

If this is only a consequence of the 'optimization' option 'needing' both [1], we shouldn't couple ourselves to that option.

Also I don't believe we should default to PGO builds, particularly at such an early stage, as the profiling and optimizations instrumentation on the official package build hardware, will only be specific to that hardware, and may not be appropriate for, is unlikely to be, or may result in performance regressions on, any other (user) hardware.

There was also an issue identified during my testing  with Python not detecting cc (clang) for PGO [2], which requires and additional '*)' case for defaulting to 'clang style PGO'. There may be alternate ways to address this, and this was not tested in the gcc case. We wont want to carry local patches in the long term either, so working with upstream to resolve this is important.

We'll also want to test and confirm QA across all supported freebsd versions, archs, and toolchains (gcc included) explicitly, and include Python's test suite in that QA. At a minimum the test failure delta should not change.

Unfortunately, a successful build alone is insufficient for LTO/PGO, and there are many results for LTO/PGO Python segfaults and other issues at runtime out there.

[1] https://bugs.python.org/issue29766
[2] https://464982.bugs.gentoo.org/attachment.cgi?id=344902
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2022-03-12 04:16:56 UTC
Created attachment 232398 [details]
pgo-clang-is-cc-v2

Fix patch (post merge of LTO patch to configure).
Comment 3 Piotr Kubaj freebsd_committer freebsd_triage 2022-03-16 19:05:10 UTC
Created attachment 232490 [details]
build log

Yes, for python27 the bug you pointed to is the issue.
However, it being only 27-related, is of minimal impact. If that causes an issue, we could just drop LTO and PGO there.

Regarding PGO being default, I'm not sure it would cause many issues. Official binaries by many projects already use PGO. E.g. official Firefox binaries from Mozilla use PGO: https://www.phoronix.com/scan.php?page=news_item&px=Firefox-Clang-LTO-All-Platforms

Regarding testing on GCC platforms, please have a look at the attached build log. I think problems with GCC platforms are much more grave than some mere Python issues :)