Bug 262495 - lang/python*: add option PGO and enable by default, add LTO option to python27 and enable by default
Summary: lang/python*: add option PGO and enable by default, add LTO option to python2...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-python (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2022-03-12 00:34 UTC by Piotr Kubaj
Modified: 2022-03-16 19:05 UTC (History)
2 users (show)

See Also:


Attachments
patch (6.24 KB, patch)
2022-03-12 00:34 UTC, Piotr Kubaj
pkubaj: maintainer-approval? (python)
Details | Diff
clang as cc detection for pgo (891 bytes, patch)
2022-03-12 03:16 UTC, Kubilay Kocak
no flags Details | Diff
pgo-clang-is-cc-v2 (1.41 KB, patch)
2022-03-12 04:16 UTC, Kubilay Kocak
no flags Details | Diff
build log (381.49 KB, text/plain)
2022-03-16 19:05 UTC, Piotr Kubaj
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Piotr Kubaj freebsd_committer freebsd_triage 2022-03-12 00:34:10 UTC
Created attachment 232396 [details]
patch

All the python ports build fine.

python27 needs PGO to use LTO.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2022-03-12 03:16:32 UTC
Created attachment 232397 [details]
clang as cc detection for pgo

Thanks Piotr

What is the nature of the dependence on PGO for LTO? 

If this is only a consequence of the 'optimization' option 'needing' both [1], we shouldn't couple ourselves to that option.

Also I don't believe we should default to PGO builds, particularly at such an early stage, as the profiling and optimizations instrumentation on the official package build hardware, will only be specific to that hardware, and may not be appropriate for, is unlikely to be, or may result in performance regressions on, any other (user) hardware.

There was also an issue identified during my testing  with Python not detecting cc (clang) for PGO [2], which requires and additional '*)' case for defaulting to 'clang style PGO'. There may be alternate ways to address this, and this was not tested in the gcc case. We wont want to carry local patches in the long term either, so working with upstream to resolve this is important.

We'll also want to test and confirm QA across all supported freebsd versions, archs, and toolchains (gcc included) explicitly, and include Python's test suite in that QA. At a minimum the test failure delta should not change.

Unfortunately, a successful build alone is insufficient for LTO/PGO, and there are many results for LTO/PGO Python segfaults and other issues at runtime out there.

[1] https://bugs.python.org/issue29766
[2] https://464982.bugs.gentoo.org/attachment.cgi?id=344902
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2022-03-12 04:16:56 UTC
Created attachment 232398 [details]
pgo-clang-is-cc-v2

Fix patch (post merge of LTO patch to configure).
Comment 3 Piotr Kubaj freebsd_committer freebsd_triage 2022-03-16 19:05:10 UTC
Created attachment 232490 [details]
build log

Yes, for python27 the bug you pointed to is the issue.
However, it being only 27-related, is of minimal impact. If that causes an issue, we could just drop LTO and PGO there.

Regarding PGO being default, I'm not sure it would cause many issues. Official binaries by many projects already use PGO. E.g. official Firefox binaries from Mozilla use PGO: https://www.phoronix.com/scan.php?page=news_item&px=Firefox-Clang-LTO-All-Platforms

Regarding testing on GCC platforms, please have a look at the attached build log. I think problems with GCC platforms are much more grave than some mere Python issues :)