Please test pkg 2.1.99.1 in the cluster environment, my guess on the performance penalty introduce in pkg 2.1.0 which is highly dependent on how many build we have in parallel is the pressure on memory. I have run some local tests using metasploit in particular and my results are the following: with pkg 2.0: build takes 1m57 with pkg 2.1: build takes 2m20 with pkg 2.1.00.1: build takes 1m58 (and this is with debug flags on) I would like to see how it performs in the cluster.
I have released 2.1.99.2 to address an important regression, should be ok now.
I just launched the exp-run. So far, it looks a lot better on random ports: ==> 2025-04-11_14h44m25s/logs/rubygem-aws-sdk-elasticsearchservice-1.100.0.log <== build time: 00:02:14 ==> 2025-04-17_19h35m10s/logs/rubygem-aws-sdk-elasticsearchservice-1.100.0.log <== build time: 00:00:09 ==> 2025-04-11_14h44m25s/logs/rubygem-web-console-rails80-4.2.1.log <== build time: 00:05:06 ==> 2025-04-15_12h37m38s/logs/rubygem-web-console-rails80-4.2.1.log <== build time: 00:01:54
(In reply to Antoine Brodin from comment #2) FYI for performance of pkg 2.1.0 vs. normal: Early package build and long running "build" step package builds do not make for great data for pkg 2.1.0 being involved and its time-taking issue. Later package builds with more dependencies took more time in build-depends, lib-depends, and run-depends, even for "small build time" package builds. But, also true in my experiments, was that as the number of packages already built grew, the times for those 3 grew, even for packages not having lots of dependencies: lots of time was spent in those 3 steps anyway. This was visible in top by later builders that had short "build" step times running for long times in one or more of these steps. Most builders most of the time would show such later in the overall "bulk -a" activity. After the full "bulk -a" context was in place, I could do single builder bulk runs with "bulk -J1 -v -C EXAMPLE/ORIGIN" and see the long times for the 3 *-depends steps mentioned.
(In reply to Mark Millard from comment #3) I should have mentioned that times were noticeably longer when the 15000?? kernel in use was a debug build kernel, but long time were present with a kernel-NODEBUG as well. This was true in my experiments. main-aarch64 on ampere2, which uses a debug kernel as far as I can tell, has been running is 423:50:45 at with over 3144 remaining as of when I just looked, so part way through day 17 of the "bulk -c -a" like run. (Probably the worst-case official builder context for pkg 2.1.0 use.)
(In reply to Mark Millard from comment #4) Information for later comparison: 141amd64PR284529 2025-02-03_14h15m15s used pkg 2.0.5 on gohan06 and had: Queued 36418 Built 36058 Skipped 32 Elapsed 51:15:54 142amd64 2025-04-05_06h57m27s used pkg 2.1.0 on gohan06 and had: Queued 36452 Built 34776 Skipped 1285 Elapsed 61:31:21 (The large skipped was during the lang/go* and www/webkit2-gtk@40 build failures.)
(In reply to Mark Millard from comment #4) Typo: "so part way through day 17" should have been: "so part way through day 18" (Day 17 was finished already.) Note: It has finished day 19 and is part way into day 20 --and has 1688 to go as stands for ampere2's main-arm64 .
(In reply to Mark Millard from comment #5) 142amd64 2025-04-17_19h35m10ss used pkg 2.1.99.2 on gohan06 and had: Queued 36498 Built 36055 Skipped 68 Elapsed 62:10:34 That as vs. pkg 2.1.0 building lots fewer in: 142amd64 2025-04-05_06h57m27s used pkg 2.1.0 on gohan06 and had: Queued 36452 Built 34776 Skipped 1285 Elapsed 61:31:21 So improved vs. pkg 2.1.0. But compared to 2.0.5 (14.1 instead of 14.2 context): 141amd64PR284529 2025-02-03_14h15m15s used pkg 2.0.5 on gohan06 and had: Queued 36418 Built 36058 Skipped 32 Elapsed 51:15:54 So somewhat under 11 hr longer. It will be interesting to see what 142amd64 on ampere2 does with the 2.1.99.2 like code. Based on pkg 2.1.0 and a debug 150035 kernel for Host (and 1500035 world for Jail) it is at: Queued 36447 Built 33513 Skipped 406 Remaining 1356 Elapsed 469:00:40 as of when I just looked. So somewhat over 19.5 days so far.
Exp-run seems fine security/metasploit goes from 00:49:26 to 00:22:29 so it's better than with 2.1.0
ok thanks I will then make a 2.1.1 with this changes
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=2bb7e518317c68231c43d382180239be809e2928 commit 2bb7e518317c68231c43d382180239be809e2928 Author: Baptiste Daroussin <bapt@FreeBSD.org> AuthorDate: 2025-04-23 13:57:09 +0000 Commit: Baptiste Daroussin <bapt@FreeBSD.org> CommitDate: 2025-04-23 13:59:25 +0000 ports-mgmt/pkg: update to 2.1.1 Changes from 2.1.0 to 2.1.1 - fix cases where upgrade detection was missed - improvement performances for pkg add (aka poudriere build time as well) PR: 286130 Exp-run: antoine ports-mgmt/pkg/Makefile | 2 +- ports-mgmt/pkg/distinfo | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
(In reply to commit-hook from comment #10) Given that with pkg 2.1.0 in use: ) beefy18's main-adm64 from-scratch builds take the likes of 287:14:05 (a little under 12 days) instead of the likes of 96:07:25 ) beefy16's 134amd64-default from-scratch builds take the likes of 167:34:49 (a little under 7 days) instead of the likes of 109:34:31 ) beefy14's 134amd64-quarterly from-scratch builds take the likes of 166:14:32 (a little under 7 days) instead of the likes of 105:11:15 ) ampere2's main-arm64 from-scratch builds take the likes of 508:02:38 (a little over 21 days) instead of the likes of 163:20:35 ) and so on and so on, I'd recommend that existing builder activity be stopped and new builders started that are based on using pkg 2.1.1 instead. Who is the right person to ask about having such done?
In trying to update my normal ports environment, I'm getting\ huge pkg-static memory use: PID JID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 47564 324 root 45 0 121577Mi 17271Mi pfault 31 1:02 31.40% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 52346 305 root 45 0 95847Mi 16669Mi pfault 2 0:53 31.40% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 47630 295 root 47 0 121577Mi 16519Mi pfault 4 1:01 31.39% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 50134 327 root 47 0 114281Mi 16181Mi pfault 23 1:01 31.39% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 48303 311 root 47 0 121577Mi 15929Mi pfault 0 1:00 31.38% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 49894 300 root 47 0 114281Mi 15670Mi pfault 7 1:00 31.33% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 49751 297 root 45 0 114281Mi 15657Mi pfault 8 0:59 31.34% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 48416 299 root 45 0 114409Mi 15604Mi pfault 26 1:01 31.33% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 47876 312 root 47 0 114281Mi 15447Mi pfault 10 1:00 31.42% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 46970 315 root 45 0 114281Mi 15368Mi pfault 20 1:00 31.41% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg 48702 316 root 47 0 114281Mi 15179Mi pfault 29 0:59 31.42% /usr/local/sbin/pkg-static add -A /packages/All/gettext-tools-0.23.1.pkg . . .
I did: # bulk -jmain-ZNV4 -c -f ~/origins/ZNV4-origins.txt . . . [00:00:03] -c specified, cleaning all packages... done . . . [00:00:04] [01] [00:00:00] Building ports-mgmt/pkg | pkg-2.1.1 [00:00:42] [01] [00:00:38] Finished ports-mgmt/pkg | pkg-2.1.1: Success . . . but ended up with the recursive package install again: =======================<phase: build-depends >============================ ===== env: USE_PACKAGE_DEPENDS_ONLY=1 USER=root UID=0 GID=0 ===> coreutils-9.7 depends on package: gettext-runtime>=0.22_1 - not found ===> Installing existing package /packages/All/gettext-runtime-0.23.1.pkg [ZNV4optb_ZFS] Installing gettext-runtime-0.23.1... [ZNV4optb_ZFS] `-- Installing indexinfo-0.3.1_1... [ZNV4optb_ZFS] `-- Extracting indexinfo-0.3.1_1: .... done [ZNV4optb_ZFS] Extracting gettext-runtime-0.23.1: .......... done ===> coreutils-9.7 depends on package: gettext-runtime>=0.22_1 - found ===> Returning to build of coreutils-9.7 ===> coreutils-9.7 depends on executable: msgfmt - not found ===> Installing existing package /packages/All/gettext-tools-0.23.1.pkg [ZNV4optb_ZFS] Installing gettext-tools-0.23.1... [ZNV4optb_ZFS] `-- Installing libtextstyle-0.23.1... [ZNV4optb_ZFS] `-- Extracting libtextstyle-0.23.1: .......... done [ZNV4optb_ZFS] `-- Installing gettext-tools-0.23.1... [ZNV4optb_ZFS] | `-- Installing gettext-tools-0.23.1... [ZNV4optb_ZFS] | | `-- Installing gettext-tools-0.23.1... . . .
(In reply to Mark Millard from comment #13) On the aarch64 system, I did not use -c . I've not seen the problem there so far. It also is based on: . . . [00:00:09] [01] [00:00:00] Building ports-mgmt/pkg | pkg-2.1.1 [00:00:47] [01] [00:00:38] Finished ports-mgmt/pkg | pkg-2.1.1: Success . . .
(In reply to Mark Millard from comment #14) I have replicated the problem on the aarch64 system via removal of the old, previously built (Feb), .pkg and then trying: # poudriere bulk -jmain-CA76 -i devel/gettext-tools When it attempted to install gettext-tools for the interactive session after the build, it got the recursive installation problem. I'm reverting my /usr/ports/ports-mgmt/pkg/ materials to pkg 2.0.6 content.
(In reply to commit-hook from comment #10) The 2.1.1 comment does not lead to including the recursive install fix . . . https://github.com/freebsd/pkg/commits/2.1.1/ shows: Release 2.1.1 (as c3a391e431b788500ace2da21259389d077aa325) prevent undefined behaviour (detected by UBSAN) (as d3bc2b95860d06192aaebb4e7ff20e1137f7f004) It does not show an: abi: fix shlibs_required skipped by accident By contrast: https://github.com/freebsd/pkg/commits/main/ does show that abi fix (as 01165121d076dfd090b101ce2915d786fea85381) It also shows an earlier: prevent undefined behaviour (detected by UBSAN) (as f8eb92999ee8e0509ceef357329c32517dca6436)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=3e75105eed521939cf0d39daee0ea4f30e068e35 commit 3e75105eed521939cf0d39daee0ea4f30e068e35 Author: Baptiste Daroussin <bapt@FreeBSD.org> AuthorDate: 2025-04-24 03:44:50 +0000 Commit: Baptiste Daroussin <bapt@FreeBSD.org> CommitDate: 2025-04-24 03:44:50 +0000 ports-mgmt/pkg: update to 2.1.2 Changes: - fix an important regression introduced in 2.1.1 - more performance improvement on pkg add Because of a bad merge between the main and the release branch of pkg some commits were missed and a regression was introduced in pkg 2.1.1 It results in some packages dependending on themselves which breaks early the various bulks. Those packages built with 2.1.1 needs to be recreated with 2.1.2 PR: 286130 ports-mgmt/pkg/Makefile | 2 +- ports-mgmt/pkg/distinfo | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
(In reply to commit-hook from comment #17) pkg 2.1.2 looks to be working, building what pkg 2.1.1 failed to for both amd64 and aarch64 . . . amd64: # poudriere bulk -jmain-ZNV4 -v -c -f ~/origins/ZNV4-origins.txt got: [01:12:02] [main-ZNV4-default] [2025-04-23_23h07m24s] [committing] Time: 01:12:00 Queued: 452 Inspected: 0 Ignored: 1 Built: 438 Failed: 3 Skipped: 10 Fetched: 0 Remaining: 0 where the 3 failures: [01:12:02] Failed ports: devel/linux-ltp:fetch lang/gcc14:build graphics/openexr:package were expected. aarch64: poudriere bulk -jmain-CA76 -v -c -f ~/origins/CA76-origins.txt got: [01:16:17] [main-CA76-default] [2025-04-23_23h06m47s] [committing] Time: 01:16:16 Queued: 266 Inspected: 0 Ignored: 0 Built: 264 Failed: 1 Skipped: 1 Fetched: 0 Remaining: 0 where the 1 failure: [01:16:17] Failed ports: lang/gcc14:build was expected. (A clang specific compiler option was involved in the build experiments.)
(In reply to commit-hook from comment #17) ports r2025Q2 has ports-mgmt/pkg 2.1.0, which can make the "bulk -c -a" like runs take up much longer than normal. For example, for quarterly,: 142amd64 building 34702 in 45:21:29 using pkg 1.21.3 vs. 142amd64 building 26082 in 61:43:35 using pkg 2.1.0 I'd recommend that 2025Q2 get pkg 2.1.2 to avoid this.
(In reply to Mark Millard from comment #19) Hmm: Intended was no "r" in front of "2025Q2"
(In reply to commit-hook from comment #17) beefy15's "bulk -c -a" like from=-scratch builds of 134i386-default (latest) do not show a notable difference between pkg 2.1.2 and 2.1.0: beefy15, i386 13.4 latest pkg 2.1.2 then 2.1.0: 4de34b986043 36505 34433 110 893 1069 0 done: Thu, 24 Apr 2025 07:19:31 GMT 141:11:49 7a82a8f2ddcc 36495 34424 101 893 1077 0 done: Thu, 17 Apr 2025 01:03:15 GMT 143:23:13 pkg 2.0.5 then 1.21.3: 182ff2d0ad1b 36393 34458 79 838 1018 0 done: Thu, 30 Jan 2025 01:02:37 GMT 87:51:54 6e326af2b884 36295 34476 89 712 1018 0 done: Thu, 02 Jan 2025 01:03:46 GMT 86:23:37
A commit in branch 2025Q2 references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=752409f0936f97604a2e80a8359016dd38277111 commit 752409f0936f97604a2e80a8359016dd38277111 Author: Baptiste Daroussin <bapt@FreeBSD.org> AuthorDate: 2025-04-23 13:57:09 +0000 Commit: Baptiste Daroussin <bapt@FreeBSD.org> CommitDate: 2025-04-30 08:43:39 +0000 ports-mgmt/pkg: update to 2.1.2 Changes from 2.1.0 to 2.1.1 - fix cases where upgrade detection was missed - improvement performances for pkg add (aka poudriere build time as well) Changes from 2.1.1 to 2.1.2 - fix an important regression introduced in 2.1.1 - more performance improvement on pkg add PR: 286130 Exp-run: antoine (cherry picked from commit 2bb7e518317c68231c43d382180239be809e2928) (cherry picked from commit 3e75105eed521939cf0d39daee0ea4f30e068e35) ports-mgmt/pkg/Makefile | 2 +- ports-mgmt/pkg/distinfo | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
(In reply to Mark Millard from comment #21) beefy16's "bulk -c -a" like from-scratch builds of 134amd64-default (latest) show a potential small improvement between pkg 2.1.2 and 2.1.0: beefy16, amd64 13.4 latest pkg 2.1.2 then 2.1.0: dc1d367f2961 36505 35999 80 104 322 0 done: Thu, 24 Apr 2025 07:24:46 GMT 159:17:19 2e18ed262338 36486 35978 72 110 326 0 done: Sat, 12 Apr 2025 01:05:21 GMT 167:34:49 pkg 2.0.6. then 2.0.5 then 1.21.3: 6ec2dc26dc50 36464 35977 47 106 334 0 done: Thu, 27 Feb 2025 01:08:57 GMT 109:34:31 7b2be625f599 36402 35961 55 92 294 0 done: Sat, 01 Feb 2025 01:07:52 GMT 112:05:39 5659b8dd8870 36295 35894 54 45 302 0 done: Thu, 02 Jan 2025 01:03:35 GMT 109:19:49 (It is going to be some time before there is such a comparison for an ampere* .)
Some notes on the non-linear nature of poudriere(-devel) "bulk -c -a" v builds and others that are nearly from-scratch full builds. The notes use the beefy18 (main-amd64-default) context as an example of the more general issue. (My notes are only about how things are, not about future planned changes for how things will work.) beefy18's most recent completed build of packages started on Tue, 15 Apr 2025 01:22:32 GMT and took 292:07:47, so a little under 12.2 days. 36494 package builds were queued and 35892 actually built. (162 failed to build. The rest were skipped or ignored.) That build started before pkg 2.1.2 was available. The ongoing build is based on pkg 2.1.2 and has taken 134:30:55 so far. It started on Tue, 29 Apr 2025 01:22:26 GMT. It queued 36512 and has built 22222 (and 79 have failed so far). 13758 Remain for it to try to build. Some have been skipped or ignored. I'm expecting something like 292 hrs to 300 hrs for everything to have completed. So far it looks like pkg 2.1.0 vs. 2.1.2 make little difference in the overall time for such builds. I decided to submit these notes before the eventual confirmation/denial of my expectations. Because of how pkg works, it takes more time to build the last 13758 than it takes to build those first 22222: more time analyzing dependencies and already built packages. In part that is because of more dependencies being involved (mean rates per package) and (if I understand right) also because of more already built local packages existing, even ones that are not dependencies for the specific pkg being built. Some of time scale is related to beefy18 configuration choices that fit better before pkg 2.1.* got involved, such as having only 13 builders active at most but up to 3 processes per builder. The build-depends, lib-depends, and run-depends stages of pkg that are taking so long later in the overall build sequence use only 1 "FreeBSD cpu". So most of the cpus are idle during this when most of the builds are in one of those 3 stages most of the time. The 10s of thousands of not-large pkg builds now take more overall time than the comparatively few huge-pkg builds do for the style of configuration used on beefy18 (and many other builder systems).
ampere2 builds of main for aarch64 and armv7: main-arm64-default with pkg 2.1.0 then 2.0.6: 36447 34856 173 406 1012 0 done: 508:02:38 36466 34853 174 427 1012 0 done: 163:20:35 main-armv7-default with pkg 2.1.0 then 2.0.6: 36447 31041 269 3382 1755 0 done: 319:12:09 36409 30985 259 3421 1744 0 done: 94:23:30 So a complete cycle of the two pkg 2.1.0 based builds was somewhat under 34.5 days, ignoring non-build-time between builds. Also, the build times were somewhat more than 3 times what they were when pkg 2.0.6 was in use. (The times for pkg 2.1.0 are also worse than linear vs. the number of packages built.)
beefy17's "bulk -c -a" like from-scratch builds of main-i386 (latest) shows a potential small improvement between pkg 2.1.2 and 2.1.0: beefy17, main-i386 13.4 latest pkg 2.1.2 then 2.1.0: pc49d6c914459_s6014596899c 36506 34342 186 889 1089 0 done: Sun, 27 Apr 2025 01:18:06 GMT 276:03:45 pe3d1564d6c13_s992b18a9ec9 36494 34332 178 883 1101 0 done: Tue, 15 Apr 2025 01:20:15 GMT 285:14:57 pkg 2.0.6. then 2.0.5 then 1.21.3: p9229caa5b2ac_s780a4667bbd 36461 34299 156 912 1094 0 done: Sat, 08 Mar 2025 01:02:25 GMT 71:52:11 p5beddb013d54_s078e8b34b13 36379 34280 153 872 1074 0 done: Tue, 04 Feb 2025 01:04:12 GMT 70:38:09 pa921cfa68370_s0f7d8b71b45 36331 34364 167 748 1052 0 done: Sun, 05 Jan 2025 01:04:08 GMT 70:45:36 Note: 276:03:45 is 33072 sec shorter than 285:14:57 despite building 10 more packages: a mean time of somewhat under 1 sec less per built package, somewhat over 9 hrs total.
ampere3's "bulk -c -a" like from-scratch builds of 142arm64-default or 141arm64-default ampere3, 142arm64-default 14.2 latest pkg 2.1.0: (It appears that it will be around a month before there is a pkg 2.1.2 based ampere3 142arm64-default build finished. So I'm not waiting for that here.) 2e18ed262338 36486 35001 108 385 992 0 done: Tue, 29 Apr 2025 08:19:51 GMT 244:49:20 pkg 2.0.6. then 2.0.5 then 1.21.3 (all 141arm64-default): 413ee6de2f09 36459 34959 69 431 1000 0 done: Tue, 04 Mar 2025 16:36:30 GMT 136:01:11 5beddb013d54 36379 34472 60 843 1004 0 done: Thu, 13 Feb 2025 06:02:03 GMT 124:50:31 203aa1081a24 36122 34617 82 443 980 0 done: Sun, 17 Nov 2024 18:22:02 GMT 140:54:46 Note: 244:49:20 vs. 136:01:11 suggests time scale factors of, say, around 1.8 for pkg 2.1.0 involved vs. earlier pkg versions.
beefy18's "bulk -c -a" like from-scratch builds of main-amd64 (latest) shows a potential small improvement between pkg 2.1.2 and 2.1.0: beefy18, main-amd64 latest pkg 2.1.2 then 2.1.0: (beefy17's "13.4" also being listed was a misedit.) pcc14b952338d_s34b43f4b26e 36512 35886 173 114 339 0 done: Tue, 29 Apr 2025 01:22:26 GMT 281:05:33 pe3d1564d6c13_s992b18a9ec9 36494 35892 162 95 345 0 done: Tue, 15 Apr 2025 01:22:32 GMT 292:07:47 pkg 2.0.6. then 2.0.5 then 1.21.3: p9229caa5b2ac_s780a4667bbd 36461 35771 152 186 352 0 done: Sat, 08 Mar 2025 01:02:42 GMT 96:28:10 p5beddb013d54_s078e8b34b13 36379 35812 145 76 346 0 done: Tue, 04 Feb 2025 01:05:05 GMT 96:07:25 pa921cfa68370_s0f7d8b71b45 36331 35414 153 426 338 0 done: Sun, 05 Jan 2025 01:04:57 GMT 92:22:59 Notes: pkg 2.1.2 based example near the 50% of the elapsed time: 36512 22222 79 ??? ??? 13758 parallel_build: Tue, 29 Apr 2025 01:22:26 GMT 134:30:55 From 134:30:55 to 281:05:33 spans a little over 52% of the 281:05:33 From 22222 to 34856 spans a little over 36% of the 34856. So, roughly, the last 52% of the time (pkgrepo: stage included) spanned just the last 36% of the package builds finishing. So, then, about 64% of the packages finished during the first 48% of the time. Crudely: About the first 2/3rds of the builds took about 1/2 the time. About the last 1/3rd of the builds took about 1/2 the time. Also: 292:07:47 - 281:05:33 = 11:02:14, which suggests that pkg 2.1.2 was an improvement over pkg 2.1.0 but there are other sources of variability.