MAXCPU=256 is too low for contemporary high end server hardware. I am opening this PR to track the dependencies required to increase MAXCPU.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e72f7ed43eefaf305c33c232bc2c33d997427f58 commit e72f7ed43eefaf305c33c232bc2c33d997427f58 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-04-26 14:09:31 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-04-26 14:09:31 +0000 buf: Dynamically allocate per-CPU buffer queues To reduce static bloat. No functional change intended. PR: 269572 Reviewed by: mjg, kib, emaste MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39808 sys/kern/vfs_bio.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=74ac712f72cfd6d7b3db3c9d3b72ccf2824aa183 commit 74ac712f72cfd6d7b3db3c9d3b72ccf2824aa183 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-04-26 14:08:42 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-04-26 14:08:42 +0000 vmm: Dynamically allocate a couple of per-CPU state save areas This avoids bloating the BSS when MAXCPU is large. No functional change intended. PR: 269572 Reviewed by: corvink, rew Tested by: rew MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39805 sys/amd64/vmm/amd/svm.c | 11 ++++++----- sys/amd64/vmm/intel/vmx.c | 12 ++++++++---- 2 files changed, 14 insertions(+), 9 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=78cfa762ebf2afc821e49e41c7bc72cc9044c233 commit 78cfa762ebf2afc821e49e41c7bc72cc9044c233 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-04-26 14:09:09 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-04-26 14:09:09 +0000 callout: Move per-CPU callout state into the dpcpu region This eliminates some static bloat in amd64 kernels and reduces the penalty of increasing MAXCPU. The structures now also maintain NUMA affinity. No functional change intended. PR: 269572 Reviewed by: mjg, kib MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39807 sys/kern/kern_timeout.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=76887e84be975698b14699d7d0dfb157d73e9990 commit 76887e84be975698b14699d7d0dfb157d73e9990 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-05-02 20:57:12 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-05-08 18:25:15 +0000 cpuset: increase userland maximum size to 1024 Hardware with more than 256 CPU cores is now available and will become increasingly common. Bump CPU_MAXSIZE (used for userland cpuset_t sizing) to 1024 to define the ABI for FreeBSD 14. PR: 269572, 271213 [exp-run] Reviewed by: mjg, jhb Relnotes: yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39941 sys/sys/_cpuset.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ea6dd3d1d4519c2798f7417c826afd6739b84383 commit ea6dd3d1d4519c2798f7417c826afd6739b84383 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-05-09 13:40:27 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-05-09 17:05:00 +0000 Revert "cpuset: increase userland maximum size to 1024" This reverts commit 76887e84be975698b14699d7d0dfb157d73e9990. struct vm_exit currently requires that cpuset_t be identical in userland and kernel. This will be recommitted after these are decoupled. PR: 271330, 269572 Reported by: corvink Sponsored by: The FreeBSD Foundation sys/sys/_cpuset.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=aec3c884057ab3f95e8c88a346694966df637303 commit aec3c884057ab3f95e8c88a346694966df637303 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-05-10 14:05:32 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-05-11 13:32:32 +0000 buf: Dynamically allocate per-CPU buffer queues To reduce static bloat. No functional change intended. PR: 269572 Reviewed by: mjg, kib, emaste MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39808 (cherry picked from commit e72f7ed43eefaf305c33c232bc2c33d997427f58) sys/kern/vfs_bio.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=0a300a95c10f079d4735c9c13bd6bea90402a87a commit 0a300a95c10f079d4735c9c13bd6bea90402a87a Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-05-10 13:19:09 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-05-11 13:32:32 +0000 vmm: Dynamically allocate a couple of per-CPU state save areas This avoids bloating the BSS when MAXCPU is large. No functional change intended. PR: 269572 Reviewed by: corvink, rew Tested by: rew MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39805 (cherry picked from commit 74ac712f72cfd6d7b3db3c9d3b72ccf2824aa183) sys/amd64/vmm/amd/svm.c | 11 ++++++----- sys/amd64/vmm/intel/vmx.c | 12 ++++++++---- 2 files changed, 14 insertions(+), 9 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d1639e43c589644510198dc53aef87e8908a1507 commit d1639e43c589644510198dc53aef87e8908a1507 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-05-02 20:57:12 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-05-24 14:59:05 +0000 cpuset: increase userland maximum size to 1024 Hardware with more than 256 CPU cores is now available and will become increasingly common. Bump CPU_MAXSIZE (used for userland cpuset_t sizing) to 1024 to define the ABI for FreeBSD 14. This change is reapplied after a change to decouple cpuset from bhyve: commit e17eca327633 ("vmm: Avoid embedding cpuset_t ioctl ABIs"). PR: 269572, 271213 [exp-run] Reviewed by: mjg, jhb Relnotes: yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39941 (cherry picked from commit 76887e84be975698b14699d7d0dfb157d73e9990) sys/sys/_cpuset.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=9fb6718d1b180c341373b25c03dbfc221ddee311 commit 9fb6718d1b180c341373b25c03dbfc221ddee311 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-04-25 16:09:24 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-05-25 22:09:55 +0000 smp: Dynamically allocate the stoppcbs array This avoids bloating the kernel image when MAXCPU is large. A follow-up patch for kgdb and other kernel debuggers is needed since the stoppcbs symbol is now a pointer. Bump __FreeBSD_version so that debuggers can use osreldate to figure out how to handle stoppcbs. PR: 269572 MFC after: never Reviewed by: mjg, emaste Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39806 sys/amd64/include/kdb.h | 2 -- sys/arm/arm/mp_machdep.c | 1 - sys/arm/include/kdb.h | 2 -- sys/arm/include/smp.h | 3 --- sys/arm64/arm64/mp_machdep.c | 2 -- sys/arm64/include/smp.h | 3 --- sys/i386/include/kdb.h | 2 -- sys/kern/subr_kdb.c | 6 +++--- sys/kern/subr_smp.c | 7 +++++++ sys/powerpc/include/kdb.h | 2 -- sys/powerpc/include/smp.h | 2 -- sys/powerpc/powerpc/mp_machdep.c | 1 - sys/riscv/include/kdb.h | 2 -- sys/riscv/include/smp.h | 2 -- sys/riscv/riscv/mp_machdep.c | 2 -- sys/sys/param.h | 2 +- sys/sys/smp.h | 3 +++ sys/x86/include/x86_smp.h | 1 - sys/x86/x86/mp_x86.c | 1 - 19 files changed, 14 insertions(+), 32 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=9051987e40c5a1f2396c5ceecff119fb4da28160 commit 9051987e40c5a1f2396c5ceecff119fb4da28160 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-09-30 12:14:22 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-08-03 21:41:26 +0000 amd64: Bump MAXCPU to 1024 (from 256) Hardware with more than 256 CPU cores is currently available and will become increasingly common over FreeBSD 14's lifetime. Increase MAXCPU in the amd64 GENERIC kernel configuration to 1024. Earlier commits increased some related limits. These prerequisite commits include at least: - d7ed40243769 Increase MAX_APIC_ID safeguard to 0x800 - d1639e43c589 cpuset: increase userland maximum size to 1024 Global and allocated arrays sized by MAXCPU result in excessive bloat on systems with lower core counts. In addition, some code used u_char (8 bits) to hold a CPU index, which is not valid if MAXCPU is greater than 256. A number of recent commits addressed these sorts of issues, including at least: - 133935d26f20 pf: atomically increment state ids - 74ac712f72cf vmm: Dynamically allocate a couple of per-CPU state save areas - 78cfa762ebf2 callout: Move per-CPU callout state into the dpcpu region - 42f722e721cd amd64: store pcids pmap data in pcpu zone - 9801e7c275f6 smp_topo: dynamically allocate group array - 9fb6718d1b18 smp: Dynamically allocate the stoppcbs array - 2bb16c635249 x86: retire use of intr_bind There are some additional allocations still to be converted and more scalability work is required to make effective use of very high core count systems, but this change allows us to boot on these systems and provides a Kernel Binary Interface (KBI) for the FreeBSD 14 release that supports these configurations. Special thanks to AMD for providing hardware to test these changes. PR: 269572 Reviewed by: des Relnotes: Yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D36838 UPDATING | 4 ++++ sys/amd64/include/param.h | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-)
Example of related functionality that would benefit from automatic CPU count tuning: commit 6da16e3eb002c469593826e1701a44e55b7a9657 Author: Andrew Gallatin <gallatin@FreeBSD.org> Date: Fri Jun 12 18:41:12 2020 +0000 x86: Bump default msi/msix vector limit to 2048 ... This bump costs 12KB on amd64 (and 6KB on i386), which seems worth the trade off for a better out of the box experience on high end hardware.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e0c6e8910898e49c3f83a0131dbdb1a9e774de44 commit e0c6e8910898e49c3f83a0131dbdb1a9e774de44 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-08-03 21:47:06 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-08-15 12:22:02 +0000 arm64: increase MAXCPU to 1024, following amd64 As in commit 9051987e40c5 for amd64, support up to 1024 CPU cores. arm64 hardware with more than 256 CPU cores is currently available and will become increasingly common over FreeBSD 14's lifetime. PR: 269572 Reviewed by: andrew Relnotes: Yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D41319 sys/arm64/include/param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)