Created attachment 263484 [details] acpidump -dt (zipped version, too big) Hi, We have two brand new servers with 2x Intel 6787p, 86 cores / 172 threads each, total of 344 threads. A user from FreeBSD forum has a good idea on what is happening : The problem is that an attempt is made to use APIC IDs larger than 255. With 2 CPUs, each having 86 cores and each core having 2 threads, the number of CPUs as seen by FreeBSD becomes quite large. With each CPU having its own APIC, APIC IDs larger than 255 have to be used. There are some problems reported with that. See <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288122>, <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=287492> and <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273022> And he suggested that I am opening this ticket and attaching the acpidump -dt Thanks !
Please provide the kernel panic and backtrace text here
Sorry I don't have a physical access any time soon, but I am attaching the photo of kernel panics I had and the one with kern.smp.disabled=1
Created attachment 263684 [details] panic pic1
Created attachment 263685 [details] panic pic2
Created attachment 263686 [details] panic with kern.smp.disabled=1
The first panic looks like a bug in em(4) and/or iflib where it tries to free a resource that hasn't been allocated. I wonder if the relevant structure isn't allocated with M_ZERO and iflib sees the 0xdeadc0de junk as non-NULL and tries to use it as a struct resource pointer. The second panic is also in iflib, perhaps not handling kern.smp.disabled=1 correctly?
Yes this is due to a Intel quad ports issue, we removed it and kept the Intel 4x10GbE which is working fine. I am attaching dmesg -a which is showing a bunch of APIC issues which is impacting the nvme drives, I hope it could help troubleshoot it
Created attachment 263842 [details] dmesg -a
(In reply to blt from comment #8) These apic remap errors are kind of expected. Try to enable DMAR and interrupt remapping, but disable dma support.
Thanks, can you guide me on how to test that ? The first would be hw.dmar.enable="1" but I am not sure how to enable interrupt remapping and disable dma support. Thanks
hw.dmar.enable=1 hw.iommu.dma=0 hw.iommu.ir=1
Thanks Konstantin, sorry I was not able to get access sooner. I tried to to enable DMAR & interrupt remapping and disable dma support, it makes the boot process last more than 15 minutes with a huge number of errors. I attached the latest dmesg -a
Created attachment 264565 [details] dmesg -a DMAR & interrupt remapping enabled, dma support disabled
(In reply to blt from comment #13) Can you get the verbose dmesg from the very start of the boot up to the point where NVME startup begins to complain about 'unsupported destination APIC'? I need/want to see the DMAR initialization log. I am also curious what is the pci109:0:5 which generates interrupts in compat format that were not remapped through the DMAR driver.
Konstantin do you want the logs with "DMAR & interrupt remapping enabled, dma support disabled" or without changing the default ?
(In reply to blt from comment #15) DMAR must be enabled. Try to get with both dma and ir enabled. Otherwise, at least ir must be enabled.
Konstantin full log at dmesg_full.txt (I understand why, I had to increase the buffer size) I tried to add verbose_loading="YES" and boot_verbose="YES" when I do that my server is not able to boot and reboot after a couple of minutes ! Theses are with DMAR & interrupt remapping enabled and dma support disabled. Let me know if it is enough or if you need anything else Many thanks
Created attachment 264603 [details] dmesg_full.txt
(In reply to blt from comment #18) I need verbose dmesg, even if it is not full. But it should be from the start of boot up to the point of reboot (panic ?).
Konstantin it is too fast I am not able to see what's happen remotely through IPMI.. Perhaps I can try just one, which one you need between verbose_loading="YES" and boot_verbose="YES" ?
(In reply to blt from comment #20) verbose_loading is irrelevant, it controls loader messages. Do you use serial console or its emulation over IPMI? It is the easiest, if not the only, working way to grab the console output.
I am using the Supermicro BMC "Remote Control" Launch Console through HTML5 but I can only see the screen and not capture the logs...
Would it work for you if I can do that : https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/#config-dumpdev
(In reply to blt from comment #22) You should be able to enable SoL (serial-over-LAN) console redirect in BIOS settings. Then you would be able to access the console using, e.g., ipmitool sol activate. https://www.supermicro.com/support/faqs/faq.cfm?faq=21816
Konstantin, was not easy to have physical access, and also had to tweak a little bit my loader.conf to have the right console after the boot. Latest attachement dmesg_sol.txt is boot_verbose="YES" got from SOL/IPMI with hopefuly what do you need ? (a bunch of logs and just before the panic). Don't hesitate to let me know
Created attachment 265829 [details] dmesg_sol.txt
(In reply to blt from comment #26) https://reviews.freebsd.org/D54464 show remedy the panic during the AP launch, so that you can properly get the verbose boot dmesg. Please re-try with the HEAD kernel (no need to rebuild world) together with the patch from the review applied.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=55305b590797524dd1cecfc9406869700e925e51 commit 55305b590797524dd1cecfc9406869700e925e51 Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2026-01-03 01:09:32 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2026-01-05 23:12:22 +0000 x86: change ap_boot_mtx from spinlock mutex to naive lock Problem is that the printfs done under the spinlock might be very slow, then spinlock triggers the panic due to some AP holding the mutex too long. We do want to ensure that the printfs are serialized, still. Also, reduce the scope of the lock. The local APIC and MTRR initializations are local to the AP, while mca_init() protects the shared state with mca-private spinlock. PR: 289297 Reviewed by: markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54464 sys/amd64/amd64/mp_machdep.c | 1 - sys/i386/i386/mp_machdep.c | 2 -- sys/x86/include/x86_smp.h | 1 - sys/x86/x86/mp_x86.c | 17 ++++++++++++----- 4 files changed, 12 insertions(+), 9 deletions(-)
Thanks. I am not familiar with the process, would you be able to guide me? I see that you committed the patch in main, so does that mean I just have to download and install 16.0-CURRENT (when the next one will be available) to test without any other change ?
(In reply to blt from comment #29) Something like # git clone <location of your preferred git src repo> # cd src # make -s -j $(nproc) -s buildworld buildkernel # make -s installkernel <reboot into the new kernel> FreeBSD Handbook contains detailed and correct description of the process.
(In reply to Konstantin Belousov from comment #30) It is also possible to wait for a new weekly snapshot to be available and just install from it. That said, getting experience with installing & testing kernels will be useful for the future in case Konstantin has a patch to collect additional diagnostic information or a potential fix to test prior to commit.
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=fe3001bde4849757064e8bb63fdab8b3b797dd6d commit fe3001bde4849757064e8bb63fdab8b3b797dd6d Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2026-01-03 01:09:32 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2026-01-12 04:05:08 +0000 x86: change ap_boot_mtx from spinlock mutex to naive lock PR: 289297 (cherry picked from commit 55305b590797524dd1cecfc9406869700e925e51) sys/amd64/amd64/mp_machdep.c | 1 - sys/i386/i386/mp_machdep.c | 2 -- sys/x86/include/x86_smp.h | 1 - sys/x86/x86/mp_x86.c | 17 ++++++++++++----- 4 files changed, 12 insertions(+), 9 deletions(-)
Many thanks Konstantin. I am in the DC today, so I downloaded the latest 16-CURRENT (from Jan-12) which is including your changes. The boot_verbose is not crashing anymore (and the boot without verbose is way quicker!), so you can see it in the attachment files under dmesg_sol_full.txt Just for your information, perhaps not related but I noticed that doing a service netif restart makes a kernel panic, attached as well under netif_panic.txt Let me know if you need me to do something else or if you catched something in the full log
Created attachment 267167 [details] dmesg_sol_full.txt debug verbose full
Created attachment 267168 [details] netif_panic.txt netif panic
(In reply to blt from comment #34) The latest dmesg_sol_full is from the boot with DMAR not enabled. The reported errors are expected. _Enable DMAR_ and give me the full verbose dmesg log.
Created attachment 267506 [details] dmar_full.txt part one
Created attachment 267507 [details] dmar_full.txt part two
Created attachment 267508 [details] dmar_and_iommu_full.txt part one
Created attachment 267509 [details] dmar_and_iommu_full.txt part two
Hi Konstantin, I had to split the files in two as they are >1000KB dmar_full.txt with hw.dmar.enable=1 dmar_and_iommu_full.txt with hw.dmar.enable=1 & hw.iommu.dma=0 & hw.iommu.ir=1 (don't know if you needed it) Which takes approximately 1h to finish for each scenario Let me know
Hi Konstantin, Had you have a chance to have a look? Please don't hesitate to let me know if what I sent was useful or if you need me to do further tests. Cheers
(In reply to blt from comment #42) Yes, I looked. It seems that the DMAR ACPI parser mis-matches the devices against the DMAR units. I do not have an immediate fix, will plan to work on it. FWIW, does the acpidump -dt output from the very first attachment is from the same machine and same BIOS as the verbose dmesg in the last two attachments? In other words, I am asking did you updated the BIOS between taking these two dumps?
Thanks for that. As far as I remember I updated prior to the first bug report, but I can be wrong as I made a lot of tests and I am not 100% about the sequence of events..
Hi Konstantin, Do you think this could be a candidate for a potential fix in 15.2 release ? Our servers are now in production, it would be more difficult to make some tests but we are still very interested if some improvements/fixes could be made :) Many thanks