pciconf -lv:

hostb0@pci0:0:0:0:  class=0x060000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2020 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E DMI3 Registers'
    class      = bridge
    subclass   = HOST-PCI
ioat0@pci0:0:4:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2021 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
ioat1@pci0:0:4:1:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2021 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
ioat2@pci0:0:4:2:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2021 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
ioat3@pci0:0:4:3:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2021 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
ioat4@pci0:0:4:4:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2021 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
ioat5@pci0:0:4:5:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2021 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
ioat6@pci0:0:4:6:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2021 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
ioat7@pci0:0:4:7:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2021 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CBDMA Registers'
    class      = base peripheral
none0@pci0:0:5:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2024 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E MM/Vt-d Configuration Registers'
    class      = base peripheral
none1@pci0:0:5:2:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2025 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS'
    class      = base peripheral
ioapic0@pci0:0:5:4: class=0x080020 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2026 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E IOAPIC'
    class      = base peripheral
    subclass   = interrupt controller
none2@pci0:0:8:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2014 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = base peripheral
none3@pci0:0:8:1:   class=0x110100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2015 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = dasp
    subclass   = performance counters
none4@pci0:0:8:2:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2016 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Ubox Registers'
    class      = base peripheral
xhci0@pci0:0:20:0:  class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x8086 device=0xa2af subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series/Z370 Chipset Family USB 3.0 xHCI Controller'
    class      = serial bus
    subclass   = USB
none5@pci0:0:20:2:  class=0x118000 rev=0x00 hdr=0x00 vendor=0x8086 device=0xa2b1 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH Thermal Subsystem'
    class      = dasp
none6@pci0:0:22:0:  class=0x078000 rev=0x00 hdr=0x00 vendor=0x8086 device=0xa2ba subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH CSME HECI'
    class      = simple comms
ahci0@pci0:0:23:0:  class=0x010400 rev=0x00 hdr=0x00 vendor=0x8086 device=0x2826 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'SATA Controller [RAID Mode]'
    class      = mass storage
    subclass   = RAID
pcib1@pci0:0:28:0:  class=0x060400 rev=0xf0 hdr=0x01 vendor=0x8086 device=0xa290 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:28:6:  class=0x060400 rev=0xf0 hdr=0x01 vendor=0x8086 device=0xa296 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:29:0:  class=0x060400 rev=0xf0 hdr=0x01 vendor=0x8086 device=0xa298 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:  class=0x060100 rev=0x00 hdr=0x00 vendor=0x8086 device=0xa2d3 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'C422 Chipset LPC/eSPI Controller'
    class      = bridge
    subclass   = PCI-ISA
none7@pci0:0:31:2:  class=0x058000 rev=0x00 hdr=0x00 vendor=0x8086 device=0xa2a1 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series/Z370 Chipset Family Power Management Controller'
    class      = memory
hdac0@pci0:0:31:3:  class=0x040300 rev=0x00 hdr=0x00 vendor=0x8086 device=0xa2f0 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH HD Audio'
    class      = multimedia
    subclass   = HDA
ichsmb0@pci0:0:31:4:    class=0x0c0500 rev=0x00 hdr=0x00 vendor=0x8086 device=0xa2a3 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = '200 Series/Z370 Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
em0@pci0:0:31:6:    class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x15e3 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection (5) I219-LM'
    class      = network
    subclass   = ethernet
pcib3@pci0:2:0:0:   class=0x060400 rev=0x00 hdr=0x01 vendor=0x104c device=0x8240 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Texas Instruments'
    device     = 'XIO2001 PCI Express-to-PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
ix0@pci0:4:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 subvendor=0x8086 subdevice=0x0001
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller X550'
    class      = network
    subclass   = ethernet
ix1@pci0:4:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 subvendor=0x8086 subdevice=0x0001
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller X550'
    class      = network
    subclass   = ethernet
none8@pci0:22:5:0:  class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2034 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E VT-d'
    class      = base peripheral
none9@pci0:22:5:2:  class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2035 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic1@pci0:22:5:4:    class=0x080020 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2036 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E IOxAPIC Configuration Registers'
    class      = base peripheral
    subclass   = interrupt controller
none10@pci0:22:8:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none11@pci0:22:8:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none12@pci0:22:8:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none13@pci0:22:8:3: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none14@pci0:22:8:4: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none15@pci0:22:8:5: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none16@pci0:22:8:6: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none17@pci0:22:8:7: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none18@pci0:22:9:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none19@pci0:22:9:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none20@pci0:22:14:0:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none21@pci0:22:14:1:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none22@pci0:22:14:2:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none23@pci0:22:14:3:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none24@pci0:22:14:4:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none25@pci0:22:14:5:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none26@pci0:22:14:6:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none27@pci0:22:14:7:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none28@pci0:22:15:0:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none29@pci0:22:15:1:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x208e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none30@pci0:22:29:0:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2054 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none31@pci0:22:29:1:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2055 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none32@pci0:22:29:2:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2056 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none33@pci0:22:29:3:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2057 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E CHA Registers'
    class      = base peripheral
none34@pci0:22:30:0:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2080 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none35@pci0:22:30:1:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2081 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none36@pci0:22:30:2:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2082 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none37@pci0:22:30:3:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2083 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none38@pci0:22:30:4:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2084 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none39@pci0:22:30:5:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2085 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
none40@pci0:22:30:6:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2086 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCU Registers'
    class      = base peripheral
pcib7@pci0:100:0:0: class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x2030 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCI Express Root Port A'
    class      = bridge
    subclass   = PCI-PCI
none41@pci0:100:5:0:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2034 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E VT-d'
    class      = base peripheral
none42@pci0:100:5:2:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2035 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic2@pci0:100:5:4:   class=0x080020 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2036 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E IOxAPIC Configuration Registers'
    class      = base peripheral
    subclass   = interrupt controller
none43@pci0:100:8:0:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2066 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none44@pci0:100:9:0:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2066 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none45@pci0:100:10:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2040 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none46@pci0:100:10:1:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2041 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none47@pci0:100:10:2:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2042 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none48@pci0:100:10:3:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2043 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none49@pci0:100:10:4:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2044 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none50@pci0:100:10:5:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2045 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LM Channel 1'
    class      = base peripheral
none51@pci0:100:10:6:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2046 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LMS Channel 1'
    class      = base peripheral
none52@pci0:100:10:7:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2047 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LMDP Channel 1'
    class      = base peripheral
none53@pci0:100:11:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2048 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E DECS Channel 2'
    class      = base peripheral
none54@pci0:100:11:1:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2049 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LM Channel 2'
    class      = base peripheral
none55@pci0:100:11:2:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x204a subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LMS Channel 2'
    class      = base peripheral
none56@pci0:100:11:3:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x204b subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LMDP Channel 2'
    class      = base peripheral
none57@pci0:100:12:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2040 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none58@pci0:100:12:1:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2041 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none59@pci0:100:12:2:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2042 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none60@pci0:100:12:3:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2043 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none61@pci0:100:12:4:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2044 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E Integrated Memory Controller'
    class      = base peripheral
none62@pci0:100:12:5:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2045 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LM Channel 1'
    class      = base peripheral
none63@pci0:100:12:6:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2046 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LMS Channel 1'
    class      = base peripheral
none64@pci0:100:12:7:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2047 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LMDP Channel 1'
    class      = base peripheral
none65@pci0:100:13:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2048 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E DECS Channel 2'
    class      = base peripheral
none66@pci0:100:13:1:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2049 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LM Channel 2'
    class      = base peripheral
none67@pci0:100:13:2:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x204a subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LMS Channel 2'
    class      = base peripheral
none68@pci0:100:13:3:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x204b subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E LMDP Channel 2'
    class      = base peripheral
vgapci0@pci0:101:0:0:   class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1cb2 subvendor=0x1028 subdevice=0x11bd
    vendor     = 'NVIDIA Corporation'
    device     = 'GP107GL [Quadro P600]'
    class      = display
    subclass   = VGA
hdac1@pci0:101:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x0fb9 subvendor=0x1028 subdevice=0x11bd
    vendor     = 'NVIDIA Corporation'
    device     = 'GP107GL High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA
none69@pci0:178:5:0:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2034 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E VT-d'
    class      = base peripheral
none70@pci0:178:5:2:    class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2035 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E RAS Configuration Registers'
    class      = base peripheral
ioapic3@pci0:178:5:4:   class=0x080020 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2036 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E IOxAPIC Configuration Registers'
    class      = base peripheral
    subclass   = interrupt controller
pcib9@pci0:178:5:5: class=0x010400 rev=0x04 hdr=0x00 vendor=0x8086 device=0x201d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Volume Management Device NVMe RAID Controller'
    class      = mass storage
    subclass   = RAID
none71@pci0:178:18:0:   class=0x110100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x204c subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = dasp
    subclass   = performance counters
none72@pci0:178:18:1:   class=0x110100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x204d subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = dasp
    subclass   = performance counters
none73@pci0:178:18:2:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x204e subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M3KTI Registers'
    class      = base peripheral
none74@pci0:178:21:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2018 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
none75@pci0:178:22:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2018 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
none76@pci0:178:22:4:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2018 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
none77@pci0:178:23:0:   class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x2018 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E M2PCI Registers'
    class      = base peripheral
pcib10@pci65526:0:2:0:  class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x2032 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCI Express Root Port C'
    class      = bridge
    subclass   = PCI-PCI
pcib11@pci65526:0:3:0:  class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x2033 subvendor=0x1028 subdevice=0x0738
    vendor     = 'Intel Corporation'
    device     = 'Sky Lake-E PCI Express Root Port D'
    class      = bridge
    subclass   = PCI-PCI
nvme0@pci65526:1:0:0:   class=0x010802 rev=0x00 hdr=0x00 vendor=0x1179 device=0x0116 subvendor=0x1179 subdevice=0x0001
    vendor     = 'Toshiba Corporation'
    device     = 'XG5 NVMe SSD Controller'
    class      = mass storage
    subclass   = NVM

Please let me know what else I can provide.

Thank you!

Updating the src tree to the latest STABLE, reversing the patch from commit 25df691800f08756228d3ac52ac64d80e9fe998d, and building the kernel results in a bootable kernel which correctly sees the nvme controller on the Precision 5820.

nvme0@pci65526:1:0:0:   class=0x010802 rev=0x00 hdr=0x00 vendor=0x1179 device=0x0116 subvendor=0x1179 subdevice=0x0001
    vendor     = 'Toshiba Corporation'
    device     = 'XG5 NVMe SSD Controller'
    class      = mass storage
    subclass   = NVM


FreeBSD 14.2-STABLE #10 stable/14-n271053-e6a470ffcbd7-dirty: Wed Apr  9 16:50:01 PDT 2025     root@buildhost.example.edu:/usr/obj/usr/src/amd64.amd64/sys/GENERIC


Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.2-STABLE #10 stable/14-n271053-e6a470ffcbd7-dirty: Wed Apr  9 16:50:01 PDT 2025
    root@spare-diatom3.smate.wwu.edu:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)
VT(efifb): resolution 1920x1200
CPU: Intel(R) Xeon(R) W-2155 CPU @ 3.30GHz (3300.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0xd39ffffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,MPX,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,AVX512BW,AVX512VL>
  Structured Extended Features3=0xbc002400<MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x2000c04<RSBA>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 66447085568 (63368 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <DELL\134x CBX3   >
FreeBSD/SMP: Multiprocessor System Detected: 20 CPUs
FreeBSD/SMP: 1 package(s) x 10 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23
ioapic1 <Version 2.0> irqs 24-31
ioapic2 <Version 2.0> irqs 32-39
ioapic3 <Version 2.0> irqs 40-47
ioapic4 <Version 2.0> irqs 48-55
Launching APs: 1 13 19 12 6 18 8 10 9 7 11 2 14 16 4 15 5 17 3
random: entropy device external interface
kbd1 at kbdmux0
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x4c50b000-0x4c50b017
smbios0: Entry point: v3 (64-bit), Version: 3.2
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <DELL CBX3   >
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> numa-domain 0 on acpi0
atrtc1: <AT realtime clock> on acpi0
atrtc1: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 350
Event timer "HPET1" frequency 24000000 Hz quality 340
Event timer "HPET2" frequency 24000000 Hz quality 340
Event timer "HPET3" frequency 24000000 Hz quality 340
Event timer "HPET4" frequency 24000000 Hz quality 340
Event timer "HPET5" frequency 24000000 Hz quality 340
Event timer "HPET6" frequency 24000000 Hz quality 340
Event timer "HPET7" frequency 24000000 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
acpi_syscontainer0: <System Container> on acpi0
apei0: <ACPI Platform Error Interface> on acpi0
acpi_button0: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
pci0: <dasp, performance counters> at device 8.1 (no driver attached)
xhci0: <Intel Union Point USB 3.0 controller> mem 0x38003ff10000-0x38003ff1ffff at device 20.0 numa-domain 0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 numa-domain 0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Patsburg+ (RAID) AHCI SATA controller> port 0x3050-0x3057,0x3040-0x3043,0x3020-0x303f mem 0x92f24000-0x92f25fff,0x92f27000-0x92f270ff,0x92f26000-0x92f267ff at device 23.0 numa-domain 0 on pci0
ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich6: <AHCI channel> at channel 6 on ahci0
ahcich7: <AHCI channel> at channel 7 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib1: <ACPI PCI-PCI bridge> at device 28.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 28.6 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 numa-domain 0 on pci2
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 29.0 numa-domain 0 on pci0
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
ix0: <Intel(R) X550-T2> mem 0x38003f800000-0x38003fbfffff,0x38003fc04000-0x38003fc07fff at device 0.0 numa-domain 0 on pci4
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 10 RX queues 10 TX queues
ix0: Using MSI-X interrupts with 11 vectors
ix0: allocated for 10 queues
ix0: allocated for 10 rx queues
ix0: Ethernet address: ec:e7:a7:01:4c:cc
ix0: PCI Express Bus: Speed 8.0GT/s Width x4
ix0: fw 2.11.11 nvm 1.86.0 Option ROM V1-b1276-p0 eTrack 0x80000c67
ix0: netmap queues/slots: TX 10/2048, RX 10/2048
ix1: <Intel(R) X550-T2> mem 0x38003f400000-0x38003f7fffff,0x38003fc00000-0x38003fc03fff at device 0.1 numa-domain 0 on pci4
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 10 RX queues 10 TX queues
ix1: Using MSI-X interrupts with 11 vectors
ix1: allocated for 10 queues
ix1: allocated for 10 rx queues
ix1: Ethernet address: ec:e7:a7:01:4c:cd
ix1: PCI Express Bus: Speed 8.0GT/s Width x4
ix1: fw 2.11.11 nvm 1.86.0 Option ROM V1-b1276-p0 eTrack 0x80000c67
ix1: netmap queues/slots: TX 10/2048, RX 10/2048
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
pci0: <memory> at device 31.2 (no driver attached)
hdac0: <Intel Kaby Lake-H HDA Controller> mem 0x38003ff20000-0x38003ff23fff,0x38003ff00000-0x38003ff0ffff at device 31.3 numa-domain 0 on pci0
em0: <Intel(R) I219-LM SPT(5)> mem 0x92f00000-0x92f1ffff at device 31.6 numa-domain 0 on pci0
em0: EEPROM V0.1-4
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 54:bf:64:7f:5c:04
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib5: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
pcib6: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 0.0 numa-domain 0 on pci6
pci7: <ACPI PCI bus> numa-domain 0 on pcib7
vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xd7000000-0xd7ffffff,0xc0000000-0xcfffffff,0xd0000000-0xd1ffffff at device 0.0 numa-domain 0 on pci7
hdac1: <NVIDIA (0x0fb9) HDA Controller> mem 0xd8080000-0xd8083fff at device 0.1 numa-domain 0 on pci7
pcib8: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <Intel Volume Management Device> mem 0x3800fc000000-0x3800fdffffff,0xf8000000-0xf9ffffff,0x3800fff00000-0x3800ffffffff at device 5.5 numa-domain 0 on pci8
pci9: <PCI bus> numa-domain 0 on pcib9
pcib10: <PCI-PCI bridge> irq 0 at device 2.0 numa-domain 0 on pci9
pcib10: failed to allocate initial I/O port window: 0-0xafff
pcib11: <PCI-PCI bridge> irq 0 at device 3.0 numa-domain 0 on pci9
pcib11: failed to allocate initial I/O port window: 0-0xbfff
pci8: <dasp, performance counters> at device 18.0 (no driver attached)
pci8: <dasp, performance counters> at device 18.1 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
atrtc0: Can't map interrupt.
atrtc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 15.
hwpstate_intel0: <Intel Speed Shift> numa-domain 0 on cpu0
cpufreq0: <CPU frequency control> numa-domain 0 on cpu0
hwpstate_intel1: <Intel Speed Shift> numa-domain 0 on cpu1
cpufreq1: <CPU frequency control> numa-domain 0 on cpu1
hwpstate_intel2: <Intel Speed Shift> numa-domain 0 on cpu2
cpufreq2: <CPU frequency control> numa-domain 0 on cpu2
hwpstate_intel3: <Intel Speed Shift> numa-domain 0 on cpu3
cpufreq3: <CPU frequency control> numa-domain 0 on cpu3
hwpstate_intel4: <Intel Speed Shift> numa-domain 0 on cpu4
cpufreq4: <CPU frequency control> numa-domain 0 on cpu4
hwpstate_intel5: <Intel Speed Shift> numa-domain 0 on cpu5
cpufreq5: <CPU frequency control> numa-domain 0 on cpu5
hwpstate_intel6: <Intel Speed Shift> numa-domain 0 on cpu6
cpufreq6: <CPU frequency control> numa-domain 0 on cpu6
hwpstate_intel7: <Intel Speed Shift> numa-domain 0 on cpu7
cpufreq7: <CPU frequency control> numa-domain 0 on cpu7
hwpstate_intel8: <Intel Speed Shift> numa-domain 0 on cpu8
cpufreq8: <CPU frequency control> numa-domain 0 on cpu8
hwpstate_intel9: <Intel Speed Shift> numa-domain 0 on cpu9
cpufreq9: <CPU frequency control> numa-domain 0 on cpu9
hwpstate_intel10: <Intel Speed Shift> numa-domain 0 on cpu10
cpufreq10: <CPU frequency control> numa-domain 0 on cpu10
hwpstate_intel11: <Intel Speed Shift> numa-domain 0 on cpu11
cpufreq11: <CPU frequency control> numa-domain 0 on cpu11
hwpstate_intel12: <Intel Speed Shift> numa-domain 0 on cpu12
cpufreq12: <CPU frequency control> numa-domain 0 on cpu12
hwpstate_intel13: <Intel Speed Shift> numa-domain 0 on cpu13
cpufreq13: <CPU frequency control> numa-domain 0 on cpu13
hwpstate_intel14: <Intel Speed Shift> numa-domain 0 on cpu14
cpufreq14: <CPU frequency control> numa-domain 0 on cpu14
hwpstate_intel15: <Intel Speed Shift> numa-domain 0 on cpu15
cpufreq15: <CPU frequency control> numa-domain 0 on cpu15
hwpstate_intel16: <Intel Speed Shift> numa-domain 0 on cpu16
cpufreq16: <CPU frequency control> numa-domain 0 on cpu16
hwpstate_intel17: <Intel Speed Shift> numa-domain 0 on cpu17
cpufreq17: <CPU frequency control> numa-domain 0 on cpu17
hwpstate_intel18: <Intel Speed Shift> numa-domain 0 on cpu18
cpufreq18: <CPU frequency control> numa-domain 0 on cpu18
hwpstate_intel19: <Intel Speed Shift> numa-domain 0 on cpu19
cpufreq19: <CPU frequency control> numa-domain 0 on cpu19
pcib10: Power Fault Detected
pci10: <PCI bus> numa-domain 0 on pcib10
nvme0: <Generic NVMe Device> irq 0 at device 0.0 numa-domain 0 on pci10
hdacc0: <Realtek ALC255 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC255 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC255 (Analog 2.0+HP/2.0)> at nid 20,27 and 26 on hdaa0
pcm1: <Realtek ALC255 (Analog)> at nid 33 and 25 on hdaa0
hdacc1: <NVIDIA (0x0080) HDA CODEC> at cad 0 on hdac1
hdaa1: <NVIDIA (0x0080) Audio Function Group> at nid 1 on hdacc1
pcm2: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 4 on hdaa1
pcm3: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 5 on hdaa1
pcm4: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 6 on hdaa1
pcm5: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 7 on hdaa1
pcib10: Power Fault Detected
Timecounter "TSC-low" frequency 1655996632 Hz quality 1000
Timecounters tick every 1.000 msec
pcib10: Power Fault Detected
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 numa-domain 0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
nda0 at nvme0 bus 0 scbus9 target 0 lun 1
nda0: <KXG50ZNV512G NVMe TOSHIBA 512GB AADA4105 78MS1223T8HQ>
nda0: Serial Number 78MS1223T8HQ
nda0: nvme version 1.2
nda0: 488386MB (1000215216 512 byte sectors)
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST3000DM008-2DM166 CC26> ACS-2 ATA SATA 3.x device
ada0: Serial Number Z505F4XJ
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors)
ada0: quirks=0x1<4K>
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <ST3000DM008-2DM166 CC26> ACS-2 ATA SATA 3.x device
ada1: Serial Number Z505FXBL
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors)
ada1: quirks=0x1<4K>
ses0 at ahciem0 bus 0 scbus8 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses0: ada0,pass0 in 'Slot 00', SATA Slot: scbus0 target 0
ses0: ada1,pass1 in 'Slot 01', SATA Slot: scbus1 target 0
ses0: cd0,pass2 in 'Slot 07', SATA Slot: scbus7 target 0
cd0 at ahcich7 bus 0 scbus7 target 0 lun 0
cd0: <HL-DT-ST DVD+-RW GHB0N A1C0> Removable CD-ROM SCSI device
cd0: Serial Number KEDI66E2431
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: 0MB (1 0 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
uhub0: 26 ports with 26 removable, self powered
ugen0.2: <vendor 0x0461 DELL Laser Mouse> at usbus0
ugen0.3: <Generic 4-Port USB 2.0 Hub> at usbus0
uhub1 numa-domain 0 on uhub0
uhub1: <Generic 4-Port USB 2.0 Hub, class 9/0, rev 2.10/1.23, addr 2> on usbus0
uhub1: MTT enabled
uhub1: 2 ports with 2 removable, self powered
ugen0.4: <vendor 0x046d Logitech BRIO> at usbus0
ugen0.5: <Generic USB3.0-CRW> at usbus0
umass0 numa-domain 0 on uhub0
umass0: <Generic USB3.0-CRW, class 0/0, rev 3.00/29.08, addr 4> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x4000
umass0:10:0: Attached to scbus10
(probe0:umass-sim0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00 
(probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(probe0:umass-sim0:0:0:0): SCSI status: Check Condition
(probe0:umass-sim0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB)
(probe0:umass-sim0:0:0:0): Error 22, Unretryable error
da0 at umass-sim0 bus 0 scbus10 target 0 lun 0
da0: <Generic- SD/MMC CRW 1.00> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 28203008282014000
da0: 400.000MB/s transfers
da0: Attempt to query device size failed: NOT READY, Medium not present
da0: quirks=0x2<NO_6_BYTE>
ugen0.6: <Generic 4-Port USB 3.0 Hub> at usbus0
uhub2 numa-domain 0 on uhub0
uhub2: <Generic 4-Port USB 3.0 Hub, class 9/0, rev 3.00/1.23, addr 5> on usbus0
uhub2: 2 ports with 2 removable, self powered
usb_msc_auto_quirk: UQ_MSC_NO_TEST_UNIT_READY set for USB mass storage device Generic USB Storage (0x05e3:0x0747)
usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device Generic USB Storage (0x05e3:0x0747)
usb_msc_auto_quirk: UQ_MSC_NO_SYNC_CACHE set for USB mass storage device Generic USB Storage (0x05e3:0x0747)
ugen0.7: <Generic USB Storage> at usbus0
umass1 numa-domain 0 on uhub2
umass1: <Generic USB Storage, class 0/0, rev 3.00/8.19, addr 6> on usbus0
umass1:  SCSI over Bulk-Only; quirks = 0xc001
umass1:11:1: Attached to scbus11
(probe0:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00 
(probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error
(probe0:umass-sim1:1:0:0): SCSI status: Check Condition
(probe0:umass-sim1:1:0:0): SCSI sense: UNIT ATTENTION asc:28,0 (Not ready to ready change, medium may have changed)
(probe0:umass-sim1:1:0:0): Retrying command (per sense data)
(probe0:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00 
(probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error
(probe0:umass-sim1:1:0:0): SCSI status: Check Condition
(probe0:umass-sim1:1:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
(probe0:umass-sim1:1:0:0): Error 22, Unretryable error
da1 at umass-sim1 bus 1 scbus11 target 0 lun 0
da1: <Generic STORAGE DEVICE 0819> Removable Direct Access SPC-4 SCSI device
da1: Serial Number 000000000819
da1: 400.000MB/s transfers
da1: 29820MB (61071360 512 byte sectors)
da1: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>
nvidia0: <Quadro P600> numa-domain 0 on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  570.124.04  Tue Feb 25 03:39:05 UTC 2025
sysctl_add_oid: can't re-use a leaf (hw.dri.debug)!
[drm] [nvidia-drm] [GPU ID 0x00006500] Loading driver
[drm] Initialized nvidia-drm 0.0.0 20160202 for nvidia0 on minor 0
ioat0: <SKX IOAT> mem 0x38003ff40000-0x38003ff43fff at device 4.0 numa-domain 0 on pci0
ioat0: Capabilities: f0006f1<DMA_with_DIF,PQ,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Page_Break>
ioat1: <SKX IOAT> mem 0x38003ff3c000-0x38003ff3ffff at device 4.1 numa-domain 0 on pci0
ioat1: Capabilities: f0006f1<DMA_with_DIF,PQ,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Page_Break>
ioat2: <SKX IOAT> mem 0x38003ff38000-0x38003ff3bfff at device 4.2 numa-domain 0 on pci0
ioat2: Capabilities: f0004f1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Page_Break>
ioat3: <SKX IOAT> mem 0x38003ff34000-0x38003ff37fff at device 4.3 numa-domain 0 on pci0
ioat3: Capabilities: f0004f1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Page_Break>
ioat4: <SKX IOAT> mem 0x38003ff30000-0x38003ff33fff at device 4.4 numa-domain 0 on pci0
ioat4: Capabilities: f0004f1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Page_Break>
ioat5: <SKX IOAT> mem 0x38003ff2c000-0x38003ff2ffff at device 4.5 numa-domain 0 on pci0
ioat5: Capabilities: f0004f1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Page_Break>
ioat6: <SKX IOAT> mem 0x38003ff28000-0x38003ff2bfff at device 4.6 numa-domain 0 on pci0
ioat6: Capabilities: f0004f1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Page_Break>
ioat7: <SKX IOAT> mem 0x38003ff24000-0x38003ff27fff at device 4.7 numa-domain 0 on pci0
ioat7: Capabilities: f0004f1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Page_Break>
ichsmb0: <Intel Kaby Lake SMBus controller> port 0x3000-0x301f mem 0x38003ff44000-0x38003ff440ff at device 31.4 numa-domain 0 on pci0
smbus0: <System Management Bus> numa-domain 0 on ichsmb0
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: cannot find EC device
acpi_wmi1: <ACPI-WMI mapping> on acpi0
acpi_wmi1: cannot find EC device
acpi_wmi2: <ACPI-WMI mapping> on acpi0
acpi_wmi2: cannot find EC device
acpi_wmi2: Embedded MOF found
ACPI: \134_SB.AMW0.WQMO: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
lo0: link state changed to UP
ix0: link state changed to UP
vlan0: changing name to 'private'
vlan2: changing name to 'public'
ix0: link state changed to DOWN
vlan1: link state changed to DOWN
private: link state changed to DOWN
public: link state changed to DOWN
ums0 numa-domain 0 on uhub0
ums0: <vendor 0x0461 DELL Laser Mouse, class 0/0, rev 2.00/7.17, addr 1> on usbus0
ums0: 5 buttons and [XYZ] coordinates ID=0
uhid0 numa-domain 0 on uhub0
uhid0: <vendor 0x046d Logitech BRIO, class 239/2, rev 3.10/3.17, addr 3> on usbus0
Cuse v0.1.37 @ /dev/cuse
uaudio0 numa-domain 0 on uhub0
uaudio0: <vendor 0x046d Logitech BRIO, class 239/2, rev 3.10/3.17, addr 3> on usbus0
uaudio0: No playback.
uaudio0: Record[0]: 48000 Hz, 2 ch, 16-bit S-LE PCM format, 2x4ms buffer. (selected)
uaudio0: Record[0]: 32000 Hz, 2 ch, 16-bit S-LE PCM format, 2x4ms buffer.
uaudio0: Record[0]: 24000 Hz, 2 ch, 16-bit S-LE PCM format, 2x4ms buffer.
uaudio0: Record[0]: 16000 Hz, 2 ch, 16-bit S-LE PCM format, 2x4ms buffer.
uaudio0: No MIDI sequencer.
pcm6 numa-domain 0 on uaudio0
uaudio0: No HID volume keys found.
ix0: link state changed to UP
vlan1: link state changed to UP
private: link state changed to UP
public: link state changed to UP
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to deny, logging disabled

Comment 3 Colin Percival 2025-04-10 22:43:18 UTC

The "pcib10: Power Fault Detected" makes me wonder if there's something else going on here and Warner's commit merely exposed a separate problem.  The commit you reverted is about cleanly detaching a disk when it is being unplugged, so the obvious question here is "why does the system think the disk is being unplugged".

Warner, any ideas?

Comment 4 Warner Losh 2025-04-10 23:22:51 UTC

Power fault suggests the card is losing power, triggering our hotplug code because the card is gone and we read all ff's. I'll have to study this more when i get home.

Comment 5 Warner Losh 2025-04-10 23:29:02 UTC

(In reply to Warner Losh from comment #4)
So 25df691800f08756228d3ac52ac64d80e9fe998d is the commit I made to ask the bus if the device was present. I'll bet that's returning FALSE for some reason.

Comment 6 Warner Losh 2025-04-10 23:30:49 UTC

I've cc'd jhb since he suggested that I add this test. Normally it's in the tear-down path. The power failure indication may make us think that the device isn't there when, but that's wild speculation I don't have time right at the moment to chase down.

Comment 7 Colin Percival 2025-04-11 01:08:20 UTC

Thanks Warner!

Jim, what sort of device is this?  Is it possible that it's coming loose and making the system think it's being unplugged?  Can you add a printf to check if bus_child_present(dev) is returning 0?

The fact that we're getting to this point -- as Warner points out, this is in the teardown path -- suggests that something has gone wrong before this patch is implicated since we shouldn't be calling nvme_ctrlr_destruct while the device is running.

Thank you for taking a look.  Yes, bus_child_present is returning 0.  The device is a sled-mounted SSD, as mentioned here: https://www.dell.com/support/kbdoc/en-us/000185631/how-to-switch-nvme-for-precision-5820-and-7820-tower   It doesn't seem like a loose connection, since it's consistently working through n270866 and broken since n270867, which I sussed out using 'git bisect' on a kernel build.  I have one theory to try, to see if I just get lucky.

Meanwhile ....

Adding a printf to stable/14-n271055-f9afcbff02a2-dirty:

13:02:18 /usr/src# git diff
diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c
index 6f5d6ae74add..1850a5f0a389 100644
--- a/sys/dev/nvme/nvme_ctrlr.c
+++ b/sys/dev/nvme/nvme_ctrlr.c
@@ -1532,6 +1532,7 @@ nvme_ctrlr_destruct(struct nvme_controller *ctrlr, device_t dev)
         * do hotplug), so failsafe with detecting all ff's (impossible with
         * this hardware) as the device being gone.
         */
+nvme_printf(ctrlr, "bus_child_present(dev) == %d\n", bus_child_present(dev));
        gone = bus_child_present(dev) == 0 ||
            (nvme_mmio_read_4(ctrlr, csts) == NVME_GONE);
        if (gone)


dmesg then shows:

# egrep -1 'nvme|pcib?10' /var/run/dmesg.boot 
pci9: <PCI bus> numa-domain 0 on pcib9
pcib10: <PCI-PCI bridge> irq 0 at device 2.0 numa-domain 0 on pci9
pcib10: failed to allocate initial I/O port window: 0-0xafff
pcib11: <PCI-PCI bridge> irq 0 at device 3.0 numa-domain 0 on pci9
--
cpufreq19: <CPU frequency control> numa-domain 0 on cpu19
pcib10: Power Fault Detected
pci10: <PCI bus> numa-domain 0 on pcib10
nvme0: <Generic NVMe Device> irq 0 at device 0.0 numa-domain 0 on pci10
pcib10: Power Fault Detected
hdacc0: <Realtek ALC255 HDA CODEC> at cad 0 on hdac0
--
pcm5: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 7 on hdaa1
pcib10: Power Fault Detected
Timecounter "TSC-low" frequency 1655996617 Hz quality 1000
--
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
nvme0: bus_child_present(dev) == 0
nvme0: failing outstanding i/o
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:14 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:14 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:13 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:13 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:12 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:12 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:11 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:11 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:10 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:10 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:9 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:9 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:8 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:8 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:7 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:7 cdw0:0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
--
Trying to mount root from zfs:thumb/ROOT/default []...
nvme0: detached
pci10: detached
uhub0: 26 ports with 26 removable, self powered

I'm exploring an NVME controller firmware upgrade at https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=7yxtm

(In reply to Jim Long from comment #9)

No joy.  The Toshiba SSD only has Windows/GUI firmware update tools.

Relative to n271055, I have reduced the necessary patch for my hardware to boot to be:

06:41:08 /usr/src# git diff
diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c
index 6f5d6ae74add..ea1de2b98e9f 100644
--- a/sys/dev/nvme/nvme_ctrlr.c
+++ b/sys/dev/nvme/nvme_ctrlr.c
@@ -1524,16 +1524,10 @@ nvme_ctrlr_destruct(struct nvme_controller *ctrlr, device_t dev)
                goto noadminq;
 
        /*
-        * Check whether it is a hot unplug or a clean driver detach.  If device
-        * is not there any more, skip any shutdown commands.  Some hotplug
-        * bridges will return zeros instead of ff's when the device is
-        * departing, so ask the bridge if the device is gone. Some systems can
-        * remove the drive w/o the bridge knowing its gone (they don't really
-        * do hotplug), so failsafe with detecting all ff's (impossible with
-        * this hardware) as the device being gone.
+        * Check whether it is a hot unplug or a clean driver detach.
+        * If device is not there any more, skip any shutdown commands.
         */
-       gone = bus_child_present(dev) == 0 ||
-           (nvme_mmio_read_4(ctrlr, csts) == NVME_GONE);
+       gone = (nvme_mmio_read_4(ctrlr, csts) == NVME_GONE);
        if (gone)
                nvme_ctrlr_fail(ctrlr);
        else


I'm trying to look into why my hardware results in returning (bus_child_present(dev) == 0) but I'm not finding where the *definition* of BUS_CHILD_PRESENT is.  Plenty of references, but no definition, to my novice eyes.

Clues appreciated, as always.

Comment 11 Warner Losh 2025-04-14 15:49:53 UTC

bus_child_present is a newbus dispatch method.
sys/dev/pci/pci_pci.c:    DEVMETHOD(bus_child_present,	pcib_child_present),
sys/dev/pccbb/pccbb_pci.c:	DEVMETHOD(bus_child_present,		cbb_child_present),
is where it's implemented. Since there are no cardbus NVMe drive, the first one is relevant.

int
pcib_child_present(device_t dev, device_t child)
{
#ifdef PCI_HP
        struct pcib_softc *sc = device_get_softc(dev);
        int retval;

        retval = bus_child_present(dev);
        if (retval != 0 && sc->flags & PCIB_HOTPLUG)
                retval = pcib_hotplug_present(sc);
        return (retval);
#else
        return (bus_child_present(dev));
#endif
}

which right away suggests you could work around this with a nooptions PCI_HP kernel.

But that code says that if all my parents think the child is there, then check its hotplug status:
/*
 * Returns -1 if the card is fully inserted, powered, and ready for
 * access.  Otherwise, returns 0.
 */
static int
pcib_hotplug_present(struct pcib_softc *sc)
{

        /* Card must be inserted. */
        if (!pcib_hotplug_inserted(sc))
                return (0);

        /* Require the Data Link Layer to be active. */
        if (!(sc->pcie_link_sta & PCIEM_LINK_STA_DL_ACTIVE))
                return (0);

        return (-1);
}

(yes, I checked, this is the right protocol to use).

So the next debugging point is to add code in pcib_hotplug_present to see which of these conditions fails.

Comment 12 Warner Losh 2025-04-14 16:04:02 UTC

judging by the comments, I'd expect:

        /* A power fault implicitly turns off power to the slot. */
	if (sc->pcie_slot_sta & PCIEM_SLOT_STA_PFD)
                return (false);

is the condition we're tripping over.

Your hunch seems correct.  So I shot it full of 'printf's:

diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c
index 06e8c2bc8433..b94d27314f49 100644
--- a/sys/dev/pci/pci_pci.c
+++ b/sys/dev/pci/pci_pci.c
@@ -1071,20 +1071,32 @@ pcib_hotplug_inserted(struct pcib_softc *sc)
 
        /* Pretend the card isn't present if a detach is forced. */
        if (sc->flags & PCIB_DETACHING)
+{
+device_printf(sc->dev, "(1) PCIB_DETACHING\n");
                return (false);
+}
 
        /* Card must be present in the slot. */
        if ((sc->pcie_slot_sta & PCIEM_SLOT_STA_PDS) == 0)
+{
+device_printf(sc->dev, "(2) PCIEM_SLOT_STA_PDS\n");
                return (false);
+}
 
        /* A power fault implicitly turns off power to the slot. */
        if (sc->pcie_slot_sta & PCIEM_SLOT_STA_PFD)
+{
+device_printf(sc->dev, "(3) PCIEM_SLOT_STA_PFD\n");
                return (false);
+}
 
        /* If the MRL is disengaged, the slot is powered off. */
        if (sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP &&
            (sc->pcie_slot_sta & PCIEM_SLOT_STA_MRLSS) != 0)
+{
+device_printf(sc->dev, "(4) CAP_MRLSP && STA_MRLSS\n");
                return (false);
+}
 
        return (true);
 }
@@ -1868,8 +1880,10 @@ pcib_child_present(device_t dev, device_t child)
        int retval;
 
        retval = bus_child_present(dev);
+device_printf(sc->dev, "(1) retval == %d\n", retval);
        if (retval != 0 && sc->flags & PCIB_HOTPLUG)
                retval = pcib_hotplug_present(sc);
+device_printf(sc->dev, "(2) retval == %d\n", retval);
        return (retval);
 #else
        return (bus_child_present(dev));


And this is what dmesg shows re: pcib10 and friends:

...
pcib9: <Intel Volume Management Device> mem 0x3800fc000000-0x3800fdffffff,0xf8000000-0xf9ffffff,0x3800fff00000-0x3800ffffffff at device 5.5 numa-domain 0 on pci8
pci9: <PCI bus> numa-domain 0 on pcib9
pcib10: <PCI-PCI bridge> irq 0 at device 2.0 numa-domain 0 on pci9
pcib10: failed to allocate initial I/O port window: 0-0xafff
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib11: <PCI-PCI bridge> irq 0 at device 3.0 numa-domain 0 on pci9
pcib11: failed to allocate initial I/O port window: 0-0xbfff
pcib11: (2) PCIEM_SLOT_STA_PDS
pcib11: (2) PCIEM_SLOT_STA_PDS
pci8: <dasp, performance counters> at device 18.0 (no driver attached)
pci8: <dasp, performance counters> at device 18.1 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
atrtc0: Can't map interrupt.
atrtc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 15.
....
hwpstate_intel19: <Intel Speed Shift> numa-domain 0 on cpu19
cpufreq19: <CPU frequency control> numa-domain 0 on cpu19
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pci10: <PCI bus> numa-domain 0 on pcib10
nvme0: <Generic NVMe Device> irq 0 at device 0.0 numa-domain 0 on pci10
hdacc0: <Realtek ALC255 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC255 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC255 (Analog 2.0+HP/2.0)> at nid 20,27 and 26 on hdaa0
pcm1: <Realtek ALC255 (Analog)> at nid 33 and 25 on hdaa0
hdacc1: <NVIDIA (0x0080) HDA CODEC> at cad 0 on hdac1
hdaa1: <NVIDIA (0x0080) Audio Function Group> at nid 1 on hdacc1
pcm2: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 4 on hdaa1
pcm3: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 5 on hdaa1
pcm4: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 6 on hdaa1
pcm5: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 7 on hdaa1
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
Timecounter "TSC-low" frequency 1655996686 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ugen0.1: <Intel XHCI root HUB> at usbus0
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
uhub0 numa-domain 0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
nda0 at nvme0 bus 0 scbus9 target 0 lun 1
nda0: <KXG50ZNV512G NVMe TOSHIBA 512GB AADA4105 78MS1223T8HQ>
nda0: Serial Number 78Mpcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
S1223T8HQ
nda0: nvme version 1.2
nda0: 488386MB (1000215216 512 byte sectors)
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST3000DM008-2DM166 CC26> ACS-2 ATA SATA 3.x device
ada0: Serial Number Z505F4XJ
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors)
ada0: quirks=0x1<4K>
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <ST3000DM008-2DM166 CC26> ACS-2 ATA SATA 3.x device
ada1: Serial Number Z505FXBL
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors)
ada1: quirks=0x1<4K>
pcib10: (3) PCIEM_SLOT_STA_PFD
ses0 at ahciem0 bus 0 scbus8 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses0: ada0,pass0 in 'Slot 00', SATA Slot: scbus0 target 0
ses0: ada1,pass1 in 'Slot 01', SATA Slot: scbus1 target 0
ses0: cd0,pass2 in 'Slot 07', SATA Slot: scbus7 target 0
cd0 at ahcich7 bus 0 scbus7 target 0 lun 0
cd0: <HL-DT-ST DVD+-RW GHB0N A1C0> Removable CD-ROM SCSI device
cd0: Serial Number KEDI66E2431
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: 0MB (1 0 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
....

I've got a verbose dmesg if that will help.  With that level of detail, I'm not as clear on what is or isn't relevant.  If you want all 2200-some lines, just say the word.  Otherwise, here's the redacted version.

Looking at my printf's, is there perhaps a clue in the sequence:

pcib10: (1) retval == -1
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (2) retval == 0

I appreciate your assistance with this.  Please don't hesitate to suggest more enlightening "printf" probes, since I am not familiar with the kernel source.

Thank you,

Jim


Diffs to n271055:

11:47:43 /usr/src# git diff
diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c
index 06e8c2bc8433..b94d27314f49 100644
--- a/sys/dev/pci/pci_pci.c
+++ b/sys/dev/pci/pci_pci.c
@@ -1071,20 +1071,32 @@ pcib_hotplug_inserted(struct pcib_softc *sc)

        /* Pretend the card isn't present if a detach is forced. */
        if (sc->flags & PCIB_DETACHING)
+{
+device_printf(sc->dev, "(1) PCIB_DETACHING\n");
                return (false);
+}

        /* Card must be present in the slot. */
        if ((sc->pcie_slot_sta & PCIEM_SLOT_STA_PDS) == 0)
+{
+device_printf(sc->dev, "(2) PCIEM_SLOT_STA_PDS\n");
                return (false);
+}

        /* A power fault implicitly turns off power to the slot. */
        if (sc->pcie_slot_sta & PCIEM_SLOT_STA_PFD)
+{
+device_printf(sc->dev, "(3) PCIEM_SLOT_STA_PFD\n");
                return (false);
+}

        /* If the MRL is disengaged, the slot is powered off. */
        if (sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP &&
            (sc->pcie_slot_sta & PCIEM_SLOT_STA_MRLSS) != 0)
+{
+device_printf(sc->dev, "(4) CAP_MRLSP && STA_MRLSS\n");
                return (false);
+}

        return (true);
 }
@@ -1868,8 +1880,10 @@ pcib_child_present(device_t dev, device_t child)
        int retval;

        retval = bus_child_present(dev);
+device_printf(sc->dev, "(1) retval == %d\n", retval);
        if (retval != 0 && sc->flags & PCIB_HOTPLUG)
                retval = pcib_hotplug_present(sc);
+device_printf(sc->dev, "(2) retval == %d\n", retval);
        return (retval);
 #else
        return (bus_child_present(dev));



Trimmed verbose dmesg:

...
pcm5: Soft PCM mixer ENABLED
pcm5: Playback channel matrix is: unknown, assuming 7.1 (disconnected)
pcib10: HotPlug interrupt: 0x52
pcib10: Power Fault Detected
pcib10: Command Completed
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: HotPlug command: 15f8 -> 17f8
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: HotPlug interrupt: 0x52
pcib10: Power Fault Detected
pcib10: Command Completed
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
procfs registered
Statistical TSC calibration took 18175 us and 4035 data points
Timecounter "TSC-low" frequency 1655996465 Hz quality 1000
Statistical lapic calibration took 8083 us and 41665 data points
...
AcpiOsExecute: enqueue 20 pending tasks
ugen0.1: <Intel XHCI root HUB> at usbus0
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
msi: Assigning MSI-X IRQ 94 to local APIC 5 vector 48
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
msi: Assigning MSI-X IRQ 95 to local APIC 21 vector 48
uhub0 numa-domain 0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (1) retval == -1
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (2) retval == 0
nvme0: failing outstanding i/o
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:14 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:14 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:13 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:13 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:12 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:12 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:11 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:11 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:10 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:10 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:9 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:9 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:8 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:8 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:7 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:7 cdw0:0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST3000DM008-2DM166 CC26> ACS-2 ATA SATA 3.x device
ada0: Serial Number Z505F4XJ
....
pass0: Command Queueing enabled
pci0: driver added
....
pci1: driver added
pci2: driver added
pci3: driver added
pci4: driver added
pci5: driver added
....
pci6: driver added
....
pci7: driver added
pci8: driver added
....
pci9: driver added
pci10: driver added
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
nvme0: detached
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
Trying to mount root from zfs:thumb/ROOT/default []...
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (1) retval == -1
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (2) retval == 0
pci10: detached
....

Comment 15 Warner Losh 2025-04-15 13:55:17 UTC

John Baldwin pointed me at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256264 which has patches from him to disable hot-plug when we see certain inconsistencies in the status reported between different parts of the system about the ports.

Comment 16 John Baldwin 2025-04-16 13:30:30 UTC

So it's kind of bizarre that pcib10 reports PFD (power fault detected) at the very beginning of its probe.  This is why we think a card isn't present.  At the start of attach for a pci bridge, we clear any pending events in the slot status register (these bits are W1C):

static void
pcib_setup_hotplug(struct pcib_softc *sc)
{
...
	sc->pcie_link_sta = pcie_read_config(dev, PCIER_LINK_STA, 2);
	sc->pcie_slot_sta = pcie_read_config(dev, PCIER_SLOT_STA, 2);

	/* Clear any events previously pending. */
	pcie_write_config(dev, PCIER_SLOT_STA, sc->pcie_slot_sta, 2);


It would be handy I think to see the output of `pciconf -lc pcib10`.  The raw register values of the slot capability register would be useful, it is at offset 0x14 in the PCI-e capability registers.  If you aren't familiar with using pciconf to read individual registers I can give you a pciconf command after seeing the output of `pciconf -lc pcib10`.

Thank you, John, for chiming in.

No, I'm not real familiar with examining PCI-E registers, but I've been on a crash course with the pciconf man page lately, so I'll give it a whack:

pcib10@pci65526:0:2:0:  class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x2032 subvendor=0x1028 subdevice=0x0738
    cap 0d[40] = PCI Bridge subvendor=0x1028 subdevice=0x0738
    cap 05[60] = MSI supports 2 messages, vector masks 
    cap 10[90] = PCI-Express 2 root port max data 128(256) ARI disabled
                 max read 128
                 link x4(x4) speed 8.0(8.0) ASPM L1(L1)
                 slot 91 power limit 0 mW HotPlug(present) surprise
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
    ecap 000b[100] = Vendor [1] ID 0002 Rev 0 Length 12
    ecap 000d[110] = ACS 1 Source Validation disabled, Translation Blocking disabled
                     P2P Req Redirect disabled, P2P Cmpl Redirect disabled
                     P2P Upstream Forwarding disabled, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
    ecap 0001[148] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 000b[1d0] = Vendor [1] ID 0003 Rev 1 Length 10
    ecap 0019[250] = PCIe Sec 1 lane errors 0
    ecap 000b[280] = Vendor [1] ID 0005 Rev 3 Length 24
    ecap 000b[300] = Vendor [1] ID 0008 Rev 0 Length 56

Is this helpful:

# pciconf -br pcib10@pci65526:0:2:0 0x14:0x1f
00 00 00 00 00 01 01 00  f0 00 00 00

I'm current testing against 14-STABLE n271070, plus jhb's patch from issue 256264, plus some debugging printf's:

diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c
index 06e8c2bc8433..f70f04d15759 100644
--- a/sys/dev/pci/pci_pci.c
+++ b/sys/dev/pci/pci_pci.c
@@ -979,21 +979,30 @@ pcib_probe_hotplug(struct pcib_softc *sc)
        return;
 
    /*
-    * Some devices report that they have an MRL when they actually
-    * do not.  Since they always report that the MRL is open, child
-    * devices would be ignored.  Try to detect these devices and
-    * ignore their claim of HotPlug support.
-    *
-    * If there is an open MRL but the Data Link Layer is active,
-    * the MRL is not real.
+    * Some bridges use HotPlug-capable chips but are not really
+    * HotPlug bridges.  The bridges can advertise inconsistent
+    * statuses such as cards not being present or an MRL that is
+    * always open, but the Data Link Layer is active.  Ignore
+    * HotPlug on such bridges.
     */
-   if ((sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP) != 0) {
-       link_sta = pcie_read_config(dev, PCIER_LINK_STA, 2);
+   link_sta = pcie_read_config(dev, PCIER_LINK_STA, 2);
+device_printf(dev, "link_sta == 0x%x\n", link_sta);
+   if ((link_sta & PCIEM_LINK_STA_DL_ACTIVE) != 0) {
        slot_sta = pcie_read_config(dev, PCIER_SLOT_STA, 2);
-       if ((slot_sta & PCIEM_SLOT_STA_MRLSS) != 0 &&
-           (link_sta & PCIEM_LINK_STA_DL_ACTIVE) != 0) {
+device_printf(dev, "slot_sta == 0x%x\n", slot_sta);
+
+       if ((slot_sta & PCIEM_SLOT_STA_PDS) == 0)
+{
+device_printf(dev, "slot_sta (1) bailing\n");
            return;
-       }
+}
+
+       if ((sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP) != 0 &&
+           (slot_sta & PCIEM_SLOT_STA_MRLSS) != 0)
+{
+device_printf(dev, "slot_sta (2) bailing\n");
+           return;
+}
    }
 
    /*
@@ -1071,20 +1080,32 @@ pcib_hotplug_inserted(struct pcib_softc *sc)
 
    /* Pretend the card isn't present if a detach is forced. */
    if (sc->flags & PCIB_DETACHING)
+{
+device_printf(sc->dev, "(1) PCIB_DETACHING\n");
        return (false);
+}
 
    /* Card must be present in the slot. */
    if ((sc->pcie_slot_sta & PCIEM_SLOT_STA_PDS) == 0)
+{
+device_printf(sc->dev, "(2) PCIEM_SLOT_STA_PDS\n");
        return (false);
+}
 
    /* A power fault implicitly turns off power to the slot. */
    if (sc->pcie_slot_sta & PCIEM_SLOT_STA_PFD)
+{
+device_printf(sc->dev, "(3) PCIEM_SLOT_STA_PFD\n");
        return (false);
+}
 
    /* If the MRL is disengaged, the slot is powered off. */
    if (sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP &&
        (sc->pcie_slot_sta & PCIEM_SLOT_STA_MRLSS) != 0)
+{
+device_printf(sc->dev, "(4) CAP_MRLSP && STA_MRLSS\n");
        return (false);
+}
 
    return (true);
 }
@@ -1868,8 +1889,10 @@ pcib_child_present(device_t dev, device_t child)
    int retval;
 
    retval = bus_child_present(dev);
+device_printf(sc->dev, "(1) retval == %d\n", retval);
    if (retval != 0 && sc->flags & PCIB_HOTPLUG)
        retval = pcib_hotplug_present(sc);
+device_printf(sc->dev, "(2) retval == %d\n", retval);
    return (retval);
 #else
    return (bus_child_present(dev));

With those patches applied, here is "egrep -5 pcib10 /var/run/dmesg.boot":

hdac1: <NVIDIA (0x0fb9) HDA Controller> mem 0xd8080000-0xd8083fff at device 0.1 numa-domain 0 on pci7
pcib8: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <Intel Volume Management Device> mem 0x3800fc000000-0x3800fdffffff,0xf8000000-0xf9ffffff,0x3800fff00000-0x3800ffffffff at device 5.5 numa-domain 0 on pci8
pci9: <PCI bus> numa-domain 0 on pcib9
pcib10: <PCI-PCI bridge> irq 0 at device 2.0 numa-domain 0 on pci9
pcib10: link_sta == 0x3043
pcib10: slot_sta == 0x15a
pcib10: failed to allocate initial I/O port window: 0-0xafff
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib11: <PCI-PCI bridge> irq 0 at device 3.0 numa-domain 0 on pci9
pcib11: link_sta == 0x1001
pcib11: failed to allocate initial I/O port window: 0-0xbfff
pcib11: (2) PCIEM_SLOT_STA_PDS
pcib11: (2) PCIEM_SLOT_STA_PDS
--
cpufreq17: <CPU frequency control> numa-domain 0 on cpu17
hwpstate_intel18: <Intel Speed Shift> numa-domain 0 on cpu18
cpufreq18: <CPU frequency control> numa-domain 0 on cpu18
hwpstate_intel19: <Intel Speed Shift> numa-domain 0 on cpu19
cpufreq19: <CPU frequency control> numa-domain 0 on cpu19
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pci10: <PCI bus> numa-domain 0 on pcib10
nvme0: <Generic NVMe Device> irq 0 at device 0.0 numa-domain 0 on pci10
hdacc0: <Realtek ALC255 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC255 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC255 (Analog 2.0+HP/2.0)> at nid 20,27 and 26 on hdaa0
pcm1: <Realtek ALC255 (Analog)> at nid 33 and 25 on hdaa0
--
hdaa1: <NVIDIA (0x0080) Audio Function Group> at nid 1 on hdacc1
pcm2: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 4 on hdaa1
pcm3: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 5 on hdaa1
pcm4: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 6 on hdaa1
pcm5: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 7 on hdaa1
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
Timecounter "TSC-low" frequency 1655996684 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ugen0.1: <Intel XHCI root HUB> at usbus0
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
uhub0 numa-domain 0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (1) retval == -1
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (2) retval == 0
nvme0: failing outstanding i/o
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:14 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:14 cdw0:0
nvme0: ASYNC_EVENT_REQUEST (0c) sqid:0 cid:13 nsid:0 cdw10:00000000 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:0 cid:13 cdw0:0
--
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses0: ada0,pass0 in 'Slot 00', SATA Slot: scbus0 target 0
ses0: ada1,pass1 in 'Slot 01', SATA Slot: scbus1 target 0
ses0: cd0,pass2 in 'Slot 07', SATA Slot: scbus7 target 0
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
nvme0: detached
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
Trying to mount root from zfs:thumb/ROOT/default []...
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (1) retval == -1
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (2) retval == 0
pci10: detached
uhub0: 26 ports with 26 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <vendor 0x0461 DELL Laser Mouse> at usbus0
ugen0.3: <Generic 4-Port USB 2.0 Hub> at usbus0

Again, I'm a novice, but my naive impression is that the intent of jhb's patch is to detect some anomalous hardware conditions which represent edge cases where hot plugging is claimed, but not truly supported (or implemented) well.  Rather than set the PCIB_HOTPLUG capability flag, the patch directs the code to return from the function with no action taken.  Labeling those exits with printf's "bailing 1" and "bailing 2", neither of those paths is taken.

Always happy to provide more info, and learning a lot from you both.

Thank you,

Jim

Just to wrap up an open question, Warner had asked for some tracepoints in pcib_hotplug_present so I added:

@@ -1099,12 +1120,19 @@ pcib_hotplug_present(struct pcib_softc *sc)
 
    /* Card must be inserted. */
    if (!pcib_hotplug_inserted(sc))
+{
+device_printf(sc->dev, "card not inserted\n");
        return (0);
+}
 
    /* Require the Data Link Layer to be active. */
    if (!(sc->pcie_link_sta & PCIEM_LINK_STA_DL_ACTIVE))
+{
+device_printf(sc->dev, "Data Link Layer not active\n");
        return (0);
+}
 
+device_printf(sc->dev, "pcib_hotplug_present\n");
    return (-1);
 }
 
And saw:

---<<BOOT>>---
...
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <Intel Volume Management Device> mem 0x3800fc000000-0x3800fdffffff,0xf8000000-0xf9ffffff,0x3800fff00000-0x3800ffffffff at device 5.5 numa-domain 0 on pci8
pci9: <PCI bus> numa-domain 0 on pcib9
pcib10: <PCI-PCI bridge> irq 0 at device 2.0 numa-domain 0 on pci9
pcib10: link_sta == 0x3043
pcib10: slot_sta == 0x15a
pcib10: failed to allocate initial I/O port window: 0-0xafff
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib11: <PCI-PCI bridge> irq 0 at device 3.0 numa-domain 0 on pci9
pcib11: link_sta == 0x1001
pcib11: failed to allocate initial I/O port window: 0-0xbfff
pcib11: (2) PCIEM_SLOT_STA_PDS
pcib11: (2) PCIEM_SLOT_STA_PDS
pcib11: card not inserted
...
hwpstate_intel19: <Intel Speed Shift> numa-domain 0 on cpu19
cpufreq19: <CPU frequency control> numa-domain 0 on cpu19
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: pcib_hotplug_present
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
Timecounter "TSC-low" frequency 1655996628 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
...

Comment 20 John Baldwin 2025-04-17 13:20:31 UTC

(In reply to Jim Long from comment #17)
In this case, the PCI-e registers start at offset 0x90:

    cap 10[90] = PCI-Express 2 root port max data 128(256) ARI disabled

The SLOT_CAP register is at a relative offset of 0x14 from there, so can you please do:

pciconf -r pcib10 0xa4

You can also perhaps try this (fine to do on top of your existing patches):

diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c
index 5e71a376604b..27aea3ebd2ed 100644
--- a/sys/dev/pci/pci_pci.c
+++ b/sys/dev/pci/pci_pci.c
@@ -1294,9 +1294,12 @@ pcib_setup_hotplug(struct pcib_softc *sc)
 
 	sc->pcie_link_sta = pcie_read_config(dev, PCIER_LINK_STA, 2);
 	sc->pcie_slot_sta = pcie_read_config(dev, PCIER_SLOT_STA, 2);
+	device_printf(dev, "initial SLOT_STA %#x\n", sc->pcie_slot_sta);
 
 	/* Clear any events previously pending. */
 	pcie_write_config(dev, PCIER_SLOT_STA, sc->pcie_slot_sta, 2);
+	sc->pcie_slot_sta = pcie_read_config(dev, PCIER_SLOT_STA, 2);
+	device_printf(dev, "cleared SLOT_STA %#x\n", sc->pcie_slot_sta);
 
 	/* Enable HotPlug events. */
 	mask = PCIEM_SLOT_CTL_DLLSCE | PCIEM_SLOT_CTL_HPIE |

The one difference here beyond the printf's is that it is explicitly discarding any pending events from POST.

Quite a bit of output this time:

---<<BOOT>>---
...
pcib5: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
pcib6: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 0.0 numa-domain 0 on pci6
pci7: <ACPI PCI bus> numa-domain 0 on pcib7
vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xd7000000-0xd7ffffff,0xc0000000-0xcfffffff,0xd0000000-0xd1ffffff at device 0.0 numa-domain 0 on pci7
hdac1: <NVIDIA (0x0fb9) HDA Controller> mem 0xd8080000-0xd8083fff at device 0.1 numa-domain 0 on pci7
pcib8: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <Intel Volume Management Device> mem 0x3800fc000000-0x3800fdffffff,0xf8000000-0xf9ffffff,0x3800fff00000-0x3800ffffffff at device 5.5 numa-domain 0 on pci8
pci9: <PCI bus> numa-domain 0 on pcib9
pcib10: <PCI-PCI bridge> irq 0 at device 2.0 numa-domain 0 on pci9
pcib10: link_sta == 0x3043
pcib10: slot_sta == 0x15a
pcib10: failed to allocate initial I/O port window: 0-0xafff
pcib10: initial SLOT_STA 0x15a
pcib10: cleared SLOT_STA 0x40
pcib10: pcib_hotplug_present
pci10: <PCI bus> numa-domain 0 on pcib10
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
nvme0: <Generic NVMe Device> irq 0 at device 0.0 numa-domain 0 on pci10
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib11: <PCI-PCI bridge> irq 0 at device 3.0 numa-domain 0 on pci9
pcib11: link_sta == 0x1001
pcib11: failed to allocate initial I/O port window: 0-0xbfff
pcib11: initial SLOT_STA 0x12
pcib11: cleared SLOT_STA 0
pcib11: (2) PCIEM_SLOT_STA_PDS
pcib11: (2) PCIEM_SLOT_STA_PDS
pcib11: card not inserted
pci8: <dasp, performance counters> at device 18.0 (no driver attached)
pci8: <dasp, performance counters> at device 18.1 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
atrtc0: Can't map interrupt.
atrtc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 15.
hwpstate_intel0: <Intel Speed Shift> numa-domain 0 on cpu0
cpufreq0: <CPU frequency control> numa-domain 0 on cpu0
hwpstate_intel1: <Intel Speed Shift> numa-domain 0 on cpu1
cpufreq1: <CPU frequency control> numa-domain 0 on cpu1
hwpstate_intel2: <Intel Speed Shift> numa-domain 0 on cpu2
cpufreq2: <CPU frequency control> numa-domain 0 on cpu2
hwpstate_intel3: <Intel Speed Shift> numa-domain 0 on cpu3
cpufreq3: <CPU frequency control> numa-domain 0 on cpu3
hwpstate_intel4: <Intel Speed Shift> numa-domain 0 on cpu4
cpufreq4: <CPU frequency control> numa-domain 0 on cpu4
hwpstate_intel5: <Intel Speed Shift> numa-domain 0 on cpu5
cpufreq5: <CPU frequency control> numa-domain 0 on cpu5
hwpstate_intel6: <Intel Speed Shift> numa-domain 0 on cpu6
cpufreq6: <CPU frequency control> numa-domain 0 on cpu6
hwpstate_intel7: <Intel Speed Shift> numa-domain 0 on cpu7
cpufreq7: <CPU frequency control> numa-domain 0 on cpu7
hwpstate_intel8: <Intel Speed Shift> numa-domain 0 on cpu8
cpufreq8: <CPU frequency control> numa-domain 0 on cpu8
hwpstate_intel9: <Intel Speed Shift> numa-domain 0 on cpu9
cpufreq9: <CPU frequency control> numa-domain 0 on cpu9
hwpstate_intel10: <Intel Speed Shift> numa-domain 0 on cpu10
cpufreq10: <CPU frequency control> numa-domain 0 on cpu10
hwpstate_intel11: <Intel Speed Shift> numa-domain 0 on cpu11
cpufreq11: <CPU frequency control> numa-domain 0 on cpu11
hwpstate_intel12: <Intel Speed Shift> numa-domain 0 on cpu12
cpufreq12: <CPU frequency control> numa-domain 0 on cpu12
hwpstate_intel13: <Intel Speed Shift> numa-domain 0 on cpu13
cpufreq13: <CPU frequency control> numa-domain 0 on cpu13
hwpstate_intel14: <Intel Speed Shift> numa-domain 0 on cpu14
cpufreq14: <CPU frequency control> numa-domain 0 on cpu14
hwpstate_intel15: <Intel Speed Shift> numa-domain 0 on cpu15
cpufreq15: <CPU frequency control> numa-domain 0 on cpu15
hwpstate_intel16: <Intel Speed Shift> numa-domain 0 on cpu16
cpufreq16: <CPU frequency control> numa-domain 0 on cpu16
hwpstate_intel17: <Intel Speed Shift> numa-domain 0 on cpu17
cpufreq17: <CPU frequency control> numa-domain 0 on cpu17
hwpstate_intel18: <Intel Speed Shift> numa-domain 0 on cpu18
cpufreq18: <CPU frequency control> numa-domain 0 on cpu18
hwpstate_intel19: <Intel Speed Shift> numa-domain 0 on cpu19
cpufreq19: <CPU frequency control> numa-domain 0 on cpu19
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (1) retval == -1
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (2) retval == 0
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
nvme0: detached
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (1) retval == -1
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
pcib10: (2) retval == 0
pci10: detached
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted
Timecounter "TSC-low" frequency 1655996800 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ugen0.1: <Intel XHCI root HUB> at usbus0
ZFS storage pool version: features support (5000)
hdacc0: <Realtek ALC255 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC255 Audio Function Group> at nid 1 on hdacc0
...

# pciconf -r pcib10 0xa4
02d80078

Comment 22 John Baldwin 2025-04-18 18:40:24 UTC

Hmm, so two things stand out to me:

1) We were able to clear the PFD bit (0x02):

pcib10: initial SLOT_STA 0x15a
pcib10: cleared SLOT_STA 0x40

yet the device set it again later:

pcib10: Power Fault Detected
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: (3) PCIEM_SLOT_STA_PFD
pcib10: card not inserted

It was also set for pcib11 initially:

pcib11: initial SLOT_STA 0x12
pcib11: cleared SLOT_STA 0

2) The slot capabilities register (the one read at offset 0xa4) does not include support for a power controller:

#define	PCIEM_SLOT_CAP_PCP		0x00000002

is not set in 0x02d80078.

Without a power controller present, there shouldn't be power fault events by my reading of the PCI-e spec.  That suggests that this might fix your system:

diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c
index 5e71a376604b..10de719e020d 100644
--- a/sys/dev/pci/pci_pci.c
+++ b/sys/dev/pci/pci_pci.c
@@ -930,7 +930,8 @@ pcib_hotplug_inserted(struct pcib_softc *sc)
 		return (false);
 
 	/* A power fault implicitly turns off power to the slot. */
-	if (sc->pcie_slot_sta & PCIEM_SLOT_STA_PFD)
+	if (sc->pcie_slot_cap & PCIEM_SLOT_CAP_PCP &&
+	    sc->pcie_slot_sta & PCIEM_SLOT_STA_PFD)
 		return (false);
 
 	/* If the MRL is disengaged, the slot is powered off. */

As soon as I saw that test, a bell started ringing.  If the controller doesn't have power controller capability, then don't pay any heed to power faults.

Yes, it works just dandy.  I had been getting by with a /boot/loader.conf hack:

hw.pci.enable_pcie_hp=0

Your patch now makes that unnecessary.

Thank you very much, Everyone!

Sorry, latest dmesg excerpts:

# egrep -4 'pcib?1[01]|nvme|nda' dmesg.n271070 
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.2-STABLE #4 stable/14-n271070-4317d0aa8c21-dirty: Mon Apr 21 07:38:34 PDT 2025
    root@spare-diatom3.smate.wwu.edu:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)
VT(efifb): resolution 1920x1200
--
pcib8: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <Intel Volume Management Device> mem 0x3800fc000000-0x3800fdffffff,0xf8000000-0xf9ffffff,0x3800fff00000-0x3800ffffffff at device 5.5 numa-domain 0 on pci8
pci9: <PCI bus> numa-domain 0 on pcib9
pcib10: <PCI-PCI bridge> irq 0 at device 2.0 numa-domain 0 on pci9
pcib10: link_sta == 0x3043
pcib10: slot_sta == 0x15a
pcib10: failed to allocate initial I/O port window: 0-0xafff
pcib10: initial SLOT_STA 0x15a
pcib10: cleared SLOT_STA 0x40
pcib10: pcib_hotplug_present
pci10: <PCI bus> numa-domain 0 on pcib10
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
nvme0: <Generic NVMe Device> irq 0 at device 0.0 numa-domain 0 on pci10
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib11: <PCI-PCI bridge> irq 0 at device 3.0 numa-domain 0 on pci9
pcib11: link_sta == 0x1001
pcib11: failed to allocate initial I/O port window: 0-0xbfff
pcib11: initial SLOT_STA 0x12
pcib11: cleared SLOT_STA 0
pcib11: (2) PCIEM_SLOT_STA_PDS
pcib11: (2) PCIEM_SLOT_STA_PDS
pcib11: card not inserted
pci8: <dasp, performance counters> at device 18.0 (no driver attached)
pci8: <dasp, performance counters> at device 18.1 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
--
hwpstate_intel18: <Intel Speed Shift> numa-domain 0 on cpu18
cpufreq18: <CPU frequency control> numa-domain 0 on cpu18
hwpstate_intel19: <Intel Speed Shift> numa-domain 0 on cpu19
cpufreq19: <CPU frequency control> numa-domain 0 on cpu19
pcib10: Power Fault Detected
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
Timecounter "TSC-low" frequency 1655996684 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
--
pcm2: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 4 on hdaa1
pcm3: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 5 on hdaa1
pcm4: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 6 on hdaa1
pcm5: <NVIDIA (0x0080) (HDMI/DP 8ch)> at nid 7 on hdaa1
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
nda0 at nvme0 bus 0 scbus9 target 0 lun 1
nda0: <KXG50ZNV512G NVMe TOSHIBA 512GB AADA4105 78MS1223T8HQ>
nda0: Serial Number 78Mpcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
pcib10: pcib_hotplug_present
S1223T8HQ
nda0: nvme version 1.2
nda0: 488386MB (1000215216 512 byte sectors)
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST3000DM008-2DM166 CC26> ACS-2 ATA SATA 3.x device
ada0: Serial Number Z505F4XJ
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
--
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors)
ada1: quirks=0x1<4K>
pcib10: pcib_hotplug_present
ses0 at ahciem0 bus 0 scbus8 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses0: ada0,pass0 in 'Slot 00', SATA Slot: scbus0 target 0

Comment 25 commit-hook 2025-04-22 02:58:48 UTC

A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=4c8b9f40cda7dd6b2de547b288d712c85998ce11

commit 4c8b9f40cda7dd6b2de547b288d712c85998ce11
Author:     John Baldwin <jhb@FreeBSD.org>
AuthorDate: 2025-04-22 02:08:01 +0000
Commit:     John Baldwin <jhb@FreeBSD.org>
CommitDate: 2025-04-22 02:08:01 +0000

    pcib: Ignore power fault detected if a power controller is not present

    At least some PCI-express bridges exposed by Intel Volume Management
    Devices report bogus power faults in their slot status register.  As a
    workaround, ignore the power fault detected flag if the slot
    capability register does not indicate support for a power controller.

    While the PCI-e spec does not explicitly state that the PFD flag
    should always be zero if a power controller is not present, parts of
    the spec do seem to assume this.  For example, the flag is included in
    the "Power Controller Registers" register group in 6.7.2.4, and the
    definition of the value for this flag seems to presume the presence of
    a power controller:

        If a Power Controller that supports power fault detection is
        implemented, this bit is Set when the Power Controller detects a
        power fault at this slot. Note that, depending on hardware
        capability, it is possible that a power fault can be detected at
        any time, independent of the Power Controller Control setting or
        the occupancy of the slot. If power fault detection is not
        supported, this bit must not be Set.

    PR:             285993
    Reported by:    Jim Long <freebsd-bugzilla@umpquanet.com>
    Reviewed by:    imp
    Differential Revision:  https://reviews.freebsd.org/D49949

 sys/dev/pci/pci_pci.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)