I upgraded our fleet of ~20 NFS servers today from 12.1 to 12.2-RELEASE-p2. Three of the newest servers, which have X722 interfaces on the motherboard, crash at attach time with the following trace: --- trap 0xc, rip = 0xffffffff80757efd, rsp = 0xfffffe0075dff2d0, rbp = 0xfffffe0075dff2e0 --- grouptaskqueue_enqueue() at grouptaskqueue_enqueue+0xd/frame 0xfffffe0075dff2e0 ixl_admin_timer() at ixl_admin_timer+0x11/frame 0xfffffe0075dff300 softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe0075dff3b0 softclock() at softclock+0x79/frame 0xfffffe0075dff3d0 ithread_loop() at ithread_loop+0x23c/frame 0xfffffe0075dff430 fork_exit() at fork_exit+0x7e/frame 0xfffffe0075dff470 fork_trampoline() at f (There's an omitted frame where ixl_admin_timer() calls iflib_admin_intr_deferred() -- apparently pf->vsi.ctx is not initialized yet, or there's a missing memory barrier?) The problem is not observed on older servers with X710 interfaces. (We don't use the 1G interfaces for anything, and if we didn't have servers that need it, I would just yank ixl(4) from our standard kernel configuration. I might still do that and just load the module on the three servers that need it.) The systems that panic have: ixl0@pci0:26:0:0: class=0x020000 card=0x37d115d9 chip=0x37d18086 rev=0x09 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection X722 for 1GbE' class = network subclass = ethernet The systems that boot (and operate properly) have: ixl0@pci0:3:0:0: class=0x020000 card=0x00088086 chip=0x15728086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Controller X710 for 10GbE SFP+' class = network subclass = ethernet
(In reply to Garrett Wollman from comment #0) vsi->ctx is set at the beginning of ixl_if_attach_pre and timer is started at the end of ixl_if_attach_post so it looks a bit strange. I haven't seen anything like that before. I'm trying to get a reproduction. Is there a chance for a core dump? Also could you, please, check the FW version (sysctl dev.ixl.0.fw_version) and provide exact model of the motherboard?
(In reply to Krzysztof Galazka from comment #1) Since these are production file servers, I removed the ixl driver so that I could complete the upgrade within the scheduled window, and won't be able to tell you the firmware version any time soon. The machine is an iXsystems IXC-4224P-IXN, and the motherboard is a Supermicro X11DPH-i rev 1.10. According to our inventory, these servers were delivered in October, 2018.