Bug 252626 - ixl: panic on attach of X722 on-motherboard interfaces
Summary: ixl: panic on attach of X722 on-motherboard interfaces
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.2-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-net (Nobody)
URL:
Keywords: IntelNetworking, panic
Depends on:
Blocks:
 
Reported: 2021-01-12 23:49 UTC by Garrett Wollman
Modified: 2021-01-15 18:23 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Garrett Wollman freebsd_committer 2021-01-12 23:49:32 UTC
I upgraded our fleet of ~20 NFS servers today from 12.1 to 12.2-RELEASE-p2. Three of the newest servers, which have X722 interfaces on the motherboard, crash at attach time with the following trace:

--- trap 0xc, rip = 0xffffffff80757efd, rsp = 0xfffffe0075dff2d0, rbp = 0xfffffe0075dff2e0 ---
grouptaskqueue_enqueue() at grouptaskqueue_enqueue+0xd/frame 0xfffffe0075dff2e0
ixl_admin_timer() at ixl_admin_timer+0x11/frame 0xfffffe0075dff300
softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe0075dff3b0
softclock() at softclock+0x79/frame 0xfffffe0075dff3d0
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe0075dff430
fork_exit() at fork_exit+0x7e/frame 0xfffffe0075dff470
fork_trampoline() at f

(There's an omitted frame where ixl_admin_timer() calls iflib_admin_intr_deferred() -- apparently pf->vsi.ctx is not initialized yet, or there's a missing memory barrier?)

The problem is not observed on older servers with X710 interfaces.  (We don't use the 1G interfaces for anything, and if we didn't have servers that need it, I would just yank ixl(4) from our standard kernel configuration.  I might still do that and just load the module on the three servers that need it.)

The systems that panic have:

ixl0@pci0:26:0:0:       class=0x020000 card=0x37d115d9 chip=0x37d18086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X722 for 1GbE'
    class      = network
    subclass   = ethernet

The systems that boot (and operate properly) have:

ixl0@pci0:3:0:0:        class=0x020000 card=0x00088086 chip=0x15728086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller X710 for 10GbE SFP+'
    class      = network
    subclass   = ethernet
Comment 1 Krzysztof Galazka 2021-01-14 14:20:08 UTC
(In reply to Garrett Wollman from comment #0)
vsi->ctx is set at the beginning of ixl_if_attach_pre and timer is started at the end of ixl_if_attach_post so it looks a bit strange. I haven't seen anything like that before. I'm trying to get a reproduction. Is there a chance for a core dump? Also could you, please, check the FW version (sysctl dev.ixl.0.fw_version) and provide exact model of the motherboard?
Comment 2 Garrett Wollman freebsd_committer 2021-01-15 18:23:40 UTC
(In reply to Krzysztof Galazka from comment #1)
Since these are production file servers, I removed the ixl driver so that I could complete the upgrade within the scheduled window, and won't be able to tell you the firmware version any time soon.

The machine is an iXsystems IXC-4224P-IXN, and the motherboard is a Supermicro X11DPH-i rev 1.10.  According to our inventory, these servers were delivered in October, 2018.