Hello, As reported on twitter on FreeBSD 12.0-ALPHA2 r338240 on Supermicro X11DPU with 2 10G interfaces traffic going thru ix1 using VLAN interface doesn't work completly. Have to set the ix1 in native mode to make this work. Dmesg (not on r338240, bug still present) : ---<<BOOT>>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-ALPHA2 #0 r338182: Wed Aug 22 09:59:11 CEST 2018 root@rsv-filer:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) WARNING: WITNESS option enabled, expect reduced performance. VT(efifb): resolution 1024x768 CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz (2100.05-MHz K8-class CPU) Origin="GenuineIntel" Id=0x50654 Family=0x6 Model=0x55 Stepping=4 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0xd39ffffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,MPX,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,AVX512BW,AVX512VL> Structured Extended Features2=0x8<PKU> Structured Extended Features3=0xc000000<IBPB,STIBP> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 66447601664 (63369 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <SUPERM SMCI--MB> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 hardware threads random: unblocking device. ioapic6: Changing APIC ID to 16 ioapic7: Changing APIC ID to 17 ioapic8: Changing APIC ID to 18 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-31 on motherboard ioapic2 <Version 2.0> irqs 32-39 on motherboard ioapic3 <Version 2.0> irqs 40-47 on motherboard ioapic4 <Version 2.0> irqs 48-55 on motherboard ioapic5 <Version 2.0> irqs 72-79 on motherboard ioapic6 <Version 2.0> irqs 80-87 on motherboard ioapic7 <Version 2.0> irqs 88-95 on motherboard ioapic8 <Version 2.0> irqs 96-103 on motherboard Launching APs: 1 26 6 8 4 20 10 2 12 16 14 24 22 18 9 15 31 29 30 27 3 28 25 7 5 17 11 21 13 23 19 Timecounter "TSC" frequency 2100054984 Hz quality 1000 random: entropy device external interface [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0xffffffff811011c0, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" kbd1 at kbdmux0 netmap: loaded module nexus0 efirtc0: <EFI Realtime Clock> on motherboard efirtc0: registered as a time-of-day clock, resolution 1.000000s cryptosoft0: <software crypto> on motherboard acpi0: <SUPERM SUPERM> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 24000000 Hz quality 950 Event timer "HPET" frequency 24000000 Hz quality 350 Event timer "HPET1" frequency 24000000 Hz quality 340 Event timer "HPET2" frequency 24000000 Hz quality 340 Event timer "HPET3" frequency 24000000 Hz quality 340 Event timer "HPET4" frequency 24000000 Hz quality 340 Event timer "HPET5" frequency 24000000 Hz quality 340 Event timer "HPET6" frequency 24000000 Hz quality 340 Event timer "HPET7" frequency 24000000 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0 acpi_syscontainer0: <System Container> on acpi0 acpi_syscontainer1: <System Container> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pci0: <dasp, performance counters> at device 8.1 (no driver attached) pci0: <unknown> at device 17.0 (no driver attached) pci0: <unknown> at device 17.1 (no driver attached) ahci0: <Intel Lewisburg AHCI SATA controller> port 0x3070-0x3077,0x3060-0x3063,0x3020-0x303f mem 0x9d206000-0x9d207fff,0x9d209000-0x9d2090ff,0x9d180000-0x9d1fffff irq 16 at device 17.5 on pci0 ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 ahciem0: <AHCI enclosure management bridge> on ahci0 xhci0: <Intel Lewisburg USB 3.0 controller> mem 0x383ffff00000-0x383ffff0ffff irq 16 at device 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0: waiting for BIOS to give up control xhci_interrupt: host controller halted usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci0: <simple comms> at device 22.0 (no driver attached) pci0: <simple comms> at device 22.1 (no driver attached) pci0: <simple comms> at device 22.4 (no driver attached) ahci1: <Intel Lewisburg AHCI SATA controller> port 0x3050-0x3057,0x3040-0x3043,0x3000-0x301f mem 0x9d204000-0x9d205fff,0x9d208000-0x9d2080ff,0x9d100000-0x9d17ffff irq 16 at device 23.0 on pci0 ahci1: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported ahcich6: <AHCI channel> at channel 0 on ahci1 ahcich7: <AHCI channel> at channel 1 on ahci1 ahcich8: <AHCI channel> at channel 2 on ahci1 ahcich9: <AHCI channel> at channel 3 on ahci1 ahcich10: <AHCI channel> at channel 4 on ahci1 ahcich11: <AHCI channel> at channel 5 on ahci1 ahcich12: <AHCI channel> at channel 6 on ahci1 ahcich13: <AHCI channel> at channel 7 on ahci1 ahciem1: <AHCI enclosure management bridge> on ahci1 pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0 pci2: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> irq 17 at device 0.0 on pci2 pci3: <ACPI PCI bus> on pcib3 vgapci0: <VGA-compatible display> port 0x2000-0x207f mem 0x9c000000-0x9cffffff,0x9d000000-0x9d01ffff irq 17 at device 0.0 on pci3 vgapci0: Boot video device isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 pci0: <memory> at device 31.2 (no driver attached) pci0: <serial bus> at device 31.5 (no driver attached) pcib4: <ACPI Host-PCI bridge> on acpi0 pci4: <ACPI PCI bus> on pcib4 pcib5: <ACPI PCI-PCI bridge> irq 39 at device 0.0 on pci4 pci5: <ACPI PCI bus> on pcib5 ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0x5020-0x503f mem 0xaab80000-0xaabfffff,0xaac04000-0xaac07fff irq 32 at device 0.0 on pci5 ix0: using 2048 tx descriptors and 2048 rx descriptors ix0: msix_init qsets capped at 16 ix0: pxm cpus: 16 queue msgs: 63 admincnt: 1 ix0: using 16 rx queues 16 tx queues ix0: Using MSIX interrupts with 17 vectors ix0: allocated for 16 queues ix0: allocated for 16 rx queues ix0: Ethernet address: 0c:c4:7a:f4:34:a0 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: netmap queues/slots: TX 16/2048, RX 16/2048 ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0x5000-0x501f mem 0xaab00000-0xaab7ffff,0xaac00000-0xaac03fff irq 36 at device 0.1 on pci5 ix1: using 2048 tx descriptors and 2048 rx descriptors ix1: msix_init qsets capped at 16 ix1: pxm cpus: 16 queue msgs: 63 admincnt: 1 ix1: using 16 rx queues 16 tx queues ix1: Using MSIX interrupts with 17 vectors ix1: allocated for 16 queues ix1: allocated for 16 rx queues ix1: Ethernet address: 0c:c4:7a:f4:34:a1 ix1: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: netmap queues/slots: TX 16/2048, RX 16/2048 pcib6: <ACPI PCI-PCI bridge> irq 39 at device 2.0 on pci4 pci6: <ACPI PCI bus> on pcib6 nvme0: <Samsung PM1725a> mem 0xaae10000-0xaae13fff irq 39 at device 0.0 on pci6 pcib7: <ACPI PCI-PCI bridge> irq 39 at device 3.0 on pci4 pci7: <ACPI PCI bus> on pcib7 nvme1: <Samsung PM1725a> mem 0xaad10000-0xaad13fff irq 39 at device 0.0 on pci7 pcib8: <ACPI Host-PCI bridge> on acpi0 pci8: <ACPI PCI bus> on pcib8 pcib9: <ACPI PCI-PCI bridge> irq 47 at device 0.0 on pci8 pci9: <ACPI PCI bus> on pcib9 pcib10: <ACPI PCI-PCI bridge> irq 47 at device 1.0 on pci8 pci10: <ACPI PCI bus> on pcib10 pcib11: <ACPI Host-PCI bridge> on acpi0 pci11: <ACPI PCI bus> on pcib11 pcib12: <ACPI PCI-PCI bridge> irq 55 at device 2.0 on pci11 pci12: <ACPI PCI bus> on pcib12 mpr0: <Avago Technologies (LSI) SAS3008> port 0x9000-0x90ff mem 0xc5e40000-0xc5e4ffff,0xc5e00000-0xc5e3ffff irq 55 at device 0.0 on pci12 mpr0: Firmware: 15.00.03.00, Driver: 18.03.00.00-fbsd mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray> pci11: <dasp, performance counters> at device 14.0 (no driver attached) pci11: <dasp, performance counters> at device 15.0 (no driver attached) pci11: <dasp, performance counters> at device 18.0 (no driver attached) pci11: <dasp, performance counters> at device 18.1 (no driver attached) pcib13: <ACPI Host-PCI bridge> on acpi0 pci13: <ACPI PCI bus> on pcib13 pci13: <dasp, performance counters> at device 8.1 (no driver attached) pcib14: <ACPI Host-PCI bridge> on acpi0 pci14: <ACPI PCI bus> on pcib14 pcib15: <ACPI PCI-PCI bridge> irq 87 at device 2.0 on pci14 pci15: <ACPI PCI bus> on pcib15 pcib16: <ACPI PCI-PCI bridge> irq 87 at device 3.0 on pci14 pci16: <ACPI PCI bus> on pcib16 pcib17: <ACPI Host-PCI bridge> on acpi0 pci17: <ACPI PCI bus> on pcib17 pcib18: <ACPI Host-PCI bridge> on acpi0 pci18: <ACPI PCI bus> on pcib18 pci18: <dasp, performance counters> at device 14.0 (no driver attached) pci18: <dasp, performance counters> at device 15.0 (no driver attached) pci18: <dasp, performance counters> at device 18.0 (no driver attached) pci18: <dasp, performance counters> at device 18.1 (no driver attached) uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xce000-0xcefff pnpid ORM0000 on isa0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec ugen0.1: <0x8086 XHCI root HUB> at usbus0 uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 nvd0: <SAMSUNG MZWLL800HEHP-00003> NVMe namespace nvd0: 763097MB (1562824368 512 byte sectors) nvd1: <SAMSUNG MZWLL800HEHP-00003> NVMe namespace nvd1: 763097MB (1562824368 512 byte sectors) uhub0: 26 ports with 26 removable, self powered mpr0: Found device <c01<SspTarg,Direct>,End Device> <6.0Gbps> handle<0x0009> enclosureHandle<0x0001> slot 0 mpr0: At enclosure level 0 and connector name ( ) lock order reversal: mpr0: 1st 0xfffff802d0d254d0 CAM device lock (CAM device lock) @ /usr/src/sys/cam/cam_xpt.c:831 2nd 0xfffffe00befa2218 MPR lock (MPR lock) @ /usr/src/sys/cam/cam_xpt.c:2808 stack backtrace: #0 0xffffffff80c02553 at witnesFound device <c01<SspTarg,Direct>,End Device> <3.0Gbps> handle<0x000a> enclosureHandle<0x0001> slot 1 mpr0: At enclosure level 0 and connector name ( ) s_debugger+0x73 #1 0xffffffff80c023d4 at witness_checkorder+0xe34 #2 0xffffffff80b74c8c at __mtx_lock_flags+0x9c #3 0xffffffugen0.2: <vendor 0x0557 product 0x7000> at usbus0 ff8034a24a at xpt_action_default+0x1ba #4 0xffffffff8035f1e8 at scsi_scan_bus+0x68 mpr0: #5 0xffffffff80350e51 at xpt_scanner_thread+0x141 #6 0xffffffff80b55334 at fork_exit+0x84 #7 0xffffffff8105941e at fork_trampoline+0xe Found device <c01<SspTarg,Direct>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0001> slot 2 mpr0: At enclosure level 0 and connector name ( ) mpr0: uhub1Found device <c01<SspTarg,Direct>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0001> slot 3 mpr0: At enclosure level 0 and connector name ( ) mpr0: on uhub0 uhub1: Found device <c01<SspTarg,Direct>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0001> slot 4 mpr0: At enclosure level 0 and connector name ( ) mpr0: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 1> on usbus0 Found device <c01<SspTarg,Direct>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0001> slot 5 mpr0: At enclosure level 0 and connector name ( ) uhub1: 4 ports with 3 removable, self powered ugen0.3: <vendor 0x0557 product 0x2419> at usbus0 ukbd0 on uhub1 ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0 kbd2 at ukbd0 ugen0.4: <vendor 0x048d product 0x1172> at usbus0 umass0 on uhub0 umass0: <vendor 0x048d product 0x1172, class 0/0, rev 2.00/1.00, addr 3> on usbus0 umass0: SCSI over Bulk-Only; quirks = 0x8100 umass0:17:0: Attached to scbus17 da2 at mpr0 bus 0 scbus16 target 4 lun 0 da2: <SEAGATE ST1000NX0453 N004> Fixed Direct Access SPC-4 SCSI device da2: Serial Number W4714PYH0000E8110EQE da2: 1200.000MB/s transfers da2: Command Queueing enabled da2: 953869MB (1953525168 512 byte sectors) da3 at mpr0 bus 0 scbus16 target 5 lun 0 da3: <SEAGATE ST1000NX0453 N004> Fixed Direct Access SPC-4 SCSI device da3: Serial Number W4714Q0R0000E810FKY7 da3: 1200.000MB/s transfers da3: Command Queueing enabled da3: 953869MB (1953525168 512 byte sectors) da4 at mpr0 bus 0 scbus16 target 6 lun 0 da4: <SEAGATE ST1000NX0453 N004> Fixed Direct Access SPC-4 SCSI device da4: Serial Number W4714PYY0000E81158Q4 da4: 1200.000MB/s transfers da4: Command Queueing enabled da4: 953869MB (1953525168 512 byte sectors) da5 at mpr0 bus 0 scbus16 target 7 lun 0 da5: <SEAGATE ST1000NX0453 N004> Fixed Direct Access SPC-4 SCSI device da5: Serial Number W4714QZ40000E8111C4T da5: 1200.000MB/s transfers da5: Command Queueing enabled da5: 953869MB (1953525168 512 byte sectors) da1 at mpr0 bus 0 scbus16 target 3 lun 0 da1: <HP DG072A9BB7 HPD0> Fixed Direct Access SPC-3 SCSI device da1: Serial Number B365P710BTFJ0702 da1: 300.000MB/s transfers da1: Command Queueing enabled da1: 70007MB (143374738 512 byte sectors) da6 at umass-sim0 bus 0 scbus17 target 0 lun 0 da6: < 0.00> Removable Direct Access SPC-2 SCSI device da6: Serial Number B1311020000020 da6: 40.000MB/s transfers da6: 3854MB (7892992 512 byte sectors) da6: quirks=0x2<NO_6_BYTE> da0 at mpr0 bus 0 scbus16 target 2 lun 0 da0: <HP EG0300FCSPH HPD0> Fixed Direct Access SPC-3 SCSI device da0: Serial Number 14V0A0JHFTM91405 da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 286102MB (585937500 512 byte sectors) ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus15 target 0 lun 0 ses1: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device ses1: SEMB SES Device WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from zfs:rsvfiler/ROOT/default []... GEOM_ELI: Device da1p2.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software lo0: link state changed to UP lock order reversal: (sleepable after non-sleepable) 1st 0xfffff802e1167988 vlan (vlan) @ /usr/src/sys/net/if_vlan.c:1931 2nd 0xfffff800059b0160 iflib ctx lock (iflib ctx lock) @ /usr/src/sys/net/iflib.c:4198 stack backtrace: #0 0xffffffff80c02553 at witness_debugger+0x73 #1 0xffffffff80c023d4 at witness_checkorder+0xe34 #2 0xffffffff80b9fe48 at _sx_xlock+0x68 #3 0xffffffff80cbeeac at iflib_if_ioctl+0x8c #4 0xffffffff80ca01da at if_addmulti+0x24a #5 0xffffffff80cb11c6 at vlan_setmulti+0x196 #6 0xffffffff80cb0349 at vlan_ioctl+0xa9 #7 0xffffffff80ca01da at if_addmulti+0x24a #8 0xffffffff80d26a9a at in_joingroup_locked+0x23a #9 0xffffffff80d26832 at in_joingroup+0x42 #10 0xffffffff80d21b28 at in_control+0xb08 #11 0xffffffff80c9f1f1 at ifioctl+0x1861 #12 0xffffffff80c07d79 at kern_ioctl+0x2b9 #13 0xffffffff80c07a4e at sys_ioctl+0x15e #14 0xffffffff8107ea21 at amd64_syscall+0x281 #15 0xffffffff81058d1d at fast_syscall_common+0x101 ix0: link state changed to UP vlan3401: link state changed to UP ix1: link state changed to UP vlan3402: link state changed to UP ioat0: <SKX IOAT> mem 0x383ffff2c000-0x383ffff2ffff irq 26 at device 4.0 on pci0 ioat0: Capabilities: f0006e1<DMA_with_DIF,PQ,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat1: <SKX IOAT> mem 0x383ffff28000-0x383ffff2bfff irq 27 at device 4.1 on pci0 ioat1: Capabilities: f0006e1<DMA_with_DIF,PQ,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat2: <SKX IOAT> mem 0x383ffff24000-0x383ffff27fff irq 26 at device 4.2 on pci0 ioat2: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat3: <SKX IOAT> mem 0x383ffff20000-0x383ffff23fff irq 27 at device 4.3 on pci0 ioat3: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat4: <SKX IOAT> mem 0x383ffff1c000-0x383ffff1ffff irq 26 at device 4.4 on pci0 ioat4: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat5: <SKX IOAT> mem 0x383ffff18000-0x383ffff1bfff irq 27 at device 4.5 on pci0 ioat5: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat6: <SKX IOAT> mem 0x383ffff14000-0x383ffff17fff irq 26 at device 4.6 on pci0 ioat6: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat7: <SKX IOAT> mem 0x383ffff10000-0x383ffff13fff irq 27 at device 4.7 on pci0 ioat7: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat8: <SKX IOAT> mem 0x393ffff1c000-0x393ffff1ffff irq 74 at device 4.0 on pci13 ioat8: Capabilities: f0006e1<DMA_with_DIF,PQ,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat9: <SKX IOAT> mem 0x393ffff18000-0x393ffff1bfff irq 75 at device 4.1 on pci13 ioat9: Capabilities: f0006e1<DMA_with_DIF,PQ,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat10: <SKX IOAT> mem 0x393ffff14000-0x393ffff17fff irq 74 at device 4.2 on pci13 ioat10: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat11: <SKX IOAT> mem 0x393ffff10000-0x393ffff13fff irq 75 at device 4.3 on pci13 ioat11: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat12: <SKX IOAT> mem 0x393ffff0c000-0x393ffff0ffff irq 74 at device 4.4 on pci13 ioat12: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat13: <SKX IOAT> mem 0x393ffff08000-0x393ffff0bfff irq 75 at device 4.5 on pci13 ioat13: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat14: <SKX IOAT> mem 0x393ffff04000-0x393ffff07fff irq 74 at device 4.6 on pci13 ioat14: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ioat15: <SKX IOAT> mem 0x393ffff00000-0x393ffff03fff irq 75 at device 4.7 on pci13 ioat15: Capabilities: f0004e1<DMA_with_DIF,Extended_APIC_ID,Block_Fill,Move_CRC,Page_Break> ums0 on uhub1 ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 ix1: promiscuous mode enabled vlan3402: promiscuous mode enabled lock order reversal: 1st 0xffffffff81fc99e8 bpf global lock (bpf global lock) @ /usr/src/sys/net/bpf.c:792 2nd 0xffffffff81fcac20 vlan_sx (vlan_sx) @ /usr/src/sys/net/if_vlan.c:1913 stack backtrace: #0 0xffffffff80c02553 at witness_debugger+0x73 #1 0xffffffff80c023d4 at witness_checkorder+0xe34 #2 0xffffffff80b9fe48 at _sx_xlock+0x68 #3 0xffffffff80cb04eb at vlan_ioctl+0x24b #4 0xffffffff80c9fca8 at if_setflag+0xf8 #5 0xffffffff80c9fb4c at ifpromisc+0x2c #6 0xffffffff80c940c5 at bpf_detachd_locked+0x225 #7 0xffffffff80c96eeb at bpf_dtor+0x8b #8 0xffffffff80a4c85e at devfs_destroy_cdevpriv+0x8e #9 0xffffffff80a4ffc5 at devfs_close_f+0x65 #10 0xffffffff80b40a65 at closef+0x1f5 #11 0xffffffff80b3dae0 at closefp+0xa0 #12 0xffffffff8107ea21 at amd64_syscall+0x281 #13 0xffffffff81058d1d at fast_syscall_common+0x101 ix1: promiscuous mode disabled vlan3402: promiscuous mode disabled ix1: promiscuous mode enabled lock order reversal: 1st 0xffffffff81fc99e8 bpf global lock (bpf global lock) @ /usr/src/sys/net/bpf.c:792 2nd 0xfffff800080be960 iflib ctx lock (iflib ctx lock) @ /usr/src/sys/net/iflib.c:4180 stack backtrace: #0 0xffffffff80c02553 at witness_debugger+0x73 #1 0xffffffff80c023d4 at witness_checkorder+0xe34 #2 0xffffffff80b9fe48 at _sx_xlock+0x68 #3 0xffffffff80cbf10b at iflib_if_ioctl+0x2eb #4 0xffffffff80c9fca8 at if_setflag+0xf8 #5 0xffffffff80c9fb4c at ifpromisc+0x2c #6 0xffffffff80c940c5 at bpf_detachd_locked+0x225 #7 0xffffffff80c96eeb at bpf_dtor+0x8b #8 0xffffffff80a4c85e at devfs_destroy_cdevpriv+0x8e #9 0xffffffff80a4ffc5 at devfs_close_f+0x65 #10 0xffffffff80b40a65 at closef+0x1f5 #11 0xffffffff80b3dae0 at closefp+0xa0 #12 0xffffffff8107ea21 at amd64_syscall+0x281 #13 0xffffffff81058d1d at fast_syscall_common+0x101 ix1: promiscuous mode disabled ix1: promiscuous mode enabled vlan3402: promiscuous mode enabled ix1: promiscuous mode disabled vlan3402: promiscuous mode disabled ix1: promiscuous mode enabled vlan3402: promiscuous mode enabled ix1: promiscuous mode disabled vlan3402: promiscuous mode disabled
Can you please check if reducing MTU to 1468 (on the if_vlan(4) child) has any influence? If so, please cross-link https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230510 -harry
vlan3402: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1468 options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:f4:34:a1 inet 10.194.101.101 netmask 0xffffff00 broadcast 10.194.101.255 groups: vlan vlan: 3402 vlanpcp: 0 parent interface: ix1 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Reducting the MTU doesn't seems to have any influence... Note that r338333 have same issue :(
> FreeBSD 12.0-ALPHA2 #0 r338182: Wed Aug 22 09:59:11 CEST 2018 VLAN tx was broken in r337943 and then fixed in r338305, so this particular revision is definitely affected. That bug was tracked via PR 230794. But the submitter mentioned that r338333 doesn't work either so this may not be a dup.
Hello, does this problem persist? If so, have you had a chance to check it on 12.0-ALPHA4? What NIC are you using? Would you please show your configuration of parent interface (ix1)? Is the machine connected to switch or do you use back-to-back configuration? If the former, would you please check if the problem occurs also with a back-to-back connection? Peter
Hi, # uname -a FreeBSD rsv-filer 12.0-ALPHA5 FreeBSD 12.0-ALPHA5 #0 r338557: Mon Sep 10 12:25:04 CEST 2018 root@rsv-filer:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 Problem "seems" to have changed from pass no traffic to do not detect link on parent interface (I have this behavior on lagg on 11.2) : ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:f4:34:a1 media: Ethernet autoselect status: no carrier nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> vlan3402: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:f4:34:a1 inet 10.194.101.101 netmask 0xffffff00 broadcast 10.194.101.255 groups: vlan vlan: 3402 vlanpcp: 0 parent interface: ix1 media: Ethernet autoselect status: no carrier nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Configuration is done using rc.conf : ifconfig_ix1="up" vlans_ix1="vlan3402" create_args_vlan3402="vlan 3402" ifconfig_vlan3402="10.194.101.101/24" This machine is connected to a Cisco 6800 thru a Twinax cable, so the media should be : media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) On the switch the configuration is pretty simple : interface TenGigabitEthernet3/5 description Server: rsv-filer switchport switchport trunk native vlan 3402 switchport mode trunk switchport trunk allowed vlan 3401-3403 switchport nonegotiate mtu 9216 logging event link-status flowcontrol send on spanning-tree portfast edge trunk spanning-tree bpduguard enable ! On the switch side the port is not shut because any security stuff... Regards
Hum... Addition: # kldstat Id Refs Address Size Name 1 23 0xffffffff80200000 2450510 kernel 2 1 0xffffffff82652000 22b0 cc_htcp.ko 3 1 0xffffffff82655000 3a7ca8 zfs.ko 4 2 0xffffffff829fd000 a4f0 opensolaris.ko 5 1 0xffffffff82f12000 7d08 ioat.ko 6 1 0xffffffff82f1a000 1800 uhid.ko 7 1 0xffffffff82f1c000 23a8 ums.ko 8 1 0xffffffff82f1f000 acf mac_ntpd.ko Seems that if_vlan.ko is not loaded. Added it into loader.conf (and rebooted), link is up but issue is *still* there unfortunatly.
Hello, we have trouble reproducing this bug in our environment. Could you please provide more details? 1. What NIC is causing the problem? 2. Have you tried connecting it back-to-back? If so, were there any differences in behaviour you previously described? 3. Is there any additional configuration applied on this interface?
Hello 1. Both nic make me problems. ix0 and ix1 (see the first dmesg). I use usualy only ix1 to avoid time consuming issues to get back access to this servers (it is in a remote location, even with the IPMi, it can be a pain in the ass have access) 2. back-to-back is not possible to test, the servers is in a remote location :( and is "more or less" in production (shared storage dor esxi) 3. Only /etc/rc.conf configuration is done. The motherboard is a X11DPU from supermicro (with latest BIOS used, which wad a pain to update as well). Notice I get 4 servers like this (on 2 differents datacenter), 2 of them are currently working like a charm on ESX 6.7 (with vlan / mtu ....) the other two are used as storage on Freebsd. These 2 freebsd servers was never being stable on FreeBSD 11.2 (nvme issues with zfs cache / log stuff) and network on freebsd 11.2 was a bit tricky (lagg that doesn't up ix1 for example). Either this is an issue on FreeBSD with Supermicro X11DPU (maybe the hardware has a bug?) or something else ? Anyway on ESXi 6.7 (and 6.5) it working without any pain. Rergards, Xavier
Is this a regression from an earlier install, or is this a new install?
Well... Since I didn't try for more than 2 years I close this bug... Seems to be fixed... somewhere.