Hello, I tried to setup VF interfaces using SR-IOV in intel-ix-kmod, it kind of works but as soon as I create the SF interfaces, the connection on the primary interfaces hangs. Hardware : X520-DA1 OS : FreeBSD-13.1 I have in my iov config file (/etc/iovctl-ix0.conf) : <pre> PF { device: ix0; num_vfs: 3; } DEFAULT { passthrough: false; allow-set-mac: true; allow-promisc: true; } VF-1 { passthrough: true; } VF-2 { passthrough: true; } </pre> I start creating VFs with : <pre> iovctl -C -f /etc/iovctl-ix0.conf </pre> and get one ixv0 plus two ppt (which is expected), but the connection i lost on ix0. No messages in log file except initialization messages : <pre> Dec 5 16:02:00 amber kernel: ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.32> port 0x5000-0x501f mem 0x6000000000-0x600007ffff,0x6000080000-0x6000083fff at device 0.0 on pci2 Dec 5 16:02:00 amber kernel: ix0: fw version unknown, nvm version unknown, etid 80000707 Dec 5 16:02:00 amber kernel: ix0: Using MSI-X interrupts with 9 vectors Dec 5 16:02:00 amber kernel: ix0: Ethernet address: 80:61:5f:0a:47:2e Dec 5 16:02:00 amber kernel: ix0: PCI Express Bus: Speed 5.0GT/s Width x8 Dec 5 16:02:00 amber kernel: ppt0 at device 0.130 on pci2 Dec 5 16:02:00 amber kernel: ppt1 at device 0.132 on pci2 </pre> Any ideas (bug in driver, too cheap hardware ?). NB : I tried with different versions of the intel driver (compiled manually), the last one is 3.3.32, but the symptoms remain the same. Thanks Bruno
Hello, I tried with the updated port of intel-ix-kmod and there is a progress but still failing for me, maybe misonfiguration on my side ? If I set SR-iov in rc.conf it starts and I see in the logs of my dhcp server that it requests an ip, and the dhcp sends an offer, but it doesn’t seem to be received (dhclient reports no offers received) so connectivity still seems partial (one way ?), so that is still failing to work properly. Bruno
I forgot to say I have upgraded to 13.2 since my first report. Otherwise is the configuration the same.
Some logs to document : /etc/rc.conf #ifconfig_ix0="dhcp up" #ifconfig_ix0_ipv6="inet6 accept_rtadv" ifconfig_ixv0="dhcp up" ifconfig_ixv0_ipv6="inet6 accept_rtadv" iovctl_files="/etc/iovctl-ix0.conf" /var/log/messages Jun 21 14:16:12 amber kernel: ---<<BOOT>>--- Jun 21 14:16:12 amber kernel: Copyright (c) 1992-2021 The FreeBSD Project. Jun 21 14:16:12 amber kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Jun 21 14:16:12 amber kernel: The Regents of the University of California. All rights reserved. Jun 21 14:16:12 amber kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Jun 21 14:16:12 amber kernel: FreeBSD 13.2-RELEASE-p1 releng/13.2-n254621-08b87f63a046 AMBER_KERNEL amd64 Jun 21 14:16:12 amber kernel: FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c) [...] Jun 21 14:16:12 amber kernel: module_register: cannot register pci/ix from kernel; already loaded from if_ix_updated.ko Jun 21 14:16:12 amber kernel: Module pci/ix failed to register: 17 [...] Jun 21 14:16:12 amber kernel: ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.33> port 0x6000-0x601f mem 0x4000000000-0x400007ffff,0x4000280000-0x4000283fff at device 0.0 on pci2 Jun 21 14:16:12 amber kernel: ix0: fw version unknown, nvm version unknown, etid 80000707 Jun 21 14:16:12 amber kernel: ix0: Using MSI-X interrupts with 9 vectors Jun 21 14:16:12 amber kernel: ix0: Ethernet address: 80:61:5f:0a:47:2e Jun 21 14:16:12 amber kernel: ix0: PCI Express Bus: Speed 5.0GT/s Width x8 [...] Jun 21 14:16:12 amber kernel: ixv0: <Intel(R) X520 82599 Virtual Function> at device 0.128 on pci2 Jun 21 14:16:12 amber kernel: ixv0: Using 2048 TX descriptors and 2048 RX descriptors Jun 21 14:16:12 amber kernel: ixv0: Using 1 RX queues 1 TX queues Jun 21 14:16:12 amber kernel: ixv0: Using MSI-X interrupts with 2 vectors Jun 21 14:16:12 amber kernel: ixv0: allocated for 1 queues Jun 21 14:16:12 amber kernel: ixv0: allocated for 1 rx queues Jun 21 14:16:12 amber kernel: ixv0: Ethernet address: 58:9c:fc:10:ff:cd Jun 21 14:16:12 amber kernel: ixv0: netmap queues/slots: TX 1/2048, RX 1/2048 Jun 21 14:16:12 amber kernel: ppt0 at device 0.130 on pci2 Jun 21 14:16:12 amber kernel: ppt1 at device 0.132 on pci2 [...] Jun 21 14:16:14 amber kernel: ix0: link state changed to UP Jun 21 14:16:15 amber kernel: ixv0: link state changed to UP </pre> /var/log/daemon.log <pre> Jun 21 14:16:15 amber dhclient[1672]: DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 8 Jun 21 14:16:23 amber dhclient[1672]: DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 13 Jun 21 14:16:36 amber dhclient[1672]: DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 7 Jun 21 14:16:43 amber dhclient[1672]: DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 8 Jun 21 14:16:51 amber dhclient[1672]: DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 12 Jun 21 14:17:03 amber dhclient[1672]: DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 13 Jun 21 14:17:16 amber dhclient[1672]: No DHCPOFFERS received. Jun 21 14:17:16 amber dhclient[1672]: No working leases in persistent database - sleeping. /var/dnsmasq.log on dhcp server side Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.10 -- 192.168.32.49 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.200 -- 192.168.32.219 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 client provides name: amber Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 DHCPDISCOVER(eth0) 58:9c:fc:10:ff:cd Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 tags: known, eth0 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 DHCPOFFER(eth0) 192.168.32.91 58:9c:fc:10:ff:cd Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route, Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 requested options: 3:router, 15:domain-name, 6:dns-server, 12:hostname, Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 requested options: 119:domain-search, 26:mtu Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 next server: 192.168.32.10 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 1 option: 53 message-type 2 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 54 server-identifier 192.168.32.10 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 51 lease-time 1d Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 58 T1 12h Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 59 T2 21h Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 1 netmask 255.255.255.0 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 28 broadcast 192.168.32.255 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 29 option:121 classless-static-route 00:c0:a8:20:01:18:c0:a8:21:c0:a8:20:0a:18... Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 8 option: 6 dns-server 192.168.32.10, 192.168.33.10 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 10 option: 15 domain-name ruomad.lan Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 12 option:119 domain-search 06:72:75:6f:6d:61:64:03:6c:61:6e:00 Jun 21 14:16:23 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 3 router 192.168.32.1 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.10 -- 192.168.32.49 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.200 -- 192.168.32.219 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 client provides name: amber Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 DHCPDISCOVER(eth0) 58:9c:fc:10:ff:cd Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 tags: known, eth0 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 DHCPOFFER(eth0) 192.168.32.91 58:9c:fc:10:ff:cd Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route, Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 requested options: 3:router, 15:domain-name, 6:dns-server, 12:hostname, Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 requested options: 119:domain-search, 26:mtu Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 next server: 192.168.32.10 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 1 option: 53 message-type 2 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 54 server-identifier 192.168.32.10 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 51 lease-time 1d Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 58 T1 12h Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 59 T2 21h Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 1 netmask 255.255.255.0 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 28 broadcast 192.168.32.255 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 29 option:121 classless-static-route 00:c0:a8:20:01:18:c0:a8:21:c0:a8:20:0a:18... Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 8 option: 6 dns-server 192.168.32.10, 192.168.33.10 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 10 option: 15 domain-name ruomad.lan Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 12 option:119 domain-search 06:72:75:6f:6d:61:64:03:6c:61:6e:00 Jun 21 14:16:36 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 3 router 192.168.32.1 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.10 -- 192.168.32.49 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.200 -- 192.168.32.219 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 client provides name: amber Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 DHCPDISCOVER(eth0) 58:9c:fc:10:ff:cd Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 tags: known, eth0 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 DHCPOFFER(eth0) 192.168.32.91 58:9c:fc:10:ff:cd Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route, Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 requested options: 3:router, 15:domain-name, 6:dns-server, 12:hostname, Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 requested options: 119:domain-search, 26:mtu Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 next server: 192.168.32.10 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 1 option: 53 message-type 2 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 54 server-identifier 192.168.32.10 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 51 lease-time 1d Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 58 T1 12h Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 59 T2 21h Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 1 netmask 255.255.255.0 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 28 broadcast 192.168.32.255 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 29 option:121 classless-static-route 00:c0:a8:20:01:18:c0:a8:21:c0:a8:20:0a:18... Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 8 option: 6 dns-server 192.168.32.10, 192.168.33.10 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 10 option: 15 domain-name ruomad.lan Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 12 option:119 domain-search 06:72:75:6f:6d:61:64:03:6c:61:6e:00 Jun 21 14:16:43 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 3 router 192.168.32.1 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.10 -- 192.168.32.49 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.200 -- 192.168.32.219 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 client provides name: amber Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 DHCPDISCOVER(eth0) 58:9c:fc:10:ff:cd Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 tags: known, eth0 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 DHCPOFFER(eth0) 192.168.32.91 58:9c:fc:10:ff:cd Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route, Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 requested options: 3:router, 15:domain-name, 6:dns-server, 12:hostname, Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 requested options: 119:domain-search, 26:mtu Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 next server: 192.168.32.10 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 1 option: 53 message-type 2 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 54 server-identifier 192.168.32.10 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 51 lease-time 1d Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 58 T1 12h Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 59 T2 21h Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 1 netmask 255.255.255.0 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 28 broadcast 192.168.32.255 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 29 option:121 classless-static-route 00:c0:a8:20:01:18:c0:a8:21:c0:a8:20:0a:18... Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 8 option: 6 dns-server 192.168.32.10, 192.168.33.10 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 10 option: 15 domain-name ruomad.lan Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 12 option:119 domain-search 06:72:75:6f:6d:61:64:03:6c:61:6e:00 Jun 21 14:16:51 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 3 router 192.168.32.1 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.10 -- 192.168.32.49 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 available DHCP range: 192.168.32.200 -- 192.168.32.219 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 client provides name: amber Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 DHCPDISCOVER(eth0) 58:9c:fc:10:ff:cd Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 tags: known, eth0 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 DHCPOFFER(eth0) 192.168.32.91 58:9c:fc:10:ff:cd Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route, Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 requested options: 3:router, 15:domain-name, 6:dns-server, 12:hostname, Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 requested options: 119:domain-search, 26:mtu Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 next server: 192.168.32.10 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 1 option: 53 message-type 2 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 54 server-identifier 192.168.32.10 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 51 lease-time 1d Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 58 T1 12h Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 59 T2 21h Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 1 netmask 255.255.255.0 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 28 broadcast 192.168.32.255 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 29 option:121 classless-static-route 00:c0:a8:20:01:18:c0:a8:21:c0:a8:20:0a:18... Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 8 option: 6 dns-server 192.168.32.10, 192.168.33.10 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 10 option: 15 domain-name ruomad.lan Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 12 option:119 domain-search 06:72:75:6f:6d:61:64:03:6c:61:6e:00 Jun 21 14:17:03 dnsmasq-dhcp[2344]: 3649208241 sent size: 4 option: 3 router 192.168.32.1
Can you try with https://reviews.freebsd.org/D19279? You will need to adjust paths.
Hello, I looked at this patch but it seems quite old. It seems to already have been applied in the current version (ix-3.3.33), and there has been some refactoring since. In the current source I see : if_sr_iov.c line 384: vf->flags |= IXGBE_VF_CTS; resp[0] = IXGBE_VF_RESET | ack; bcopy(vf->ether_addr, &resp[1], ETHER_ADDR_LEN); resp[3] = hw->mac.mc_filter_type; ixgbe_write_mbx(hw, resp, IXGBE_VF_PERMADDR_MSG_LEN, vf->pool); if_sr_iov.c line 716: ixgbe_init_mbx_params_pf(&sc->hw); /* set the SRIOV flag now as it's needed * by ixgbe_init_locked() */ sc->feat_en |= IXGBE_FEATURE_SRIOV; ixgbe_init_locked(sc);
Hello, Tried with the newer updated driver ix-3.3.35, no luck, the symptoms remain the same... Bruno