my setup is a virtualized OPNsense Firewall 20.1.7 with FreeBSD 11.2-RELEASE-p20-HBSD.
The host is Ubuntu 18.04.04 with Xen 4.9 and Intel i350 NIC with several guests (OPNsense and ubuntu).
I created SR-IOV virtual function, VF on physical interface enp5s0 and
tagged each VF with VLAN ID 20 on the host.
The NIC supports transparent vlan which will automatically tag and untag packet at the VF before leave or reach the guest. Currently, FreeBSD igb driver does not correctly handle the vlan tag. Packet inbound is not untagged as supposed to be, it has a wrong vlan tag (1024). The other ubuntu guests do not have this problem - traffic is untagged.
I tried with additional VLAN tagging in the OPNSense VM, but then I get error message on the host, that tagging is not allowed in guest.
Looking at the traffic in case no VLAN tagging in the VMs, which should be the right way using VLANs with SR-IOV. All traffic coming in to OPNSense VM is tagged with 1024 as seen in the tcpdump.
tcpdump ubuntu VM
PING 192.168.20.1 (192.168.20.1) 56(84) bytes of data.
18:06:03.256121 00:16:3e:a0:18:f1 > 00:16:3e:a0:16:f1, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 18868, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.20.250 > 192.168.20.1: ICMP echo request, id 767, seq 1, length 64
tcpdump opnsense VM
18:10:09.974113 00:16:3e:a0:18:f1 > 00:16:3e:a0:16:f1, ethertype 802.1Q (0x8100), length 102: vlan 1024, p 0, DEI, ethertype IPv4, (tos 0x0, ttl 64, id 39437, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.20.250 > 192.168.20.1: ICMP echo request, id 778, seq 1, length 64
There is an old bug report with a similar behavior https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209581
Any chance this bug could be fixed?