Created attachment 250217 [details]
pciconf -lv outout

The virtio-net-pci network interface (eg. vtnet1) placed on not-root pcie bus is unusable on arm64 FreeBSD as a qemu guest with virt platform, gic-version=3 and its=on.

There are 3 requirements to produce this bug.
1. The GIC version must be 3, "gic-version=3"; (vtnet works fine with gic-version=2)
2. The ITS must be on, "its=on"; (vtnet works fine with its=off).
3. The virtio-net-pci must be placed on a QEMU PCI-PCI bridge instead of the pcie.0 HOST-PCI root bus. virtio-net-pci works fine when it is placed at virtio_pci1@pci0:0:4:0 but it becomes unusable on virtio_pci2@pci0:1:5:0.

This is an example command to produce this bug.

sudo qemu-system-aarch64 \
 -m 1024 \
 -nographic \
 -drive if=pflash,format=raw,file=efi.img,readonly=on \
 -drive if=pflash,format=raw,file=varstore.img \
 -device pci-bridge,id=pcie.1,chassis_nr=1,bus=pcie.0,addr=0xa \
 -device qemu-xhci,id=qemu-xhci,bus=pcie.0,addr=3 \
 -drive if=none,format=qcow2,file=FreeBSD-14.0-RELEASE-arm64-aarch64.qcow2,id=hd0 \
 -device virtio-blk-pci,bus=pcie.0,addr=2,drive=hd0 \
 -netdev type=tap,id=net0,ifname=tap0,script=no,downscript=no \
 -device virtio-net-pci,mac=02:01:07:00:00:10,netdev=net0,bus=pcie.0,addr=4,id=net0 \
 -netdev type=tap,id=net1,ifname=tap1,script=no,downscript=no \
 -device virtio-net-pci,mac=02:01:07:00:00:11,netdev=net1,bus=pcie.1,addr=5,id=net1 \
 -cpu max -M virt,gic-version=3,its=on

To use KVM, replace `-cpu max -M virt,gic-version=3,its=on` with `-cpu host -M virt,gic-version=3,accel=kvm,its=on`. KVM does not affect the result (make sure your host is GICv3).

Manually set up tap0 and tap1 and add them to a host bridge br0.
(host) ifconfig br0 192.168.3.1/24
(vm) ifconfig vtnet0 up
(vm) ifconfig vtnet0 192.168.3.10/24
(vm) ifconfig vtnet1 up
(vm) ifconfig vtnet1 192.168.3.11/24

Actual result:
There will be two network interfaces vtnet0 and vtnet1. vtnet0 on virtio_pci1@pci0:0:4:0 works fine, but vtnet1 on virtio_pci2@pci0:1:5:0 is usable. You can ping 192.168.3.10 and get replies, but ping 192.168.3.11 gets no reply.

This bug is reproducible on the following FreeBSD images:
- FreeBSD-13.2-RELEASE-arm64-aarch64.qcow2
- FreeBSD-14.0-RELEASE-arm64-aarch64.qcow2
- FreeBSD-15.0-CURRENT-arm64-aarch64-20240404-112783ebbc31-269103-disc1.iso

With the exactly same qemu configuration (except system image, of course), all two network adapters works fine in Linux guest: Linux debian 6.1.0-20-arm64 #1 SMP Debian 6.1.85-1 (2024-04-11) aarch64 GNU/Linux.

I have tested on the two host platforms:
- qemu-system-aarch64 QEMU emulator version 8.2.2 on Linux ArchLinux 6.8.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024 15:20:28 +0000 x86_64 GNU/Linux
- qemu-system-aarch64 QEMU emulator version 8.1.5 (pve-qemu-kvm_8.1.5-4) on Linux armbian 6.6.16-current-rockchip64 #1 SMP PREEMPT Fri Feb 23 08:25:28 UTC 2024 aarch64 GNU/Linux

About efi.img and varstore.img:
efi image is from qemu-efi-aarch64_2024.02-2_all.deb (you can get it from https://packages.debian.org/sid/all/qemu-efi-aarch64/download), and expand it to 64M, according to https://ubuntu.com/server/docs/boot-arm64-virtual-machines-on-qemu.
varstore.img is 64M of zeros.

The descriptions of "gic-version=3" and "its=on" are available at https://qemu-project.gitlab.io/qemu/system/arm/virt.html.