With IPv6 traffic, if it matters. root@freebsd12-test:~ # tcpdump -ni vnic0 ip6 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vnic0, link-type EN10MB (Ethernet), capture size 262144 bytes Kernel page fault with the following non-sleepable locks held: shared rw bpf interface lock (bpf interface lock) r = 0 (0xfffffd002e32d4a8) locked @ /root/freebsd/sys/net/bpf.c:2207 exclusive sleep mutex vnic0: SQ(6) lock (vnic0: SQ(6) lock) r = 0 (0xffff00004b9a15d0) locked @ /root/freebsd/sys/dev/vnic/nicvf_main.c:696 exclusive rw tcpinp (tcpinp) r = 0 (0xfffffd009212bd60) locked @ /root/freebsd/sys/netinet/tcp_usrreq.c:902 stack backtrace: #0 0xffff0000003741a0 at witness_debugger+0x64 #1 0xffff0000003754b0 at witness_warn+0x3fc #2 0xffff000000626488 at data_abort+0xe0 #3 0xffff0000006262a4 at do_el1h_sync+0xf8 #4 0xffff00000060e874 at handle_el1h_sync+0x74 #5 0xffff0000003f5158 at bpf_mtap+0xc4 #6 0xffff0000003f5158 at bpf_mtap+0xc4 #7 0xffff000000646a84 at nicvf_xmit_locked+0x4a0 #8 0xffff000000645720 at nicvf_if_transmit+0x214 #9 0xffff000000405498 at ether_output+0x6b4 #10 0xffff000000445840 at ip_output+0x1120 #11 0xffff0000004bb1e8 at tcp_output+0x19ec #12 0xffff0000004cb2fc at tcp_usr_send+0x2dc #13 0xffff0000003a2d6c at sosend_generic+0x3d0 #14 0xffff0000003a2fb4 at sosend+0x5c #15 0xffff000000380b5c at soo_write+0x40 #16 0xffff000000379448 at dofilewrite+0xb4 #17 0xffff00000037908c at kern_writev+0x6c x0: 66 x1: fffffd009244c600 x2: 66 x3: 0 x4: 68 x5: 0 x6: ffff000ba48da310 x7: 80 x8: deadc0dedeadc0ea x9: fffffd009244c600 x10: deadc0dedeadc0de x11: ffffffffdeadc0d2 x12: ffff000000a75678 x13: fffffd00922445b0 x14: 1 x15: 1 x16: 0 x17: 4071e768 x18: ffff000ba48da1c0 x19: 0 x20: 66 x21: fffffd009244c600 x22: fffffd00bdff7940 x23: ffff00000072ecc0 x24: 0 x25: 0 x26: 2 x27: 7b x28: 66 x29: ffff000ba48da240 sp: ffff000ba48da1c0 lr: ffff0000003f515c elr: ffff0000003f9270 spsr: a0000345 far: deadc0dedeadc0ea esr: 96000004 panic: data abort in critical section or under mutex cpuid = 88 time = 1508412796 KDB: stack backtrace: db_trace_self() at db_trace_self_wrapper+0x28 pc = 0xffff00000060c848 lr = 0xffff000000086b8c sp = 0xffff000ba48d9be0 fp = 0xffff000ba48d9df0 db_trace_self_wrapper() at vpanic+0x184 pc = 0xffff000000086b8c lr = 0xffff000000315818 sp = 0xffff000ba48d9e00 fp = 0xffff000ba48d9e80 vpanic() at panic+0x44 pc = 0xffff000000315818 lr = 0xffff0000003158a0 sp = 0xffff000ba48d9e90 fp = 0xffff000ba48d9f10 panic() at data_abort+0x250 pc = 0xffff0000003158a0 lr = 0xffff0000006265f8 sp = 0xffff000ba48d9f20 fp = 0xffff000ba48d9fd0 data_abort() at do_el1h_sync+0xf8 pc = 0xffff0000006265f8 lr = 0xffff0000006262a4 sp = 0xffff000ba48d9fe0 fp = 0xffff000ba48da010 do_el1h_sync() at handle_el1h_sync+0x74 pc = 0xffff0000006262a4 lr = 0xffff00000060e874 sp = 0xffff000ba48da020 fp = 0xffff000ba48da130 handle_el1h_sync() at bpf_mtap+0xc4 pc = 0xffff00000060e874 lr = 0xffff0000003f5158 sp = 0xffff000ba48da140 fp = 0xffff000ba48da240 bpf_mtap() at bpf_mtap+0xc4 pc = 0xffff0000003f5158 lr = 0xffff0000003f5158 sp = 0xffff000ba48da250 fp = 0xffff000ba48da2c0 bpf_mtap() at nicvf_xmit_locked+0x4a0 pc = 0xffff0000003f5158 lr = 0xffff000000646a84 sp = 0xffff000ba48da2d0 fp = 0xffff000ba48db360 nicvf_xmit_locked() at nicvf_if_transmit+0x214 pc = 0xffff000000646a84 lr = 0xffff000000645720 sp = 0xffff000ba48db370 fp = 0xffff000ba48db3b0 nicvf_if_transmit() at ether_output+0x6b4 pc = 0xffff000000645720 lr = 0xffff000000405498 sp = 0xffff000ba48db3c0 fp = 0xffff000ba48db450 ether_output() at ip_output+0x1120 pc = 0xffff000000405498 lr = 0xffff000000445840 sp = 0xffff000ba48db460 fp = 0xffff000ba48db580 ip_output() at tcp_output+0x19ec pc = 0xffff000000445840 lr = 0xffff0000004bb1e8 sp = 0xffff000ba48db590 fp = 0xffff000ba48db740 tcp_output() at tcp_usr_send+0x2dc pc = 0xffff0000004bb1e8 lr = 0xffff0000004cb2fc sp = 0xffff000ba48db750 fp = 0xffff000ba48db7b0 tcp_usr_send() at sosend_generic+0x3d0 pc = 0xffff0000004cb2fc lr = 0xffff0000003a2d6c sp = 0xffff000ba48db7c0 fp = 0xffff000ba48db860 sosend_generic() at sosend+0x5c pc = 0xffff0000003a2d6c lr = 0xffff0000003a2fb4 sp = 0xffff000ba48db870 fp = 0xffff000ba48db890 sosend() at soo_write+0x40 pc = 0xffff0000003a2fb4 lr = 0xffff000000380b5c sp = 0xffff000ba48db8a0 fp = 0xffff000ba48db8b0 soo_write() at dofilewrite+0xb4 pc = 0xffff000000380b5c lr = 0xffff000000379448 sp = 0xffff000ba48db8c0 fp = 0xffff000ba48db900 dofilewrite() at kern_writev+0x6c pc = 0xffff000000379448 lr = 0xffff00000037908c sp = 0xffff000ba48db910 fp = 0xffff000ba48db950 kern_writev() at sys_write+0x84 pc = 0xffff00000037908c lr = 0xffff000000379010 sp = 0xffff000ba48db960 fp = 0xffff000ba48db9a0 sys_write() at do_el0_sync+0x890 pc = 0xffff000000379010 lr = 0xffff000000626e8c sp = 0xffff000ba48db9b0 fp = 0xffff000ba48dba70 do_el0_sync() at handle_el0_sync+0x74 pc = 0xffff000000626e8c lr = 0xffff00000060e9f4 sp = 0xffff000ba48dba80 fp = 0xffff000ba48dbb90 handle_el0_sync() at 0x40a91fd0 pc = 0xffff00000060e9f4 lr = 0x0000000040a91fd0 sp = 0xffff000ba48dbba0 fp = 0x0000ffffffffa5a0 KDB: enter: panic [ thread pid 911 tid 100655 ] Stopped at bpf_filter+0x98: ldrb w9, [x8] db> ~
Looking at nicvf_xmit_locked() it seems the mbuf is passed to bufdma and the ring is advanced and only then a copy is sent to BPF. By that time DMA might have happened and the reference to the mbuf is no longer valid. The BPF_MTAP() call needs to happen before dma is initiated so probably before nicvf_tx_mbuf_locked() and not after.
A commit references this bug: Author: emaste Date: Fri Nov 10 22:17:30 UTC 2017 New revision: 325683 URL: https://svnweb.freebsd.org/changeset/base/325683 Log: vnic: apply BPF tap before passing packet to hardware Previously we passed tx packets to hardware via nicvf_tx_mbuf_locked and then to the BPF tap, with a possibly invalid mbuf which would result in a panic. PR: 223600 Discussed with: bz MFC after: 1 week Sponsored by: The FreeBSD Foundation, Packet.net (hardware) Changes: head/sys/dev/vnic/nicvf_queues.c
A commit references this bug: Author: emaste Date: Fri Nov 17 00:38:01 UTC 2017 New revision: 325916 URL: https://svnweb.freebsd.org/changeset/base/325916 Log: MFC r325683: vnic: apply BPF tap before passing packet to hardware Previously we passed tx packets to hardware via nicvf_tx_mbuf_locked and then to the BPF tap, with a possibly invalid mbuf which would result in a panic. PR: 223600 Discussed with: bz Sponsored by: The FreeBSD Foundation, Packet.net (hardware) Changes: _U stable/11/ stable/11/sys/dev/vnic/nicvf_queues.c