This is a crafted bug from WIP fix https://reviews.freebsd.org/D39536. A setup: em0 <---> igc0 Steps to repeat: Capture on igc0 ``` # ifconfig igc0 up # tcpdump -nevi igc0 ``` Disable PCP and hardware VLAN tag insertion, then send packets with priority ``` # ifconfig em0 -vlanhwtag -pcp # ifconfig em0 inet 192.0.2.1/24 # arp -s 192.0.2.2 0:1:2:3:4:5 # ping -c1 -C3 192.0.2.2 ``` Packets captured on igc0: ``` 09:53:57.397415 00:0c:29:73:4f:98 > 00:01:02:03:04:05, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 19249, offset 0, flags [none], proto ICMP (1), length 84) 192.0.2.1 > 192.0.2.2: ICMP echo request, id 5382, seq 0, length 64 ``` When either hardware VLAN tag insertion is enabled, or enable default PCP on interface ``` # ifconfig em0 vlanhwtag ### or # ifconfig em0 pcp 0 ``` then the outbound packets looks good: ``` 10:00:03.189436 00:0c:29:73:4f:98 > 00:01:02:03:04:05, ethertype 802.1Q (0x8100), length 102: vlan 0, p 3, ethertype IPv4, (tos 0x0, ttl 64, id 33532, offset 0, flags [none], proto ICMP (1), length 84) 192.0.2.1 > 192.0.2.2: ICMP echo request, id 6662, seq 0, length 64 10:02:17.070757 00:0c:29:73:4f:98 > 00:01:02:03:04:05, ethertype 802.1Q (0x8100), length 102: vlan 0, p 3, ethertype IPv4, (tos 0x0, ttl 64, id 19250, offset 0, flags [none], proto ICMP (1), length 84) 192.0.2.1 > 192.0.2.2: ICMP echo request, id 8198, seq 0, length 64 ``` Expected behavior: outbound traffic should always have priority when application requires.
Can repeat this on Realtek USB 2.5G network dongle with cdce(4) driver, which does not support hardware VLAN tag insertion.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=49d6743da15fe378782e43776df8b4fd4f84c8d0 commit 49d6743da15fe378782e43776df8b4fd4f84c8d0 Author: Zhenlei Huang <zlei@FreeBSD.org> AuthorDate: 2023-09-06 10:15:14 +0000 Commit: Zhenlei Huang <zlei@FreeBSD.org> CommitDate: 2023-09-06 10:15:14 +0000 net: Check per-flow priority code point for untagged traffic Commit 868aabb4708d introduced per-flow priority. There's a defect in the logic for untagged traffic, it does not check M_VLANTAG set in the mbuf packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then can result missing desired priority in the outbound packets. For mbuf packet with M_VLANTAG in header, some interfaces happen to work due to bug in the drivers mentioned in D39499. As modern interfaces have VLAN hardware offloading, the defect is barely noticeable unless the feature per-flow priority is widely tested. As a side effect of this defect, the soft padding to work around buggy bridges is bypassed. That may result in regression if soft padding is requested. PR: 273431 Discussed with: kib Fixes: 868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39536 sys/net/if_ethersubr.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c750055382f73db964c20f8eba855a9ac9e19591 commit c750055382f73db964c20f8eba855a9ac9e19591 Author: Zhenlei Huang <zlei@FreeBSD.org> AuthorDate: 2023-09-06 10:15:14 +0000 Commit: Zhenlei Huang <zlei@FreeBSD.org> CommitDate: 2023-09-20 04:04:57 +0000 net: Check per-flow priority code point for untagged traffic Commit 868aabb4708d introduced per-flow priority. There's a defect in the logic for untagged traffic, it does not check M_VLANTAG set in the mbuf packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then can result missing desired priority in the outbound packets. For mbuf packet with M_VLANTAG in header, some interfaces happen to work due to bug in the drivers mentioned in D39499. As modern interfaces have VLAN hardware offloading, the defect is barely noticeable unless the feature per-flow priority is widely tested. As a side effect of this defect, the soft padding to work around buggy bridges is bypassed. That may result in regression if soft padding is requested. PR: 273431 Discussed with: kib Fixes: 868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39536 (cherry picked from commit 49d6743da15fe378782e43776df8b4fd4f84c8d0) sys/net/if_ethersubr.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8303afca1765148d0069ce5144072b3ae9cab61e commit 8303afca1765148d0069ce5144072b3ae9cab61e Author: Zhenlei Huang <zlei@FreeBSD.org> AuthorDate: 2023-09-06 10:15:14 +0000 Commit: Zhenlei Huang <zlei@FreeBSD.org> CommitDate: 2023-09-20 10:00:37 +0000 net: Check per-flow priority code point for untagged traffic Commit 868aabb4708d introduced per-flow priority. There's a defect in the logic for untagged traffic, it does not check M_VLANTAG set in the mbuf packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then can result missing desired priority in the outbound packets. For mbuf packet with M_VLANTAG in header, some interfaces happen to work due to bug in the drivers mentioned in D39499. As modern interfaces have VLAN hardware offloading, the defect is barely noticeable unless the feature per-flow priority is widely tested. As a side effect of this defect, the soft padding to work around buggy bridges is bypassed. That may result in regression if soft padding is requested. PR: 273431 Discussed with: kib Fixes: 868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39536 (cherry picked from commit 49d6743da15fe378782e43776df8b4fd4f84c8d0) (cherry picked from commit c750055382f73db964c20f8eba855a9ac9e19591) sys/net/if_ethersubr.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d5dbed2f1e2ab14f54acbe14c30ccac9f9fee29f commit d5dbed2f1e2ab14f54acbe14c30ccac9f9fee29f Author: Zhenlei Huang <zlei@FreeBSD.org> AuthorDate: 2023-09-06 10:15:14 +0000 Commit: Zhenlei Huang <zlei@FreeBSD.org> CommitDate: 2023-09-20 10:05:14 +0000 net: Check per-flow priority code point for untagged traffic Commit 868aabb4708d introduced per-flow priority. There's a defect in the logic for untagged traffic, it does not check M_VLANTAG set in the mbuf packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then can result missing desired priority in the outbound packets. For mbuf packet with M_VLANTAG in header, some interfaces happen to work due to bug in the drivers mentioned in D39499. As modern interfaces have VLAN hardware offloading, the defect is barely noticeable unless the feature per-flow priority is widely tested. As a side effect of this defect, the soft padding to work around buggy bridges is bypassed. That may result in regression if soft padding is requested. PR: 273431 Discussed with: kib Fixes: 868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39536 (cherry picked from commit 49d6743da15fe378782e43776df8b4fd4f84c8d0) (cherry picked from commit c750055382f73db964c20f8eba855a9ac9e19591) (cherry picked from commit 8303afca1765148d0069ce5144072b3ae9cab61e) sys/net/if_ethersubr.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=0d648a59fceda79106fe66347b1df5cc11a7fa00 commit 0d648a59fceda79106fe66347b1df5cc11a7fa00 Author: Zhenlei Huang <zlei@FreeBSD.org> AuthorDate: 2023-09-06 10:15:14 +0000 Commit: Zhenlei Huang <zlei@FreeBSD.org> CommitDate: 2023-09-21 04:49:53 +0000 net: Check per-flow priority code point for untagged traffic Commit 868aabb4708d introduced per-flow priority. There's a defect in the logic for untagged traffic, it does not check M_VLANTAG set in the mbuf packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then can result missing desired priority in the outbound packets. For mbuf packet with M_VLANTAG in header, some interfaces happen to work due to bug in the drivers mentioned in D39499. As modern interfaces have VLAN hardware offloading, the defect is barely noticeable unless the feature per-flow priority is widely tested. As a side effect of this defect, the soft padding to work around buggy bridges is bypassed. That may result in regression if soft padding is requested. PR: 273431 Approved by: re (cperciva) Discussed with: kib Fixes: 868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39536 (cherry picked from commit 49d6743da15fe378782e43776df8b4fd4f84c8d0) (cherry picked from commit c750055382f73db964c20f8eba855a9ac9e19591) sys/net/if_ethersubr.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)