Bug 273431

Summary: No PCP (priority code point) on outbound traffic via em(4) when PCP and hardware VLAN tag insertion is disabled
Product: Base System Reporter: Zhenlei Huang <zlei>
Component: kernAssignee: Zhenlei Huang <zlei>
Status: Closed FIXED    
Severity: Affects Some People Flags: zlei: mfc-stable14+
zlei: mfc-stable13+
zlei: mfc-stable12+
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   
URL: https://reviews.freebsd.org/D39536

Description Zhenlei Huang freebsd_committer freebsd_triage 2023-08-30 02:07:16 UTC
This is a crafted bug from WIP fix https://reviews.freebsd.org/D39536.

A setup:

em0 <---> igc0

Steps to repeat:

Capture on igc0
```
# ifconfig igc0 up
# tcpdump -nevi igc0
```

Disable PCP and hardware VLAN tag insertion, then send packets with priority
```
# ifconfig em0 -vlanhwtag -pcp
# ifconfig em0 inet 192.0.2.1/24
# arp -s 192.0.2.2 0:1:2:3:4:5
# ping -c1 -C3 192.0.2.2
```

Packets captured on igc0:
```
09:53:57.397415 00:0c:29:73:4f:98 > 00:01:02:03:04:05, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 19249, offset 0, flags [none], proto ICMP (1), length 84)
    192.0.2.1 > 192.0.2.2: ICMP echo request, id 5382, seq 0, length 64
```

When either hardware VLAN tag insertion is enabled, or enable default PCP on interface
```
# ifconfig em0 vlanhwtag
### or
# ifconfig em0 pcp 0
```

then the outbound packets looks good:
```
10:00:03.189436 00:0c:29:73:4f:98 > 00:01:02:03:04:05, ethertype 802.1Q (0x8100), length 102: vlan 0, p 3, ethertype IPv4, (tos 0x0, ttl 64, id 33532, offset 0, flags [none], proto ICMP (1), length 84)
    192.0.2.1 > 192.0.2.2: ICMP echo request, id 6662, seq 0, length 64

10:02:17.070757 00:0c:29:73:4f:98 > 00:01:02:03:04:05, ethertype 802.1Q (0x8100), length 102: vlan 0, p 3, ethertype IPv4, (tos 0x0, ttl 64, id 19250, offset 0, flags [none], proto ICMP (1), length 84)
    192.0.2.1 > 192.0.2.2: ICMP echo request, id 8198, seq 0, length 64
```


Expected behavior: outbound traffic should always have priority when application requires.
Comment 1 Zhenlei Huang freebsd_committer freebsd_triage 2023-08-30 07:16:26 UTC
Can repeat this on Realtek USB 2.5G network dongle with cdce(4) driver, which does not support hardware VLAN tag insertion.
Comment 2 commit-hook freebsd_committer freebsd_triage 2023-09-06 10:17:28 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=49d6743da15fe378782e43776df8b4fd4f84c8d0

commit 49d6743da15fe378782e43776df8b4fd4f84c8d0
Author:     Zhenlei Huang <zlei@FreeBSD.org>
AuthorDate: 2023-09-06 10:15:14 +0000
Commit:     Zhenlei Huang <zlei@FreeBSD.org>
CommitDate: 2023-09-06 10:15:14 +0000

    net: Check per-flow priority code point for untagged traffic

    Commit 868aabb4708d introduced per-flow priority. There's a defect in the
    logic for untagged traffic, it does not check M_VLANTAG set in the mbuf
    packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then
    can result missing desired priority in the outbound packets.

    For mbuf packet with M_VLANTAG in header, some interfaces happen to work
    due to bug in the drivers mentioned in D39499. As modern interfaces have
    VLAN hardware offloading, the defect is barely noticeable unless the
    feature per-flow priority is widely tested.

    As a side effect of this defect, the soft padding to work around buggy
    bridges is bypassed. That may result in regression if soft padding is
    requested.

    PR:             273431
    Discussed with: kib
    Fixes:  868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D39536

 sys/net/if_ethersubr.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
Comment 3 commit-hook freebsd_committer freebsd_triage 2023-09-20 04:06:18 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=c750055382f73db964c20f8eba855a9ac9e19591

commit c750055382f73db964c20f8eba855a9ac9e19591
Author:     Zhenlei Huang <zlei@FreeBSD.org>
AuthorDate: 2023-09-06 10:15:14 +0000
Commit:     Zhenlei Huang <zlei@FreeBSD.org>
CommitDate: 2023-09-20 04:04:57 +0000

    net: Check per-flow priority code point for untagged traffic

    Commit 868aabb4708d introduced per-flow priority. There's a defect in the
    logic for untagged traffic, it does not check M_VLANTAG set in the mbuf
    packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then
    can result missing desired priority in the outbound packets.

    For mbuf packet with M_VLANTAG in header, some interfaces happen to work
    due to bug in the drivers mentioned in D39499. As modern interfaces have
    VLAN hardware offloading, the defect is barely noticeable unless the
    feature per-flow priority is widely tested.

    As a side effect of this defect, the soft padding to work around buggy
    bridges is bypassed. That may result in regression if soft padding is
    requested.

    PR:             273431
    Discussed with: kib
    Fixes:  868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D39536

    (cherry picked from commit 49d6743da15fe378782e43776df8b4fd4f84c8d0)

 sys/net/if_ethersubr.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
Comment 4 commit-hook freebsd_committer freebsd_triage 2023-09-20 10:02:27 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=8303afca1765148d0069ce5144072b3ae9cab61e

commit 8303afca1765148d0069ce5144072b3ae9cab61e
Author:     Zhenlei Huang <zlei@FreeBSD.org>
AuthorDate: 2023-09-06 10:15:14 +0000
Commit:     Zhenlei Huang <zlei@FreeBSD.org>
CommitDate: 2023-09-20 10:00:37 +0000

    net: Check per-flow priority code point for untagged traffic

    Commit 868aabb4708d introduced per-flow priority. There's a defect in the
    logic for untagged traffic, it does not check M_VLANTAG set in the mbuf
    packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then
    can result missing desired priority in the outbound packets.

    For mbuf packet with M_VLANTAG in header, some interfaces happen to work
    due to bug in the drivers mentioned in D39499. As modern interfaces have
    VLAN hardware offloading, the defect is barely noticeable unless the
    feature per-flow priority is widely tested.

    As a side effect of this defect, the soft padding to work around buggy
    bridges is bypassed. That may result in regression if soft padding is
    requested.

    PR:             273431
    Discussed with: kib
    Fixes:  868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D39536

    (cherry picked from commit 49d6743da15fe378782e43776df8b4fd4f84c8d0)
    (cherry picked from commit c750055382f73db964c20f8eba855a9ac9e19591)

 sys/net/if_ethersubr.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
Comment 5 commit-hook freebsd_committer freebsd_triage 2023-09-20 10:09:30 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=d5dbed2f1e2ab14f54acbe14c30ccac9f9fee29f

commit d5dbed2f1e2ab14f54acbe14c30ccac9f9fee29f
Author:     Zhenlei Huang <zlei@FreeBSD.org>
AuthorDate: 2023-09-06 10:15:14 +0000
Commit:     Zhenlei Huang <zlei@FreeBSD.org>
CommitDate: 2023-09-20 10:05:14 +0000

    net: Check per-flow priority code point for untagged traffic

    Commit 868aabb4708d introduced per-flow priority. There's a defect in the
    logic for untagged traffic, it does not check M_VLANTAG set in the mbuf
    packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then
    can result missing desired priority in the outbound packets.

    For mbuf packet with M_VLANTAG in header, some interfaces happen to work
    due to bug in the drivers mentioned in D39499. As modern interfaces have
    VLAN hardware offloading, the defect is barely noticeable unless the
    feature per-flow priority is widely tested.

    As a side effect of this defect, the soft padding to work around buggy
    bridges is bypassed. That may result in regression if soft padding is
    requested.

    PR:             273431
    Discussed with: kib
    Fixes:  868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D39536

    (cherry picked from commit 49d6743da15fe378782e43776df8b4fd4f84c8d0)
    (cherry picked from commit c750055382f73db964c20f8eba855a9ac9e19591)
    (cherry picked from commit 8303afca1765148d0069ce5144072b3ae9cab61e)

 sys/net/if_ethersubr.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
Comment 6 commit-hook freebsd_committer freebsd_triage 2023-09-21 04:54:41 UTC
A commit in branch releng/14.0 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=0d648a59fceda79106fe66347b1df5cc11a7fa00

commit 0d648a59fceda79106fe66347b1df5cc11a7fa00
Author:     Zhenlei Huang <zlei@FreeBSD.org>
AuthorDate: 2023-09-06 10:15:14 +0000
Commit:     Zhenlei Huang <zlei@FreeBSD.org>
CommitDate: 2023-09-21 04:49:53 +0000

    net: Check per-flow priority code point for untagged traffic

    Commit 868aabb4708d introduced per-flow priority. There's a defect in the
    logic for untagged traffic, it does not check M_VLANTAG set in the mbuf
    packet header or MTAG_8021Q/MTAG_8021Q_PCP_OUT tag set by firewall, then
    can result missing desired priority in the outbound packets.

    For mbuf packet with M_VLANTAG in header, some interfaces happen to work
    due to bug in the drivers mentioned in D39499. As modern interfaces have
    VLAN hardware offloading, the defect is barely noticeable unless the
    feature per-flow priority is widely tested.

    As a side effect of this defect, the soft padding to work around buggy
    bridges is bypassed. That may result in regression if soft padding is
    requested.

    PR:             273431
    Approved by:    re (cperciva)
    Discussed with: kib
    Fixes:  868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D39536

    (cherry picked from commit 49d6743da15fe378782e43776df8b4fd4f84c8d0)
    (cherry picked from commit c750055382f73db964c20f8eba855a9ac9e19591)

 sys/net/if_ethersubr.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)