Bug 254623 - traceroute6: ICMP6 no longer works due to Capsicum'ization: data too short (-1 bytes) from invalid
Summary: traceroute6: ICMP6 no longer works due to Capsicum'ization: data too short (-...
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 13.0-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Mark Johnston
URL: https://reviews.freebsd.org/D29523
Keywords: needs-qa, regression
Depends on:
Blocks:
 
Reported: 2021-03-28 13:08 UTC by rashey
Modified: 2021-04-05 13:52 UTC (History)
5 users (show)

See Also:
koobs: maintainer-feedback+
koobs: mfc-stable13?


Attachments
Patch for traceroute6.c (401 bytes, patch)
2021-03-29 08:28 UTC, Zhenlei Huang
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description rashey 2021-03-28 13:08:34 UTC
Hi,

The traceroute6 command with -I argument does not work anymore.

I did not notice any other issues with data transfer or ping response.

I also restored 12.1 from backup to make sure that there is no any issues with tunnel broker service and everything works well with the same configuration.


# freebsd-version
13.0-RC3


# fetch -6 https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/13.0/CHECKSUM.SHA256-FreeBSD-13.0-RC3-amd64
CHECKSUM.SHA256-FreeBSD-13.0-RC3-amd64                1131  B 6071 kBps    00s


# ping6 -c1 www.google.com
PING6(56=40+8+8 bytes) 2001:X:X:X::2 --> 2a00:1450:401b:807::2004
16 bytes from 2a00:1450:401b:807::2004, icmp_seq=0 hlim=121 time=11.376 ms

--- www.google.com ping6 statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 11.376/11.376/11.376/0.000 ms


# tcpdump -nvi gif0
tcpdump: listening on gif0, link-type NULL (BSD loopback), capture size 262144 bytes
13:42:08.397237 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:X:X:X::2 > 2a00:1450:401b:807::2004: [icmp6 sum ok] ICMP6, echo request, seq 0
13:42:08.408567 IP6 (hlim 121, next-header ICMPv6 (58) payload length: 16) 2a00:1450:401b:807::2004 > 2001:X:X:X::2: [icmp6 sum ok] ICMP6, echo reply, seq 0
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel


# traceroute6 -Iq1 www.google.com
traceroute6 to www.google.com (2a00:1450:401b:807::2004) from 2001:X:X:X::2, 64 hops max, 20 byte packets
 1  *
 2  *
 3  *
 4  *
 5  *
 6  *
^C


# tcpdump -nvi gif0
tcpdump: listening on gif0, link-type NULL (BSD loopback), capture size 262144 bytes
13:46:12.095569 IP6 (hlim 1, next-header ICMPv6 (58) payload length: 20) 2001:X:X:X::2 > 2a00:1450:401b:807::2004: [icmp6 sum ok] ICMP6, echo request, seq 1
13:46:12.108615 IP6 (flowlabel 0xbcb91, hlim 64, next-header ICMPv6 (58) payload length: 68) 2001:470:70:402::1 > 2001:X:X:X::2: [icmp6 sum ok] ICMP6, time exceeded in-transit for 2a00:1450:401b:807::2004
13:46:17.095798 IP6 (hlim 2, next-header ICMPv6 (58) payload length: 20) 2001:X:X:X::2 > 2a00:1450:401b:807::2004: [icmp6 sum ok] ICMP6, echo request, seq 2
13:46:17.107274 IP6 (hlim 63, next-header ICMPv6 (58) payload length: 68) 2001:470:0:222::1 > 2001:X:X:X::2: [icmp6 sum ok] ICMP6, time exceeded in-transit for 2a00:1450:401b:807::2004
13:46:22.096031 IP6 (hlim 3, next-header ICMPv6 (58) payload length: 20) 2001:X:X:X::2 > 2a00:1450:401b:807::2004: [icmp6 sum ok] ICMP6, echo request, seq 3
13:46:22.107907 IP6 (hlim 62, next-header ICMPv6 (58) payload length: 68) 2001:7f8:42::a501:5169:1 > 2001:X:X:X::2: [icmp6 sum ok] ICMP6, time exceeded in-transit for 2a00:1450:401b:807::2004
13:46:27.096194 IP6 (hlim 4, next-header ICMPv6 (58) payload length: 20) 2001:X:X:X::2 > 2a00:1450:401b:807::2004: [icmp6 sum ok] ICMP6, echo request, seq 4
13:46:27.112026 IP6 (hlim 60, next-header ICMPv6 (58) payload length: 68) 2001:4860:0:1185::1 > 2001:X:X:X::2: [icmp6 sum ok] ICMP6, time exceeded in-transit for 2a00:1450:401b:807::2004
13:46:32.096428 IP6 (hlim 5, next-header ICMPv6 (58) payload length: 20) 2001:X:X:X::2 > 2a00:1450:401b:807::2004: [icmp6 sum ok] ICMP6, echo request, seq 5
13:46:32.108899 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 68) 2001:4860:0:1::1645 > 2001:X:X:X::2: [icmp6 sum ok] ICMP6, time exceeded in-transit for 2a00:1450:401b:807::2004
13:46:37.096667 IP6 (hlim 6, next-header ICMPv6 (58) payload length: 20) 2001:X:X:X::2 > 2a00:1450:401b:807::2004: [icmp6 sum ok] ICMP6, echo request, seq 6
13:46:37.108037 IP6 (hlim 121, next-header ICMPv6 (58) payload length: 20) 2a00:1450:401b:807::2004 > 2001:X:X:X::2: [icmp6 sum ok] ICMP6, echo reply, seq 6
13:46:42.096917 IP6 (hlim 7, next-header ICMPv6 (58) payload length: 20) 2001:X:X:X::2 > 2a00:1450:401b:807::2004: [icmp6 sum ok] ICMP6, echo request, seq 7
13:46:42.108278 IP6 (hlim 121, next-header ICMPv6 (58) payload length: 20) 2a00:1450:401b:807::2004 > 2001:X:X:X::2: [icmp6 sum ok] ICMP6, echo reply, seq 7
^C
14 packets captured
14 packets received by filter
0 packets dropped by kernel


# traceroute6 -Ivq1 www.google.com
data too short (-1 bytes) from invalid
data too short (-1 bytes) from invalid
data too short (-1 bytes) from invalid
data too short (-1 bytes) from invalid
data too short (-1 bytes) from invalid
data too short (-1 bytes) from invalid
^C


Please let me know if you need any additional information.
Comment 1 Zhenlei Huang 2021-03-29 07:22:46 UTC
The traceroute6.c from stable/12 works great with stable/13 kernel.
Comment 2 Zhenlei Huang 2021-03-29 08:28:10 UTC
Created attachment 223678 [details]
Patch for traceroute6.c

Hi rashey,
  Can you please test if this patch works? Thanks :)


CC Mariusz Zaborski
Comment 3 Zhenlei Huang 2021-03-29 08:46:28 UTC
The review D25604 capsicumize traceroute6, and use connect / send instead of sendto. For ICMPV6 there may be ICMP6_DST_UNREACH type packets from intermediate gateways, and they will not be captured by the pre-connected socket IIUC.

Here comes the solution, let's separate the connected socket (sndsock) from receive socket (rcvsock) as same as UDP / TCP / SCTP routines.

CC Mariusz Zaborski
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2021-03-30 03:26:55 UTC
^Triage: Review author doesn't appear to have an account in Bugzilla, request feedback from one of the reviews reviewers
Comment 5 Mark Johnston freebsd_committer 2021-03-31 22:31:07 UTC
Looking.  The patch works for me in a simple test but I don't quite understand why.
Comment 6 Mark Johnston freebsd_committer 2021-03-31 22:41:34 UTC
Ugh, we don't limit rights on rcvsock.  Thanks for catching this.  Please give this a try: https://reviews.freebsd.org/D29523
Comment 7 Zhenlei Huang 2021-04-01 04:44:34 UTC
(In reply to Mark Johnston from comment #6)
The patch D29523 works greatly :)

I do not have a FreeBSD phabricator account, and just registered one and the account is not approved yet. So I comment directly here.

Summary from review D29523:
> For ICMP6 we were using the same socket for both, and we limited rights
on the socket such that it's impossible to receive anything.

At first glance it seems the regression was due to no sufficient rights on receiving socket, and I tried setting CAP_RECV on the receiving socket without luck, I also tried disabling capsicum entirely and it behaves the same. So the root cause is not no sufficient rights on receiving socket. 

Limit rights on the recv socket is great :)

PS, man of cap_rights_limit gives an example entering capability mode before limiting rights. I tried setting CAP_RECV on recv socket after entering capability mode it also works greatly :-)  I'm not familiar with capsicum and it's pleasant if someone clarify this.
Comment 8 commit-hook freebsd_committer 2021-04-01 14:01:53 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b8ae450f05e62a851f444edaf7db2506ff99aa37

commit b8ae450f05e62a851f444edaf7db2506ff99aa37
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2021-04-01 13:58:32 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2021-04-01 14:00:29 +0000

    traceroute6: Fix Capsicum rights for rcvsock

    - Always use distinct sockets for send and recv
    - Limit rights on the recv socket

    For ICMP6 we were using the same socket for both send and receive, and
    we limited rights on the socket such that it's impossible to receive
    anything.

    PR:             254623
    Diagnosed by:   Zhenlei Huang <zlei.huang@gmail.com>
    Reviewed by:    oshogbo
    MFC after:      3 days
    Differential Revision:  https://reviews.freebsd.org/D29523

 usr.sbin/traceroute6/traceroute6.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
Comment 9 commit-hook freebsd_committer 2021-04-05 13:52:47 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=d3f2c31b43b726ffbb180a42cee4b9f00c5ad5ed

commit d3f2c31b43b726ffbb180a42cee4b9f00c5ad5ed
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2021-04-01 13:58:32 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2021-04-05 13:51:56 +0000

    traceroute6: Fix Capsicum rights for rcvsock

    - Always use distinct sockets for send and recv
    - Limit rights on the recv socket

    For ICMP6 we were using the same socket for both send and receive, and
    we limited rights on the socket such that it's impossible to receive
    anything.

    PR:             254623
    Diagnosed by:   Zhenlei Huang <zlei.huang@gmail.com>
    Reviewed by:    oshogbo
    Differential Revision:  https://reviews.freebsd.org/D29523

    (cherry picked from commit b8ae450f05e62a851f444edaf7db2506ff99aa37)

 usr.sbin/traceroute6/traceroute6.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)