Bug 223817
| Summary: | mbuf leaking with tcpmd5 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Dirk Meyer <dinoex> | ||||
| Component: | kern | Assignee: | Andrey V. Elsukov <ae> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Only Me | CC: | ae, zarychtam | ||||
| Priority: | --- | Keywords: | regression | ||||
| Version: | 11.1-RELEASE | ||||||
| Hardware: | i386 | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
|
Description
Dirk Meyer
2017-11-23 13:06:41 UTC
Can you show the output of `netstat -sp tcp` command? uptime 3 min
tcp:
13140 packets sent
208 data packets (15004 bytes)
0 data packets (0 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
8423 ack-only packets (457 delayed)
0 URG only packets
0 window probe packets
4485 window update packets
24 control packets
18601 packets received
163 acks (for 15011 bytes)
35 duplicate acks
0 acks for unsent data
17810 packets (21631400 bytes) received in-sequence
35 completely duplicate packets (26221 bytes)
0 old duplicate packets
9 packets with some dup. data (11927 bytes duped)
464 out-of-order packets (625887 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
18 connection requests
1 connection accept
0 bad connection attempts
0 listen queue overflows
0 ignored RSTs in the windows
11 connections established (including accepts)
0 times used RTT from hostcache
0 times used RTT variance from hostcache
0 times used slow-start threshold from hostcache
17 connections closed (including 0 drops)
1 connection updated cached RTT on close
1 connection updated cached RTT variance on close
0 connections updated cached ssthresh on close
6 embryonic connections dropped
163 segments updated rtt (of 170 attempts)
62 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
6 keepalive timeouts
0 keepalive probes sent
6 connections dropped by keepalive
92 correct ACK header predictions
17774 correct data packet header predictions
1 syncache entry added
0 retransmitted
0 dupsyn
0 dropped
1 completed
0 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
1 cookie sent
0 cookies received
1 hostcache entry added
0 bucket overflow
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks) received
0 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
0 packets with matching signature received
0 packets with bad signature received
143 times failed to make signature due to no SA
0 times unexpected signature received
10 times no signature provided by segment
TCP connection count by state:
0 connections in CLOSED state
4 connections in LISTEN state
2 connections in SYN_SENT state
0 connections in SYN_RCVD state
5 connections in ESTABLISHED state
0 connections in CLOSE_WAIT state
0 connections in FIN_WAIT_1 state
0 connections in CLOSING state
0 connections in LAST_ACK state
0 connections in FIN_WAIT_2 state
0 connections in TIME_WAIT state
uptime 30 min
tcp:
19129 packets sent
673 data packets (32758 bytes)
0 data packets (0 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
13871 ack-only packets (5040 delayed)
0 URG only packets
0 window probe packets
4485 window update packets
100 control packets
25085 packets received
572 acks (for 32765 bytes)
35 duplicate acks
0 acks for unsent data
23055 packets (22835499 bytes) received in-sequence
35 completely duplicate packets (26221 bytes)
0 old duplicate packets
9 packets with some dup. data (11927 bytes duped)
464 out-of-order packets (625887 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
94 connection requests
1 connection accept
0 bad connection attempts
0 listen queue overflows
1 ignored RSTs in the window
11 connections established (including accepts)
0 times used RTT from hostcache
0 times used RTT variance from hostcache
0 times used RTT from hostcache
0 times used RTT variance from hostcache
0 times used slow-start threshold from hostcache
93 connections closed (including 0 drops)
1 connection updated cached RTT on close
1 connection updated cached RTT variance on close
0 connections updated cached ssthresh on close
82 embryonic connections dropped
572 segments updated rtt (of 655 attempts)
669 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
82 keepalive timeouts
0 keepalive probes sent
82 connections dropped by keepalive
455 correct ACK header predictions
22992 correct data packet header predictions
1 syncache entry added
0 retransmitted
0 dupsyn
0 dropped
1 completed
0 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
1 cookie sent
0 cookies received
1 hostcache entry added
0 bucket overflow
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks) received
0 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
0 packets with matching signature received
0 packets with bad signature received
1209 times failed to make signature due to no SA
0 times unexpected signature received
10 times no signature provided by segment
TCP connection count by state:
0 connections in CLOSED state
4 connections in LISTEN state
2 connections in SYN_SENT state
0 connections in SYN_RCVD state
5 connections in ESTABLISHED state
0 connections in CLOSE_WAIT state
0 connections in FIN_WAIT_1 state
0 connections in CLOSING state
0 connections in LAST_ACK state
0 connections in FIN_WAIT_2 state
0 connections in TIME_WAIT state
The same command on FreeeBSD 10.3
shows data for tcpmd5.
uptime 2 days:
root@hbw2:~# netstat -sp tcp
tcp:
404820 packets sent
16566 data packets (809618 bytes)
9 data packets (171 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
381530 ack-only packets (351256 delayed)
0 URG only packets
0 window probe packets
6703 window update packets
12 control packets
453693 packets received
15429 acks (for 809625 bytes)
6189 duplicate acks
0 acks for unsent data
420555 packets (130396892 bytes) received in-sequence
42 completely duplicate packets (32133 bytes)
0 old duplicate packets
9 packets with some dup. data (9812 bytes duped)
222 out-of-order packets (240797 bytes)
0 packets (0 bytes) of data after window
0 window probes
25 window update packets
0 packets received after close
3 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
8 connection requests
5 connection accepts
0 bad connection attempts
0 listen queue overflows
0 ignored RSTs in the windows
11 connections established (including accepts)
11 connections closed (including 0 drops)
4 connections updated cached RTT on close
4 connections updated cached RTT variance on close
0 connections updated cached ssthresh on close
0 embryonic connections dropped
15426 segments updated rtt (of 15426 attempts)
7 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
8 keepalive timeouts
8 keepalive probes sent
0 connections dropped by keepalive
13389 correct ACK header predictions
419108 correct data packet header predictions
10 syncache entries added
15 retransmitted
0 dupsyn
0 dropped
5 completed
0 bucket overflow
0 cache overflow
0 reset
5 stale
0 aborted
0 badack
0 unreach
0 zone failures
10 cookies sent
0 cookies received
3 hostcache entries added
0 bucket overflow
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks) received
0 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
252573 packets with valid tcp-md5 signature received
2 packets with invalid tcp-md5 signature received
0 packets with tcp-md5 signature mismatch
0 packets with unexpected tcp-md5 signature received
2 packets without expected tcp-md5 signature received
Can you try to see what will show the following dtrace script?
# kldload dtraceall
# dtrace -n 'fbt::tcp_ipsec_output:return {printf("%d", arg1);}'
Also, it seems your MD5-signed connections doesn't work. Probably you need to check that SAs for both directions are created.
Created attachment 188313 [details]
Proposed patch
If dtrace script will show 13 return code, then probably my assumption is correct and you can try this patch.
I rebuild the kernel with dtrace.
# kldload dtraceall
# dtrace -n 'fbt::tcp_ipsec_output:return {printf("%d", arg1);}'
dtrace: description 'fbt::tcp_ipsec_output:return ' matched 1 probe
no further output for 50 mins.
Indeed the connection does not come up.
# setkey -D
output looks good.
setkey shows 4 SA, 2 IPv4 for each direction, 2 IPv6 for each direction.
The 11.1 kernel compiled with the patch runs now with stable mbufs. The memory leak I saw is fixed. but MD5-signed connections doesn't work with 11.1 I have in my kernel: options IPSEC decive crypto options TCP_SIGNATURE MD5-signed connections doesn't work I will try to rebuild the kernel with option options IPSEC_SUPPORT adding IPSEC_SUPPORT does not have any effect on the issue. Has the TCP_MD5 API changed in 11.x? A commit references this bug: Author: ae Date: Thu Dec 14 12:54:21 UTC 2017 New revision: 326847 URL: https://svnweb.freebsd.org/changeset/base/326847 Log: Fix mbuf leak when TCPMD5_OUTPUT() method returns error. PR: 223817 MFC after: 1 week Changes: head/sys/netinet/tcp_output.c (In reply to Dirk Meyer from comment #9) > adding IPSEC_SUPPORT does not have any effect on the issue. > > Has the TCP_MD5 API changed in 11.x? The IPSEC_SUPPORT option just allows use tcpmd5.ko and ipsec.ko kernel modules. The main change in 11.1+ is that now security association for both directions are required to initiate MD5 signed TCP connection. Also, now TCP-MD5 implementation fully conforms to RFC and it is impossible to send unsigned TCP segments or segments with empty MD5 signature. If you have both SAs and connection still does not created, look at the netstat -sp tcp statistics. Probably you need to create SAs by hand (via setkey or ipsec.conf) using endpoint's IP addresses. I think source addresses in SAs is incorrect due to bird has bound socket to any IP address and then uses this address to create SA. Also there were several reports that disabling RX/TX checksum offloading on NIC can help. A commit references this bug: Author: ae Date: Thu Dec 21 14:09:06 UTC 2017 New revision: 327061 URL: https://svnweb.freebsd.org/changeset/base/327061 Log: MFC r326847: Fix mbuf leak when TCPMD5_OUTPUT() method returns error. PR: 223817 Changes: _U stable/11/ stable/11/sys/netinet/tcp_output.c Fixed in head/ and stable/11. Thanks! |