When TCP socket goes to LAST_ACK state & remote host do not respone ACK forever, socket would stay at LAST_ACK forever and never be removed. It would be a bug expolit by DDoS attacker. Patch file to fix the problem is following.
Hello thinker, Friday, March 23, 2001, 12:50:53 AM, you wrote: >>Number: 25986 >>Category: kern >>Synopsis: Socket would hang at LAST_ACK forever. >>Confidential: no >>Severity: serious >>Priority: high >>Responsible: freebsd-bugs >>State: open >>Quarter: >>Keywords: >>Date-Required: >>Class: change-request >>Submitter-Id: current-users >>Arrival-Date: Thu Mar 22 00:50:00 PST 2001 >>Closed-Date: >>Last-Modified: >>Originator: thinker >>Release: FreeBSD 5.0-CURRENT i386 >>Organization: >>Environment: t> System: FreeBSD hell.branda.to 5.0-CURRENT FreeBSD 5.0-CURRENT #1: Wed Mar 7 20:41:54 GMT 2001 thinker@hell.branda.to:/usr/src/sys/compile/thk i386 >>Description: t> When TCP socket goes to LAST_ACK state & remote host do not respone t> ACK forever, socket would stay at LAST_ACK forever and never be removed. t> It would be a bug expolit by DDoS attacker. Patch file to fix the problem t> is following. >>How-To-Repeat: >>Fix: t> --- sys/netinet/tcp_usrreq.c.orig Thu Mar 22 14:59:45 2001 t> +++ sys/netinet/tcp_usrreq.c Thu Mar 22 15:04:49 2001 t> @@ -1139,13 +1139,15 @@ t> tp->t_state = TCPS_LAST_ACK; t> break; t> } t> - if (tp && tp->t_state >= TCPS_FIN_WAIT_2) { t> + if (tp && tp->t_state >= TCPS_FIN_WAIT_2) t> soisdisconnected(tp->t_inpcb->inp_socket); t> - /* To prevent the connection hanging in FIN_WAIT_2 forever. */ t> - if (tp->t_state == TCPS_FIN_WAIT_2) t> - callout_reset(tp->tt_2msl, tcp_maxidle, t> - tcp_timer_2msl, tp); t> - } t> + /* t> + * To prevent the connection hanging in FIN_WAIT_2 & t> + * TCPS_LAST_ACK forever. t> + */ + if (tp->>t_state == TCPS_FIN_WAIT_2 || tp->t_state == TCPS_LAST_ACK) t> + callout_reset(tp->tt_2msl, tcp_maxidle, t> + tcp_timer_2msl, tp); t> return (tp); t> } Thank you! it's a long standing bug, but seems no one want to fix it, don't know why! -- Best regards, David Xu
<<On Thu, 22 Mar 2001 01:10:02 -0800 (PST), David Xu <bsddiy@21cn.com> said: > Thank you! > it's a long standing bug, but seems no one want to fix it, don't know why! Probably because this ``fix'' would violate the TCP specification. (Not that we don't do that already for FINWAIT-2.) See the state diagram in RFC 793, page 23. There are three states in TCP in which the specification requires that we wait indefinitely for the other end to say something on the way to shutdown. In FINWAIT-2, we are waiting for a FIN from the other side, but they have already acknowledged our FIN). Since TCP models a dual-simplex octet stream, it is entirely legitimate for the other end to continue to transmit data after it has acknowledged our FIN and before it gives up itself. (They will be in CLOSE-WAIT when we are in FINWAIT-2.) In CLOSING and LAST-ACK we are waiting for the other end to acknowledge our FIN, them having already sent a FIN of their own. CLOSING is an exceedingly difficult state to get into, since most TCPs will ACK a FIN immediately, before delivering the end-of-session indication to the listening process. It is more dangerous to time out LAST-ACK than FINWAIT-2, because FINWAIT-2 sockets normally transition into TIME-WAIT whereas LAST-ACK sockets are destroyed immediately. -GAWollman
> It is more dangerous to time out LAST-ACK than FINWAIT-2, because > FINWAIT-2 sockets normally transition into TIME-WAIT whereas LAST-ACK > sockets are destroyed immediately. Yes, timeout at LAST-ACK make we can't make sure that FIN had been received by the other end. But, if we can't not receive any ACK for a long time, it can be thinked as a broken host or network. Diagram in RFC 793, page 23, donot draw out what should we do when we had detect a connection broken (by any way) at ESTAB state, too. It tell us what is right but what is wrong. What I want to said is "Can we be imcompatable with dead site?". -- thinker@branda.to Branda Open Site (BOS) thinker.bbs@bbs.yzu.edu.tw http://www.branda.to/
Responsible Changed From-To: freebsd-bugs->silby I plan to get around to fixing this some day.
*just a little follow-up to give this bug attention again* This PR has been here since March 2001, could this please be fixed? This is a serious bug which allows for a easy DoS, we really don't want it to remain unpatched, do we?
Silby, any news on this one? May/shall I look into it in more detail? -- Andre
Responsible Changed From-To: silby->andre Take over after conversation with silby. I'll look into this and how to fix it.
State Changed From-To: open->analyzed I have analyzed the situation and what is going on here. The result has been communicated to silby to check. I'll close the case when he confirms my analysis.
andre 2006-09-28 18:02:46 UTC FreeBSD src repository Modified files: sys/netinet tcp_output.c Log: When tcp_output() receives an error upon sending a packet it reverts parts of its internal state to ignore the failed send and try again a bit later. If the error is EPERM the packet got blocked by the local firewall and the revert may cause the session to get stuck and retry indefinitely. This way we treat it like a packet loss and let the retransmit timer and timeouts do their work over time. The correct behavior is to drop a connection that gets an EPERM error. However this _may_ introduce some POLA problems and a two commit approach was chosen. Discussed with: glebius PR: kern/25986 PR: kern/102653 Revision Changes Path 1.120 +15 -2 src/sys/netinet/tcp_output.c _______________________________________________ cvs-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/cvs-all To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
State Changed From-To: analyzed->patched A fix has been committed. (andre: has this been MFCed yet?)
Responsible Changed From-To: andre->silby silby is currently the more active of the two
Please check this! Regarding to CVS history, this is patched but never MFCed. rwatson suggests to have that MFCed.
State Changed From-To: patched->feedback Set to 'feedback' as there's a followup to Silby WRT MFC'ing.
On Sat, 26 Jan 2008, rwatson@FreeBSD.org wrote: > Synopsis: Socket would hang at LAST_ACK forever. > > State-Changed-From-To: patched->feedback > State-Changed-By: rwatson > State-Changed-When: Sat Jan 26 12:41:48 UTC 2008 > State-Changed-Why: > Set to 'feedback' as there's a followup to Silby WRT MFC'ing. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=25986 <time passes> Looking at this PR and the committed patch, I think that the wrong PR was referenced in the commit: fix for the TCPS_LAST_ACK problem doesn't seem to have been committed. The commit referenced in this PR should be MFC'd (if it hasn't been already), but for unrelated reasons. The location where the fix for this PR would now be required has moved from tcp_output.c to tcp_usrreq.c, tcp_usrclosed() (I think, anyway). I believe the observation about needing a timeout there is correct, but we need to get someone with adequate TCP protocol and timer clue to render an opinion on both the nature of the fix and how to make it. I've added John Baldwin and Lawrence Stewart to the CC line, since both have expertise in these areas. Robert
State Changed From-To: feedback->open Submitter provided requested information
Hi, Hopefully this can help, where I work we have been using the patch below for years now. Please note the slight difference with the proposed patch. We found that after a connection close (which sends a FIN and puts the tcpcb in FIN_WAIT_1 state) if the other end never ACK the FIN or sends a FIN for that matter we would also see lingering FIN_WAIT_1 connections. @@ -1543,16 +1926,19 @@ tp->t_state = TCPS_LAST_ACK; break; } + if (tp->t_state >= TCPS_FIN_WAIT_2) { soisdisconnected(tp->t_inpcb->inp_socket); - /* Prevent the connection hanging in FIN_WAIT_2 forever. */ - if (tp->t_state == TCPS_FIN_WAIT_2) { - int timeout; - - timeout = (tcp_fast_finwait2_recycle) ? - tcp_finwait2_timeout : tcp_maxidle; - tcp_timer_activate(tp, TT_2MSL, timeout); - } + } + + /* kfl: To prevent the connection hanging in half closed states forever. */ + if ((tp->t_state == TCPS_FIN_WAIT_1) || + (tp->t_state == TCPS_FIN_WAIT_2) || + (tp->t_state == TCPS_LAST_ACK)) { + int timeout; + + timeout = (tcp_fast_finwait2_recycle) ? tcp_finwait2_timeout : tcp_maxidle; + tcp_timer_activate(tp, TT_2MSL, timeout); } }
Responsible Changed From-To: silby->freebsd-net Reassign to freebsd-net@.
This should be fixed by FreeBSD-SA-15:13.tcp.
We're seeing this problem on a set of servers running head from March 2017. The patch suggested above does not appear to be committed. What was the fix that was supposed to solve this? Apparently it has been left out or is not working as expected.
(In reply to Johannes Lundberg from comment #19) 47a8e86509b258d16c4a36e2e475cb946bbc447c / r285777 .
I believe I see the same issue with 13.0. When someone tries to brute force SSH and "sshguard" blocks the connections then the connections "stuck" in LAST_ACK state. Also the same happens with some connections related to http.