While stress testing I see this a lot. Seems related to low value of v_free_count. panic: SACK scoreboard must not be empty cpuid = 10 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0131bcd560 vpanic() at vpanic+0x186/frame 0xfffffe0131bcd5e0 kassert_panic() at kassert_panic+0x126/frame 0xfffffe0131bcd650 tcp_sack_doack() at tcp_sack_doack+0x4d4/frame 0xfffffe0131bcd6d0 tcp_do_segment() at tcp_do_segment+0x1a06/frame 0xfffffe0131bcd7c0 tcp_input() at tcp_input+0xd7c/frame 0xfffffe0131bcd920 ip_input() at ip_input+0x17b/frame 0xfffffe0131bcd980 swi_net() at swi_net+0x175/frame 0xfffffe0131bcd9e0 intr_event_execute_handlers() at intr_event_execute_handlers+0x96/frame 0xfffffe0131bcda20 ithread_loop() at ithread_loop+0xa6/frame 0xfffffe0131bcda70 fork_exit() at fork_exit+0x84/frame 0xfffffe0131bcdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0131bcdab0 https://people.freebsd.org/~pho/stress/log/full.txt
Peter, which test causes this? Any way I can reprod this?
(In reply to Peter Holm from comment #0) Any change to get a pcap or pcapng tracefile which triggers the issue? I would like to reproduce this with packetdrill...
(In reply to Hiren Panchasara from comment #1) Sure: OK set hw.physmem=6g : $ svnlite checkout -q svn://svn.freebsd.org/base/user/pho/stress2 $ cd stress2 $ make > /dev/null $ echo "testuser=stress" > `hostname` $ cd misc $ ./all.sh full.sh Details @ https://people.freebsd.org/~pho/stress/log/full-2.txt
(In reply to Michael Tuexen from comment #2) Michael, the panic is very easy to reproduce on my hardware. What do I need to do?
By looking at your Details, your system is low on memory when the problem occurs. Can you just remove the KASSERT http://fxr.watson.org/fxr/source/netinet/tcp_sack.c#L474 and retest? I think if tcp_sackhole_insert in http://fxr.watson.org/fxr/source/netinet/tcp_sack.c#L446 returns NULL due to memory shortage, you run into the KASSERT. Best regards Michael
(In reply to Michael Tuexen from comment #5) Looks promising. I'll let the test run a bit longer, just to be sure.
(In reply to Peter Holm from comment #6) No problems seen in four hours. LGTM.
A commit references this bug: Author: tuexen Date: Sun Dec 25 17:37:18 UTC 2016 New revision: 310547 URL: https://svnweb.freebsd.org/changeset/base/310547 Log: Remove a KASSERT which is not always true. In case of the empty queue tp->snd_holes and tcp_sackhole_insert() failing due to memory shortage, tp->snd_holes will be empty. This problem was hit when stress tests where performed by pho. PR: 215513 Reported by: pho Tested by: pho Sponsored by: Netflix, Inc. Changes: head/sys/netinet/tcp_sack.c
A commit references this bug: Author: tuexen Date: Thu Dec 29 11:25:41 UTC 2016 New revision: 310772 URL: https://svnweb.freebsd.org/changeset/base/310772 Log: MFC r310547: Remove a KASSERT which is not always true. In case of the empty queue tp->snd_holes and tcp_sackhole_insert() failing due to memory shortage, tp->snd_holes will be empty. This problem was hit when stress tests where performed by pho. PR: 215513 Reported by: pho Tested by: pho Sponsored by: Netflix, Inc. Changes: _U stable/11/ stable/11/sys/netinet/tcp_sack.c