Bug 219476 - [i386] 11.1-PRERELEASE double faults due to low kern.kstack_pages default
Summary: [i386] 11.1-PRERELEASE double faults due to low kern.kstack_pages default
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-STABLE
Hardware: i386 Any
: --- Affects Some People
Assignee: freebsd-qa (Nobody)
URL:
Keywords: crash, easy
: 185427 (view as bug list)
Depends on:
Blocks:
 
Reported: 2017-05-23 19:35 UTC by Eugene Grosbein
Modified: 2018-12-26 10:13 UTC (History)
6 users (show)

See Also:
koobs: mfc-stable11+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Eugene Grosbein freebsd_committer freebsd_triage 2017-05-23 19:35:34 UTC
stable/11 got lots of kernel level code that abuses small default stack of i386 kernel threads: ZFS, IPSEC, SCTP, device drivers etc. Overflow of kernel stack produces "double fault" panics.

kern.kstack_pages is loader tunnable now for i386. The system should be stable out of the box, so we should increase default for kern.kstack_pages. Loader tunnable can serve users of i386 systems that are unhappy with new default, they can decrease it with /boot/loader.conf

Some examples:

* sys/netinet/sctp_pcb.c, function sctp_load_addresses_from_init() allocates 2184 bytes on stack (disassemble: sub $0x888,%esp);

* sys/netinet/sctp_auth.c: sctp_auth_get_cookie_params(): 1592 bytes on stack;

* src/sys/contrib/dev/ath/ath_hal/ar9300/ar9300_eeprom.c: ar9300_eeprom_restore_internal_address(): 2092 bytes on stack;

* sys/contrib/dev/ath/ath_hal/ar9300/ar9300_paprd.c: create_pa_curve(): 1416 bytes;

* sys/libkern/zlib.c: huft_build(): 1420 bytes;

And so on. Here is "top-list" for my home router custom kernel (1GB RAM, no swap):

Bytes-on-stack  In-module
2184            sctp_pcb.o
2092            ar9300_eeprom.o
2080            kern_linker.o
1664            cryptosoft.o
1592            sctp_auth.o
1536            glxsb_hash.o
1420            zlib.o
1416            ar9300_paprd.o
1352            scsi_da.o
1344            nfs_nfsdport.o
1328            vm_object.o
1312            fortuna.o
1232            cam_periph.o
1224            zlib.o
1192            cam_xpt.o
1192            ata_da.o
1184            cam_xpt.o
1168            ata_da.o
1160            sctp_output.o

Some of "network hot path" subroutines not shown here do abuse kernel stack too, for example SHA256_Transform() from sys/crypto/sha2/sha256c.c that may be actively used with IPSEC processing.

ae@'s https://reviews.freebsd.org/D10869 deals with IPSEC in part and helps in my case, at least while there is no ZFS involved. However, it is impossible to perform similar cleanup for each and every kernel subsystem in near future and double panics with current defaults are bad too.
Comment 1 Ed Maste freebsd_committer freebsd_triage 2017-05-27 01:42:44 UTC
This should probably have Hardware=i386, yes?
Comment 2 Eugene Grosbein freebsd_committer freebsd_triage 2017-05-27 10:32:25 UTC
(In reply to Ed Maste from comment #1)

Fixed, thanks!
Comment 3 commit-hook freebsd_committer freebsd_triage 2017-05-29 09:31:12 UTC
A commit references this bug:

Author: ae
Date: Mon May 29 09:30:39 UTC 2017
New revision: 319118
URL: https://svnweb.freebsd.org/changeset/base/319118

Log:
  Disable IPsec debugging code by default when IPSEC_DEBUG kernel option
  is not specified.

  Due to the long call chain IPsec code can produce the kernel stack
  exhaustion on the i386 architecture. The debugging code usually is not
  used, but it requires a lot of stack space to keep buffers for strings
  formatting. This patch conditionally defines macros to disable building
  of IPsec debugging code.

  IPsec currently has two sysctl variables to configure debug output:
   * net.key.debug variable is used to enable debug output for PF_KEY
     protocol. Such debug messages are produced by KEYDBG() macro and
     usually they can be interesting for developers.
   * net.inet.ipsec.debug variable is used to enable debug output for
     DPRINTF() macro and ipseclog() function. DPRINTF() macro usually
     is used for development debugging. ipseclog() function is used for
     debugging by administrator.

  The patch disables KEYDBG() and DPRINTF() macros, and formatting buffers
  declarations when IPSEC_DEBUG is not present in kernel config. This reduces
  stack requirement for up to several hundreds of bytes.
  The net.inet.ipsec.debug variable still can be used to enable ipseclog()
  messages by administrator.

  PR:		219476
  Reported by:	eugen
  No objection from:	#network
  MFC after:	1 week
  Differential Revision:	https://reviews.freebsd.org/D10869

Changes:
  head/sys/netipsec/ipsec.h
  head/sys/netipsec/ipsec_input.c
  head/sys/netipsec/ipsec_output.c
  head/sys/netipsec/key_debug.h
  head/sys/netipsec/xform_ah.c
  head/sys/netipsec/xform_esp.c
  head/sys/netipsec/xform_ipcomp.c
Comment 4 commit-hook freebsd_committer freebsd_triage 2017-06-05 11:12:14 UTC
A commit references this bug:

Author: ae
Date: Mon Jun  5 11:11:08 UTC 2017
New revision: 319599
URL: https://svnweb.freebsd.org/changeset/base/319599

Log:
  MFC r319118:
    Disable IPsec debugging code by default when IPSEC_DEBUG kernel option
    is not specified.

    Due to the long call chain IPsec code can produce the kernel stack
    exhaustion on the i386 architecture. The debugging code usually is not
    used, but it requires a lot of stack space to keep buffers for strings
    formatting. This patch conditionally defines macros to disable building
    of IPsec debugging code.

    IPsec currently has two sysctl variables to configure debug output:
     * net.key.debug variable is used to enable debug output for PF_KEY
       protocol. Such debug messages are produced by KEYDBG() macro and
       usually they can be interesting for developers.
     * net.inet.ipsec.debug variable is used to enable debug output for
       DPRINTF() macro and ipseclog() function. DPRINTF() macro usually
       is used for development debugging. ipseclog() function is used for
       debugging by administrator.

    The patch disables KEYDBG() and DPRINTF() macros, and formatting buffers
    declarations when IPSEC_DEBUG is not present in kernel config. This
    reduces stack requirement for up to several hundreds of bytes.
    The net.inet.ipsec.debug variable still can be used to enable ipseclog()
    messages by administrator.

    PR:		219476

  MFC r319412:
    Build kdebug_secreplay() function only when IPSEC_DEBUG is defined.
    This should fix the build on sparc.

  Approved by: re (kib)

Changes:
_U  stable/11/
  stable/11/sys/netipsec/ipsec.h
  stable/11/sys/netipsec/ipsec_input.c
  stable/11/sys/netipsec/ipsec_output.c
  stable/11/sys/netipsec/key_debug.c
  stable/11/sys/netipsec/key_debug.h
  stable/11/sys/netipsec/xform_ah.c
  stable/11/sys/netipsec/xform_esp.c
  stable/11/sys/netipsec/xform_ipcomp.c
Comment 5 Eugene Grosbein freebsd_committer freebsd_triage 2017-08-31 11:16:57 UTC
ae@ did its job and this PR not longer concerns him but needs general discussion.
Comment 6 commit-hook freebsd_committer freebsd_triage 2017-12-11 04:32:59 UTC
A commit references this bug:

Author: cem
Date: Mon Dec 11 04:32:37 UTC 2017
New revision: 326758
URL: https://svnweb.freebsd.org/changeset/base/326758

Log:
  i386: Bump KSTACK_PAGES default to match amd64

  Logically, extend r286288 to cover all threads, by default.

  The world has largely moved on from i386.  Most FreeBSD users and developers
  test on amd64 hardware.  For better or worse, we have written a non-trivial
  amount of kernel code that relies on stacks larger than 8 kB, and it "just
  works" on amd64, so there has been little incentive to shrink it.

  amd64 had its KSTACK_PAGES bumped to 4 back in Peter's initial AMD64 commit,
  r114349, in 2003.  Since that time, i386 has limped along on a stack half
  the size.  We've even observed the stack overflows years ago, but neglected
  to fix the issue; see the 20121223 and 20150728 entries in UPDATING.

  If anyone is concerned with this change, I suggest they configure their
  AMD64 kernels with KSTACK_PAGES 2 and fix the fallout there first.  Eugene
  has identified a list of high stack usage functions in the first PR below.

  PR:		219476, 224218
  Reported by:	eugen@, Shreesh Holla <hshreesh AT yahoo.com>
  Relnotes:	maybe
  Sponsored by:	Dell EMC Isilon

Changes:
  head/sys/i386/conf/NOTES
  head/sys/i386/include/param.h
Comment 7 Eugene Grosbein freebsd_committer freebsd_triage 2018-05-28 21:19:00 UTC
*** Bug 185427 has been marked as a duplicate of this bug. ***
Comment 8 Eugene Grosbein freebsd_committer freebsd_triage 2018-12-26 10:13:16 UTC
Fixed and MFC'd.