Comment 1 Andrey V. Elsukov 2019-02-12 11:20:08 UTC

(In reply to Sergey Anokhin from comment #0)
> I see kernel panic during racoon restart.
> 
> # uname -rv
> 12.0-STABLE FreeBSD 12.0-STABLE r343904 SERVER

Please, show the content of your kernel config and what sysctl variables do you changed against default configuration.

(In reply to Andrey V. Elsukov from comment #1)

kernel config:

(pts/2)[root@server:~]# cat /usr/src/sys/amd64/conf/SERVER
#
# GENERIC -- Generic kernel configuration file for FreeBSD/amd64
#
# For more information on this file, please read the config(5) manual page,
# and/or the handbook section on Kernel Configuration Files:
#
#    https://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
#
# The handbook is also available locally in /usr/share/doc/handbook
# if you've installed the doc distribution, otherwise always see the
# FreeBSD World Wide Web server (https://www.FreeBSD.org/) for the
# latest information.
#
# An exhaustive list of options and more detailed explanations of the
# device lines is also present in the ../../conf/NOTES and NOTES files.
# If you are in doubt as to the purpose or necessity of a line, check first
# in NOTES.
#
# $FreeBSD: stable/12/sys/amd64/conf/GENERIC 340695 2018-11-20 19:37:09Z zeising $

cpu             HAMMER
ident           SERVER

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols
makeoptions     WITH_CTF=1              # Run ctfconvert(1) for DTrace support

options         SCHED_ULE               # ULE scheduler
options         NUMA                    # Non-Uniform Memory Architecture support
options         PREEMPTION              # Enable kernel thread preemption
options         VIMAGE                  # Subsystem virtualization, e.g. VNET
options         INET                    # InterNETworking
options         INET6                   # IPv6 communications protocols
options         IPSEC                   # IP (v4/v6) security
options         IPSEC_SUPPORT           # Allow kldload of ipsec and tcpmd5
options         TCP_OFFLOAD             # TCP offload
options         TCP_BLACKBOX            # Enhanced TCP event logging
options         TCP_HHOOK               # hhook(9) framework for TCP
options         TCP_RFC7413             # TCP Fast Open
options         SCTP                    # Stream Control Transmission Protocol
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         UFS_ACL                 # Support for access control lists
options         UFS_DIRHASH             # Improve performance on big directories
options         UFS_GJOURNAL            # Enable gjournal-based UFS journaling
options         QUOTA                   # Enable disk quotas for UFS
options         MD_ROOT                 # MD is a potential root device
options         NFSCL                   # Network Filesystem Client
options         NFSD                    # Network Filesystem Server
options         NFSLOCKD                # Network Lock Manager
options         NFS_ROOT                # NFS usable as /, requires NFSCL
options         MSDOSFS                 # MSDOS Filesystem
options         CD9660                  # ISO 9660 Filesystem
options         PROCFS                  # Process filesystem (requires PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         GEOM_RAID               # Soft RAID functionality.
options         GEOM_LABEL              # Provides labelization
options         EFIRT                   # EFI Runtime Services support
options         COMPAT_FREEBSD32        # Compatible with i386 binaries
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         COMPAT_FREEBSD5         # Compatible with FreeBSD5
options         COMPAT_FREEBSD6         # Compatible with FreeBSD6
options         COMPAT_FREEBSD7         # Compatible with FreeBSD7
options         COMPAT_FREEBSD9         # Compatible with FreeBSD9
options         COMPAT_FREEBSD10        # Compatible with FreeBSD10
options         COMPAT_FREEBSD11        # Compatible with FreeBSD11
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI
options         KTRACE                  # ktrace(1) support
options         STACK                   # stack(9) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options         PRINTF_BUFR_SIZE=128    # Prevent printf output being interspersed.
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options         HWPMC_HOOKS             # Necessary kernel hooks for hwpmc(4)
options         AUDIT                   # Security event auditing
options         CAPABILITY_MODE         # Capsicum capability mode
options         CAPABILITIES            # Capsicum capabilities
options         MAC                     # TrustedBSD MAC Framework
options         KDTRACE_FRAME           # Ensure frames are compiled in
options         KDTRACE_HOOKS           # Kernel DTrace hooks
options         DDB_CTF                 # Kernel ELF linker loads CTF data
options         INCLUDE_CONFIG_FILE     # Include this file in kernel
options         RACCT                   # Resource accounting framework
options         RACCT_DEFAULT_TO_DISABLED # Set kern.racct.enable=0 by default
options         RCTL                    # Resource limits

# Debugging support.  Always need this:
options         KDB                     # Enable kernel debugger support.
options         KDB_TRACE               # Print a stack trace for a panic.

# Kernel dump features.
options         EKCD                    # Support for encrypted kernel dumps
options         GZIO                    # gzip-compressed kernel and user dumps
options         ZSTDIO                  # zstd-compressed kernel and user dumps
options         NETDUMP                 # netdump(4) client support

# Make an SMP-capable kernel by default
options         SMP                     # Symmetric MultiProcessor Kernel
options         EARLY_AP_STARTUP

# CPU frequency control
device          cpufreq

# Bus support.
device          acpi
options         ACPI_DMAR
device          pci
options         PCI_HP                  # PCI-Express native HotPlug
options         PCI_IOV                 # PCI SR-IOV support

# Floppy drives
device          fdc

# ATA controllers
device          ahci                    # AHCI-compatible SATA controllers
device          ata                     # Legacy ATA/SATA controllers
device          mvs                     # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA
device          siis                    # SiliconImage SiI3124/SiI3132/SiI3531 SATA

# SCSI Controllers
device          ahc                     # AHA2940 and onboard AIC7xxx devices
device          ahd                     # AHA39320/29320 and onboard AIC79xx devices
device          esp                     # AMD Am53C974 (Tekram DC-390(T))
device          hptiop                  # Highpoint RocketRaid 3xxx series
device          isp                     # Qlogic family
#device         ispfw                   # Firmware for QLogic HBAs- normally a module
device          mpt                     # LSI-Logic MPT-Fusion
device          mps                     # LSI-Logic MPT-Fusion 2
device          mpr                     # LSI-Logic MPT-Fusion 3
#device         ncr                     # NCR/Symbios Logic
device          sym                     # NCR/Symbios Logic (newer chipsets + those of `ncr')
device          trm                     # Tekram DC395U/UW/F DC315U adapters
device          isci                    # Intel C600 SAS controller
device          ocs_fc                  # Emulex FC adapters

# ATA/SCSI peripherals
device          scbus                   # SCSI bus (required for ATA/SCSI)
device          ch                      # SCSI media changers
device          da                      # Direct Access (disks)
device          sa                      # Sequential Access (tape etc)
device          cd                      # CD
device          pass                    # Passthrough device (direct ATA/SCSI access)
device          ses                     # Enclosure Services (SES and SAF-TE)
#device         ctl                     # CAM Target Layer

# RAID controllers interfaced to the SCSI subsystem
device          amr                     # AMI MegaRAID
device          arcmsr                  # Areca SATA II RAID
device          ciss                    # Compaq Smart RAID 5*
device          dpt                     # DPT Smartcache III, IV - See NOTES for options
device          hptmv                   # Highpoint RocketRAID 182x
device          hptnr                   # Highpoint DC7280, R750
device          hptrr                   # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx
device          hpt27xx                 # Highpoint RocketRAID 27xx
device          iir                     # Intel Integrated RAID
device          ips                     # IBM (Adaptec) ServeRAID
device          mly                     # Mylex AcceleRAID/eXtremeRAID
device          twa                     # 3ware 9000 series PATA/SATA RAID
device          smartpqi                # Microsemi smartpqi driver
device          tws                     # LSI 3ware 9750 SATA+SAS 6Gb/s RAID controller

# RAID controllers
device          aac                     # Adaptec FSA RAID
device          aacp                    # SCSI passthrough for aac (requires CAM)
device          aacraid                 # Adaptec by PMC RAID
device          ida                     # Compaq Smart RAID
device          mfi                     # LSI MegaRAID SAS
device          mlx                     # Mylex DAC960 family
device          mrsas                   # LSI/Avago MegaRAID SAS/SATA, 6Gb/s and 12Gb/s
device          pmspcv                  # PMC-Sierra SAS/SATA Controller driver
#XXX pointer/int warnings
#device         pst                     # Promise Supertrak SX6000
device          twe                     # 3ware ATA RAID

# NVM Express (NVMe) support
device          nvme                    # base NVMe driver
device          nvd                     # expose NVMe namespaces as disks, depends on nvme

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc                  # AT keyboard controller
device          atkbd                   # AT keyboard
device          psm                     # PS/2 mouse

device          kbdmux                  # keyboard multiplexer

device          vga                     # VGA video card driver
options         VESA                    # Add support for VESA BIOS Extensions (VBE)

device          splash                  # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device          sc
options         SC_PIXEL_MODE           # add support for the raster text mode

# vt is the new video console driver
device          vt
device          vt_vga
device          vt_efifb

device          agp                     # support several AGP chipsets

# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
device          cbb                     # cardbus (yenta) bridge
device          pccard                  # PC Card (16-bit) bus
device          cardbus                 # CardBus (32-bit) bus

# Serial (COM) ports
device          uart                    # Generic UART driver

# Parallel port
device          ppc
device          ppbus                   # Parallel port bus (required)
device          lpt                     # Printer
device          ppi                     # Parallel port interface device
#device         vpo                     # Requires scbus and da

device          puc                     # Multi I/O cards and multi-channel UARTs

# PCI Ethernet NICs.
device          bxe                     # Broadcom NetXtreme II BCM5771X/BCM578XX 10GbE
device          de                      # DEC/Intel DC21x4x (``Tulip'')
device          em                      # Intel PRO/1000 Gigabit Ethernet Family
device          ix                      # Intel PRO/10GbE PCIE PF Ethernet
device          ixv                     # Intel PRO/10GbE PCIE VF Ethernet
device          ixl                     # Intel 700 Series Physical Function
device          iavf                    # Intel Adaptive Virtual Function
device          le                      # AMD Am7900 LANCE and Am79C9xx PCnet
device          ti                      # Alteon Networks Tigon I/II gigabit Ethernet
device          txp                     # 3Com 3cR990 (``Typhoon'')
device          vx                      # 3Com 3c590, 3c595 (``Vortex'')

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device          miibus                  # MII bus support
device          ae                      # Attansic/Atheros L2 FastEthernet
device          age                     # Attansic/Atheros L1 Gigabit Ethernet
device          alc                     # Atheros AR8131/AR8132 Ethernet
device          ale                     # Atheros AR8121/AR8113/AR8114 Ethernet
device          bce                     # Broadcom BCM5706/BCM5708 Gigabit Ethernet
device          bfe                     # Broadcom BCM440x 10/100 Ethernet
device          bge                     # Broadcom BCM570xx Gigabit Ethernet
device          cas                     # Sun Cassini/Cassini+ and NS DP83065 Saturn
device          dc                      # DEC/Intel 21143 and various workalikes
device          et                      # Agere ET1310 10/100/Gigabit Ethernet
device          fxp                     # Intel EtherExpress PRO/100B (82557, 82558)
device          gem                     # Sun GEM/Sun ERI/Apple GMAC
device          hme                     # Sun HME (Happy Meal Ethernet)
device          jme                     # JMicron JMC250 Gigabit/JMC260 Fast Ethernet
device          lge                     # Level 1 LXT1001 gigabit Ethernet
device          msk                     # Marvell/SysKonnect Yukon II Gigabit Ethernet
device          nfe                     # nVidia nForce MCP on-board Ethernet
device          nge                     # NatSemi DP83820 gigabit Ethernet
device          pcn                     # AMD Am79C97x PCI 10/100 (precedence over 'le')
device          re                      # RealTek 8139C+/8169/8169S/8110S
device          rl                      # RealTek 8129/8139
device          sf                      # Adaptec AIC-6915 (``Starfire'')
device          sge                     # Silicon Integrated Systems SiS190/191
device          sis                     # Silicon Integrated Systems SiS 900/SiS 7016
device          sk                      # SysKonnect SK-984x & SK-982x gigabit Ethernet
device          ste                     # Sundance ST201 (D-Link DFE-550TX)
device          stge                    # Sundance/Tamarack TC9021 gigabit Ethernet
device          tl                      # Texas Instruments ThunderLAN
device          tx                      # SMC EtherPower II (83c170 ``EPIC'')
device          vge                     # VIA VT612x gigabit Ethernet
device          vr                      # VIA Rhine, Rhine II
device          wb                      # Winbond W89C840F
device          xl                      # 3Com 3c90x (``Boomerang'', ``Cyclone'')

# Wireless NIC cards
device          wlan                    # 802.11 support
options         IEEE80211_DEBUG         # enable debug msgs
options         IEEE80211_AMPDU_AGE     # age frames in AMPDU reorder q's
options         IEEE80211_SUPPORT_MESH  # enable 802.11s draft support
device          wlan_wep                # 802.11 WEP support
device          wlan_ccmp               # 802.11 CCMP support
device          wlan_tkip               # 802.11 TKIP support
device          wlan_amrr               # AMRR transmit rate control algorithm
device          an                      # Aironet 4500/4800 802.11 wireless NICs.
device          ath                     # Atheros NICs
device          ath_pci                 # Atheros pci/cardbus glue
device          ath_hal                 # pci/cardbus chip support
options         AH_SUPPORT_AR5416       # enable AR5416 tx/rx descriptors
options         AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation
options         ATH_ENABLE_11N          # Enable 802.11n support for AR5416 and later
device          ath_rate_sample         # SampleRate tx rate control for ath
#device         bwi                     # Broadcom BCM430x/BCM431x wireless NICs.
#device         bwn                     # Broadcom BCM43xx wireless NICs.
device          ipw                     # Intel 2100 wireless NICs.
device          iwi                     # Intel 2200BG/2225BG/2915ABG wireless NICs.
device          iwn                     # Intel 4965/1000/5000/6000 wireless NICs.
device          malo                    # Marvell Libertas wireless NICs.
device          mwl                     # Marvell 88W8363 802.11n wireless NICs.
device          ral                     # Ralink Technology RT2500 wireless NICs.
device          wi                      # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
device          wpi                     # Intel 3945ABG wireless NICs.

# Pseudo devices.
device          crypto                  # core crypto support
device          loop                    # Network loopback
device          random                  # Entropy device
device          padlock_rng             # VIA Padlock RNG
device          rdrand_rng              # Intel Bull Mountain RNG
device          ether                   # Ethernet support
device          vlan                    # 802.1Q VLAN support
device          tun                     # Packet tunnel.
device          md                      # Memory "disks"
device          gif                     # IPv6 and IPv4 tunneling
device          firmware                # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device          bpf                     # Berkeley packet filter

# USB support
options         USB_DEBUG               # enable debug msgs
device          uhci                    # UHCI PCI->USB interface
device          ohci                    # OHCI PCI->USB interface
device          ehci                    # EHCI PCI->USB interface (USB 2.0)
device          xhci                    # XHCI PCI->USB interface (USB 3.0)
device          usb                     # USB Bus (required)
device          ukbd                    # Keyboard
device          umass                   # Disks/Mass storage - Requires scbus and da

# Sound support
device          sound                   # Generic sound driver (required)
device          snd_cmi                 # CMedia CMI8338/CMI8738
device          snd_csa                 # Crystal Semiconductor CS461x/428x
device          snd_emu10kx             # Creative SoundBlaster Live! and Audigy
device          snd_es137x              # Ensoniq AudioPCI ES137x
device          snd_hda                 # Intel High Definition Audio
device          snd_ich                 # Intel, NVidia and other ICH AC'97 Audio
device          snd_via8233             # VIA VT8233x Audio

# MMC/SD
device          mmc                     # MMC/SD bus
device          mmcsd                   # MMC/SD memory card
device          sdhci                   # Generic PCI SD Host Controller

# VirtIO support
device          virtio                  # Generic VirtIO bus (required)
device          virtio_pci              # VirtIO PCI device
device          vtnet                   # VirtIO Ethernet device
device          virtio_blk              # VirtIO Block device
device          virtio_scsi             # VirtIO SCSI device
device          virtio_balloon          # VirtIO Memory Balloon device

# HyperV drivers and enhancement support
device          hyperv                  # HyperV drivers

# Xen HVM Guest Optimizations
# NOTE: XENHVM depends on xenpci.  They must be added or removed together.
options         XENHVM                  # Xen HVM kernel infrastructure
device          xenpci                  # Xen HVM Hypervisor services driver

# VMware support
device          vmx                     # VMware VMXNET3 Ethernet

# Netmap provides direct access to TX/RX rings on supported NICs
device          netmap                  # netmap(4) support

# evdev interface
options         EVDEV_SUPPORT           # evdev support in legacy drivers
device          evdev                   # input event device support
device          uinput                  # install /dev/uinput cdev

#CUSTOM KERNEL FOLLOWING...
options         NETGRAPH
options         NETGRAPH_PPP
options         NETGRAPH_PPTPGRE
options         NETGRAPH_ETHER
options         NETGRAPH_SOCKET
options         NETGRAPH_TEE
options         NETGRAPH_ASYNC
options         NETGRAPH_IFACE
options         NETGRAPH_MPPC_ENCRYPTION
options         NETGRAPH_MPPC_COMPRESSION
options         NETGRAPH_BPF
options         NETGRAPH_KSOCKET
options         NETGRAPH_TCPMSS
options         NETGRAPH_VJC
options         NETGRAPH_ONE2MANY
options         NETGRAPH_RFC1490
options         NETGRAPH_TTY
options         NETGRAPH_UI
options         LIBALIAS
options         MROUTING
options         NETGRAPH_PPPOE
options         NETGRAPH_HOLE
options         NETGRAPH_ECHO
options         NETGRAPH_L2TP

# By Executor (vlad.admin@mail.ru)
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=999
options IPFILTER
options IPFILTER_LOG
options IPDIVERT
options DUMMYNET
options DEVICE_POLLING
#options IPFIREWALL_FORWARD
options IPFIREWALL_NAT
options IPFIREWALL_DEFAULT_TO_ACCEPT

#colortag
options SC_NORM_ATTR="(FG_GREEN|BG_BLACK)"
options SC_NORM_REV_ATTR="(FG_YELLOW|BG_GREEN)"
options SC_KERNEL_CONS_REV_ATTR="(FG_BLACK|BG_RED)"
options SC_KERNEL_CONS_ATTR="(FG_RED|BG_BLACK)"

# For HTTP Server
maxusers 512

#

options HZ=1000

# PF support
device          pf
device          pflog
device          pfsync
options         ALTQ
options         ALTQ_CBQ
options         ALTQ_RED
options         ALTQ_RIO
options         ALTQ_HFSC
options         ALTQ_PRIQ
options         ALTQ_NOPCC
options         SHMMAXPGS=65536
options         SEMMNI=40
options         SEMMNS=240
options         SEMUME=40
options         SEMMNU=120


#options RADIX_MPATH
#options COMPAT_FREEBSD8 # Compatible with FreeBSD8

#22-08-2012 for ZFS
#options         KVA_PAGES=160

#03-10-2013
# IPSec
#options         IPSEC_FILTERTUNNEL
#options         IPSEC_NAT_T
options         IPSEC_DEBUG
device          enc

#19-11-2013
device          tap

#28-02-2014
options MAC_PORTACL

sysctl config:

# cat /etc/sysctl.conf
# $FreeBSD: stable/12/sbin/sysctl/sysctl.conf 337624 2018-08-11 13:28:03Z brd $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
net.inet6.ip6.v6only=0

kern.maxfiles=65536
kern.maxfilesperproc=32768
kern.ipc.somaxconn=32768
kern.ipc.shmmax=204800000
kern.ipc.shmall=409600
#kern.ipc.nmbclusters=65535
net.inet.ip.random_id=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.mssdflt=1500
#kern.kstack_pages=4
nen.inet.ip.portrange.reservedlow=0
net.inet.ip.portrange.reservedhigh=0
security.mac.portacl.port_high=1023
security.mac.portacl.suser_exempt=1
security.mac.portacl.rules=uid:53:tcp:53,uid:53:udp:53
#vfs.zfs.arc_max=2000000000
debug.debugger_on_panic=0

Comment 3 Andrey V. Elsukov 2019-02-12 14:21:14 UTC

KEYDBG() macro executed only when net.key.debug is set to non-zero value. It looks like your sysctl.conf didn't set it. Also, it looks impossible to get page fault with fault address 0x28 in this line of code. I suspect, that you have some sort of memory corruption. Not sure, is it hardware related or it is overwritten by some code.

(In reply to Andrey V. Elsukov from comment #3)

There is a mind that if turn off

options         IPSEC_DEBUG

kernel panic will disappear

Comment 5 Andrey V. Elsukov 2019-02-12 16:18:41 UTC

(In reply to Sergey Anokhin from comment #4)
> (In reply to Andrey V. Elsukov from comment #3)
> 
> There is a mind that if turn off
> 
> options         IPSEC_DEBUG
> 
> kernel panic will disappear

Disabling IPSEC_DEBUG also reduces the requirement to kernel stack size.

Can you try again with IPSEC_DEBUG and a doubled kernel stack size?

btw, perhaps it can be helpful: if port security/ipsec-tools was built with default options (make rmconfig), so the bug doesn't reproduced

(In reply to Jan Bramkamp from comment #6)

Did you mean try to set kern.maxssiz into /boot/loader.conf?

Comment 9 Andrey V. Elsukov 2019-02-12 20:25:31 UTC

Can you try to remove `option VIMAGE` from your kernel config? It looks like the problem is similar to the one described in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235699

(In reply to Jan Bramkamp from comment #6)

Will it ok?

(pts/1)[root@server:~]# sysctl kern.maxssiz=1073741824
kern.maxssiz: 536870912 -> 1073741824
(pts/1)[root@server:~]# /usr/local/etc/rc.d/racoon onestart
Starting racoon.
(pts/1)[root@server:~]# /usr/local/etc/rc.d/racoon onestop
Stopping racoon.
Waiting for PIDS: 5662

kernel panic

btw, I've noticed that kernel panic during stopping racoon.

# kgdb kernel /var/crash/vmcore.last
GNU gdb (GDB) 8.2.1 [GDB v8.2.1 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kernel...Reading symbols from /usr/obj/usr/src/amd64.amd64/sys/SERVER/kernel.debug...done.
done.

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address   = 0x28
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80ecd31d
stack pointer           = 0x28:0xfffffe003fca7a40
frame pointer           = 0x28:0xfffffe003fca7a60
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (softirq_2)
trap number             = 12
panic: page fault
cpuid = 2
time = 1550009599
KDB: stack backtrace:
#0 0xffffffff80c531c7 at kdb_backtrace+0x67
#1 0xffffffff80c07143 at vpanic+0x1a3
#2 0xffffffff80c06f93 at panic+0x43
#3 0xffffffff8118d9ff at trap_fatal+0x35f
#4 0xffffffff8118da59 at trap_pfault+0x49
#5 0xffffffff8118d07e at trap+0x29e
#6 0xffffffff81168ac5 at calltrap+0x8
#7 0xffffffff80eca240 at ipsec_delete_pcbpolicy+0x20
#8 0xffffffff80dbaeec at in_pcbfree_deferred+0x6c
#9 0xffffffff80c4db1a at epoch_call_task+0x1ca
#10 0xffffffff80c51a54 at gtaskqueue_run_locked+0x144
#11 0xffffffff80c516b8 at gtaskqueue_thread_loop+0x98
#12 0xffffffff80bc6f23 at fork_exit+0x83
#13 0xffffffff81169abe at fork_trampoline+0xe
Uptime: 8m33s
Dumping 950 out of 8077 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at ./machine/pcpu.h:230
230             __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));
(kgdb) bt
#0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80c06d2b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:446
#3  0xffffffff80c071a3 in vpanic (fmt=<optimized out>, ap=0xfffffe003fca7790) at /usr/src/sys/kern/kern_shutdown.c:872
#4  0xffffffff80c06f93 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:799
#5  0xffffffff8118d9ff in trap_fatal (frame=0xfffffe003fca7980, eva=40) at /usr/src/sys/amd64/amd64/trap.c:929
#6  0xffffffff8118da59 in trap_pfault (frame=0xfffffe003fca7980, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:765
#7  0xffffffff8118d07e in trap (frame=0xfffffe003fca7980) at /usr/src/sys/amd64/amd64/trap.c:441
#8  <signal handler called>
#9  0xffffffff80ecd31d in key_freesp (spp=0xfffff80211241880) at /usr/src/sys/netipsec/key.c:1199
#10 0xffffffff80eca240 in ipsec_delete_pcbpolicy (inp=0xfffff800151aa1e8) at /usr/src/sys/netipsec/ipsec_pcb.c:176
#11 0xffffffff80dbaeec in in_pcbfree_deferred (ctx=0xfffff800151aa3c0) at /usr/src/sys/netinet/in_pcb.c:1576
#12 0xffffffff80c4db1a in epoch_call_task (arg=<optimized out>) at /usr/src/sys/kern/subr_epoch.c:507
#13 0xffffffff80c51a54 in gtaskqueue_run_locked (queue=0xfffff80003363c00) at /usr/src/sys/kern/subr_gtaskqueue.c:376
#14 0xffffffff80c516b8 in gtaskqueue_thread_loop (arg=<optimized out>) at /usr/src/sys/kern/subr_gtaskqueue.c:557
#15 0xffffffff80bc6f23 in fork_exit (callout=0xffffffff80c51620 <gtaskqueue_thread_loop>, arg=0xfffffe00025f5038, frame=0xfffffe003fca7c00)
    at /usr/src/sys/kern/kern_fork.c:1059
#16 <signal handler called>
(kgdb) frame 9
#9  0xffffffff80ecd31d in key_freesp (spp=0xfffff80211241880) at /usr/src/sys/netipsec/key.c:1199
1199            KEYDBG(IPSEC_STAMP,
(kgdb)

Comment 11 Andrey V. Elsukov 2019-02-12 22:21:22 UTC

Created attachment 201968 [details]
Proposed patch

Also, you can test this patch instead, it should fix panic with VIMAGE option.
The problem is due to introduced deferred PCB destroying via epoch_call(). Since this code is executed from gtaskqueue, it has no VNET context.

(In reply to Andrey V. Elsukov from comment #9)

Sure, now I'm building kernel without VIMAGE. I'll let you know about testing result

(In reply to Andrey V. Elsukov from comment #11)

I'd preferred to try to rebuild kernel if it's no difference between turning off VIMAGE from kernel config and applying patch because kernel building more faster then "world" building. As far as I understand, you are propose patch for "world" component, right?

Comment 14 Andrey V. Elsukov 2019-02-12 22:45:09 UTC

(In reply to Sergey Anokhin from comment #13)
> (In reply to Andrey V. Elsukov from comment #11)
> 
> I'd preferred to try to rebuild kernel if it's no difference between turning
> off VIMAGE from kernel config and applying patch because kernel building
> more faster then "world" building. As far as I understand, you are propose
> patch for "world" component, right?

No, the patch is for kernel.

Comment 15 Rodney W. Grimes 2019-02-13 01:46:31 UTC

Please do not put bugs on stable@, current@, hackers@, etc

Comment 16 commit-hook 2019-02-13 15:47:08 UTC

A commit references this bug:

Author: ae
Date: Wed Feb 13 15:46:05 UTC 2019
New revision: 344103
URL: https://svnweb.freebsd.org/changeset/base/344103

Log:
  In r335015 PCB destroing was made deferred using epoch_call().

  But ipsec_delete_pcbpolicy() uses some VNET-virtualized variables,
  and thus it needs VNET context, that is missing during gtaskqueue
  executing. Use inp_vnet context to set curvnet in in_pcbfree_deferred().

  PR:		235684
  MFC after:	1 week

Changes:
  head/sys/netinet/in_pcb.c

(In reply to Andrey V. Elsukov from comment #14)

I've tested your patch. The bug disappeared.

Thanks.

Comment 18 Andrey V. Elsukov 2019-02-20 10:23:37 UTC

Fixed in head/ and stable/12. Thanks!

Comment 19 commit-hook 2019-02-20 10:23:42 UTC

A commit references this bug:

Author: ae
Date: Wed Feb 20 10:22:48 UTC 2019
New revision: 344356
URL: https://svnweb.freebsd.org/changeset/base/344356

Log:
  MFC r344103:
    In r335015 PCB destroing was made deferred using epoch_call().

    But ipsec_delete_pcbpolicy() uses some VNET-virtualized variables,
    and thus it needs VNET context, that is missing during gtaskqueue
    executing. Use inp_vnet context to set curvnet in in_pcbfree_deferred().

    PR:		235684

Changes:
_U  stable/12/
  stable/12/sys/netinet/in_pcb.c

Comment 20 Kubilay Kocak 2019-02-20 12:10:11 UTC

Correct classification