Bug 242626 - www/nginx High CPU usage, Nginx crashes, unable to restart daemon, only hard reset helps
Summary: www/nginx High CPU usage, Nginx crashes, unable to restart daemon, only hard ...
Status: Closed Feedback Timeout
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.1-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Jochen Neumeister
: 236961 (view as bug list)
Depends on:
Reported: 2019-12-13 19:34 UTC by Andris Vasers
Modified: 2020-04-03 13:45 UTC (History)
8 users (show)

See Also:
linimon: maintainer-feedback? (joneum)

top daemon with 100% nginx cpu usage (87.60 KB, image/png)
2019-12-13 19:34 UTC, Andris Vasers
no flags Details
top process list (136.97 KB, image/png)
2019-12-13 20:05 UTC, Andris Vasers
no flags Details
2 minute stuck after init 6 (102.34 KB, image/png)
2019-12-13 20:17 UTC, Andris Vasers
no flags Details
stuck on sync (102.34 KB, image/png)
2019-12-13 20:19 UTC, Andris Vasers
no flags Details
Problem still exists core process stuck on latest stable kernel (445.49 KB, image/png)
2019-12-14 12:03 UTC, Andris Vasers
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andris Vasers 2019-12-13 19:34:56 UTC
Created attachment 209918 [details]
top daemon with 100% nginx cpu usage

Hello. Using FreeBSD for many years as production environment.
Upgraded webserver to 12.1 release and many problems begun. This is worst BSD release ever I have used before starting 4.11. Very unstable.
WIth exact configuration using on freebsd 11.1 wanted to migrate to bsd 12.1 and now nginx stucks and I'm unable to kill the process only restart will help
It seems that nginx causes kernel / process panic. So after init 6 or reboot sometimes system halts un i need to hard restart whole system on virtual machine.

I, using VMware Vsphere 6.0 environment with open-vm-tools-nox11 installed
16GB of ram, 2 x 4 CPU cores.
em and vmx network interfaces.

tried to play with several nginx configurations - no success
logged out nginx for error logs - no error outputs, seems everything is fine

configured for 8 cores 8 vworker processes and after 8 or 10 hours few cores are in stuck on nginx till nginx daeemon crashes

Im unable to kill process, even kill x -9 wont help.

Tried build kernel with latest source without any success.

So thinking return back to 11.x release. I don't know why this release has been submitted. Next generation approvals i think gets more lazy to check out bugs.
i'm using following packages:

ImageMagick7-        Image processing tools
apr-                Apache Portability Library
asciidoc-8.6.10_1              Text document format for writing short documents and man pages
autoconf-2.69_3                Automatically configure source code on many Un*x platforms
autoconf-wrapper-20131203      Wrapper script for GNU autoconf
automake-1.16.1_2              GNU Standards-compliant Makefile generator
avahi-app-0.7_2                Service discovery on a local network
bash-5.0.11                    GNU Project's Bourne Again SHell
bdftopcf-1.1_2                 Convert X font from BDF to PCF
binutils-2.33.1,1              GNU binary tools
bison-3.4.2,1                  Parser generator from FSF, (mostly) compatible with Yacc
boehm-gc-8.0.4_1               Garbage collection and memory leak detection for C and C++
boost-jam-1.71.0               Build tool from the boost.org
boost-libs-1.71.0_2            Free portable C++ libraries (without Boost.Python)
brotli-1.0.7_2,1               Generic-purpose lossless compression algorithm
c-ares-1.15.0_1                Asynchronous DNS resolver library
ca_root_nss-3.47.1             Root certificate bundle from the Mozilla Project
cairo-1.16.0,2                 Vector graphics library with cross-device output support
check-0.13.0                   Unit test framework for C
cmake-3.15.5                   Cross-platform Makefile generator
cups-2.2.12                    Common UNIX Printing System
curl-7.67.0                    Command line tool and library for transferring data with URLs
cvsps-2.1_2                    Create patchset information from CVS
db5-5.3.28_7                   Oracle Berkeley DB, revision 5.3
dbus-1.12.16                   Message bus system for inter-application communication
dbus-glib-0.110                GLib bindings for the D-BUS messaging system
dejavu-2.37_1                  Bitstream Vera Fonts clone with a wider range of characters
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
docbook-1.5                    Meta-port for the different versions of the DocBook DTD
docbook-sgml-4.5_1             DocBook SGML DTD
docbook-xml-5.0_3              DocBook XML DTD
docbook-xsl-1.79.1_1,1         XSL DocBook stylesheets
encodings-1.0.5,1              X.Org Encoding fonts
expat-2.2.8                    XML 1.0 parser written in C
fftw3-3.3.8_5                  Fast C routines to compute the Discrete Fourier Transform
font-bh-ttf-1.0.3_4            X.Org Bigelow & Holmes TTF font
font-misc-ethiopic-1.0.3_4     X.Org miscellaneous Ethiopic font
font-misc-meltho-1.0.3_4       X.Org miscellaneous Meltho font
font-util-1.3.2                Create an index of X font files in a directory
fontconfig-2.12.6,1            XML-based font configuration API for X Windows
freetype2-2.10.1               Free and portable TrueType font rendering engine
fribidi-0.19.7                 Free Implementation of the Unicode Bidirectional Algorithm
fusefs-libs-2.9.9_1            FUSE allows filesystem implementation in userspace
gdbm-1.18.1_1                  GNU database manager
gdk-pixbuf2-2.38.1             Graphic library for GTK+
getopt-1.1.6                   Replacement for getopt(1) that supports GNU-style long options
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
ghostscript9-agpl-base-9.27_3  PostScript and PDF interpreter
ghostscript9-agpl-x11-9.27     PostScript and PDF interpreter, X11 support
giflib-5.2.1                   Tools and library routines for working with GIF images
git-2.24.0                     Distributed source code management tool
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
gmp-6.1.2_1                    Free library for arbitrary precision arithmetic
gnome_subr-1.0                 Common startup and shutdown subroutines used by GNOME scripts
gnutls-3.6.10                  GNU Transport Layer Security library
gobject-introspection-1.56.1,1 Generate interface introspection data for GObject libraries
gperf-3.0.3_2                  Generates perfect hash functions for sets of keywords
graphite2-1.3.13_1             Rendering capabilities for complex non-Roman writing systems
graphviz-2.42.2_3              Graph Visualization Software from AT&T and Bell Labs
groff-1.22.4_3                 Software typesetting package
gsfonts-8.11_8                 Standard Fonts for Ghostscript
gtk-doc-1.29                   Code documentation system, originally for GTK+
harfbuzz-2.6.4                 OpenType text shaping engine
help2man-1.47.11               Automatically generating simple manual pages from program output
highlight-3.43_7,3             Customizable syntax highlighter
html2text-1.3.2a               Converts HTML documents into plain text
icu-65.1,1                     International Components for Unicode (from IBM)
indexinfo-0.3.1                Utility to regenerate the GNU info page index
intltool-0.51.0_1              Tools to internationalize various kinds of data files
iso8879-1986_3                 Character entity sets from ISO 8879:1986 (SGML)
itstool-2.0.5_2                Make XML documents translatable through po files
jade-1.2.1_10                  Object-oriented SGML/XML parser toolkit and DSSSL engine
jbig2dec-0.17                  Decoder implementation of the JBIG2 image compression format
jbigkit-2.1_1                  Lossless compression for bi-level images such as scanned pages, faxes
jpeg-turbo-2.0.3               SIMD-accelerated JPEG codec which replaces libjpeg
jsoncpp-1.8.1_7                JSON reader and writer library for C++
lcms2-2.9                      Accurate, fast, and small-footprint color management engine
libICE-1.0.10,1                Inter Client Exchange library for X11
libSM-1.2.3,1                  Session Management library for X11
libX11-1.6.9,1                 X11 library
libXau-1.0.9                   Authentication Protocol library for X11
libXaw-1.0.13_3,2              X Athena Widgets library
libXdamage-1.1.5               X Damage extension library
libXdmcp-1.1.3                 X Display Manager Control Protocol library
libXext-1.3.4,1                X11 Extension library
libXfixes-5.0.3_2              X Fixes extension library
libXft-2.3.3                   Client-sided font API for X applications
libXmu-1.1.3,1                 X Miscellaneous Utilities libraries
libXpm-3.5.12_2                X Pixmap library
libXrender-0.9.10_2            X Render extension library
libXt-1.2.0,1                  X Toolkit library
libXxf86vm-1.1.4_3             X Vidmode Extension
libarchive-3.4.0,1             Library to create and read several streaming archive formats
libargon2-20190702             Memory hard password hashing program and library
libatomic_ops-7.6.10           Atomic operations access library
libcroco-0.6.12                CSS2 parsing library
libdaemon-0.14_1               Lightweight C library that eases the writing of UNIX daemons
libdnet-1.13_3                 Simple interface to low level networking routines
libdrm-2.4.99,1                Userspace interface to kernel Direct Rendering Module services
libedit-3.1.20190324,1         Command line editor library
libepoll-shim-0.0.20190311     epoll shim implemented using kevent
libffi-3.2.1_3                 Foreign Function Interface
libfontenc-1.1.4               The fontenc Library
libgcrypt-1.8.5                General purpose cryptographic library based on the code from GnuPG
libgd-2.2.5_2,1                Graphics library for fast creation of images
libgpg-error-1.36              Common error values for all GnuPG components
libgsf-1.14.44                 Extensible I/O abstraction for dealing with structured file formats
libiconv-1.14_11               Character set conversion library
libidn-1.35                    Internationalized Domain Names command line tool
libidn2-2.3.0                  Implementation of IDNA2008 internationalized domain names
liblqr-1-0.4.2                 Easy to use C/C++ seam carving library
libltdl-2.4.6                  System independent dlopen wrapper
liblz4-1.9.2,1                 LZ4 compression library, lossless and very fast
libmspack-0.9.1                Library for Microsoft compression formats
libnghttp2-1.40.0              HTTP/2.0 C Library
libpaper-              Library providing routines for paper size management
libpciaccess-0.16              Generic PCI access library
libpthread-stubs-0.4           This library provides weak aliases for pthread functions
libraqm-0.6.0                  Library that encapsulates complex text layout logic
libraw-0.19.2_1                Library for manipulating raw images
librsvg2-2.40.20               Library for parsing and rendering SVG vector-graphic files
libsigsegv-2.12                Handling page faults in user mode
libslang2-2.3.2_1              Routines for rapid alpha-numeric terminal applications development
libssh2-1.8.2,3                Library implementing the SSH2 protocol
libtasn1-4.15.0                ASN.1 structure parser library
libtextstyle-0.20.1            Text styling library
libtool-2.4.6_1                Generic shared library support script
libunistring-0.9.10_1          Unicode string library
libunwind-20170615             Generic stack unwinding library
libuv-1.34.0                   Multi-platform support library with a focus on asynchronous I/O
libwmf-              Tools and library for converting Microsoft WMF (windows metafile)
libxcb-1.13.1                  The X protocol C-language Binding (XCB) library
libxml2-2.9.10                 XML parser library for GNOME
libxshmfence-1.3               Shared memory 'SyncFence' synchronization primitive
libxslt-1.1.34                 The XSLT C library for GNOME
libzip-1.5.2                   C library for reading, creating, and modifying ZIP archives
llvm80-8.0.1_3                 LLVM and Clang
lua52-5.2.4                    Small, compilable scripting language providing easy access to C code
lzo2-2.10_1                    Portable speedy, lossless data compression library
m4-1.4.18_1,1                  GNU M4
mc-4.8.22_2                    Midnight Commander, a free Norton Commander Clone
mesa-libs-18.3.2_3             OpenGL libraries that support GLX and EGL clients
meson-0.52.0                   High performance build system
minixmlto-0.0.2_1              Minimalistic alternative to xmlto
mkfontscale-1.2.1              Creates an index of scalable font files for X
mpfr-4.0.2                     Library for multiple-precision floating-point computations
nasm-2.14.02,1                 General-purpose multi-platform x86 and amd64 assembler
nettle-3.5.1_1                 Low-level cryptographic library
nginx-devel-1.17.6_1           Robust and small WWW server
ninja-1.9.0,2                  Ninja is a small build system closest in spirit to Make
node10-10.17.0                 V8 JavaScript for client and server
npm-6.12.1                     Node package manager
oniguruma-6.9.3                Regular expressions library compatible with POSIX/GNU/Perl
open-vm-tools-nox11-11.0.1_2,2 Open VMware tools for FreeBSD VMware guests
openjpeg-2.3.1                 Open-source JPEG 2000 codec
p11-kit-              Library for loading and enumerating of PKCS#11 modules
p5-Authen-SASL-2.16_1          Perl5 module for SASL authentication
p5-CGI-4.44                    Handle Common Gateway Interface requests and responses
p5-Digest-HMAC-1.03_1          Perl5 interface to HMAC Message-Digest Algorithms
p5-Error-0.17028               Error/exception handling in object-oriented programming style
p5-GSSAPI-0.28_1               Perl extension providing access to the GSSAPIv2 library
p5-HTML-Parser-3.72            Perl5 module for parsing HTML documents
p5-HTML-Tagset-3.20_1          Some useful data table in parsing HTML
p5-IO-Socket-INET6-2.72_1      Perl module with object interface to AF_INET6 domain sockets
p5-IO-Socket-SSL-2.066         Perl5 interface to SSL sockets
p5-Locale-gettext-1.07         Message handling functions
p5-Locale-libintl-1.31         Internationalization library for Perl
p5-Mozilla-CA-20180117         Perl extension for Mozilla CA cert bundle in PEM format
p5-Net-SSLeay-1.85             Perl5 interface to SSL
p5-Socket6-0.29                IPv6 related part of the C socket.h defines and structure manipulators
p5-Term-ReadKey-2.38_1         Perl5 module for simple terminal control
p5-Text-Unidecode-1.30         Text::Unidecode -- US-ASCII transliterations of Unicode text
p5-Unicode-EastAsianWidth-12.0 East Asian Width properties
p5-XML-Parser-2.44             Perl extension interface to James Clark's XML parser, expat
p5-subversion-1.13.0           Perl bindings for Version control system
pango-1.42.4_3                 Open-source framework for the layout and rendering of i18n text
pciids-20191012                Database of all known IDs used in PCI devices
pcre-8.43_2                    Perl Compatible Regular Expressions library
pcre2-10.33                    Perl Compatible Regular Expressions library, version 2
pdflib-7.0.5_5                 C library for dynamically generating PDF
perl5-5.30.1                   Practical Extraction and Report Language
php73-7.3.12                   PHP Scripting Language
php73-bcmath-7.3.12            The bcmath shared extension for php
php73-bz2-7.3.12               The bz2 shared extension for php
php73-calendar-7.3.12          The calendar shared extension for php
php73-composer-1.9.0           Dependency Manager for PHP
php73-ctype-7.3.12             The ctype shared extension for php
php73-curl-7.3.12              The curl shared extension for php
php73-dom-7.3.12               The dom shared extension for php
php73-exif-7.3.12              The exif shared extension for php
php73-extensions-1.0           "meta-port" to install PHP extensions
php73-fileinfo-7.3.12          The fileinfo shared extension for php
php73-filter-7.3.12            The filter shared extension for php
php73-ftp-7.3.12               The ftp shared extension for php
php73-gd-7.3.12                The gd shared extension for php
php73-gettext-7.3.12           The gettext shared extension for php
php73-hash-7.3.12              The hash shared extension for php
php73-iconv-7.3.12             The iconv shared extension for php
php73-intl-7.3.12              The intl shared extension for php
php73-json-7.3.12              The json shared extension for php
php73-mbstring-7.3.12          The mbstring shared extension for php
php73-opcache-7.3.12           The opcache shared extension for php
php73-openssl-7.3.12           The openssl shared extension for php
php73-pdo-7.3.12               The pdo shared extension for php
php73-pdo_pgsql-7.3.12         The pdo_pgsql shared extension for php
php73-pdo_sqlite-7.3.12        The pdo_sqlite shared extension for php
php73-pecl-imagick-im7-3.4.4   PHP wrapper to the ImageMagick/GraphicsMagick library version 7
php73-pecl-pdflib-4.1.3        PECL extension to create PDF on the fly
php73-pecl-redis-5.1.1         Extension to access Redis
php73-pgsql-7.3.12             The pgsql shared extension for php
php73-phar-7.3.12              The phar shared extension for php
php73-posix-7.3.12             The posix shared extension for php
php73-readline-7.3.12          The readline shared extension for php
php73-session-7.3.12           The session shared extension for php
php73-simplexml-7.3.12         The simplexml shared extension for php
php73-soap-7.3.12              The soap shared extension for php
php73-sockets-7.3.12           The sockets shared extension for php
php73-sqlite3-7.3.12           The sqlite3 shared extension for php
php73-tidy-7.3.12              The tidy shared extension for php
php73-tokenizer-7.3.12         The tokenizer shared extension for php
php73-xml-7.3.12               The xml shared extension for php
php73-xmlreader-7.3.12         The xmlreader shared extension for php
php73-xmlrpc-7.3.12            The xmlrpc shared extension for php
php73-xmlwriter-7.3.12         The xmlwriter shared extension for php
php73-xsl-7.3.12               The xsl shared extension for php
php73-zip-7.3.12               The zip shared extension for php
php73-zlib-7.3.12              The zlib shared extension for php
pixman-0.38.4                  Low-level pixel manipulation library
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
png-1.6.37                     Library for manipulating PNG images
poppler-data-0.4.9_1           Poppler encoding data
postgresql11-client-11.6       PostgreSQL database (client)
psutils-1.17_5                 Utilities for manipulating PostScript documents
py27-setuptools-41.4.0         Python packages installer
py36-Babel-2.7.0               Collection of tools for internationalizing Python applications
py36-CommonMark-0.9.1          Python parser for the CommonMark Markdown spec
py36-Jinja2-2.10.1             Fast and easy to use stand-alone template engine
py36-MarkupSafe-1.1.1          Implements XML/HTML/XHTML Markup safe string for Python
py36-alabaster-0.7.6           Modified Kr Sphinx theme
py36-asn1crypto-0.24.0         ASN.1 library with a focus on performance and a pythonic API
py36-certifi-2019.9.11         Mozilla SSL certificates
py36-cffi-1.13.2               Foreign Function Interface for Python calling C code
py36-chardet-3.0.4_1           Universal encoding detector for Python 2 and 3
py36-cryptography-2.6.1        Cryptographic recipes and primitives for Python developers
py36-cython-0.29.13            Compiler for Writing C Extensions for the Python Language
py36-docutils-0.15.2           Python Documentation Utilities
py36-future-0.18.2             Clean single-source support for Python 3 and 2
py36-idna-2.8                  Internationalized Domain Names in Applications (IDNA)
py36-imagesize-1.1.0           Python image size library
py36-libxml2-2.9.10            Python interface for XML parser library for GNOME
py36-openssl-19.0.0            Python interface to the OpenSSL library
py36-pip-19.1.1                Tool for installing and managing Python packages
py36-pycparser-2.19            C parser in Python
py36-pygments-2.4.1            Syntax highlighter written in Python
py36-pypa-docs-theme-g20180622 Common base Sphinx theme for PyPA projects
py36-pysocks-1.7.1             Python SOCKS module
py36-pystemmer-1.3.0_2         Snowball Stemming Algorithms for Information Retrieval
py36-pytest-runner-2.11.1      Test support for pytest runner in setup.py
py36-python-docs-theme-2018.2  Sphinx theme for the CPython docs and related projects
py36-pytz-2019.3,1             World Timezone Definitions for Python
py36-recommonmark-0.5.0_1      CommonMark bridge for docutils and Sphinx
py36-requests-2.22.0           HTTP library written in Python for human beings
py36-setuptools-41.4.0         Python packages installer
py36-setuptools_scm-3.3.3      Setuptools plugin to manage your versions by scm tags
py36-six-1.12.0                Python 2 and 3 compatibility utilities
py36-snowballstemmer-1.2.1     Snowball stemming library collection for Python
py36-sphinx-1.6.5_2,1          Python documentation generator
py36-sphinx_rtd_theme-0.4.3    Mobile-friendly py-sphinx theme
py36-sphinxcontrib-websupport-1.1.2 Sphinx API for Web Apps
py36-sqlite3-3.6.9_7           Standard Python binding to the SQLite3 library (Python 3.6)
py36-urllib3-1.25.6,1          HTTP library with thread-safe connection pooling, file post, and more
python27-2.7.17_1              Interpreted object-oriented programming language
python36-3.6.9_1               Interpreted object-oriented programming language
rarian-0.8.1_4                 OMF help system based on the Freedesktop specification
readline-8.0.0                 Library for editing command lines as they are typed
rhash-1.3.5                    Utility and library for computing and checking of file hashes
scons-3.1.1                    Build tool alternative to make
sdocbook-xml-1.1_2,2           "Simplified" DocBook XML DTD
serf-1.3.9_4                   Serf HTTP client library
shared-mime-info-1.10_2        MIME types database from the freedesktop.org project
sqlite3-3.30.1                 SQL database engine in a C library
subversion-1.13.0              Version control system
sudo-1.8.29                    Allow others to run commands as root
swig30-3.0.12                  Generate wrappers for calling C/C++ code from other languages
tcl86-8.6.10                   Tool Command Language
texinfo-6.7_1,1                Typeset documentation system with multiple format output
tidy-lib-090315.c_3            Utility to clean up and pretty print HTML/XHTML/XML (with shared lib)
tiff-4.1.0                     Tools and library routines for working with TIFF images
tpm-emulator-0.7.4_2           Trusted Platform Module (TPM) emulator
trousers-0.3.14_2              Open-source TCG Software Stack
uchardet-0.0.6                 Universal charset detection library
utf8proc-2.4.0                 UTF-8 processing library
vala-0.40.15,1                 Programming language and compiler that converts Vala code into C code
w3m-             Pager/text-based WWW browser
wayland-1.16.0_1               Wayland composite "server"
wayland-protocols-1.17         Wayland protocols
webp-1.0.3_1                   Google WebP image format conversion tool
wget-1.20.3                    Retrieve files from the Net via HTTP(S) and FTP
xcb-proto-1.13_1               The X protocol C-language Binding (XCB) protocol
xmlcatmgr-2.2_2                SGML and XML catalog manager
xmlcharent-0.3_2               XML character entities
xmlto-0.0.28                   Front-end to an XSL toolchain
xorg-fonts-truetype-7.7_1      X.Org TrueType fonts
xorg-macros-1.19.2             X.Org development aclocal macros
xorgproto-2019.2               xorg protocol headers
xtrans-1.4.0                   Abstract network code for X
yarn-1.19.1                    Package manager for node, alternative to npm

FreeBSD web1.vilnis.dc.local 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

tried also to upgrade to latest nginx-devel-1.17.6_1, but the same shit.

I'm against to use 12.1 RELEASE in production even it classifies as STABLE, it' s unstable.

Lot of errors using in VM in dmesg:
Comment 1 Mark Johnston freebsd_committer 2019-12-13 19:37:16 UTC
Can you grab procstat -kk output from some of the stuck processes?
Comment 2 Andris Vasers 2019-12-13 19:40:11 UTC
  PID    TID COMM                TDNAME              KSTACK
 1106 100280 nginx               -                   vfs_bio_getpages ncl_getpages VOP_GETPAGES_APV vop_stdgetpages_async VOP_GETPAGES_ASYNC_APV vnode_pager_getpages_async vn_sendfile sendfile amd64_syscall fast_syscall_common
Comment 3 Andris Vasers 2019-12-13 19:48:04 UTC
PID    TID COMM                TDNAME              KSTACK
 1106 100280 nginx               -                   vfs_bio_getpages+0x1d9 ncl_getpages+0x2be VOP_GETPAGES_APV+0x7c vop_stdgetpages_async+0x49 VOP_GETPAGES_ASYNC_APV+0x7c vnode_pager_getpages_async+0x7e vn_sendfile+0xd9c sendfile+0x12b amd64_syscall+0x364 fast_syscall_common+0x101
Comment 4 Andris Vasers 2019-12-13 20:03:53 UTC
Now got 2 cores on stuck:
root@web1:~ # procstat -kk 1106
  PID    TID COMM                TDNAME              KSTACK                     
 1106 100280 nginx               -                   vfs_bio_getpages+0x1d9 ncl_getpages+0x2be VOP_GETPAGES_APV+0x7c vop_stdgetpages_async+0x49 VOP_GETPAGES_ASYNC_APV+0x7c vnode_pager_getpages_async+0x7e vn_sendfile+0xd9c sendfile+0x12b amd64_syscall+0x364 fast_syscall_common+0x101
root@web1:~ # procstat -kk 1104
  PID    TID COMM                TDNAME              KSTACK                     
 1104 100136 nginx               -                   vfs_bio_getpages+0x1d9 ncl_getpages+0x2be VOP_GETPAGES_APV+0x7c vop_stdgetpages_async+0x49 VOP_GETPAGES_ASYNC_APV+0x7c vnode_pager_getpages_async+0x7e vn_sendfile+0xd9c sendfile+0x12b amd64_syscall+0x364 fast_syscall_common+0x101
Comment 5 Andris Vasers 2019-12-13 20:05:04 UTC
Created attachment 209920 [details]
top process list
Comment 6 Andris Vasers 2019-12-13 20:17:29 UTC
Created attachment 209923 [details]
2 minute stuck after init 6
Comment 7 Andris Vasers 2019-12-13 20:19:16 UTC
Created attachment 209924 [details]
stuck on sync

process still in stuck after 5 minutes need to reboot server manually
Comment 8 Andris Vasers 2019-12-13 20:20:20 UTC
Sorry, forgot to post vmstat list
Comment 9 Mark Johnston freebsd_committer 2019-12-13 20:22:39 UTC
As a temporary workaround you can try setting vfs.nfs.use_buf_pager=0, but vfs_bio_getpages() has been the default for some time now.

Are the files that are being served also being modified or truncated?
Comment 10 Andris Vasers 2019-12-13 20:27:11 UTC
seems that files are ok
this server is frontend part, probably files are not saved just log writes in file way.
Comment 11 Konstantin Belousov freebsd_committer 2019-12-13 22:05:51 UTC
Try installing latest stable/12 kernel (no need to rebuild world).  If there is indeed a truncation in parallel with the mapping operation, then I have an expectation that it is fixed.
Comment 12 Andris Vasers 2019-12-13 23:12:07 UTC
ok, new kernel has built, so I'll check and report any issues if occured
12.1-STABLE FreeBSD 12.1-STABLE r355737 amd64
Comment 13 Andris Vasers 2019-12-14 12:03:22 UTC
Created attachment 209933 [details]
Problem still exists core process stuck on latest stable kernel
Comment 14 Andris Vasers 2019-12-14 12:05:50 UTC
No success, nginx still eating cpu time.
Process stuck and onky hard whole vm reset helped to restart nginx.
Any other ideas?
Comment 15 Jochen Neumeister freebsd_committer 2019-12-14 12:32:00 UTC
It could be the same problem as here: 
Comment 16 Konstantin Belousov freebsd_committer 2019-12-14 17:22:20 UTC
(In reply to Andris Vasers from comment #14)
I suppose you either build kernel with debug symbols and ddb, or can rebuild it.
After that, please dump core by entering into ddb:
# sysctl debug.kdb.enter=1
db> dump
.... output ...
db> c
after that run savecore(8) and provide me with kernel.full and vmcore.
Also tell me the pid of the stuck nginx.

Do not put the core file onto a public location, it might contain sensitive info.
Comment 17 Ed Maste freebsd_committer 2019-12-14 18:01:58 UTC
(In reply to Jochen Neumeister from comment #15)
> It could be the same problem as here: 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242626

It seems you pasted this bug in your comment
Comment 18 Jochen Neumeister freebsd_committer 2019-12-16 08:54:22 UTC
(In reply to Ed Maste from comment #17)

hups, i mean https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235296
Comment 19 Andris Vasers 2019-12-16 23:15:15 UTC
(In reply to Konstantin Belousov from comment #16)
Thanks Konstantin, I'll post you and attach link to data later.
Now mentioned that there's no dependency in which state is nginx, CPUX or nfs, it will stuck unexpectedly. I have tried playing also with sysctl values, but still no success. Turning off nfs pages only helps if last state was stuck on nfs.
I don't think that 11.x compatibility enabled kernel will crash with the same config as it was on 11.x RELEASE. Third party Binaries were build from latest sources. Tried different versions of nginx, even configuring and building without ports manually. Still the same.
Comment 20 Andris Vasers 2019-12-21 18:46:49 UTC
Now only workaround suggested by Konstantin is disabling vfs.nfs.use_buf_pager=0 works for me.
On 11.1 RELEASE it's disabled by default, but now on 12.1 - enabled.
Comment 21 Konstantin Belousov freebsd_committer 2020-01-04 05:50:14 UTC
(In reply to Andris Vasers from comment #19)
So are you going to provide the requested kernel.full and vmcore files ?
Comment 22 Julien Cigar 2020-03-30 12:48:54 UTC
Andris, check https://reviews.freebsd.org/D24038
Comment 23 Christos Chatzaras 2020-03-30 13:07:30 UTC
I am not sure if it's the same issue, but with sendfile enabled in nginx.conf I had the nginx process in D-state. I could not unmount the filesystem and the only option was a hard reset. I believe the issue caused by files that frequently change, for example cache files.
Comment 24 Andris Vasers 2020-03-30 13:30:14 UTC
(In reply to Christos Chatzaras from comment #23)
Hello, I have found what kind of problem causes this.
FreeBSD 12.1 has buggy VMware vmx (vmxnet3) driver, i changed it to em and everything works smoothly. If you have same issues, consider changing virtual network adapter, it will help!

Comment 25 Christos Chatzaras 2020-03-30 13:31:51 UTC
It's a bare metal server, no virtualisation.
Comment 26 Andris Vasers 2020-03-30 13:34:07 UTC
ok, then have you tried suggested option by turning off pages? What kind of network adapter you use?
Maybe buggy network subsystem not only adapter drivers...

Comment 27 Christos Chatzaras 2020-03-30 13:49:47 UTC
I have em.

The workaround for me was to disable sendfile in Nginx.
Comment 28 Andris Vasers 2020-03-30 13:56:22 UTC
(In reply to Christos Chatzaras from comment #27)
console# echo "vfs.nfs.use_buf_pager=0" >> /etc/sysctl.conf
reload everything, enable sendfile again and check.
Comment 29 Andris Vasers 2020-03-30 13:58:51 UTC
(In reply to Andris Vasers from comment #28)
Sorry, forgot to ask, do you use nfs mounts?
Comment 30 Christos Chatzaras 2020-03-30 13:59:54 UTC
No just normal UFS2 filesystem.
Comment 31 Mark Johnston freebsd_committer 2020-04-03 13:44:03 UTC
*** Bug 236961 has been marked as a duplicate of this bug. ***
Comment 32 Mark Johnston freebsd_committer 2020-04-03 13:45:19 UTC
The original issue may be fixed by r359464.