Created attachment 209918 [details] top daemon with 100% nginx cpu usage Hello. Using FreeBSD for many years as production environment. Upgraded webserver to 12.1 release and many problems begun. This is worst BSD release ever I have used before starting 4.11. Very unstable. WIth exact configuration using on freebsd 11.1 wanted to migrate to bsd 12.1 and now nginx stucks and I'm unable to kill the process only restart will help It seems that nginx causes kernel / process panic. So after init 6 or reboot sometimes system halts un i need to hard restart whole system on virtual machine. I, using VMware Vsphere 6.0 environment with open-vm-tools-nox11 installed 16GB of ram, 2 x 4 CPU cores. em and vmx network interfaces. tried to play with several nginx configurations - no success logged out nginx for error logs - no error outputs, seems everything is fine configured for 8 cores 8 vworker processes and after 8 or 10 hours few cores are in stuck on nginx till nginx daeemon crashes Im unable to kill process, even kill x -9 wont help. Tried build kernel with latest source without any success. So thinking return back to 11.x release. I don't know why this release has been submitted. Next generation approvals i think gets more lazy to check out bugs. i'm using following packages: ImageMagick7-7.0.8.57_1 Image processing tools apr-1.7.0.1.6.1 Apache Portability Library asciidoc-8.6.10_1 Text document format for writing short documents and man pages autoconf-2.69_3 Automatically configure source code on many Un*x platforms autoconf-wrapper-20131203 Wrapper script for GNU autoconf automake-1.16.1_2 GNU Standards-compliant Makefile generator avahi-app-0.7_2 Service discovery on a local network bash-5.0.11 GNU Project's Bourne Again SHell bdftopcf-1.1_2 Convert X font from BDF to PCF binutils-2.33.1,1 GNU binary tools bison-3.4.2,1 Parser generator from FSF, (mostly) compatible with Yacc boehm-gc-8.0.4_1 Garbage collection and memory leak detection for C and C++ boost-jam-1.71.0 Build tool from the boost.org boost-libs-1.71.0_2 Free portable C++ libraries (without Boost.Python) brotli-1.0.7_2,1 Generic-purpose lossless compression algorithm c-ares-1.15.0_1 Asynchronous DNS resolver library ca_root_nss-3.47.1 Root certificate bundle from the Mozilla Project cairo-1.16.0,2 Vector graphics library with cross-device output support check-0.13.0 Unit test framework for C cmake-3.15.5 Cross-platform Makefile generator cups-2.2.12 Common UNIX Printing System curl-7.67.0 Command line tool and library for transferring data with URLs cvsps-2.1_2 Create patchset information from CVS db5-5.3.28_7 Oracle Berkeley DB, revision 5.3 dbus-1.12.16 Message bus system for inter-application communication dbus-glib-0.110 GLib bindings for the D-BUS messaging system dejavu-2.37_1 Bitstream Vera Fonts clone with a wider range of characters dialog4ports-0.1.6 Console Interface to configure ports diffutils-3.7 GNU differential compare utilities docbook-1.5 Meta-port for the different versions of the DocBook DTD docbook-sgml-4.5_1 DocBook SGML DTD docbook-xml-5.0_3 DocBook XML DTD docbook-xsl-1.79.1_1,1 XSL DocBook stylesheets encodings-1.0.5,1 X.Org Encoding fonts expat-2.2.8 XML 1.0 parser written in C fftw3-3.3.8_5 Fast C routines to compute the Discrete Fourier Transform font-bh-ttf-1.0.3_4 X.Org Bigelow & Holmes TTF font font-misc-ethiopic-1.0.3_4 X.Org miscellaneous Ethiopic font font-misc-meltho-1.0.3_4 X.Org miscellaneous Meltho font font-util-1.3.2 Create an index of X font files in a directory fontconfig-2.12.6,1 XML-based font configuration API for X Windows freetype2-2.10.1 Free and portable TrueType font rendering engine fribidi-0.19.7 Free Implementation of the Unicode Bidirectional Algorithm fusefs-libs-2.9.9_1 FUSE allows filesystem implementation in userspace gdbm-1.18.1_1 GNU database manager gdk-pixbuf2-2.38.1 Graphic library for GTK+ getopt-1.1.6 Replacement for getopt(1) that supports GNU-style long options gettext-runtime-0.20.1 GNU gettext runtime libraries and programs gettext-tools-0.20.1_1 GNU gettext development and translation tools ghostscript9-agpl-base-9.27_3 PostScript and PDF interpreter ghostscript9-agpl-x11-9.27 PostScript and PDF interpreter, X11 support giflib-5.2.1 Tools and library routines for working with GIF images git-2.24.0 Distributed source code management tool glib-2.56.3_6,1 Some useful routines of C programming (current stable version) gmake-4.2.1_3 GNU version of 'make' utility gmp-6.1.2_1 Free library for arbitrary precision arithmetic gnome_subr-1.0 Common startup and shutdown subroutines used by GNOME scripts gnutls-3.6.10 GNU Transport Layer Security library gobject-introspection-1.56.1,1 Generate interface introspection data for GObject libraries gperf-3.0.3_2 Generates perfect hash functions for sets of keywords graphite2-1.3.13_1 Rendering capabilities for complex non-Roman writing systems graphviz-2.42.2_3 Graph Visualization Software from AT&T and Bell Labs groff-1.22.4_3 Software typesetting package gsfonts-8.11_8 Standard Fonts for Ghostscript gtk-doc-1.29 Code documentation system, originally for GTK+ harfbuzz-2.6.4 OpenType text shaping engine help2man-1.47.11 Automatically generating simple manual pages from program output highlight-3.43_7,3 Customizable syntax highlighter html2text-1.3.2a Converts HTML documents into plain text icu-65.1,1 International Components for Unicode (from IBM) indexinfo-0.3.1 Utility to regenerate the GNU info page index intltool-0.51.0_1 Tools to internationalize various kinds of data files iso8879-1986_3 Character entity sets from ISO 8879:1986 (SGML) itstool-2.0.5_2 Make XML documents translatable through po files jade-1.2.1_10 Object-oriented SGML/XML parser toolkit and DSSSL engine jbig2dec-0.17 Decoder implementation of the JBIG2 image compression format jbigkit-2.1_1 Lossless compression for bi-level images such as scanned pages, faxes jpeg-turbo-2.0.3 SIMD-accelerated JPEG codec which replaces libjpeg jsoncpp-1.8.1_7 JSON reader and writer library for C++ lcms2-2.9 Accurate, fast, and small-footprint color management engine libICE-1.0.10,1 Inter Client Exchange library for X11 libSM-1.2.3,1 Session Management library for X11 libX11-1.6.9,1 X11 library libXau-1.0.9 Authentication Protocol library for X11 libXaw-1.0.13_3,2 X Athena Widgets library libXdamage-1.1.5 X Damage extension library libXdmcp-1.1.3 X Display Manager Control Protocol library libXext-1.3.4,1 X11 Extension library libXfixes-5.0.3_2 X Fixes extension library libXft-2.3.3 Client-sided font API for X applications libXmu-1.1.3,1 X Miscellaneous Utilities libraries libXpm-3.5.12_2 X Pixmap library libXrender-0.9.10_2 X Render extension library libXt-1.2.0,1 X Toolkit library libXxf86vm-1.1.4_3 X Vidmode Extension libarchive-3.4.0,1 Library to create and read several streaming archive formats libargon2-20190702 Memory hard password hashing program and library libatomic_ops-7.6.10 Atomic operations access library libcroco-0.6.12 CSS2 parsing library libdaemon-0.14_1 Lightweight C library that eases the writing of UNIX daemons libdnet-1.13_3 Simple interface to low level networking routines libdrm-2.4.99,1 Userspace interface to kernel Direct Rendering Module services libedit-3.1.20190324,1 Command line editor library libepoll-shim-0.0.20190311 epoll shim implemented using kevent libffi-3.2.1_3 Foreign Function Interface libfontenc-1.1.4 The fontenc Library libgcrypt-1.8.5 General purpose cryptographic library based on the code from GnuPG libgd-2.2.5_2,1 Graphics library for fast creation of images libgpg-error-1.36 Common error values for all GnuPG components libgsf-1.14.44 Extensible I/O abstraction for dealing with structured file formats libiconv-1.14_11 Character set conversion library libidn-1.35 Internationalized Domain Names command line tool libidn2-2.3.0 Implementation of IDNA2008 internationalized domain names liblqr-1-0.4.2 Easy to use C/C++ seam carving library libltdl-2.4.6 System independent dlopen wrapper liblz4-1.9.2,1 LZ4 compression library, lossless and very fast libmspack-0.9.1 Library for Microsoft compression formats libnghttp2-1.40.0 HTTP/2.0 C Library libpaper-1.1.24.4 Library providing routines for paper size management libpciaccess-0.16 Generic PCI access library libpthread-stubs-0.4 This library provides weak aliases for pthread functions libraqm-0.6.0 Library that encapsulates complex text layout logic libraw-0.19.2_1 Library for manipulating raw images librsvg2-2.40.20 Library for parsing and rendering SVG vector-graphic files libsigsegv-2.12 Handling page faults in user mode libslang2-2.3.2_1 Routines for rapid alpha-numeric terminal applications development libssh2-1.8.2,3 Library implementing the SSH2 protocol libtasn1-4.15.0 ASN.1 structure parser library libtextstyle-0.20.1 Text styling library libtool-2.4.6_1 Generic shared library support script libunistring-0.9.10_1 Unicode string library libunwind-20170615 Generic stack unwinding library libuv-1.34.0 Multi-platform support library with a focus on asynchronous I/O libwmf-0.2.8.4_15 Tools and library for converting Microsoft WMF (windows metafile) libxcb-1.13.1 The X protocol C-language Binding (XCB) library libxml2-2.9.10 XML parser library for GNOME libxshmfence-1.3 Shared memory 'SyncFence' synchronization primitive libxslt-1.1.34 The XSLT C library for GNOME libzip-1.5.2 C library for reading, creating, and modifying ZIP archives llvm80-8.0.1_3 LLVM and Clang lua52-5.2.4 Small, compilable scripting language providing easy access to C code lzo2-2.10_1 Portable speedy, lossless data compression library m4-1.4.18_1,1 GNU M4 mc-4.8.22_2 Midnight Commander, a free Norton Commander Clone mesa-libs-18.3.2_3 OpenGL libraries that support GLX and EGL clients meson-0.52.0 High performance build system minixmlto-0.0.2_1 Minimalistic alternative to xmlto mkfontscale-1.2.1 Creates an index of scalable font files for X mpfr-4.0.2 Library for multiple-precision floating-point computations nasm-2.14.02,1 General-purpose multi-platform x86 and amd64 assembler nettle-3.5.1_1 Low-level cryptographic library nginx-devel-1.17.6_1 Robust and small WWW server ninja-1.9.0,2 Ninja is a small build system closest in spirit to Make node10-10.17.0 V8 JavaScript for client and server npm-6.12.1 Node package manager oniguruma-6.9.3 Regular expressions library compatible with POSIX/GNU/Perl open-vm-tools-nox11-11.0.1_2,2 Open VMware tools for FreeBSD VMware guests openjpeg-2.3.1 Open-source JPEG 2000 codec p11-kit-0.23.18.1 Library for loading and enumerating of PKCS#11 modules p5-Authen-SASL-2.16_1 Perl5 module for SASL authentication p5-CGI-4.44 Handle Common Gateway Interface requests and responses p5-Digest-HMAC-1.03_1 Perl5 interface to HMAC Message-Digest Algorithms p5-Error-0.17028 Error/exception handling in object-oriented programming style p5-GSSAPI-0.28_1 Perl extension providing access to the GSSAPIv2 library p5-HTML-Parser-3.72 Perl5 module for parsing HTML documents p5-HTML-Tagset-3.20_1 Some useful data table in parsing HTML p5-IO-Socket-INET6-2.72_1 Perl module with object interface to AF_INET6 domain sockets p5-IO-Socket-SSL-2.066 Perl5 interface to SSL sockets p5-Locale-gettext-1.07 Message handling functions p5-Locale-libintl-1.31 Internationalization library for Perl p5-Mozilla-CA-20180117 Perl extension for Mozilla CA cert bundle in PEM format p5-Net-SSLeay-1.85 Perl5 interface to SSL p5-Socket6-0.29 IPv6 related part of the C socket.h defines and structure manipulators p5-Term-ReadKey-2.38_1 Perl5 module for simple terminal control p5-Text-Unidecode-1.30 Text::Unidecode -- US-ASCII transliterations of Unicode text p5-Unicode-EastAsianWidth-12.0 East Asian Width properties p5-XML-Parser-2.44 Perl extension interface to James Clark's XML parser, expat p5-subversion-1.13.0 Perl bindings for Version control system pango-1.42.4_3 Open-source framework for the layout and rendering of i18n text pciids-20191012 Database of all known IDs used in PCI devices pcre-8.43_2 Perl Compatible Regular Expressions library pcre2-10.33 Perl Compatible Regular Expressions library, version 2 pdflib-7.0.5_5 C library for dynamically generating PDF perl5-5.30.1 Practical Extraction and Report Language php73-7.3.12 PHP Scripting Language php73-bcmath-7.3.12 The bcmath shared extension for php php73-bz2-7.3.12 The bz2 shared extension for php php73-calendar-7.3.12 The calendar shared extension for php php73-composer-1.9.0 Dependency Manager for PHP php73-ctype-7.3.12 The ctype shared extension for php php73-curl-7.3.12 The curl shared extension for php php73-dom-7.3.12 The dom shared extension for php php73-exif-7.3.12 The exif shared extension for php php73-extensions-1.0 "meta-port" to install PHP extensions php73-fileinfo-7.3.12 The fileinfo shared extension for php php73-filter-7.3.12 The filter shared extension for php php73-ftp-7.3.12 The ftp shared extension for php php73-gd-7.3.12 The gd shared extension for php php73-gettext-7.3.12 The gettext shared extension for php php73-hash-7.3.12 The hash shared extension for php php73-iconv-7.3.12 The iconv shared extension for php php73-intl-7.3.12 The intl shared extension for php php73-json-7.3.12 The json shared extension for php php73-mbstring-7.3.12 The mbstring shared extension for php php73-opcache-7.3.12 The opcache shared extension for php php73-openssl-7.3.12 The openssl shared extension for php php73-pdo-7.3.12 The pdo shared extension for php php73-pdo_pgsql-7.3.12 The pdo_pgsql shared extension for php php73-pdo_sqlite-7.3.12 The pdo_sqlite shared extension for php php73-pecl-imagick-im7-3.4.4 PHP wrapper to the ImageMagick/GraphicsMagick library version 7 php73-pecl-pdflib-4.1.3 PECL extension to create PDF on the fly php73-pecl-redis-5.1.1 Extension to access Redis php73-pgsql-7.3.12 The pgsql shared extension for php php73-phar-7.3.12 The phar shared extension for php php73-posix-7.3.12 The posix shared extension for php php73-readline-7.3.12 The readline shared extension for php php73-session-7.3.12 The session shared extension for php php73-simplexml-7.3.12 The simplexml shared extension for php php73-soap-7.3.12 The soap shared extension for php php73-sockets-7.3.12 The sockets shared extension for php php73-sqlite3-7.3.12 The sqlite3 shared extension for php php73-tidy-7.3.12 The tidy shared extension for php php73-tokenizer-7.3.12 The tokenizer shared extension for php php73-xml-7.3.12 The xml shared extension for php php73-xmlreader-7.3.12 The xmlreader shared extension for php php73-xmlrpc-7.3.12 The xmlrpc shared extension for php php73-xmlwriter-7.3.12 The xmlwriter shared extension for php php73-xsl-7.3.12 The xsl shared extension for php php73-zip-7.3.12 The zip shared extension for php php73-zlib-7.3.12 The zlib shared extension for php pixman-0.38.4 Low-level pixel manipulation library pkg-1.12.0 Package manager pkgconf-1.6.3,1 Utility to help to configure compiler and linker flags png-1.6.37 Library for manipulating PNG images poppler-data-0.4.9_1 Poppler encoding data postgresql11-client-11.6 PostgreSQL database (client) psutils-1.17_5 Utilities for manipulating PostScript documents py27-setuptools-41.4.0 Python packages installer py36-Babel-2.7.0 Collection of tools for internationalizing Python applications py36-CommonMark-0.9.1 Python parser for the CommonMark Markdown spec py36-Jinja2-2.10.1 Fast and easy to use stand-alone template engine py36-MarkupSafe-1.1.1 Implements XML/HTML/XHTML Markup safe string for Python py36-alabaster-0.7.6 Modified Kr Sphinx theme py36-asn1crypto-0.24.0 ASN.1 library with a focus on performance and a pythonic API py36-certifi-2019.9.11 Mozilla SSL certificates py36-cffi-1.13.2 Foreign Function Interface for Python calling C code py36-chardet-3.0.4_1 Universal encoding detector for Python 2 and 3 py36-cryptography-2.6.1 Cryptographic recipes and primitives for Python developers py36-cython-0.29.13 Compiler for Writing C Extensions for the Python Language py36-docutils-0.15.2 Python Documentation Utilities py36-future-0.18.2 Clean single-source support for Python 3 and 2 py36-idna-2.8 Internationalized Domain Names in Applications (IDNA) py36-imagesize-1.1.0 Python image size library py36-libxml2-2.9.10 Python interface for XML parser library for GNOME py36-openssl-19.0.0 Python interface to the OpenSSL library py36-pip-19.1.1 Tool for installing and managing Python packages py36-pycparser-2.19 C parser in Python py36-pygments-2.4.1 Syntax highlighter written in Python py36-pypa-docs-theme-g20180622 Common base Sphinx theme for PyPA projects py36-pysocks-1.7.1 Python SOCKS module py36-pystemmer-1.3.0_2 Snowball Stemming Algorithms for Information Retrieval py36-pytest-runner-2.11.1 Test support for pytest runner in setup.py py36-python-docs-theme-2018.2 Sphinx theme for the CPython docs and related projects py36-pytz-2019.3,1 World Timezone Definitions for Python py36-recommonmark-0.5.0_1 CommonMark bridge for docutils and Sphinx py36-requests-2.22.0 HTTP library written in Python for human beings py36-setuptools-41.4.0 Python packages installer py36-setuptools_scm-3.3.3 Setuptools plugin to manage your versions by scm tags py36-six-1.12.0 Python 2 and 3 compatibility utilities py36-snowballstemmer-1.2.1 Snowball stemming library collection for Python py36-sphinx-1.6.5_2,1 Python documentation generator py36-sphinx_rtd_theme-0.4.3 Mobile-friendly py-sphinx theme py36-sphinxcontrib-websupport-1.1.2 Sphinx API for Web Apps py36-sqlite3-3.6.9_7 Standard Python binding to the SQLite3 library (Python 3.6) py36-urllib3-1.25.6,1 HTTP library with thread-safe connection pooling, file post, and more python27-2.7.17_1 Interpreted object-oriented programming language python36-3.6.9_1 Interpreted object-oriented programming language rarian-0.8.1_4 OMF help system based on the Freedesktop specification readline-8.0.0 Library for editing command lines as they are typed rhash-1.3.5 Utility and library for computing and checking of file hashes scons-3.1.1 Build tool alternative to make sdocbook-xml-1.1_2,2 "Simplified" DocBook XML DTD serf-1.3.9_4 Serf HTTP client library shared-mime-info-1.10_2 MIME types database from the freedesktop.org project sqlite3-3.30.1 SQL database engine in a C library subversion-1.13.0 Version control system sudo-1.8.29 Allow others to run commands as root swig30-3.0.12 Generate wrappers for calling C/C++ code from other languages tcl86-8.6.10 Tool Command Language texinfo-6.7_1,1 Typeset documentation system with multiple format output tidy-lib-090315.c_3 Utility to clean up and pretty print HTML/XHTML/XML (with shared lib) tiff-4.1.0 Tools and library routines for working with TIFF images tpm-emulator-0.7.4_2 Trusted Platform Module (TPM) emulator trousers-0.3.14_2 Open-source TCG Software Stack uchardet-0.0.6 Universal charset detection library utf8proc-2.4.0 UTF-8 processing library vala-0.40.15,1 Programming language and compiler that converts Vala code into C code w3m-0.5.3.20190702 Pager/text-based WWW browser wayland-1.16.0_1 Wayland composite "server" wayland-protocols-1.17 Wayland protocols webp-1.0.3_1 Google WebP image format conversion tool wget-1.20.3 Retrieve files from the Net via HTTP(S) and FTP xcb-proto-1.13_1 The X protocol C-language Binding (XCB) protocol xmlcatmgr-2.2_2 SGML and XML catalog manager xmlcharent-0.3_2 XML character entities xmlto-0.0.28 Front-end to an XSL toolchain xorg-fonts-truetype-7.7_1 X.Org TrueType fonts xorg-macros-1.19.2 X.Org development aclocal macros xorgproto-2019.2 xorg protocol headers xtrans-1.4.0 Abstract network code for X yarn-1.19.1 Package manager for node, alternative to npm FreeBSD web1.vilnis.dc.local 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC amd64 tried also to upgrade to latest nginx-devel-1.17.6_1, but the same shit. I'm against to use 12.1 RELEASE in production even it classifies as STABLE, it' s unstable. Lot of errors using in VM in dmesg:
Can you grab procstat -kk output from some of the stuck processes?
PID TID COMM TDNAME KSTACK 1106 100280 nginx - vfs_bio_getpages ncl_getpages VOP_GETPAGES_APV vop_stdgetpages_async VOP_GETPAGES_ASYNC_APV vnode_pager_getpages_async vn_sendfile sendfile amd64_syscall fast_syscall_common
PID TID COMM TDNAME KSTACK 1106 100280 nginx - vfs_bio_getpages+0x1d9 ncl_getpages+0x2be VOP_GETPAGES_APV+0x7c vop_stdgetpages_async+0x49 VOP_GETPAGES_ASYNC_APV+0x7c vnode_pager_getpages_async+0x7e vn_sendfile+0xd9c sendfile+0x12b amd64_syscall+0x364 fast_syscall_common+0x101
Now got 2 cores on stuck: root@web1:~ # procstat -kk 1106 PID TID COMM TDNAME KSTACK 1106 100280 nginx - vfs_bio_getpages+0x1d9 ncl_getpages+0x2be VOP_GETPAGES_APV+0x7c vop_stdgetpages_async+0x49 VOP_GETPAGES_ASYNC_APV+0x7c vnode_pager_getpages_async+0x7e vn_sendfile+0xd9c sendfile+0x12b amd64_syscall+0x364 fast_syscall_common+0x101 root@web1:~ # procstat -kk 1104 PID TID COMM TDNAME KSTACK 1104 100136 nginx - vfs_bio_getpages+0x1d9 ncl_getpages+0x2be VOP_GETPAGES_APV+0x7c vop_stdgetpages_async+0x49 VOP_GETPAGES_ASYNC_APV+0x7c vnode_pager_getpages_async+0x7e vn_sendfile+0xd9c sendfile+0x12b amd64_syscall+0x364 fast_syscall_common+0x101
Created attachment 209920 [details] top process list
Created attachment 209923 [details] 2 minute stuck after init 6
Created attachment 209924 [details] stuck on sync process still in stuck after 5 minutes need to reboot server manually
Sorry, forgot to post vmstat list
As a temporary workaround you can try setting vfs.nfs.use_buf_pager=0, but vfs_bio_getpages() has been the default for some time now. Are the files that are being served also being modified or truncated?
seems that files are ok this server is frontend part, probably files are not saved just log writes in file way.
Try installing latest stable/12 kernel (no need to rebuild world). If there is indeed a truncation in parallel with the mapping operation, then I have an expectation that it is fixed.
ok, new kernel has built, so I'll check and report any issues if occured 12.1-STABLE FreeBSD 12.1-STABLE r355737 amd64
Created attachment 209933 [details] Problem still exists core process stuck on latest stable kernel
No success, nginx still eating cpu time. Process stuck and onky hard whole vm reset helped to restart nginx. Any other ideas?
It could be the same problem as here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242626
(In reply to Andris Vasers from comment #14) I suppose you either build kernel with debug symbols and ddb, or can rebuild it. After that, please dump core by entering into ddb: # sysctl debug.kdb.enter=1 db> dump .... output ... db> c after that run savecore(8) and provide me with kernel.full and vmcore. Also tell me the pid of the stuck nginx. Do not put the core file onto a public location, it might contain sensitive info.
(In reply to Jochen Neumeister from comment #15) > It could be the same problem as here: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242626 It seems you pasted this bug in your comment
(In reply to Ed Maste from comment #17) hups, i mean https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235296
(In reply to Konstantin Belousov from comment #16) Thanks Konstantin, I'll post you and attach link to data later. Now mentioned that there's no dependency in which state is nginx, CPUX or nfs, it will stuck unexpectedly. I have tried playing also with sysctl values, but still no success. Turning off nfs pages only helps if last state was stuck on nfs. I don't think that 11.x compatibility enabled kernel will crash with the same config as it was on 11.x RELEASE. Third party Binaries were build from latest sources. Tried different versions of nginx, even configuring and building without ports manually. Still the same.
Now only workaround suggested by Konstantin is disabling vfs.nfs.use_buf_pager=0 works for me. On 11.1 RELEASE it's disabled by default, but now on 12.1 - enabled.
(In reply to Andris Vasers from comment #19) So are you going to provide the requested kernel.full and vmcore files ?
Andris, check https://reviews.freebsd.org/D24038
I am not sure if it's the same issue, but with sendfile enabled in nginx.conf I had the nginx process in D-state. I could not unmount the filesystem and the only option was a hard reset. I believe the issue caused by files that frequently change, for example cache files.
(In reply to Christos Chatzaras from comment #23) Hello, I have found what kind of problem causes this. FreeBSD 12.1 has buggy VMware vmx (vmxnet3) driver, i changed it to em and everything works smoothly. If you have same issues, consider changing virtual network adapter, it will help! Regards Andris
It's a bare metal server, no virtualisation.
ok, then have you tried suggested option by turning off pages? What kind of network adapter you use? Maybe buggy network subsystem not only adapter drivers... Andris
I have em. The workaround for me was to disable sendfile in Nginx.
(In reply to Christos Chatzaras from comment #27) Try console# echo "vfs.nfs.use_buf_pager=0" >> /etc/sysctl.conf reload everything, enable sendfile again and check.
(In reply to Andris Vasers from comment #28) Sorry, forgot to ask, do you use nfs mounts?
No just normal UFS2 filesystem.
*** Bug 236961 has been marked as a duplicate of this bug. ***
The original issue may be fixed by r359464.