Bug 213940

Summary:	lang/php56 php-fpm hangs
Product:	Ports & Packages	Reporter:	Christos Chatzaras <chris>
Component:	Individual Port(s)	Assignee:	Alex Dupre <ale>
Status:	Closed FIXED
Severity:	Affects Only Me	Flags:	bugzilla: maintainer-feedback? (ale)
Priority:	---
Version:	Latest
Hardware:	Any
OS:	Any

Description Christos Chatzaras 2016-10-31 08:57:51 UTC

There is a problem with PHP 5.6.27 (maybe in previous version too) or Freebsd 10.3-STABLE (Wed Oct 26 20:16:09 EEST 2016) and randomly php-fpm (with pm=dynamic) hangs. It doesn't process requests for whole server. top shows 1 or 2 processes that use 100% CPU.

truss -p 32701

kill(18810,SIGKILL)                              ERR#3 'No such process'
fcntl(3,F_GETLK,0x7fffffffc5c0)                  = 0 (0x0)
kill(18810,SIGKILL)                              ERR#3 'No such process'
fcntl(3,F_GETLK,0x7fffffffc5c0)                  = 0 (0x0)
kill(18810,SIGKILL)                              ERR#3 'No such process'
fcntl(3,F_GETLK,0x7fffffffc5c0)                  = 0 (0x0)
kill(18810,SIGKILL)                              ERR#3 'No such process'
fcntl(3,F_GETLK,0x7fffffffc5c0)                  = 0 (0x0)

fstat -p 32701

USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
gioannou php-fpm    32701 text /usr     162750 -rwxr-xr-x  4471845  r
gioannou php-fpm    32701   wd /        321033 drwxr-xr-x     512  r
gioannou php-fpm    32701 root /             2 drwxr-xr-x     512  r
gioannou php-fpm    32701    0* local stream fffff8032888e4b0
gioannou php-fpm    32701    1 /dev         13 crw-rw-rw-    null rw
gioannou php-fpm    32701    2 /dev         13 crw-rw-rw-    null rw
gioannou php-fpm    32701    3 -          6084 -rw-rw-rw-       0 rw
gioannou php-fpm    32701    4* local stream fffff80457b1aa50

procstat -kk 32701
  PID    TID COMM             TDNAME           KSTACK
32701 101779 php-fpm          -                <running>

Comment 1 Christos Chatzaras 2016-11-01 02:41:48 UTC

gdb /usr/local/sbin/php-fpm 60638

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
Attaching to program: /usr/local/sbin/php-fpm, process 60638
Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.5
Reading symbols from /usr/local/lib/libpcre.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libpcre.so.1
Reading symbols from /lib/libm.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
[New Thread 802806400 (LWP 101470/php-fpm)]
Loaded symbols for /lib/libthr.so.3
Reading symbols from /usr/local/lib/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libxml2.so.2
Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.6
Reading symbols from /usr/lib/liblzma.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblzma.so.5
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /usr/local/lib/php/20131226/ioncube/ioncube_loader.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/ioncube/ioncube_loader.so
Reading symbols from /usr/local/lib/php/20131226/opcache.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/opcache.so
Reading symbols from /usr/local/lib/php/20131226/session.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/session.so
Reading symbols from /usr/local/lib/php/20131226/bcmath.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/bcmath.so
Reading symbols from /usr/local/lib/php/20131226/bz2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/bz2.so
Reading symbols from /usr/lib/libbz2.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libbz2.so.4
Reading symbols from /usr/local/lib/php/20131226/calendar.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/calendar.so
Reading symbols from /usr/local/lib/php/20131226/ctype.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/ctype.so
Reading symbols from /usr/local/lib/php/20131226/curl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/curl.so
Reading symbols from /usr/local/lib/libcurl.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libcurl.so.4
Reading symbols from /usr/local/lib/libssl.so.9...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libssl.so.9
Reading symbols from /usr/local/lib/libcrypto.so.9...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libcrypto.so.9
Reading symbols from /usr/local/lib/php/20131226/dom.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/dom.so
Reading symbols from /usr/local/lib/php/20131226/exif.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/exif.so
Reading symbols from /usr/local/lib/php/20131226/fileinfo.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/fileinfo.so
Reading symbols from /usr/local/lib/php/20131226/filter.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/filter.so
Reading symbols from /usr/local/lib/php/20131226/ftp.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/ftp.so
Reading symbols from /usr/local/lib/php/20131226/gd.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/gd.so
Reading symbols from /usr/local/lib/libt1.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libt1.so.5
Reading symbols from /usr/local/lib/libpng16.so.16...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libpng16.so.16
Reading symbols from /usr/local/lib/libjpeg.so.8...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libjpeg.so.8
Reading symbols from /usr/local/lib/libfreetype.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libfreetype.so.6
Reading symbols from /usr/local/lib/php/20131226/gettext.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/gettext.so
Reading symbols from /usr/local/lib/libintl.so.8...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libintl.so.8
Reading symbols from /usr/local/lib/php/20131226/hash.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/hash.so
Reading symbols from /usr/local/lib/php/20131226/iconv.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/iconv.so
Reading symbols from /usr/local/lib/libiconv.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libiconv.so.2
Reading symbols from /usr/local/lib/php/20131226/imap.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/imap.so
Reading symbols from /usr/local/lib/libc-client4.so.9...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libc-client4.so.9
Reading symbols from /usr/lib/libpam.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libpam.so.5
Reading symbols from /usr/local/lib/php/20131226/intl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/intl.so
Reading symbols from /usr/local/lib/libicui18n.so.57...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libicui18n.so.57
Reading symbols from /usr/local/lib/libicuuc.so.57...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libicuuc.so.57
Reading symbols from /usr/local/lib/libicudata.so.57...
warning: Lowest section in /usr/local/lib/libicudata.so.57 is .hash at 0000000000000120
(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libicudata.so.57
Reading symbols from /usr/local/lib/libicuio.so.57...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libicuio.so.57
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /usr/lib/libc++.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libc++.so.1
Reading symbols from /lib/libcxxrt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcxxrt.so.1
Reading symbols from /usr/local/lib/php/20131226/json.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/json.so
Reading symbols from /usr/local/lib/php/20131226/mbstring.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/mbstring.so
Reading symbols from /usr/local/lib/libonig.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libonig.so.1
Reading symbols from /usr/local/lib/php/20131226/mcrypt.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/mcrypt.so
Reading symbols from /usr/local/lib/libmcrypt.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libmcrypt.so.4
Reading symbols from /usr/local/lib/libltdl.so.7...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libltdl.so.7
Reading symbols from /usr/local/lib/php/20131226/mysql.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/mysql.so
Reading symbols from /usr/local/lib/php/20131226/mysqli.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/mysqli.so
Reading symbols from /usr/local/lib/php/20131226/openssl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/openssl.so
Reading symbols from /usr/local/lib/php/20131226/pdo.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/pdo.so
Reading symbols from /usr/local/lib/php/20131226/posix.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/posix.so
Reading symbols from /usr/local/lib/php/20131226/simplexml.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/simplexml.so
Reading symbols from /usr/local/lib/php/20131226/soap.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/soap.so
Reading symbols from /usr/local/lib/php/20131226/sockets.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/sockets.so
Reading symbols from /usr/local/lib/php/20131226/sqlite3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/sqlite3.so
Reading symbols from /usr/local/lib/libsqlite3.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libsqlite3.so.0
Reading symbols from /usr/local/lib/php/20131226/tidy.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/tidy.so
Reading symbols from /usr/local/lib/libtidy-0.99.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libtidy-0.99.so.0
Reading symbols from /usr/local/lib/php/20131226/tokenizer.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/tokenizer.so
Reading symbols from /usr/local/lib/php/20131226/xml.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/xml.so
Reading symbols from /usr/local/lib/php/20131226/xmlwriter.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/xmlwriter.so
Reading symbols from /usr/local/lib/php/20131226/zip.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/zip.so
Reading symbols from /usr/local/lib/libzip.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libzip.so.4
Reading symbols from /usr/local/lib/php/20131226/zlib.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/zlib.so
Reading symbols from /usr/local/lib/php/20131226/pdf.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/pdf.so
Reading symbols from /usr/local/lib/libpdf.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libpdf.so.6
Reading symbols from /usr/local/lib/php/20131226/pdo_mysql.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/pdo_mysql.so
Reading symbols from /usr/local/lib/php/20131226/pdo_sqlite.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/pdo_sqlite.so
Reading symbols from /usr/local/lib/php/20131226/phar.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/phar.so
Reading symbols from /usr/local/lib/php/20131226/xmlreader.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/xmlreader.so
Reading symbols from /usr/local/lib/php/20131226/xmlrpc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/xmlrpc.so
Reading symbols from /usr/local/lib/php/20131226/xsl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/php/20131226/xsl.so
Reading symbols from /usr/local/lib/libexslt.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libexslt.so.0
Reading symbols from /usr/local/lib/libxslt.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libxslt.so.1
Reading symbols from /usr/local/lib/libgcrypt.so.20...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libgcrypt.so.20
Reading symbols from /usr/local/lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libgpg-error.so.0
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
Loaded symbols for /libexec/ld-elf.so.1
[Switching to Thread 802806400 (LWP 101470/php-fpm)]
0x0000000801dda4ba in _fcntl () from /lib/libc.so.7
(gdb)
(gdb) bt
#0  0x0000000801dda4ba in _fcntl () from /lib/libc.so.7
#1  0x0000000801d88506 in __fcntl_compat () from /lib/libc.so.7
#2  0x00000008012b7a78 in pthread_suspend_all_np () from /lib/libthr.so.3
#3  0x0000000801d885c2 in fcntl () from /lib/libc.so.7
#4  0x0000000802c0989b in accel_shutdown () from /usr/local/lib/php/20131226/opcache.so
#5  0x00000000005810c7 in zend_llist_apply ()
#6  0x000000000057d641 in init_executor ()
#7  0x000000000058bae3 in zend_activate ()
#8  0x00000000005319d3 in php_request_startup ()
#9  0x0000000000617531 in main ()
(gdb)

Comment 2 Christos Chatzaras 2016-11-09 12:18:20 UTC

Looks like the issue was with opcache_reset(). Some applications had access to reset the cache and maybe they flood it with such requests. I disable this function and everything is ok now.