After updating to nginx-devel-1.19.5_3 (from nginx-devel-1.19.4_3), I noticed that the error log was chock-full with "worker process XXX exited on signal 11" (where XXX is is the PID). At first I thought there was some library incompatibility, so I rebuilt the whole jail. Unfortunately, that didn't help at all. After looking through the CHANGELOG, I downgraded to nginx-deve-1.19.4_4, which fixed the issue. I can't be certain, but as the nginx CHANGELOG says that there were some SSL fixes in 1.19.5, it could be related to LibreSSL (which is what I use).
Hi Peter, thanks for the report! Could you please provide an output of the `nginx -V' command.
(In reply to Sergey A. Osokin from comment #1) This is for the downgraded version (which works). Do you need the output for the non-working version as well? (The configuration should be the same.) nginx version: nginx/1.19.4 built with LibreSSL 3.2.2 TLS SNI support enabled configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --user=www --group=www --with-compat --modules-path=/usr/local/libexec/nginx --with-file-aio --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx/access.log --with-http_v2_module --with-http_auth_request_module --with-http_dav_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-pcre --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-http_image_filter_module=dynamic --with-stream=dynamic --add-dynamic-module=/var/ports/root/nginx-devel/work/ngx_devel_kit-0.3.1 --add-dynamic-module=/var/ports/root/nginx-devel/work/ngx_brotli-25f86f0 --add-dynamic-module=/var/ports/root/nginx-devel/work/ngx_cache_purge-2b977cf --add-dynamic-module=/var/ports/root/nginx-devel/work/nginx_cookie_flag_module-c4ff449 --add-dynamic-module=/var/ports/root/nginx-devel/work/echo-nginx-module-c65f5c6 --add-dynamic-module=/var/ports/root/nginx-devel/work/encrypted-session-nginx-module-0.08 --add-dynamic-module=/var/ports/root/nginx-devel/work/headers-more-nginx-module-55fbdab --add-dynamic-module=/var/ports/root/nginx-devel/work/nginx_accept_language_module-5683967 --add-dynamic-module=/var/ports/root/nginx-devel/work/nginx-http-auth-digest-274490c --add-dynamic-module=/var/ports/root/nginx-devel/work/nginx-dav-ext-module-3.0.0 --add-dynamic-module=/var/ports/root/nginx-devel/work/ngx_http_geoip2_module-3.2 --add-dynamic-module=/var/ports/root/nginx-devel/work/nginx-notice-3c95966 --add-dynamic-module=/var/ports/root/nginx-devel/work/set-misc-nginx-module-048e9e0 --add-dynamic-module=/var/ports/root/nginx-devel/work/ngx_http_redis-0.3.9
Peter, Thanks for the update. Could you please also provide an output of the `/usr/local/bin/openssl version' command. Thanks.
(In reply to Peter Putzer from comment #2) We definitely need an output from the non-working version, thanks.
(In reply to Sergey A. Osokin from comment #4) Here we go: nginx version: nginx/1.19.5 built with LibreSSL 3.2.2 TLS SNI support enabled configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --user=www --group=www --with-compat --modules-path=/usr/local/libexec/nginx --with-file-aio --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx/access.log --with-http_v2_module --with-http_auth_request_module --with-http_dav_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-pcre --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-http_image_filter_module=dynamic --with-stream=dynamic --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/ngx_devel_kit-0.3.1 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/ngx_brotli-25f86f0 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/ngx_cache_purge-2b977cf --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/nginx_cookie_flag_module-c4ff449 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/echo-nginx-module-c65f5c6 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/encrypted-session-nginx-module-0.08 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/headers-more-nginx-module-55fbdab --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/nginx_accept_language_module-5683967 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/nginx-http-auth-digest-274490c --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/nginx-dav-ext-module-3.0.0 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/ngx_http_geoip2_module-3.2 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/nginx-notice-3c95966 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/set-misc-nginx-module-048e9e0 --add-dynamic-module=/var/ports/usr/ports/www/nginx-devel/work/ngx_http_redis-0.3.9
(In reply to Sergey A. Osokin from comment #3) LibreSSL 3.2.2
Thanks for the update, Peter. Is there a chance to disable all third-party modules and try to reproduce the issue? Could you create and share a minimal configuration file to reproduce the issue.
(In reply to Sergey A. Osokin from comment #7) I can try to reduce the number to those that are actually used (there are some dynamic modules that are not loaded in the current configuration), but that server is hosting some live sites, so I'd prefer not do a truly minimal build there. However, I could do a debug build and try to see where exactly it is crashing.
(In reply to Peter Putzer from comment #8) That would be great to build the nginx with debugging enabled and reproduce the issue. Can we get a core file in that case? Is there a chance to see a backtrace?
(In reply to Sergey A. Osokin from comment #9) GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD] ... Reading symbols from /usr/local/sbin/nginx... [New LWP 100677] Core was generated by `nginx: worker process'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000283693 in ngx_ssl_recv (c=0x8224b7ba8, buf=0x7fffffffd7d0 "", size=4096) at src/event/ngx_event_openssl.c:2089 2089 src/event/ngx_event_openssl.c: No such file or directory. (gdb) backtrace #0 0x0000000000283693 in ngx_ssl_recv (c=0x8224b7ba8, buf=0x7fffffffd7d0 "", size=4096) at src/event/ngx_event_openssl.c:2089 #1 0x000000000029cca0 in ngx_http_lingering_close_handler (rev=0x8254b90a8) at src/http/ngx_http_request.c:3466 #2 0x000000000027f8b5 in ngx_kqueue_process_events (cycle=0x800baa650, timer=<optimized out>, flags=1) at src/event/modules/ngx_kqueue_module.c:669 #3 0x00000000002723c1 in ngx_process_events_and_timers (cycle=0x800baa650) at src/event/ngx_event.c:247 #4 0x000000000027d869 in ngx_worker_process_cycle (cycle=0x800baa650, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:740 #5 0x000000000027b5db in ngx_spawn_process (cycle=<optimized out>, proc=0x27d7e0 <ngx_worker_process_cycle>, data=0x5, name=0x221913 "worker process", respawn=-3) at src/os/unix/ngx_process.c:199 #6 0x000000000027c03d in ngx_start_worker_processes (cycle=<optimized out>, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:349 #7 ngx_master_process_cycle (cycle=0x800baa650) at src/os/unix/ngx_process_cycle.c:130 #8 0x0000000000249e63 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:383
Thanks for the update, Peter. Could you provide an output of the `ldd /usr/local/sbin/nginx` command as well.
(In reply to Sergey A. Osokin from comment #11) /usr/local/sbin/nginx: libthr.so.3 => /lib/libthr.so.3 (0x800359000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x800386000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x8003a7000) libssl.so.48 => /usr/local/lib/libssl.so.48 (0x800450000) libcrypto.so.46 => /usr/local/lib/libcrypto.so.46 (0x8004b8000) libz.so.6 => /lib/libz.so.6 (0x8006da000) libc.so.7 => /lib/libc.so.7 (0x8006f4000)
Peter, could you please provide debug log, that helps a lot on the investigation. Thanks in advance.
(In reply to Sergey A. Osokin from comment #13) I'll try to produce a sanitized debug log that can be shared. Unfortunately, I'm somewhat at a loss on how error_log directives for different server blocks interact. (The results seem rather haphazard so far.)
(In reply to Peter Putzer from comment #14) Peter, here's the document describes how to configure debug log, https://nginx.org/en/docs/debugging_log.html Please let me know if you have any questions.
(In reply to Sergey A. Osokin from comment #15) Ah, that's not what I meant, rather: which debug messages get put into which file if different ones are set for the various server blocks? That works ... in mysterious ways.
Created attachment 220354 [details] [PATCH] SSL: fixed SSL shutdown on lingering close Hi Peter, could you try the attached patch, try to reproduce the issue and report back. Appreciate your help!
A commit references this bug: Author: osa Date: Mon Dec 7 20:06:32 UTC 2020 New revision: 557244 URL: https://svnweb.freebsd.org/changeset/ports/557244 Log: Fix a worker process issue by adding the vendor's patch. Bump PORTREVISION. Reported by: Peter Putzer <freebsd@mnd.sc> <ChangeLog> SSL: fixed SSL shutdown on lingering close. Ensure c->recv is properly reset to ngx_recv if SSL_shutdown() blocks on writing. The bug had appeared in 554c6ae25ffc. </ChangeLog> PR: 251664 Changes: head/www/nginx-devel/Makefile head/www/nginx-devel/files/PR-251664.patch
(In reply to Sergey A. Osokin from comment #17) Port version 1.19.5_4 has fixed the issue for me. Thank you!
Thanks for the report, Peter!