Hi joneum, Yesterday we upgraded to 1.18.0_15, and the module upstream_check stopped working properly. You can do a test using a simple config file: <begin config file> worker_processes 1; worker_rlimit_nofile 262144; load_module "/usr/local/libexec/nginx/ngx_http_upstream_check_module.so"; error_log /tmp/test/error.log debug; events { worker_connections 16384; use kqueue; kqueue_changes 2048; } http { upstream cluster { # simple round-robin server 192.168.0.1:80; server 192.168.0.2:80; check interval=5000 rise=1 fall=3 timeout=4000; } server { listen 50080; location / { proxy_pass http://cluster; } location /status { check_status; access_log off; } } } <end config file> <begin commands> nginx -p /tmp/test -c test.conf curl http://localhost:50080/status <end commands> And logs show: <begin logs> 2020/06/23 12:35:00 [debug] 20275#101354: worker cycle 2020/06/23 12:35:00 [debug] 20275#101354: kevent timer: 60000, changes: 1 2020/06/23 12:35:00 [debug] 20275#101354: kevent events: 1 2020/06/23 12:35:00 [debug] 20275#101354: kevent: 3: ft:-1 fl:0020 ff:00000000 d:85 ud:000000080680D111 2020/06/23 12:35:00 [debug] 20275#101354: *3 http wait request handler 2020/06/23 12:35:00 [debug] 20275#101354: *3 malloc: 0000000802270000:1024 2020/06/23 12:35:00 [debug] 20275#101354: *3 recv: eof:0, avail:85, err:0 2020/06/23 12:35:00 [debug] 20275#101354: *3 recv: fd:3 85 of 1024 2020/06/23 12:35:00 [debug] 20275#101354: *3 reusable connection: 0 2020/06/23 12:35:00 [debug] 20275#101354: *3 posix_memalign: 000000080227F000:4096 @16 2020/06/23 12:35:00 [debug] 20275#101354: *3 http process request line 2020/06/23 12:35:00 [debug] 20275#101354: *3 http request line: "GET /status HTTP/1.1" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http uri: "/status" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http args: "" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http exten: "" 2020/06/23 12:35:00 [debug] 20275#101354: *3 posix_memalign: 000000080227B000:4096 @16 2020/06/23 12:35:00 [debug] 20275#101354: *3 http process request header line 2020/06/23 12:35:00 [debug] 20275#101354: *3 http header: "Host: localhost:50080" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http header: "User-Agent: curl/7.67.0" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http header: "Accept: */*" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http header done 2020/06/23 12:35:00 [debug] 20275#101354: *3 event timer del: 3: 7021985672 2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 0 2020/06/23 12:35:00 [debug] 20275#101354: *3 rewrite phase: 1 2020/06/23 12:35:00 [debug] 20275#101354: *3 test location: "/" 2020/06/23 12:35:00 [debug] 20275#101354: *3 test location: "status" 2020/06/23 12:35:00 [debug] 20275#101354: *3 using configuration "/status" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http cl:-1 max:1048576 2020/06/23 12:35:00 [debug] 20275#101354: *3 rewrite phase: 3 2020/06/23 12:35:00 [debug] 20275#101354: *3 rewrite phase: 4 2020/06/23 12:35:00 [debug] 20275#101354: *3 post rewrite phase: 5 2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 6 2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 7 2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 8 2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 9 2020/06/23 12:35:00 [debug] 20275#101354: *3 access phase: 10 2020/06/23 12:35:00 [debug] 20275#101354: *3 access phase: 11 2020/06/23 12:35:00 [debug] 20275#101354: *3 access phase: 12 2020/06/23 12:35:00 [debug] 20275#101354: *3 post access phase: 13 2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 14 2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 15 2020/06/23 12:35:00 [debug] 20275#101354: *3 http set discard body 2020/06/23 12:35:00 [error] 20275#101354: *3 http upstream check module can not find any check server, make sure you've added the check servers, client: 127.0.0.1, server: , request: "GET /status HTTP/1.1", host: "localhost:50080" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http finalize request: 500, "/status?" a:1, c:1 2020/06/23 12:35:00 [debug] 20275#101354: *3 http special response: 500, "/status?" 2020/06/23 12:35:00 [debug] 20275#101354: *3 http set discard body 2020/06/23 12:35:00 [debug] 20275#101354: *3 HTTP/1.1 500 Internal Server Error Server: nginx/1.18.0 Date: Tue, 23 Jun 2020 19:35:00 GMT Content-Type: text/html Content-Length: 177 Connection: close <end logs> Looks as if it wasn't installed properly.
The module was tested with Poudriere and a test server (FreeBSD 12.1 / amd64) before release. There were no problems. If you want me to test your configuration here in more detail, please contact me privately.
Just to confirm that using a vanilla VM and building from ports everything worked as expected. So this should be a particular problem, not related to the package. Will do better testing next time, sorry.
I would add that we see this issue as well with our our poudriere pkg repo -- anyone have any ideas what's causing this?
(In reply to Adam Strohl from comment #3) see #2
(In reply to Jochen Neumeister from comment #4) I don't see what was actually wrong in comment #2, just that building it from ports apparently works?
Can confirm that this is 100% broken if you build with Poudriere however the port itself works. Are Poudriere builds not supported? How is the FreeBSD project building this successfully for it's own repo?
r538078 works, it is just before making upstream check module dynamic, but just next r538085 is not working. r538085 supposed to convert upstream check module to dynamic, but it broke the module.
Counter to my previous comment, building from latest clean ports directly and only enabling this checkbox in make config (leave all other defaults) does NOT work. So the port itself is broken in terms of this feature. It will build and run, but not work correctly. It is not poudriere specific, doing 'make build install' via /usr/ports/www/nginx shows the issue too.
I need more information here to reproduce it on a test VM: -Are there any error messages? - Which configurations are used? -Good would be a list of "make showconfig"
(In reply to Jochen Neumeister from comment #9) Everything will look fine, but if you actually go to query the module (via it's reporting endpoint) it reports all peers down for sites using it. If you're not familiar with setting it up let me know and I can give you some examples. Overall unless you're using it for sites (you need to specifically configure it per site) and monitoring it you'd never know it's broken. The web server works fine for sites not using it, etc. Config: ===> The following configuration options are available for nginx-1.18.0_22,2: DEBUG=off: Build with debugging support DEBUGLOG=off: Enable debug log (--with-debug) DSO=on: Enable dynamic modules support FILE_AIO=off: Enable file aio IPV6=on: Enable IPv6 support NJS=off: Enable http_javascript module THREADS=on: Enable threads support WWW=on: Enable html sample files ====> Modules that require MAIL module MAIL=off: Enable IMAP4/POP3/SMTP proxy module MAIL_IMAP=off: Enable IMAP4 proxy module MAIL_POP3=off: Enable POP3 proxy module MAIL_SMTP=off: Enable SMTP proxy module MAIL_SSL=off: Enable mail_ssl module ====> Modules that require HTTP module GOOGLE_PERFTOOLS=off: Enable google perftools module HTTP=on: Enable HTTP module HTTP_ADDITION=off: Enable http_addition module HTTP_AUTH_REQ=off: Enable http_auth_request module HTTP_CACHE=on: Enable http_cache module HTTP_DAV=off: Enable http_webdav module HTTP_FLV=off: Enable http_flv module HTTP_GUNZIP_FILTER=off: Enable http_gunzip_filter module HTTP_GZIP_STATIC=off: Enable http_gzip_static module HTTP_IMAGE_FILTER=off: Enable http_image_filter module HTTP_MP4=off: Enable http_mp4 module HTTP_PERL=off: Enable http_perl module HTTP_RANDOM_INDEX=off: Enable http_random_index module HTTP_REALIP=off: Enable http_realip module HTTP_REWRITE=on: Enable http_rewrite module HTTP_SECURE_LINK=off: Enable http_secure_link module HTTP_SLICE=on: Enable http_slice module HTTP_SLICE_AHEAD=off: Enable http_slice_ahead module HTTP_SSL=on: Enable http_ssl module HTTP_STATUS=on: Enable http_stub_status module HTTP_SUB=off: Enable http_sub module HTTP_XSLT=off: Enable http_xslt module HTTPV2=on: Enable HTTP/2 protocol support (SSL req.) STREAM=on: Enable stream module STREAM_SSL=on: Enable stream_ssl module (SSL req.) STREAM_SSL_PREREAD=on: Enable stream_ssl_preread module (SSL req.) AJP=off: 3rd party ajp module AWS_AUTH=off: 3rd party aws auth module BROTLI=off: 3rd party brotli module CACHE_PURGE=on: 3rd party cache_purge module CLOJURE=off: 3rd party clojure module CT=off: 3rd party cert_transparency module (SSL req.) DEVEL_KIT=off: 3rd party Nginx Development Kit module ARRAYVAR=off: 3rd party array_var module DRIZZLE=off: 3rd party drizzle module DYNAMIC_TLS=off: 3rd party dynamic tls records patch DYNAMIC_UPSTREAM=off: 3rd party dynamic_upstream module ECHO=off: 3rd party echo module ENCRYPTSESSION=off: 3rd party encrypted_session module FORMINPUT=off: 3rd party form_input module GRIDFS=off: 3rd party gridfs module HEADERS_MORE=off: 3rd party headers_more module HTTP_ACCEPT_LANGUAGE=off: 3rd party accept_language module HTTP_AUTH_DIGEST=off: 3rd party http_authdigest module HTTP_AUTH_JWT=off: 3rd party http_auth_jwt module HTTP_AUTH_KRB5=off: 3rd party http_auth_gss module HTTP_AUTH_LDAP=on: 3rd party http_auth_ldap module HTTP_AUTH_PAM=off: 3rd party http_auth_pam module HTTP_DAV_EXT=off: 3rd party webdav_ext module HTTP_EVAL=off: 3rd party eval module HTTP_FANCYINDEX=off: 3rd party http_fancyindex module HTTP_FOOTER=off: 3rd party http_footer module HTTP_GEOIP2=off: 3rd party geoip2 module HTTP_IP2LOCATION=off: 3rd party ip2location-nginx module HTTP_IP2PROXY=off: 3rd party ip2proxy-nginx module HTTP_JSON_STATUS=off: 3rd party http_json_status module HTTP_MOGILEFS=off: 3rd party mogilefs module HTTP_MP4_H264=off: 3rd party mp4/h264 module HTTP_NOTICE=off: 3rd party notice module HTTP_PUSH=off: 3rd party push module HTTP_PUSH_STREAM=off: 3rd party push stream module HTTP_REDIS=off: 3rd party http_redis module HTTP_RESPONSE=off: 3rd party http_response module HTTP_SUBS_FILTER=off: 3rd party subs filter module HTTP_TARANTOOL=off: 3rd party tarantool upstream module HTTP_UPLOAD=off: 3rd party upload module HTTP_UPLOAD_PROGRESS=off: 3rd party uploadprogress module HTTP_UPSTREAM_CHECK=on: 3rd party upstream check module HTTP_UPSTREAM_FAIR=off: 3rd party upstream fair module HTTP_UPSTREAM_STICKY=off: 3rd party upstream sticky module HTTP_VIDEO_THUMBEXTRACTOR=off: 3rd party video_thumbextractor module HTTP_ZIP=off: 3rd party http_zip module ICONV=off: 3rd party iconv module LET=off: 3rd party let module LINK=off: 3rd party link function module LUA=off: 3rd party lua module MEMC=off: 3rd party memc (memcached) module MODSECURITY3=off: 3rd party modsecurity3 module NAXSI=off: 3rd party naxsi module OPENTRACING=off: 3rd party opentracing module PASSENGER=off: 3rd party passenger module POSTGRES=off: 3rd party postgres module RDS_CSV=off: 3rd party rds_csv module RDS_JSON=off: 3rd party rds_json module REDIS2=off: 3rd party redis2 module RTMP=off: 3rd party rtmp module SET_MISC=off: 3rd party set_misc module SFLOW=off: 3rd party sflow module SHIBBOLETH=off: 3rd party shibboleth module SLOWFS_CACHE=off: 3rd party slowfs_cache module SMALL_LIGHT=off: 3rd party small_light module SRCACHE=off: 3rd party srcache module VOD=off: 3rd party vod module VTS=off: 3rd party vts module XSS=off: 3rd party xss module WEBSOCKIFY=off: 3rd party websockify module ====> GSSAPI implementation (imply HTTP_AUTH_KRB5): you can only select none or one of them GSSAPI_BASE=off: GSSAPI support via base system (needs Kerberos) GSSAPI_HEIMDAL=off: GSSAPI support via security/heimdal GSSAPI_MIT=off: GSSAPI support via security/krb5 ===> Use 'make config' to modify these settings
I'm setting up a test environment. I'm afraid that's gonna take some time. Two questions: - has it been tested with nginx-devel? If so, what was the result here? - during the change the patch "files/extra-patch-nginx_upstream_check_module-config" was added. Was this patch deleted for test purposes?
Test environment should be easy. You can follow/use Felipe's config with nginx and log should show: 2020/06/23 12:35:00 [error] 20275#101354: *3 http upstream check module can not find any check server, make sure you've added the check servers, client: 127.0.0.1, server: , request: "GET /status HTTP/1.1", host: "localhost:50080" Please note in above log that server is empty -> "server: ,", so it is like not getting config lines or something. Without making upstream check module dynamic, it will not show above error and it would work as expected. We didn't test nginx-devel. We didn't delete files/extra-patch-nginx_upstream_check_module-config file during test. I didn't look closely the content of it, but if it makes the module dynamic then removing it may make it static and thus it would work. Also maybe Makefile.extmod needs to have upstream related changes reverted.
nginx-devel doesn't work either. deleted files/extra-patch-nginx_upstream_check_module-config and tried. It gives: pkg-static: Unable to access file /usr/ports/www/nginx/work/stage/usr/local/libexec/nginx/ngx_http_upstream_check_module.so:No such file or directory *** Error code 74 Although it didn't install properly, it can be run from work/nginx-1.18.0/objs/ and it works fine: result of curl: root@beastie:/usr/ports/www/nginx # curl http://localhost:50080/status <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>Nginx http upstream check status</title> </head> <body> <h1>Nginx http upstream check status</h1> <h2>Check upstream server number: 2, generation: 1</h2> <table style="background-color:white" cellspacing="0" cellpadding="3" border="1"> <tr bgcolor="#C0C0C0"> <th>Index</th> <th>Upstream</th> <th>Name</th> <th>Status</th> <th>Rise counts</th> <th>Fall counts</th> <th>Check type</th> <th>Check port</th> </tr> <tr bgcolor="#FF0000"> <td>0</td> <td>cluster</td> <td>192.168.1.8:80</td> <td>down</td> <td>0</td> <td>17</td> <td>tcp</td> <td>0</td> </tr> <tr bgcolor="#FF0000"> <td>1</td> <td>cluster</td> <td>192.168.1.9:80</td> <td>down</td> <td>0</td> <td>17</td> <td>tcp</td> <td>0</td> </tr> </table> </body> </html>
(In reply to Ganbold Tsagaankhuu from comment #13) ok, then the bug seems to be in the patch
According to official documentation not all modules can be converted from static to dynamic. Especially those that patching nginx core. I see several options here until we find a way how to fix that: - convert this module to static back; - use another modules, like dynamic_healthcheck, it's available as dynamic module with the www/nginx-devel port.
Please make this module static as before. That way both options would be available to users.
Created attachment 217455 [details] upstream_check static module Hi, Here is the patch to make upstream_check module static. Please go ahead and test it, let me know how it works.
Created attachment 217470 [details] Patch for nginx nginx-devel patch works for me. Here is the patch that works with nginx port.
A commit references this bug: Author: osa Date: Mon Aug 24 02:58:56 UTC 2020 New revision: 546038 URL: https://svnweb.freebsd.org/changeset/ports/546038 Log: Partially revert back r533146 cause upstream_check third-party dynamic module can't work as expected, so let's keep it static for now. Bump PORTREVISION. PR: 247523 Changes: head/www/nginx-devel/Makefile head/www/nginx-devel/Makefile.extmod head/www/nginx-devel/files/extra-patch-nginx_upstream_check_module-config head/www/nginx-devel/pkg-plist
Can you revert back nginx port? thanks,
A commit references this bug: Author: joneum Date: Mon Aug 24 12:25:08 UTC 2020 New revision: 546072 URL: https://svnweb.freebsd.org/changeset/ports/546072 Log: Partially revert back r538085 cause upstream_check third-party dynamic module can't work as expected, so let's keep it static for now. PR: 247523 Sponsored by: Netzkommune GmbH Changes: head/www/nginx/Makefile head/www/nginx/Makefile.extmod head/www/nginx/files/extra-patch-nginx_upstream_check_module-config head/www/nginx/pkg-plist