Bug 247523 - www/nginx{-devel}: module ngx_http_upstream_check stopped working after migrating to dynamic in 1.18.0_12,2
Summary: www/nginx{-devel}: module ngx_http_upstream_check stopped working after migra...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Jochen Neumeister
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-24 16:27 UTC by Felipe Zipitria
Modified: 2020-08-24 12:25 UTC (History)
4 users (show)

See Also:
bugzilla: maintainer-feedback? (joneum)


Attachments
upstream_check static module (3.98 KB, patch)
2020-08-23 15:22 UTC, Sergey A. Osokin
no flags Details | Diff
Patch for nginx (3.99 KB, patch)
2020-08-24 02:41 UTC, Ganbold Tsagaankhuu
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Felipe Zipitria 2020-06-24 16:27:22 UTC
Hi joneum,

Yesterday we upgraded to 1.18.0_15, and the module upstream_check stopped working properly.

You can do a test using a simple config file:

<begin config file>
worker_processes 1;
worker_rlimit_nofile 262144;

load_module "/usr/local/libexec/nginx/ngx_http_upstream_check_module.so";

error_log /tmp/test/error.log debug;

events {
    worker_connections  16384;
    use kqueue;
    kqueue_changes 2048;
}


http {

        upstream cluster {

            # simple round-robin
            server 192.168.0.1:80;
            server 192.168.0.2:80;
            check interval=5000 rise=1 fall=3 timeout=4000;
        }

        server {
            listen 50080;

            location / {
                proxy_pass http://cluster;
            }

            location /status {
                check_status;

                access_log   off;
           }
        }

    }
<end config file>

<begin commands>
nginx -p /tmp/test -c test.conf
curl http://localhost:50080/status
<end commands>

And logs show:
<begin logs>
2020/06/23 12:35:00 [debug] 20275#101354: worker cycle
2020/06/23 12:35:00 [debug] 20275#101354: kevent timer: 60000, changes: 1
2020/06/23 12:35:00 [debug] 20275#101354: kevent events: 1
2020/06/23 12:35:00 [debug] 20275#101354: kevent: 3: ft:-1 fl:0020 ff:00000000 d:85 ud:000000080680D111
2020/06/23 12:35:00 [debug] 20275#101354: *3 http wait request handler
2020/06/23 12:35:00 [debug] 20275#101354: *3 malloc: 0000000802270000:1024
2020/06/23 12:35:00 [debug] 20275#101354: *3 recv: eof:0, avail:85, err:0
2020/06/23 12:35:00 [debug] 20275#101354: *3 recv: fd:3 85 of 1024
2020/06/23 12:35:00 [debug] 20275#101354: *3 reusable connection: 0
2020/06/23 12:35:00 [debug] 20275#101354: *3 posix_memalign: 000000080227F000:4096 @16
2020/06/23 12:35:00 [debug] 20275#101354: *3 http process request line
2020/06/23 12:35:00 [debug] 20275#101354: *3 http request line: "GET /status HTTP/1.1"
2020/06/23 12:35:00 [debug] 20275#101354: *3 http uri: "/status"
2020/06/23 12:35:00 [debug] 20275#101354: *3 http args: ""
2020/06/23 12:35:00 [debug] 20275#101354: *3 http exten: ""
2020/06/23 12:35:00 [debug] 20275#101354: *3 posix_memalign: 000000080227B000:4096 @16
2020/06/23 12:35:00 [debug] 20275#101354: *3 http process request header line
2020/06/23 12:35:00 [debug] 20275#101354: *3 http header: "Host: localhost:50080"
2020/06/23 12:35:00 [debug] 20275#101354: *3 http header: "User-Agent: curl/7.67.0"
2020/06/23 12:35:00 [debug] 20275#101354: *3 http header: "Accept: */*"
2020/06/23 12:35:00 [debug] 20275#101354: *3 http header done
2020/06/23 12:35:00 [debug] 20275#101354: *3 event timer del: 3: 7021985672
2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 0
2020/06/23 12:35:00 [debug] 20275#101354: *3 rewrite phase: 1
2020/06/23 12:35:00 [debug] 20275#101354: *3 test location: "/"
2020/06/23 12:35:00 [debug] 20275#101354: *3 test location: "status"
2020/06/23 12:35:00 [debug] 20275#101354: *3 using configuration "/status"
2020/06/23 12:35:00 [debug] 20275#101354: *3 http cl:-1 max:1048576
2020/06/23 12:35:00 [debug] 20275#101354: *3 rewrite phase: 3
2020/06/23 12:35:00 [debug] 20275#101354: *3 rewrite phase: 4
2020/06/23 12:35:00 [debug] 20275#101354: *3 post rewrite phase: 5
2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 6
2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 7
2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 8
2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 9
2020/06/23 12:35:00 [debug] 20275#101354: *3 access phase: 10
2020/06/23 12:35:00 [debug] 20275#101354: *3 access phase: 11
2020/06/23 12:35:00 [debug] 20275#101354: *3 access phase: 12
2020/06/23 12:35:00 [debug] 20275#101354: *3 post access phase: 13
2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 14
2020/06/23 12:35:00 [debug] 20275#101354: *3 generic phase: 15
2020/06/23 12:35:00 [debug] 20275#101354: *3 http set discard body
2020/06/23 12:35:00 [error] 20275#101354: *3 http upstream check module can not find any check server, make sure you've added the check servers, client: 127.0.0.1, server: , request: "GET /status HTTP/1.1", host: "localhost:50080"
2020/06/23 12:35:00 [debug] 20275#101354: *3 http finalize request: 500, "/status?" a:1, c:1
2020/06/23 12:35:00 [debug] 20275#101354: *3 http special response: 500, "/status?"
2020/06/23 12:35:00 [debug] 20275#101354: *3 http set discard body
2020/06/23 12:35:00 [debug] 20275#101354: *3 HTTP/1.1 500 Internal Server Error
Server: nginx/1.18.0
Date: Tue, 23 Jun 2020 19:35:00 GMT
Content-Type: text/html
Content-Length: 177
Connection: close
<end logs>

Looks as if it wasn't installed properly.
Comment 1 Jochen Neumeister freebsd_committer freebsd_triage 2020-07-03 21:32:59 UTC
The module was tested with Poudriere and a test server (FreeBSD 12.1 / amd64) before release. There were no problems.

If you want me to test your configuration here in more detail, please contact me privately.
Comment 2 Felipe Zipitria 2020-07-05 00:42:57 UTC
Just to confirm that using a vanilla VM and building from ports everything worked as expected. So this should be a particular problem, not related to the package. Will do better testing next time, sorry.
Comment 3 Adam Strohl 2020-07-15 22:20:20 UTC
I would add that we see this issue as well with our our poudriere pkg repo -- anyone have any ideas what's causing this?
Comment 4 Jochen Neumeister freebsd_committer freebsd_triage 2020-07-15 23:25:36 UTC
(In reply to Adam Strohl from comment #3)

see #2
Comment 5 Adam Strohl 2020-07-16 00:58:19 UTC
(In reply to Jochen Neumeister from comment #4)

I don't see what was actually wrong in comment #2, just that building it from ports apparently works?
Comment 6 Adam Strohl 2020-08-05 01:52:26 UTC
Can confirm that this is 100% broken if you build with Poudriere however the port itself works.

Are Poudriere builds not supported?  How is the FreeBSD project building this successfully for it's own repo?
Comment 7 Ganbold Tsagaankhuu freebsd_committer freebsd_triage 2020-08-12 14:22:09 UTC
r538078 works, it is just before making upstream check module dynamic, 
but just next r538085 is not working. 
r538085 supposed to convert upstream check module to dynamic, but it broke the module.
Comment 8 Adam Strohl 2020-08-12 19:13:19 UTC
Counter to my previous comment, building from latest clean ports directly and only enabling this checkbox in make config (leave all other defaults) does NOT work.

So the port itself is broken in terms of this feature.  It will build and run, but not work correctly.

It is not poudriere specific, doing 'make build install' via /usr/ports/www/nginx shows the issue too.
Comment 9 Jochen Neumeister freebsd_committer freebsd_triage 2020-08-12 20:33:21 UTC
I need more information here to reproduce it on a test VM: 
-Are there any error messages?
- Which configurations are used?
-Good would be a list of "make showconfig"
Comment 10 Adam Strohl 2020-08-12 20:45:27 UTC
(In reply to Jochen Neumeister from comment #9)

Everything will look fine, but if you actually go to query the module (via it's reporting endpoint) it reports all peers down for sites using it.  If you're not familiar with setting it up let me know and I can give you some examples.

Overall unless you're using it for sites (you need to specifically configure it per site) and monitoring it you'd never know it's broken.  The web server works fine for sites not using it, etc.

Config:

===> The following configuration options are available for nginx-1.18.0_22,2:
     DEBUG=off: Build with debugging support
     DEBUGLOG=off: Enable debug log (--with-debug)
     DSO=on: Enable dynamic modules support
     FILE_AIO=off: Enable file aio
     IPV6=on: Enable IPv6 support
     NJS=off: Enable http_javascript module
     THREADS=on: Enable threads support
     WWW=on: Enable html sample files
====> Modules that require MAIL module
     MAIL=off: Enable IMAP4/POP3/SMTP proxy module
     MAIL_IMAP=off: Enable IMAP4 proxy module
     MAIL_POP3=off: Enable POP3 proxy module
     MAIL_SMTP=off: Enable SMTP proxy module
     MAIL_SSL=off: Enable mail_ssl module
====> Modules that require HTTP module
     GOOGLE_PERFTOOLS=off: Enable google perftools module
     HTTP=on: Enable HTTP module
     HTTP_ADDITION=off: Enable http_addition module
     HTTP_AUTH_REQ=off: Enable http_auth_request module
     HTTP_CACHE=on: Enable http_cache module
     HTTP_DAV=off: Enable http_webdav module
     HTTP_FLV=off: Enable http_flv module
     HTTP_GUNZIP_FILTER=off: Enable http_gunzip_filter module
     HTTP_GZIP_STATIC=off: Enable http_gzip_static module
     HTTP_IMAGE_FILTER=off: Enable http_image_filter module
     HTTP_MP4=off: Enable http_mp4 module
     HTTP_PERL=off: Enable http_perl module
     HTTP_RANDOM_INDEX=off: Enable http_random_index module
     HTTP_REALIP=off: Enable http_realip module
     HTTP_REWRITE=on: Enable http_rewrite module
     HTTP_SECURE_LINK=off: Enable http_secure_link module
     HTTP_SLICE=on: Enable http_slice module
     HTTP_SLICE_AHEAD=off: Enable http_slice_ahead module
     HTTP_SSL=on: Enable http_ssl module
     HTTP_STATUS=on: Enable http_stub_status module
     HTTP_SUB=off: Enable http_sub module
     HTTP_XSLT=off: Enable http_xslt module
     HTTPV2=on: Enable HTTP/2 protocol support (SSL req.)
     STREAM=on: Enable stream module
     STREAM_SSL=on: Enable stream_ssl module (SSL req.)
     STREAM_SSL_PREREAD=on: Enable stream_ssl_preread module (SSL req.)
     AJP=off: 3rd party ajp module
     AWS_AUTH=off: 3rd party aws auth module
     BROTLI=off: 3rd party brotli module
     CACHE_PURGE=on: 3rd party cache_purge module
     CLOJURE=off: 3rd party clojure module
     CT=off: 3rd party cert_transparency module (SSL req.)
     DEVEL_KIT=off: 3rd party Nginx Development Kit module
     ARRAYVAR=off: 3rd party array_var module
     DRIZZLE=off: 3rd party drizzle module
     DYNAMIC_TLS=off: 3rd party dynamic tls records patch
     DYNAMIC_UPSTREAM=off: 3rd party dynamic_upstream module
     ECHO=off: 3rd party echo module
     ENCRYPTSESSION=off: 3rd party encrypted_session module
     FORMINPUT=off: 3rd party form_input module
     GRIDFS=off: 3rd party gridfs module
     HEADERS_MORE=off: 3rd party headers_more module
     HTTP_ACCEPT_LANGUAGE=off: 3rd party accept_language module
     HTTP_AUTH_DIGEST=off: 3rd party http_authdigest module
     HTTP_AUTH_JWT=off: 3rd party http_auth_jwt module
     HTTP_AUTH_KRB5=off: 3rd party http_auth_gss module
     HTTP_AUTH_LDAP=on: 3rd party http_auth_ldap module
     HTTP_AUTH_PAM=off: 3rd party http_auth_pam module
     HTTP_DAV_EXT=off: 3rd party webdav_ext module
     HTTP_EVAL=off: 3rd party eval module
     HTTP_FANCYINDEX=off: 3rd party http_fancyindex module
     HTTP_FOOTER=off: 3rd party http_footer module
     HTTP_GEOIP2=off: 3rd party geoip2 module
     HTTP_IP2LOCATION=off: 3rd party ip2location-nginx module
     HTTP_IP2PROXY=off: 3rd party ip2proxy-nginx module
     HTTP_JSON_STATUS=off: 3rd party http_json_status module
     HTTP_MOGILEFS=off: 3rd party mogilefs module
     HTTP_MP4_H264=off: 3rd party mp4/h264 module
     HTTP_NOTICE=off: 3rd party notice module
     HTTP_PUSH=off: 3rd party push module
     HTTP_PUSH_STREAM=off: 3rd party push stream module
     HTTP_REDIS=off: 3rd party http_redis module
     HTTP_RESPONSE=off: 3rd party http_response module
     HTTP_SUBS_FILTER=off: 3rd party subs filter module
     HTTP_TARANTOOL=off: 3rd party tarantool upstream module
     HTTP_UPLOAD=off: 3rd party upload module
     HTTP_UPLOAD_PROGRESS=off: 3rd party uploadprogress module
     HTTP_UPSTREAM_CHECK=on: 3rd party upstream check module
     HTTP_UPSTREAM_FAIR=off: 3rd party upstream fair module
     HTTP_UPSTREAM_STICKY=off: 3rd party upstream sticky module
     HTTP_VIDEO_THUMBEXTRACTOR=off: 3rd party video_thumbextractor module
     HTTP_ZIP=off: 3rd party http_zip module
     ICONV=off: 3rd party iconv module
     LET=off: 3rd party let module
     LINK=off: 3rd party link function module
     LUA=off: 3rd party lua module
     MEMC=off: 3rd party memc (memcached) module
     MODSECURITY3=off: 3rd party modsecurity3 module
     NAXSI=off: 3rd party naxsi module
     OPENTRACING=off: 3rd party opentracing module
     PASSENGER=off: 3rd party passenger module
     POSTGRES=off: 3rd party postgres module
     RDS_CSV=off: 3rd party rds_csv module
     RDS_JSON=off: 3rd party rds_json module
     REDIS2=off: 3rd party redis2 module
     RTMP=off: 3rd party rtmp module
     SET_MISC=off: 3rd party set_misc module
     SFLOW=off: 3rd party sflow module
     SHIBBOLETH=off: 3rd party shibboleth module
     SLOWFS_CACHE=off: 3rd party slowfs_cache module
     SMALL_LIGHT=off: 3rd party small_light module
     SRCACHE=off: 3rd party srcache module
     VOD=off: 3rd party vod module
     VTS=off: 3rd party vts module
     XSS=off: 3rd party xss module
     WEBSOCKIFY=off: 3rd party websockify module
====> GSSAPI implementation (imply HTTP_AUTH_KRB5): you can only select none or one of them
     GSSAPI_BASE=off: GSSAPI support via base system (needs Kerberos)
     GSSAPI_HEIMDAL=off: GSSAPI support via security/heimdal
     GSSAPI_MIT=off: GSSAPI support via security/krb5
===> Use 'make config' to modify these settings
Comment 11 Jochen Neumeister freebsd_committer freebsd_triage 2020-08-17 18:02:10 UTC
I'm setting up a test environment. I'm afraid that's gonna take some time.

Two questions:

- has it been tested with nginx-devel? If so, what was the result here?

- during the change the patch "files/extra-patch-nginx_upstream_check_module-config" was added. Was this patch deleted for test purposes?
Comment 12 Ganbold Tsagaankhuu freebsd_committer freebsd_triage 2020-08-18 04:30:13 UTC
Test environment should be easy. You can follow/use Felipe's config with nginx and log should show:

2020/06/23 12:35:00 [error] 20275#101354: *3 http upstream check module can not find any check server, make sure you've added the check servers, client: 127.0.0.1, server: , request: "GET /status HTTP/1.1", host: "localhost:50080"

Please note in above log that server is empty -> "server: ,", so it is like not getting config lines or something.

Without making upstream check module dynamic, it will not show above error and it would work as expected.

We didn't test nginx-devel.

We didn't delete files/extra-patch-nginx_upstream_check_module-config file during test. I didn't look closely the content of it, but if it makes the module dynamic then removing it may make it static and thus it would work. Also maybe Makefile.extmod needs to have upstream related changes reverted.
Comment 13 Ganbold Tsagaankhuu freebsd_committer freebsd_triage 2020-08-19 06:11:39 UTC
nginx-devel doesn't work either.

deleted files/extra-patch-nginx_upstream_check_module-config and tried.
It gives:

pkg-static: Unable to access file /usr/ports/www/nginx/work/stage/usr/local/libexec/nginx/ngx_http_upstream_check_module.so:No such file or directory
*** Error code 74

Although it didn't install properly, it can be run from work/nginx-1.18.0/objs/
and it works fine:

result of curl:

root@beastie:/usr/ports/www/nginx # curl http://localhost:50080/status
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <title>Nginx http upstream check status</title>
</head>
<body>
<h1>Nginx http upstream check status</h1>
<h2>Check upstream server number: 2, generation: 1</h2>
<table style="background-color:white" cellspacing="0"        cellpadding="3" border="1">
  <tr bgcolor="#C0C0C0">
    <th>Index</th>
    <th>Upstream</th>
    <th>Name</th>
    <th>Status</th>
    <th>Rise counts</th>
    <th>Fall counts</th>
    <th>Check type</th>
    <th>Check port</th>
  </tr>
  <tr bgcolor="#FF0000">
    <td>0</td>
    <td>cluster</td>
    <td>192.168.1.8:80</td>
    <td>down</td>
    <td>0</td>
    <td>17</td>
    <td>tcp</td>
    <td>0</td>
  </tr>
  <tr bgcolor="#FF0000">
    <td>1</td>
    <td>cluster</td>
    <td>192.168.1.9:80</td>
    <td>down</td>
    <td>0</td>
    <td>17</td>
    <td>tcp</td>
    <td>0</td>
  </tr>
</table>
</body>
</html>
Comment 14 Jochen Neumeister freebsd_committer freebsd_triage 2020-08-21 10:05:01 UTC
(In reply to Ganbold Tsagaankhuu from comment #13)

ok, then the bug seems to be in the patch
Comment 15 Sergey A. Osokin freebsd_committer freebsd_triage 2020-08-23 03:07:13 UTC
According to official documentation not all modules can be converted from static to dynamic.  Especially those that patching nginx core.

I see several options here until we find a way how to fix that:
- convert this module to static back;
- use another modules, like dynamic_healthcheck, it's available as dynamic module 
  with the www/nginx-devel port.
Comment 16 Ganbold Tsagaankhuu freebsd_committer freebsd_triage 2020-08-23 08:02:29 UTC
Please make this module static as before. That way both options would be available to users.
Comment 17 Sergey A. Osokin freebsd_committer freebsd_triage 2020-08-23 15:22:21 UTC
Created attachment 217455 [details]
upstream_check static module

Hi,

Here is the patch to make upstream_check module static.
Please go ahead and test it, let me know how it works.
Comment 18 Ganbold Tsagaankhuu freebsd_committer freebsd_triage 2020-08-24 02:41:24 UTC
Created attachment 217470 [details]
Patch for nginx

nginx-devel patch works for me.

Here is the patch that works with nginx port.
Comment 19 commit-hook freebsd_committer freebsd_triage 2020-08-24 02:59:40 UTC
A commit references this bug:

Author: osa
Date: Mon Aug 24 02:58:56 UTC 2020
New revision: 546038
URL: https://svnweb.freebsd.org/changeset/ports/546038

Log:
  Partially revert back r533146 cause upstream_check third-party
  dynamic module can't work as expected, so let's keep it static
  for now.

  Bump PORTREVISION.

  PR:	247523

Changes:
  head/www/nginx-devel/Makefile
  head/www/nginx-devel/Makefile.extmod
  head/www/nginx-devel/files/extra-patch-nginx_upstream_check_module-config
  head/www/nginx-devel/pkg-plist
Comment 20 Ganbold Tsagaankhuu freebsd_committer freebsd_triage 2020-08-24 03:23:53 UTC
Can you revert back nginx port?

thanks,
Comment 21 commit-hook freebsd_committer freebsd_triage 2020-08-24 12:25:15 UTC
A commit references this bug:

Author: joneum
Date: Mon Aug 24 12:25:08 UTC 2020
New revision: 546072
URL: https://svnweb.freebsd.org/changeset/ports/546072

Log:
  Partially revert back r538085 cause upstream_check third-party
  dynamic module can't work as expected, so let's keep it static for now.

  PR:		247523
  Sponsored by:	Netzkommune GmbH

Changes:
  head/www/nginx/Makefile
  head/www/nginx/Makefile.extmod
  head/www/nginx/files/extra-patch-nginx_upstream_check_module-config
  head/www/nginx/pkg-plist