The author recently release a new version that fixes compilation with nginx 1.11.1 or higher.
A commit references this bug: Author: osa Date: Sun Jul 31 23:05:38 UTC 2016 New revision: 419419 URL: https://svnweb.freebsd.org/changeset/ports/419419 Log: Upgrade third-party rtmp module from 1.1.7 to 1.1.8. Remove needless patch. PR: 211487 Changes: head/www/nginx/Makefile head/www/nginx/distinfo head/www/nginx-devel/Makefile head/www/nginx-devel/distinfo head/www/nginx-devel/files/extra-patch-ngx_rtmp_core_module.c
Thanks for report!