Created attachment 221833 [details] ntopng build changes Hi, I have this reproducible error in our 12.1 builds and looking at the fix it seems something that should be done upstream. For one stage-qa says dependencies are missing for the gpg libraries used and additionally we need to point ndpi build to gpg-error lib as well as gcrypt. Cheers, Franco
Hi, I've made a quick test in poudriere and could not reproduce your findings. This prompts me to say that, before adding dependencies, these findings require more scrutiny. I'm not doubting what you see, but I suspect other variables are influencing your findings. First question that comes to mind is, are you building with any custom options? Also on the ports on which ntopng and ndpi depend. It is quite possible that the extra dependencies are pulled in indirectly by some other library optional code which makes it link to gpg-error or gcrypt. This is difficult to address in the ports tree, but adding an unconditional dependency would be wrong, it would make ntopng always pull in gpg-error even if not actually using it. Anyway if the dependency is indirect and the library actually directly linking to it is properly pulling in the dependency there is no risk of it missing. Unluckily the ports tree gives no instrument to properly manage the above scenario. Could you post the output of "ldd -a `which ntopng`"? The -a option will show the full tree of dependencies, so we can differentiate between direct and indirect ones, and identify which dependent library pull gpg-error and/or gcrypt in.
I'm not using poudriere. Cheers, Franco
(In reply to Franco Fichtner from comment #2) > I'm not using poudriere. Ok, that means the gpg-error and gcrypt dependencies can be triggered by the simple presence of the libaries on your system. But we still don't know exactly which ports directly links to them. It could be ntopng or any of its dependencies. Could you please provide the output of "ldd -a `which ntopng`" (adapt the command if needed) as I asked in comment #1? That would help me better understand the situation and fix it properly. Any fix I commit must work as intended both on live systems using ports and in poudriere, adding unconditional dependencies would not be fine for poudriere package building.
It's ndpi. And since ntopng links to ndpi it needs these libraries as well, also in DEPENDS as per FreeBSD ports rules. So ntopng already pulls in ndpi using -libcrypt explicitly and what other proper solution is there? Cheers, Franco
Sorry, I meant -lgcrypt.
You can inspect the error here: https://nightly.opnsense.org/20.7/amd64/logs/202101220005/08-ports-OpenSSL.log.err Specifically: In file included from src/nDPIStats.cpp:22: In file included from /usr/obj/usr/ports/net/ntopng/work/ntopng-d2f4f8f/include/ntop_includes.h:118: In file included from /usr/local/include/ndpi/ndpi_api.h:28: In file included from /usr/local/include/ndpi/ndpi_main.h:28: In file included from /usr/local/include/ndpi/ndpi_define.h:184: /usr/local/include/ndpi/ndpi_config.h:98:9: warning: 'PACKAGE_VERSION' macro redefined [-Wmacro-redefined] #define PACKAGE_VERSION "3.4.0" ^ /usr/obj/usr/ports/net/ntopng/work/ntopng-d2f4f8f/include/config.h:163:9: note: previous definition is here #define PACKAGE_VERSION "4.2.210122" ^ 4 warnings generated. 4 warnings generated. 4 warnings generated. 4 warnings generated. 4 warnings generated. 4 warnings generated. 4 warnings generated. ld: error: undefined symbol: gpg_strerror >>> referenced by visibility.c >>> libgcrypt_la-visibility.o:(gcry_strerror) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_strsource >>> referenced by visibility.c >>> libgcrypt_la-visibility.o:(gcry_strsource) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_err_code_from_errno >>> referenced by visibility.c >>> libgcrypt_la-visibility.o:(gcry_err_code_from_errno) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_err_code_from_errno >>> referenced by visibility.c >>> libgcrypt_la-visibility.o:(gcry_err_code_to_errno) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_err_code_from_errno >>> referenced by visibility.c >>> libgcrypt_la-visibility.o:(gcry_err_make_from_errno) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_err_code_from_errno >>> referenced by visibility.c >>> libgcrypt_la-visibility.o:(gcry_error_from_errno) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_strerror >>> referenced by misc.c >>> libgcrypt_la-misc.o:(_gcry_fatal_error) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_err_set_errno >>> referenced by misc.c >>> libgcrypt_la-misc.o:(_gcry_strtokenize) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_err_set_errno >>> referenced by misc.c >>> libgcrypt_la-misc.o:(_gcry_divide_by_zero) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpg_err_code_from_errno >>> referenced by misc.c >>> libgcrypt_la-misc.o:(_gcry_divide_by_zero) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_get_syscall_clamp >>> referenced by global.c >>> libgcrypt_la-global.o:(global_init) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fopenmem >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fprintf >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fprintf >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fprintf >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fprintf >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fprintf >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fprintf >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fprintf >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: undefined symbol: gpgrt_fprintf >>> referenced by global.c >>> libgcrypt_la-global.o:(_gcry_get_config) in archive /usr/local/lib/libgcrypt.a ld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors) c++: error: linker command failed with exit code 1 (use -v to see invocation) gmake[1]: *** [Makefile:151: ntopng] Error 1
(In reply to Franco Fichtner from comment #4) > It's ndpi. And since ntopng links to ndpi it needs these libraries as well, > also in DEPENDS as per FreeBSD ports rules. It would if nDPI by default links to libgcrypt, which it does not in my tests, so I need to understand why it does for you before I add any dependencies for everyone when they are nt necessarily needed by everyone. > So ntopng already pulls in ndpi using -libcrypt explicitly and what other > proper solution is there? Can you please attach a full build log showing the issue on a FreeBSD machine? (and not paste, because bugzilla tends to mangle pasted logs making them unreadable). I need to understand what is going on. I have not reproduced your issue and builds I've done show no missing libraries here. I am responsible for the commits I make and will not commit a change until I'm sure it is correct for ALL users. Also please don't send me OPNSense logs. Their build system is based on FreeBSD but adds a lot of parts. If your issue is specific to OPNSense, please report the bug to the OPNSense people. I don't use OPNSense and don't have the ability to test builds using their tools.
(In reply to Franco Fichtner from comment #6) This log is from another operating system with a custom build system, both are based respectively on FreeBSD and the FreeBSD ports system but are highly modified and customized. Also this log is only partial, not showing the actual errors and command lines used. It has no utility to me as is.
Take it or leave it but lose unprofessional attitude. ;) Cheers, Franco
Anyway I think I now understand what is going on. The build you are using (which is non standard) is explicitly building ndpi with libgcrupt support. Default builds don't do that. nDPI by default has no dependency on libgcrypt and should not link to it. Adding an unconditional dependency could be a solution but I need to investigate it. In theory the ideal solution would be to add an option to nDPI to enable/disable gcrypt support and make the software forcibly use or not use it depending on the option. Unluckily FreeBSD ports tree has no provision for checking a dependency options from another port, so the indirect dependencies on the ntopng port can only reflect the defaults set in nDPI. Adding libgcrypt as a dependency only in ntopng would not fix the issue correctly, since we would miss the direct dependency in nDPI. So you DID find a bug that needs fixing, which is triggered by non default conditions (which require being catered for anyway) but due to various reasons it's somewhat more nuanced to fix it than your patch. Please give me some time to produce a fix and test it.
With all due respect, I am not here to waste your time. When I add a ticket to bugs.freebsd.org I know the issue isn't completely local to OPNsense so I always try to add a patch, even if it is just a POC or a base for discussion. Poudriere is nice and all, but a simple "make package" even if it takes hours is something that needs to be addressed in some form. Whether or not there are scripts around "make package" does not change the fact that "make package" was called. These are the basics even poudriere requires. The trend in FreeBSD ports is worrisome to say the least, but that is just my opinion. To add context: this only happens with since latest updates of ndpi and ntopng. Other than this it's likely irrelevant to exchange FUD and only costs both of our time. Cheers, Franco
FWIW, I added the DEPENDS because stage-qa complained about them so I took some extra care verifying both libraries, but merely adjusting the library inclusion in the build patch fixes this fine on its own and is also what upstream would likely include since lgcrytp itself wants lgpg-error so either drop one or add the other. Either way not a big deal here to address I suppose. Cheers, Franco
I did not mean to sound rude. Sorry if that's what appeared. I already stated the bug report is valid and I'm addressing it. The correct solution, addressing this without regressions for all uses of the ports tree is more nuanced than only adding a dependency to ntopng. I'm working on the full patch and will commit it to the ports tree as soon as I have it ready.
A commit references this bug: Author: madpilot Date: Mon Jan 25 18:03:30 UTC 2021 New revision: 562593 URL: https://svnweb.freebsd.org/changeset/ports/562593 Log: - Add unconditional dependency on gcrypt and libgpg-error to ndpi and ntopng to ensure full feature set [1] - Patch ntopng to link correctly with libgcrypt - While here, update ntopng to latest upstream snapshot PR: 252935 [1] Submitted by: Franco Fichtner <franco@opnsense.org> MFH: 2021Q1 Changes: head/net/ndpi/Makefile head/net/ntopng/Makefile head/net/ntopng/distinfo head/net/ntopng/files/patch-configure.seed
A commit references this bug: Author: madpilot Date: Mon Jan 25 18:04:30 UTC 2021 New revision: 562594 URL: https://svnweb.freebsd.org/changeset/ports/562594 Log: MFH: r562593 - Add unconditional dependency on gcrypt and libgpg-error to ndpi and ntopng to ensure full feature set [1] - Patch ntopng to link correctly with libgcrypt - While here, update ntopng to latest upstream snapshot PR: 252935 [1] Submitted by: Franco Fichtner <franco@opnsense.org> Changes: _U branches/2021Q1/ branches/2021Q1/net/ndpi/Makefile branches/2021Q1/net/ntopng/Makefile branches/2021Q1/net/ntopng/distinfo branches/2021Q1/net/ntopng/files/patch-configure.seed
Patch adding gcrypt dependency and fixing build committed. Thanks!
Thank you, I appreciate your work.