Created attachment 232674 [details] glib addition A plain ports tree build in a clean 13-STABLE chroot fails with: clang++ -O2 -pipe -fstack-protector-strong -DLDAP_DEPRECATED -isystem /usr/local/include -fno-strict-aliasing -isystem /usr/local/include -std=c++0x -Wl,-rpath,/usr/local/lib -fstack-protector-strong src/AddressResolution.o src/AddressTree.o src/AlertCounter.o src/AlertableEntity.o src/AlertsQueue.o src/AutonomousSystem.o src/AutonomousSystemHash.o src/Bitmap128.o src/Bitmask.o src/Bloom.o src/BroadcastDomains.o src/Check.o src/ChecksLoader.o src/Condvar.o src/ContainerStats.o src/ContinuousPing.o src/ContinuousPingStats.o src/CountriesHash.o src/Country.o src/DB.o src/DSCPStats.o src/DnsStats.o src/DummyInterface.o src/ElasticSearch.o src/EthStats.o src/ExportInterface.o src/Fingerprint.o src/Flow.o src/FlowAlert.o src/FlowAlertsLoader.o src/FlowCheck.o src/FlowChecksExecutor.o src/FlowChecksLoader.o src/FlowGrouper.o src/FlowHash.o src/FlowRiskAlerts.o src/FlowStats.o src/FlowTrafficStats.o src/FrequentStringItems.o src/GenericHash.o src/GenericHashEntry.o src/GenericTrafficElement.o src/Geolocation.o src/HTTPserver.o src/HTTPstats.o src/Host.o src/HostAlert.o src/HostAlertableEntity.o src/HostCheck.o src/HostChecksExecutor.o src/HostChecksLoader.o src/HostHash.o src/HostPoolStats.o src/HostPools.o src/HostStats.o src/ICMPinfo.o src/ICMPstats.o src/IEC104Stats.o src/InfluxDBTimeseriesExporter.o src/InterarrivalStats.o src/InterfaceStatsHash.o src/IpAddress.o src/L4Stats.o src/ListeningPorts.o src/LocalHost.o src/LocalHostStats.o src/LocalTrafficStats.o src/LuaEngine.o src/LuaEngineInterface.o src/LuaEngineNetwork.o src/LuaEngineNtop.o src/MDNS.o src/Mac.o src/MacHash.o src/MacManufacturers.o src/MacStats.o src/MostVisitedList.o src/Mutex.o src/MySQLDB.o src/NetworkDiscovery.o src/NetworkInterface.o src/NetworkStats.o src/Ntop.o src/NtopGlobals.o src/ObservationPoint.o src/ObservationPointHash.o src/OperatingSystem.o src/OperatingSystemHash.o src/OtherAlertableEntity.o src/PF_RINGInterface.o src/PacketDumper.o src/PacketDumperTuntap.o src/PacketStats.o src/Paginator.o src/ParsedFlow.o src/ParsedFlowCore.o src/ParsedeBPF.o src/ParserInterface.o src/PartializableFlowTrafficStats.o src/PcapInterface.o src/PeriodicActivities.o src/PeriodicScript.o src/Ping.o src/Prefs.o src/ProtoStats.o src/RRDTimeseriesExporter.o src/RecipientQueues.o src/Recipients.o src/Redis.o src/RemoteHost.o src/RoundTripStats.o src/RwLock.o src/SNMP.o src/SQLiteAlertStore.o src/SQLiteStoreManager.o src/Score.o src/ScoreStats.o src/SerializableElement.o src/StatsManager.o src/SyslogCollectorInterface.o src/SyslogDump.o src/SyslogLuaEngine.o src/SyslogParserInterface.o src/SyslogStats.o src/TcpFlowStats.o src/TcpPacketStats.o src/ThreadPool.o src/ThreadedActivity.o src/ThreadedActivityStats.o src/ThroughputStats.o src/TimelineExtract.o src/TimeseriesExporter.o src/Trace.o src/TrafficStats.o src/Utils.o src/VLAN.o src/VLANAddressTree.o src/VLANHash.o src/ViewInterface.o src/ViewScoreStats.o src/VirtualHost.o src/VirtualHostHash.o src/ZCCollectorInterface.o src/ZMQCollectorInterface.o src/ZMQParserInterface.o src/ZMQPublisher.o src/main.o src/nDPIStats.o src/flow_checks/BlacklistedCountry.o src/flow_checks/BlacklistedFlow.o src/flow_checks/BroadcastNonUDPTraffic.o src/flow_checks/DeviceProtocolNotAllowed.o src/flow_checks/ExternalAlertCheck.o src/flow_checks/FlowRisk.o src/flow_checks/IECUnexpectedTypeId.o src/flow_checks/LowGoodputFlow.o src/flow_checks/NotPurged.o src/flow_checks/RemoteAccess.o src/flow_checks/RemoteToLocalInsecureProto.o src/flow_checks/RemoteToRemote.o src/flow_checks/TCPNoDataExchanged.o src/flow_checks/TCPZeroWindow.o src/flow_checks/UDPUnidirectional.o src/flow_checks/UnexpectedServer.o src/flow_checks/WebMining.o src/flow_alerts/BlacklistedCountryAlert.o src/flow_alerts/BlacklistedFlowAlert.o src/flow_alerts/BroadcastNonUDPTrafficAlert.o src/flow_alerts/DeviceProtocolNotAllowedAlert.o src/flow_alerts/ElephantFlowAlert.o src/flow_alerts/ExternalAlertCheckAlert.o src/flow_alerts/FlowRiskBinaryApplicationTransferAlert.o src/flow_alerts/FlowRiskKnownProtocolOnNonStandardPortAlert.o src/flow_alerts/FlowRiskSSHObsoleteClientAlert.o src/flow_alerts/FlowRiskSSHObsoleteServerAlert.o src/flow_alerts/FlowRiskSimpleAlert.o src/flow_alerts/FlowRiskSuspiciousDGADomainAlert.o src/flow_alerts/FlowRiskTLSCertificateSelfSignedAlert.o src/flow_alerts/FlowRiskTLSOldProtocolVersionAlert.o src/flow_alerts/IECInvalidTransitionAlert.o src/flow_alerts/IECUnexpectedTypeIdAlert.o src/flow_alerts/LongLivedFlowAlert.o src/flow_alerts/LowGoodputFlowAlert.o src/flow_alerts/PeriodicityChangedAlert.o src/flow_alerts/RemoteAccessAlert.o src/flow_alerts/RemoteToLocalInsecureProtoAlert.o src/flow_alerts/TLSMaliciousSignatureAlert.o src/flow_alerts/UnexpectedServerAlert.o src/host_checks/CountriesContacts.o src/host_checks/DNSServerContacts.o src/host_checks/DNSTraffic.o src/host_checks/DangerousHost.o src/host_checks/DomainNamesContacts.o src/host_checks/FlowFlood.o src/host_checks/FlowHits.o src/host_checks/ICMPFlood.o src/host_checks/NTPServerContacts.o src/host_checks/NTPTraffic.o src/host_checks/P2PTraffic.o src/host_checks/PktThreshold.o src/host_checks/RemoteConnection.o src/host_checks/SMTPServerContacts.o src/host_checks/SYNFlood.o src/host_checks/SYNScan.o src/host_checks/ScoreThreshold.o src/host_checks/ServerContacts.o src/host_alerts/CountriesContactsAlert.o src/host_alerts/DNSServerContactsAlert.o src/host_alerts/DNSTrafficAlert.o src/host_alerts/DangerousHostAlert.o src/host_alerts/DomainNamesContactsAlert.o src/host_alerts/FlowHitsAlert.o src/host_alerts/NTPServerContactsAlert.o src/host_alerts/NTPTrafficAlert.o src/host_alerts/P2PTrafficAlert.o src/host_alerts/PktThresholdAlert.o src/host_alerts/RemoteConnectionAlert.o src/host_alerts/SMTPServerContactsAlert.o src/host_alerts/ServerContactsAlert.o -Wall -Wl,-Bstatic -L/usr/local/lib -lndpi -lrrd -lm -lgcrypt -Wl,-Bdynamic -lpcap -llua-5.4 -L/usr/local/lib -lrrd /usr/local/lib/libjson-c.a -lmaxminddb -lsqlite3 /usr/local/lib/mysql/libmysqlclient.a -lexpat -L/usr/local/lib -lssl -lssl -lcrypto -Wl,-rpath,/usr/local/lib -fstack-protector-strong -L/usr/local/lib -lmaxminddb -lzmq -L/usr/local/lib -lgpg-error -lzmq -lsodium -L/usr/local/lib -lgcrypt -lldap -llber -lrt -lz -ldl -lcurl /usr/local/lib/libzstd.a -lm -lpthread -o ntopng ld: error: undefined symbol: g_list_append >>> referenced by librrdupd_la-rrd_create.o:(rrd_create) in archive /usr/local/lib/librrd.a >>> referenced by librrdupd_la-rrd_create.o:(rrd_create_r2) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_list_free_full >>> referenced by librrdupd_la-rrd_create.o:(rrd_create) in archive /usr/local/lib/librrd.a >>> referenced by librrdupd_la-rrd_create.o:(rrd_create_r2) in archive /usr/local/lib/librrd.a >>> referenced by librrdupd_la-rrd_create.o:(rrd_create_r2) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_list_length >>> referenced by librrdupd_la-rrd_create.o:(rrd_create) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_regex_new >>> referenced by librrdupd_la-rrd_create.o:(parseDS) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_match_info_free >>> referenced by librrdupd_la-rrd_create.o:(parseDS) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_regex_unref >>> referenced by librrdupd_la-rrd_create.o:(parseDS) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_regex_match >>> referenced by librrdupd_la-rrd_create.o:(parseDS) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_match_info_fetch_pos >>> referenced by librrdupd_la-rrd_create.o:(parseDS) in archive /usr/local/lib/librrd.a >>> referenced by librrdupd_la-rrd_create.o:(parseDS) in archive /usr/local/lib/librrd.a >>> referenced by librrdupd_la-rrd_create.o:(parseDS) in archive /usr/local/lib/librrd.a >>> referenced 2 more times ld: error: undefined symbol: g_tree_new_full >>> referenced by librrdupd_la-rrd_update.o:(rrd_template_update) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_tree_lookup >>> referenced by librrdupd_la-rrd_update.o:(rrd_template_update) in archive /usr/local/lib/librrd.a ld: error: undefined symbol: g_tree_insert >>> referenced by librrdupd_la-rrd_update.o:(rrd_template_update) in archive /usr/local/lib/librrd.a clang++: error: linker command failed with exit code 1 (use -v to see invocation) gmake[1]: *** [Makefile:136: ntopng] Error 1 gmake[1]: Leaving directory '/usr/obj/usr/ports/net/ntopng/work/ntopng-9d82383' Attached it is a patch meant as a POC that allows the build to continue. Cheers, Franco
(In reply to Franco Fichtner from comment #0) >ld: error: undefined symbol: g_list_append >>> referenced by librrdupd_la-rrd_create.o:(rrd_create) in archive /usr/local/lib/librrd.a >>> referenced by librrdupd_la-rrd_create.o:(rrd_create_r2) in archive /usr/local/lib/librrd.a This error message shows the symbol is undefined in the already installed library object /usr/local/lib/librrd.a Could you try forcing a rebuild and reinstall of that port and then build ntopng again? Should be databases/rrdtool Anyway the undefined reference resides in that static library installed by the databases/rrdtool port. If a rebuild does not fix it the problem could be there. Adding its maintainer to bug report CC, just in case. I'll test the build again in poudriere (which warrants a clean environment) anyway.
As you can see in the latest ntopng update it tries to statically link librrd so it cannot resolve glib on its own... this isn't a dynamic case that can be fixed by rebuilding. How Poudriere/ntopng combo works around this is irrelevant if the ports tree build fails it needs to be fixed. Cheers, Franco
(In reply to Franco Fichtner from comment #2) The fact that it works in poudriere IS relevant, poudriere uses no magic tricks, it cleans in a clean environment. I need a full log of you failing build, also a full build log of rrdtool from that machine could help. Before fixing a port which, to my present knowledge, works fine for everyone else I need to understand exactly why and how it is broken, and why and how the proposed fix fixes it. (I'm ready to revise my knowledge about who the port is broken for, but I need time to test that) If you are so sure a rebuild will not help, why not try it? It would only confirm your report. OTOH the fact that it works in poudriere and not in your case could be due to some build tool getting confused. for example configure, litool or pkgconf. One thing I suspect is pkgconf not reporting the correct set of library required for rrdtool libraries in your case, which COULD be fixed by recompiling and reinstalling the rrdtool port. This COULD also be a universsal issue, but I need to verify that. I do need to correctly understand why your patch fixes it, the fact that it fixes it is not enough, since the port is not universally broken. I hope my explanation to make clear that I'm not refusing to modify the port or apply your patch, but I need some further investigation before taking action, if you could please try doing what I asked (rebuild and reinstall rrdtool then ntopng and provide full logs of both builds if the rebuild does not fix the issue) could speed up my investigation. Otherwise it could take longer, at least until I am able to reproduce the issue, which, at present, I'm not able to reproduce.
BTW the issue could also be caused by some interaction with some otherwise unrelated port/software installed on the machine. This has happened to me a few times. Again, if this is the case, it requires fixing, but before acting I need to identify the exact issue I'm going to fix.
As stated above I'm using a clean chroot into a 13-STABLE build. I'm running "make" in net/ntopng and this is the result. I can attach the build log but I'm sure this can be reproduced from there. Cheers, Franco
(In reply to Franco Fichtner from comment #5) To reproduce it I need time, so you attaching the build log would speed up things. Please note that I don't know the details of your setup, so I made only some generic descriptions of issues I'm thinking could apply. I'm trying to understand what is going on. I'm going to run builds to verify if the issue is universal and try to reproduce it. I'm also evaluating your proposed patch, which I did not dismiss (sorry if something I said made it look that way). Please also note that even if I thought your patch unconditionally correct and required I would have required a few hours at best to test it before accepting and committing it, it would have gone the same testing process I'm using anyway.
Another generic statement I could propose: Some other commit that happened since I updated the port could have caused this, and in that case too I need to understand what has changed to cause this. Since I still have to make any tests I don't know what is going on, I'm just making hypotheses and I need to verify them.
I've been able to reproduce your issue in a VirtualBox VM. It took some time since it had to build all the dependencies. Looking at the logs I produced I found a significant difference with the poudriere build: poudriere links librrd dynamically for some reason in the VM it tries to link it statically poudriere is doing the right thing, while in the VM it is doing it right. for some reason. My first guess is the issue resides somewhere in the ntopng build system, and I need to investigate it. Adding a blind dependency to the port is not the right solution, tracking indirect dependencies is not a port Makefile job. I need some time to investigate it and force it to always do the right thing. As I said my guess is the issue is in ntopng Makefiles and they need some patching.
Created attachment 232682 [details] ntopng build fix. Actually your report uncovered a complex issue in ndpi and ntop build in ports that required some investigation to completely understand. now that I know what is going on I could create a patch, which I am going to test more thoroughly before committing, but looks correct on the live system. There is actually a missing dependency on rrdtool in net/ndpi, but adding that is not enough to fix things. Upstream build system is forcing ntopng to link statically with ndpi, but that is really not needed, so I also patch ntopng to use dynamic linking for ndpi too. The difference in how the ports build between poudriere and live system is due to ordering and the way poudriere builds each port in isolation. On live system when one runs cd /usr/ports/net/ntopng ; make install the ports system starts installing dependencies, and in this case installs rrdtool before starting to build ndpi. ndpi finds rrdtool and links to it. But in this case ntopng does not know that using static linking it also needs to explicitly link to glib and fails. In poudriere when asking it to build ntopng it also builds ndpi but in a separate process, isolated, without installing in that environment rrdtool (since it was not declared as a dependency to ndpi). ndpi happens to not link to rrdtool, and glib, so ndpi error in linking does not cause a failure later in this case. The same behaviour would show on a live system by doing: cd /usr/ports/net/ndpi; make install; make clean; cd ../ntopng; make install The patch avoids these by both adding a dependency on rrdtool to ndpi and fixing ntopng linking behaviour. This is the correct solution to get consistent builds both in poudriere and in a live system whatever the order used in compiling things.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=66e6345cf99ecb76939beba0b008e704fc7dcf5f commit 66e6345cf99ecb76939beba0b008e704fc7dcf5f Author: Guido Falsi <madpilot@FreeBSD.org> AuthorDate: 2022-03-24 23:46:14 +0000 Commit: Guido Falsi <madpilot@FreeBSD.org> CommitDate: 2022-03-24 23:48:47 +0000 net/ntopng,net/ndpi: Fix build due to missing symbols in certain circumstances When building on live machines when rrdtool happens to be already present, which often happens automatically due to the order in which the ports system builds things locally, the ntopng port could fail due to unreferenced symbols from glib. This was caused by ntopng trying to unconditionally link statically to ndpi but not providing all required static libraries references to the command line. - Add databases/rrdtool dependency in ndpi to ensure the same libraries are used in both ports - Make ntopng unconditionally use dynamic linking for ndpi too, avoiding the issue described above. PR: 262756 net/ndpi/Makefile | 4 +++- net/ntopng/Makefile | 1 + net/ntopng/files/patch-configure.ac.in | 13 ++++++++++++- 3 files changed, 16 insertions(+), 2 deletions(-)
(In reply to commit-hook from comment #10) I committed my patch, since it looks like it's working fine. Can you test and confirm so the issue can be closed? Thanks!
Works fine, thank you very much. Cheers, Franco