I am running FreeBSD 11.3 amd64 and installed the newest version of logstash6. When I try to start logstash the service does start for about 20 seconds then kills itself. I turned on trace logging and found two interesting things: 1. Logstash is looking for ciphers that are not there. Note: in my logstash conf file I tried different ciphers by the cipher_suites option and allowed them all at one point, that made no difference. 2. Java is trying to load some netty library. I have netty installed on the system too. [2019-10-17T15:42:27,065][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Cipher `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` is not available>, :backtrace=>["/usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.6-java/lib/logstash/inputs/beats.rb:182:in `create_server'", "/usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.6-java/lib/logstash/inputs/beats.rb:170:in `register'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:242:in `register_plugin'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `register_plugins'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:396:in `start_inputs'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:294:in `start_workers'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:200:in `run'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:160:in `block in start'"], :thread=>"#<Thread:0x55053c00 run>"} [2019-10-17T15:42:26,385][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.tmpdir: /tmp (java.io.tmpdir) [2019-10-17T15:42:26,385][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.bitMode: 64 (sun.arch.data.model) [2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.noPreferDirect: false [2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.maxDirectMemory: 1038876672 bytes [2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.uninitializedArrayAllocationThreshold: -1 [2019-10-17T15:42:26,387][DEBUG][io.netty.util.internal.CleanerJava6] java.nio.ByteBuffer.cleaner(): available [2019-10-17T15:42:26,390][DEBUG][io.netty.util.internal.NativeLibraryLoader] -Dio.netty.native.workdir: /tmp (io.netty.tmpdir) [2019-10-17T15:42:26,391][DEBUG][io.netty.util.internal.NativeLibraryLoader] Unable to load the library 'netty_tcnative_freebsd_x86_64', trying other loading mechanism. java.lang.UnsatisfiedLinkError: no netty_tcnative_freebsd_x86_64 in java.library.path Thank you. Note: I get the same errors trying to run logstash5.
For whats its worth I see the netty jar files here (one of two locations) /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.6-java/vendor/jar-dependencies/io/netty/netty-tcnative-boringssl-static/2.0.7.Final/netty-tcnative-boringssl-static-2.0.7.Final.jar If you open the jar files you can see libraries for all other OS's besides FreeBSD.
I can get the service to start and stay started when not using SSL. Soon as I set ssl = true and and my ssl option is when we get the errors below.
If the lack of native libraries for netty-tcnative is what we think is the root cause here (I haven't looked at the Ruby code at all) then one path forward would be to create a FreeBSD port of netty-tcnative which compiles the native code and then replace the logstash JAR with a JAR from that port. Another alternative would be to basically do all of that within the logstash port, but that seems like you'd then duplicate that across logstash versions depending on whether they all use compatible versions or not. There are some build instructions on the netty-tcnative wiki at https://netty.io/wiki/forked-tomcat-native.html (we'd follow the linux build instructions most likely)
I could help and try to build the port in-house to test. For the section "then replace the logstash JAR with a JAR from that port" where do we want to place/copy the jar files to?
There is a JAVAJARDIR where most ports should install their JAR files. I was thinking the netty-tcnative port would place it's JAR file there. Then for logstash we'd just removed the supplied JAR and symlink in the one from the netty-tcnative port. The logstash port shouldn't put its JAR files in JAVAJARDIR since they are all specific to it rather than JARs that should be shared
(In reply to Greg Lewis from comment #5) I'm working on this Greg, but I'm confused. Based on https://netty.io/wiki/forked-tomcat-native.html#wiki-h2-7 the build process includes 'mvn clean install'. I'm running into problems with that. See https://reviews.freebsd.org/D22182
Hi Dan, It looks like it is trying to download the src from Maven central and can't find it. Ideally this should be done during the fetch step, but it looks like the fetch is instead pulling the binary distribution. I'll try and take a look at this tomorrow and see if I can get a little further.
I pulled the distfile myself, and I see that the source in there. I'm confused as to why Maven would also be attempting to download a separate source bundle as well.
I need to get into this deeper, but some other potential concerns: * netty-tcnative looks like it will build native libraries against libressl, boringssl, and openssl. You can't have all of those installed at once from ports because they conflict with each other (boringssl doesn't list the conflict, but it clearly has it). So there are some gymnastics that would need to happen there to allow the full netty-tcnative build to work. * The JAR in logstash isn't the full netty-tcnative JAR, it's the one linked against boringssl based on the name (netty-tcnative-boringssl-static-2.0.12.Final.jar). That's unfortunate in that I'm guessing boringssl is the least likely of the three for someone to have installed (given the general state of the project and port). One question I have here, without knowing the answer yet, is whether LogStash is doing anything that is specific to boringssl? My guess would be no, since the tc-native code allows all three libraries to potentially be used as the native component. If that is the case, then it might be possible to build a native library for FreeBSD using whatever SSL lib is set as the default or maybe ignore ports altogether and build it against the system installed version. The dependency on apr might mean we have to use the version in ports though, I haven't checked into that. Were that the case, it might make more sense to do it within the LogStash port and just build that library and add it into the JAR rather than attempt a general netty-tcnative port since that is going to have problems. Is someone either able to test or willing to share configuration files and testing steps if I were to try and put something like that together?
Greg: we (I am cooperating with Wallace on this project) are happy to test anything you can create. So far our testing has consisted of trying to get logstash5 and logstash6 to run with SSL. Both work fine without. Both fail with.
Hi Greg, yes, we can test.
Created attachment 208961 [details] Preliminary SSL patch Pull down the appropriate version of netty-tcnative and build the native library for FreeBSD then insert it into the netty-tcnative jar that ships with logstash. This needs some clean up before it could be committed, but the bigger question is whether it works or not. Logstash starts for me but I don't have a set up to test the use of SSL. Please give it a try and report results.
Created attachment 208962 [details] Updated patch
Forgot the dependency on apr in the first patch
(In reply to Greg Lewis from comment #14) Working on this now. We will have results today. Thank you.
I'm getting compile errors on head with both FreeBSD 11.3 and FreeBSD 12.0: --- sslutils.pico --- /usr/local/bin/ccache cc -fpic -DPIC -O2 -pipe -I/usr/local/openjdk8/include -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DHAVE_OPENSSL -I/usr/include/openssl `/usr/local/bin/apr-1-config --includes` -I/usr/local/openjdk8/include -I/usr/local/openjdk8/include/freebsd -D_LARGEFILE64_SOURCE `/usr/local/bin/apr-1-config --cflags` -fvisibility=hidden -MD -MF.depend.sslutils.pico -MTsslutils.pico -std=gnu99 -fstack-protector-strong -Qunused-arguments -c sslutils.c -o sslutils.pico --- bb.pico --- In file included from bb.c:32: ./tcn.h:36:10: fatal error: 'jni.h' file not found #include <jni.h> ^~~~~~~ I tried adding this to the Makefile, but that makes no difference: CFLAGS+= "-I/usr/local/openjdk8/include" That is where I found the required file: root@120R-dvl:~ # find / -name jni.h /usr/local/openjdk8/include/jni.h Discussions on IRC speculated it might be -isystem is causing the problem. The full poudriere testport output is at: https://gist.github.com/dlangille/5a4d586507738f5acb173688080cf7c0
It seems like -isystem is something that is set in a configuration file on the system somewhere? Or maybe part of the ccache set up? Either way it isn't used on my 11.3 system when I compile. I'd suggest looking at /etc/make.conf or similar configuration files to see if it is set there and turning it off. Either that or maybe try setting NO_CCACHE= yes in the Makefile to see if it is related to the use of ccache.
I have been distracted by other tasks. Back soon, sorry.
I wish I knew more about how this code looks for .h files I found the file which fails: root@113R-dvl:~ # find / -name tcn.h /wrkdirs/usr/ports/sysutils/logstash6/work/netty-tcnative-netty-tcnative-parent-2.0.12.Final/openssl-dynamic/src/main/c/tcn.h Looking in that file, I find the include statement: # grep jni.h tcn.h #include <jni.h> It exists at /usr/local/openjdk8/include/jni.h root@113R-dvl:~ # pkg which /usr/local/openjdk8/include/jni.h /usr/local/openjdk8/include/jni.h was installed by package openjdk8-8.232.09.1_1 For fun, I changed the include line to: #include "/usr/local/openjdk8/include/jni.h" Then it builds.
(In reply to Dan Langille from comment #19) Missing from the above: I recompiled with the following added to the Makefile: NO_CCACHE= yes CFLAGS+= "-I/usr/local/openjdk8/include" The full output is at: https://gist.github.com/dlangille/e25e977e8235b70d66278a02472a6797
Dan, I think the build error is very fixable. My main question is, if you make the change to tcn.h and get ti to build (I suspect just making it "jni.h" would work) then are you able to use SSL with logstash?
I fixed that include error by adding this to the Makefile: JAVA_BUILD= yes The resulting package builds. We tried it on 12.0 - the service starts but crashes. No logs. We're going to look at this again on Monday. Than kyou.
Created attachment 209364 [details] Updated patch * Add JAVA_BUILD=yes * Update to be against 6.8.4
I also did my own update to 6.8.4, package builds, I will install it and set it up with TLS to see if it crashes or not.
It crashes because it can not find JNI library: [2019-11-23T19:14:59,308][DEBUG][io.netty.util.internal.NativeLibraryLoader] Unable to load the library 'netty_tcnative_freebsd_x86_64', trying other loading mechanism. java.lang.UnsatisfiedLinkError: null and some other related error messages.
I think we have progress, I created a new FreeBSD 11.3 server and SSL certs, installed logstash 6.8.3 from our tree and got it to run on port 5044 with my certs. See log below, some are snips from the log file since I turned on super debug and it spammed the screen and log file. [2019-11-25T16:54:31,180][DEBUG][io.netty.util.internal.NativeLibraryLoader] Unable to load the library 'netty_tcnative_freebsd_x86_64', trying other loading mechanism. [2019-11-25T16:54:31,181][DEBUG][io.netty.util.internal.NativeLibraryLoader] netty_tcnative_freebsd_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp [2019-11-25T16:54:31,187][DEBUG][io.netty.util.internal.NativeLibraryLoader] Successfully loaded the library /tmp/libnetty_tcnative_freebsd_x86_647758890754716951252.so [2019-11-25T16:54:31,187][DEBUG][io.netty.handler.ssl.OpenSsl] Initialize netty-tcnative using engine: 'default' [2019-11-25T16:54:31,189][DEBUG][io.netty.handler.ssl.OpenSsl] netty-tcnative using native library: OpenSSL 1.0.2s-freebsd 28 May 2019 ... [2019-11-25T16:54:31,354][DEBUG][io.netty.handler.ssl.ReferenceCountedOpenSslContext] ReferenceCountedOpenSslContext supports -Djdk.tls.ephemeralDHKeySize={int}, but got: matched ... [2019-11-25T16:54:31,379][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384 [2019-11-25T16:54:31,379][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384 [2019-11-25T16:54:31,380][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384 [2019-11-25T16:54:31,380][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384 [2019-11-25T16:54:31,380][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 => ECDHE-RSA-AES256-SHA384 [2019-11-25T16:54:31,380][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384 => ECDHE-RSA-AES256-SHA384 ... (This above section mapped all the SSL suites and was very long in the log file) [2019-11-25T16:54:31,404][DEBUG][io.netty.handler.ssl.OpenSsl] Supported protocols (OpenSSL): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] [2019-11-25T16:54:31,404][DEBUG][io.netty.handler.ssl.OpenSsl] Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA] [2019-11-25T16:54:31,411][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} ... I haven't got this far before, Logstash doesn't crash. It seems to be working but would like to test and view the logs a bit more, this is huge progress. Will report back after the holiday.
Created attachment 209470 [details] Updated patch Fixed the previous patch We'll need another one since there is also an update to 6.8.5 in a separate PR
That is good to hear the progress there. I'll look to start cleaning up this patch so that it could be committed. Will hear from you next week then.
I am even further. The main problem is that tcnative (its downloadable archive for linux_x86_64) uses its own builtin BoringSSL. Providing our .so library will not work as the API/ABI might difffer between BoringSSL and OpenSSL, this also JNI shims are to be updated. My first guess was that a separate port for tcnative (openssl-dynamic) will also be needed.
I was able to make a port for tcnative and was able to successfully built it with OpenSSL dynamic linkage. The next step will be to determine if/how this could be plugged into logstash.
Created attachment 209490 [details] Updated patch against 6.8.5 * Updated to be against 6.8.5 * Fixes some portlint complaints * Cleans up the patch a little
I think we might be good here. Is there anything else I should test? I used curl to validate the cert was read correctly following the https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html link. Recap: I am testing the port that Dan made for me, that does work. I think all the changes Dan made to get it to work are in the 6.8.5 version you created, Greg.
Sorry for not getting this committed. Any concerns with me going ahead and updating the patch to be against 6.8.6 and doing so?
Greg, commit away! Thanks!
Could we close this PR, please?
Fixed via other PRs I suspect.
I am still having this issue on version 6.8.12 and 6.8.13 when installing via portsnap. Was this fix applied only to 6.8.5?