Bug 241312 - sysutils/logstash6 Cipher Error
Summary: sysutils/logstash6 Cipher Error
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-elastic (Nobody)
Depends on:
Reported: 2019-10-17 16:27 UTC by Wallace
Modified: 2020-08-20 00:12 UTC (History)
4 users (show)

See Also:
bugzilla: maintainer-feedback? (elastic)

Preliminary SSL patch (6.65 KB, patch)
2019-11-08 04:55 UTC, Greg Lewis
no flags Details | Diff
Updated patch (6.82 KB, patch)
2019-11-08 05:11 UTC, Greg Lewis
no flags Details | Diff
Updated patch (3.93 KB, patch)
2019-11-23 17:16 UTC, Greg Lewis
no flags Details | Diff
Updated patch (3.93 KB, patch)
2019-11-27 05:46 UTC, Greg Lewis
no flags Details | Diff
Updated patch against 6.8.5 (10.48 KB, patch)
2019-11-27 17:34 UTC, Greg Lewis
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Wallace 2019-10-17 16:27:57 UTC
I am running FreeBSD 11.3 amd64 and installed the newest version of logstash6. 

When I try to start logstash the service does start for about 20 seconds then kills itself. I turned on trace logging and found two interesting things:

1. Logstash is looking for ciphers that are not there. Note: in my logstash conf file I tried different ciphers by the cipher_suites option and allowed them all at one point, that made no difference.
2. Java is trying to load some netty library.

I have netty installed on the system too.

[2019-10-17T15:42:27,065][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Cipher `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` is not available>, :backtrace=>["/usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.6-java/lib/logstash/inputs/beats.rb:182:in `create_server'", "/usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.6-java/lib/logstash/inputs/beats.rb:170:in `register'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:242:in `register_plugin'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `register_plugins'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:396:in `start_inputs'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:294:in `start_workers'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:200:in `run'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:160:in `block in start'"], :thread=>"#<Thread:0x55053c00 run>"}

[2019-10-17T15:42:26,385][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
[2019-10-17T15:42:26,385][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.bitMode: 64 (sun.arch.data.model)
[2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.noPreferDirect: false
[2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.maxDirectMemory: 1038876672 bytes
[2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.uninitializedArrayAllocationThreshold: -1
[2019-10-17T15:42:26,387][DEBUG][io.netty.util.internal.CleanerJava6] java.nio.ByteBuffer.cleaner(): available
[2019-10-17T15:42:26,390][DEBUG][io.netty.util.internal.NativeLibraryLoader] -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
[2019-10-17T15:42:26,391][DEBUG][io.netty.util.internal.NativeLibraryLoader] Unable to load the library 'netty_tcnative_freebsd_x86_64', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_tcnative_freebsd_x86_64 in java.library.path

Thank you.

Note: I get the same errors trying to run logstash5.
Comment 1 Wallace 2019-10-17 19:38:49 UTC
For whats its worth I see the netty jar files here (one of two locations)


If you open the jar files you can see libraries for all other OS's besides FreeBSD.
Comment 2 Wallace 2019-10-22 13:41:37 UTC
I can get the service to start and stay started when not using SSL. 

Soon as I set ssl = true and and my ssl option is when we get the errors below.
Comment 3 Greg Lewis freebsd_committer 2019-10-22 17:57:41 UTC
If the lack of native libraries for netty-tcnative is what we think is the root cause here (I haven't looked at the Ruby code at all) then one path forward would be to create a FreeBSD port of netty-tcnative which compiles the native code and then replace the logstash JAR with a JAR from that port.

Another alternative would be to basically do all of that within the logstash port, but that seems like you'd then duplicate that across logstash versions depending on whether they all use compatible versions or not.

There are some build instructions on the netty-tcnative wiki at https://netty.io/wiki/forked-tomcat-native.html (we'd follow the linux build instructions most likely)
Comment 4 Wallace 2019-10-22 21:43:11 UTC
I could help and try to build the port in-house to test.

For the section "then replace the logstash JAR with a JAR from that port" where do we want to place/copy the jar files to?
Comment 5 Greg Lewis freebsd_committer 2019-10-23 15:33:06 UTC
There is a JAVAJARDIR where most ports should install their JAR files.  I was thinking the netty-tcnative port would place it's JAR file there.  Then for logstash we'd just removed the supplied JAR and symlink in the one from the netty-tcnative port.

The logstash port shouldn't put its JAR files in JAVAJARDIR since they are all specific to it rather than JARs that should be shared
Comment 6 Dan Langille freebsd_committer 2019-10-29 21:23:37 UTC
(In reply to Greg Lewis from comment #5)
I'm working on this Greg, but I'm confused. Based on https://netty.io/wiki/forked-tomcat-native.html#wiki-h2-7 the build process includes 'mvn clean install'.

I'm running into problems with that.   See https://reviews.freebsd.org/D22182
Comment 7 Greg Lewis freebsd_committer 2019-10-30 18:44:48 UTC
Hi Dan,

It looks like it is trying to download the src from Maven central and can't find it.  Ideally this should be done during the fetch step, but it looks like the fetch is instead pulling the binary distribution.  I'll try and take a look at this tomorrow and see if I can get a little further.
Comment 8 Greg Lewis freebsd_committer 2019-10-30 19:11:40 UTC
I pulled the distfile myself, and I see that the source in there.  I'm confused as to why Maven would also be attempting to download a separate source bundle as well.
Comment 9 Greg Lewis freebsd_committer 2019-11-01 03:25:43 UTC
I need to get into this deeper, but some other potential concerns:

* netty-tcnative looks like it will build native libraries against libressl, boringssl, and openssl.  You can't have all of those installed at once from ports because they conflict with each other (boringssl doesn't list the conflict, but it clearly has it).  So there are some gymnastics that would need to happen there to allow the full netty-tcnative build to work.

* The JAR in logstash isn't the full netty-tcnative JAR, it's the one linked against boringssl based on the name (netty-tcnative-boringssl-static-2.0.12.Final.jar).  That's unfortunate in that I'm guessing boringssl is the least likely of the three for someone to have installed (given the general state of the project and port).

One question I have here, without knowing the answer yet, is whether LogStash is doing anything that is specific to boringssl?  My guess would be no, since the tc-native code allows all three libraries to potentially be used as the native component.

If that is the case, then it might be possible to build a native library for FreeBSD using whatever SSL lib is set as the default or maybe ignore ports altogether and build it against the system installed version.  The dependency on apr might mean we have to use the version in ports though, I haven't checked into that.  Were that the case, it might make more sense to do it within the LogStash port and just build that library and add it into the JAR rather than attempt a general netty-tcnative port since that is going to have problems.

Is someone either able to test or willing to share configuration files and testing steps if I were to try and put something like that together?
Comment 10 Dan Langille freebsd_committer 2019-11-01 13:33:57 UTC
Greg: we (I am cooperating with Wallace on this project) are happy to test anything you can create.

So far our testing has consisted of trying to get logstash5 and logstash6 to run with SSL. Both work fine without. Both fail with.
Comment 11 Wallace 2019-11-04 14:46:13 UTC
Hi Greg, yes, we can test.
Comment 12 Greg Lewis freebsd_committer 2019-11-08 04:55:31 UTC
Created attachment 208961 [details]
Preliminary SSL patch

Pull down the appropriate version of netty-tcnative and build the native library for FreeBSD then insert it into the netty-tcnative jar that ships with logstash.

This needs some clean up before it could be committed, but the bigger question is whether it works or not.  Logstash starts for me but I don't have a set up to test the use of SSL.

Please give it a try and report results.
Comment 13 Greg Lewis freebsd_committer 2019-11-08 05:11:22 UTC
Created attachment 208962 [details]
Updated patch
Comment 14 Greg Lewis freebsd_committer 2019-11-08 05:11:54 UTC
Forgot the dependency on apr in the first patch
Comment 15 Dan Langille freebsd_committer 2019-11-08 12:42:36 UTC
(In reply to Greg Lewis from comment #14)
Working on this now. We will have results today. Thank you.
Comment 16 Dan Langille freebsd_committer 2019-11-08 14:19:17 UTC
I'm getting compile errors on head with both FreeBSD 11.3 and FreeBSD 12.0: 

--- sslutils.pico ---
/usr/local/bin/ccache cc -fpic -DPIC  -O2 -pipe  -I/usr/local/openjdk8/include -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -DHAVE_OPENSSL -I/usr/include/openssl `/usr/local/bin/apr-1-config --includes` -I/usr/local/openjdk8/include -I/usr/local/openjdk8/include/freebsd -D_LARGEFILE64_SOURCE `/usr/local/bin/apr-1-config --cflags` -fvisibility=hidden   -MD  -MF.depend.sslutils.pico -MTsslutils.pico -std=gnu99 -fstack-protector-strong    -Qunused-arguments  -c sslutils.c -o sslutils.pico
--- bb.pico ---
In file included from bb.c:32:
./tcn.h:36:10: fatal error: 'jni.h' file not found
#include <jni.h>

I tried adding this to the Makefile, but that makes no difference:

CFLAGS+=        "-I/usr/local/openjdk8/include"

That is where I found the required file:

root@120R-dvl:~ # find / -name jni.h

Discussions on IRC speculated it might be -isystem is causing the problem.  The full poudriere testport output is at: https://gist.github.com/dlangille/5a4d586507738f5acb173688080cf7c0
Comment 17 Greg Lewis freebsd_committer 2019-11-10 06:31:55 UTC
It seems like -isystem is something that is set in a configuration file on the system somewhere?  Or maybe part of the ccache set up?

Either way it isn't used on my 11.3 system when I compile.  I'd suggest looking at /etc/make.conf or similar configuration files to see if it is set there and turning it off.  Either that or maybe try setting

NO_CCACHE=      yes

in the Makefile to see if it is related to the use of ccache.
Comment 18 Dan Langille freebsd_committer 2019-11-20 19:31:56 UTC
I have been distracted by other tasks.  Back soon, sorry.
Comment 19 Dan Langille freebsd_committer 2019-11-22 16:03:18 UTC
I wish I knew more about how this code looks for .h files 

I found the file which fails:

root@113R-dvl:~ # find / -name tcn.h

Looking in that file, I find the include statement:

# grep jni.h tcn.h
#include <jni.h>

It exists at /usr/local/openjdk8/include/jni.h

root@113R-dvl:~ # pkg which /usr/local/openjdk8/include/jni.h
/usr/local/openjdk8/include/jni.h was installed by package openjdk8-

For fun, I changed the include line to:

#include "/usr/local/openjdk8/include/jni.h"

Then it builds.
Comment 20 Dan Langille freebsd_committer 2019-11-22 16:05:02 UTC
(In reply to Dan Langille from comment #19)
Missing from the above:

I recompiled with the following added to the Makefile:

NO_CCACHE=      yes
CFLAGS+=        "-I/usr/local/openjdk8/include"

The full output is at: https://gist.github.com/dlangille/e25e977e8235b70d66278a02472a6797
Comment 21 Greg Lewis freebsd_committer 2019-11-22 19:54:31 UTC
Dan, I think the build error is very fixable.  My main question is, if you make the change to tcn.h and get ti to build (I suspect just making it "jni.h" would work) then are you able to use SSL with logstash?
Comment 22 Dan Langille freebsd_committer 2019-11-22 22:28:31 UTC
I fixed that include error by adding this to the Makefile:


The resulting package builds. We tried it on 12.0 - the service starts but crashes. No logs. We're going to look at this again on Monday.

Than kyou.
Comment 23 Greg Lewis freebsd_committer 2019-11-23 17:16:36 UTC
Created attachment 209364 [details]
Updated patch

* Add JAVA_BUILD=yes
* Update to be against 6.8.4
Comment 24 Juraj Lutter 2019-11-23 17:56:59 UTC
I also did my own update to 6.8.4, package builds, I will install it and set it up with TLS to see if it crashes or not.
Comment 25 Juraj Lutter 2019-11-23 18:16:17 UTC
It crashes because it can not find JNI library:

[2019-11-23T19:14:59,308][DEBUG][io.netty.util.internal.NativeLibraryLoader] Unable to load the library 'netty_tcnative_freebsd_x86_64', trying other loading
java.lang.UnsatisfiedLinkError: null

and some other related error messages.
Comment 26 Wallace 2019-11-25 17:36:39 UTC
I think we have progress, I created a new FreeBSD 11.3 server and SSL certs, installed logstash 6.8.3 from our tree and got it to run on port 5044 with my certs. See log below, some are snips from the log file since I turned on super debug and it spammed the screen and log file.

[2019-11-25T16:54:31,180][DEBUG][io.netty.util.internal.NativeLibraryLoader] Unable to load the library 'netty_tcnative_freebsd_x86_64', trying other loading mechanism.
[2019-11-25T16:54:31,181][DEBUG][io.netty.util.internal.NativeLibraryLoader] netty_tcnative_freebsd_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
[2019-11-25T16:54:31,187][DEBUG][io.netty.util.internal.NativeLibraryLoader] Successfully loaded the library /tmp/libnetty_tcnative_freebsd_x86_647758890754716951252.so
[2019-11-25T16:54:31,187][DEBUG][io.netty.handler.ssl.OpenSsl] Initialize netty-tcnative using engine: 'default'
[2019-11-25T16:54:31,189][DEBUG][io.netty.handler.ssl.OpenSsl] netty-tcnative using native library: OpenSSL 1.0.2s-freebsd  28 May 2019

[2019-11-25T16:54:31,354][DEBUG][io.netty.handler.ssl.ReferenceCountedOpenSslContext] ReferenceCountedOpenSslContext supports -Djdk.tls.ephemeralDHKeySize={int}, but got: matched

[2019-11-25T16:54:31,379][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
[2019-11-25T16:54:31,379][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
[2019-11-25T16:54:31,380][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
[2019-11-25T16:54:31,380][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
[2019-11-25T16:54:31,380][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 => ECDHE-RSA-AES256-SHA384
[2019-11-25T16:54:31,380][DEBUG][io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384 => ECDHE-RSA-AES256-SHA384
... (This above section mapped all the SSL suites and was very long in the log file)

[2019-11-25T16:54:31,404][DEBUG][io.netty.handler.ssl.OpenSsl] Supported protocols (OpenSSL): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[2019-11-25T16:54:31,411][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>""}

I haven't got this far before, Logstash doesn't crash. It seems to be working but would like to test and view the logs a bit more, this is huge progress. Will report back after the holiday.
Comment 27 Greg Lewis freebsd_committer 2019-11-27 05:46:42 UTC
Created attachment 209470 [details]
Updated patch

Fixed the previous patch

We'll need another one since there is also an update to 6.8.5 in a separate PR
Comment 28 Greg Lewis freebsd_committer 2019-11-27 05:48:02 UTC
That is good to hear the progress there. I'll look to start cleaning up this patch  so that it could be committed.  Will hear from you next week then.
Comment 29 Juraj Lutter 2019-11-27 09:43:27 UTC
I am even further.

The main problem is that tcnative (its downloadable archive for linux_x86_64) uses its own builtin BoringSSL. Providing our .so library will not work as the API/ABI might difffer between BoringSSL and OpenSSL, this also JNI shims are to be updated.

My first guess was that a separate port for tcnative (openssl-dynamic) will also be needed.
Comment 30 Juraj Lutter 2019-11-27 17:03:11 UTC
I was able to make a port for tcnative and was able to successfully built it with OpenSSL dynamic linkage.

The next step will be to determine if/how this could be plugged into logstash.
Comment 31 Greg Lewis freebsd_committer 2019-11-27 17:34:56 UTC
Created attachment 209490 [details]
Updated patch against 6.8.5

* Updated to be against 6.8.5
* Fixes some portlint complaints
* Cleans up the patch a little
Comment 32 Wallace 2019-12-03 20:26:31 UTC
I think we might be good here. Is there anything else I should test? 

I used curl to validate the cert was read correctly following the https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html link.

Recap: I am testing the port that Dan made for me, that does work. I think all the changes Dan made to get it to work are in the 6.8.5 version you created, Greg.
Comment 33 Greg Lewis freebsd_committer 2020-01-24 01:54:03 UTC
Sorry for not getting this committed.

Any concerns with me going ahead and updating the patch to be against 6.8.6 and doing so?
Comment 34 Wallace 2020-02-10 20:06:37 UTC
Greg, commit away! Thanks!
Comment 35 Juraj Lutter 2020-08-19 22:37:48 UTC
Could we close this PR, please?
Comment 36 Dan Langille freebsd_committer 2020-08-20 00:12:18 UTC
Fixed via other PRs I suspect.