Bug 241312 - sysutils/logstash6 Cipher Error
Summary: sysutils/logstash6 Cipher Error
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-elastic mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-10-17 16:27 UTC by Wallace
Modified: 2019-11-10 06:31 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (elastic)


Attachments
Preliminary SSL patch (6.65 KB, patch)
2019-11-08 04:55 UTC, Greg Lewis
no flags Details | Diff
Updated patch (6.82 KB, patch)
2019-11-08 05:11 UTC, Greg Lewis
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Wallace 2019-10-17 16:27:57 UTC
I am running FreeBSD 11.3 amd64 and installed the newest version of logstash6. 

When I try to start logstash the service does start for about 20 seconds then kills itself. I turned on trace logging and found two interesting things:

1. Logstash is looking for ciphers that are not there. Note: in my logstash conf file I tried different ciphers by the cipher_suites option and allowed them all at one point, that made no difference.
2. Java is trying to load some netty library.

I have netty installed on the system too.

[2019-10-17T15:42:27,065][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Cipher `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` is not available>, :backtrace=>["/usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.6-java/lib/logstash/inputs/beats.rb:182:in `create_server'", "/usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.6-java/lib/logstash/inputs/beats.rb:170:in `register'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:242:in `register_plugin'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `register_plugins'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:396:in `start_inputs'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:294:in `start_workers'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:200:in `run'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:160:in `block in start'"], :thread=>"#<Thread:0x55053c00 run>"}

[2019-10-17T15:42:26,385][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
[2019-10-17T15:42:26,385][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.bitMode: 64 (sun.arch.data.model)
[2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.noPreferDirect: false
[2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.maxDirectMemory: 1038876672 bytes
[2019-10-17T15:42:26,386][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.uninitializedArrayAllocationThreshold: -1
[2019-10-17T15:42:26,387][DEBUG][io.netty.util.internal.CleanerJava6] java.nio.ByteBuffer.cleaner(): available
[2019-10-17T15:42:26,390][DEBUG][io.netty.util.internal.NativeLibraryLoader] -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
[2019-10-17T15:42:26,391][DEBUG][io.netty.util.internal.NativeLibraryLoader] Unable to load the library 'netty_tcnative_freebsd_x86_64', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_tcnative_freebsd_x86_64 in java.library.path

Thank you.

Note: I get the same errors trying to run logstash5.
Comment 1 Wallace 2019-10-17 19:38:49 UTC
For whats its worth I see the netty jar files here (one of two locations)

/usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.6-java/vendor/jar-dependencies/io/netty/netty-tcnative-boringssl-static/2.0.7.Final/netty-tcnative-boringssl-static-2.0.7.Final.jar

If you open the jar files you can see libraries for all other OS's besides FreeBSD.
Comment 2 Wallace 2019-10-22 13:41:37 UTC
I can get the service to start and stay started when not using SSL. 

Soon as I set ssl = true and and my ssl option is when we get the errors below.
Comment 3 Greg Lewis freebsd_committer 2019-10-22 17:57:41 UTC
If the lack of native libraries for netty-tcnative is what we think is the root cause here (I haven't looked at the Ruby code at all) then one path forward would be to create a FreeBSD port of netty-tcnative which compiles the native code and then replace the logstash JAR with a JAR from that port.

Another alternative would be to basically do all of that within the logstash port, but that seems like you'd then duplicate that across logstash versions depending on whether they all use compatible versions or not.

There are some build instructions on the netty-tcnative wiki at https://netty.io/wiki/forked-tomcat-native.html (we'd follow the linux build instructions most likely)
Comment 4 Wallace 2019-10-22 21:43:11 UTC
I could help and try to build the port in-house to test.

For the section "then replace the logstash JAR with a JAR from that port" where do we want to place/copy the jar files to?
Comment 5 Greg Lewis freebsd_committer 2019-10-23 15:33:06 UTC
There is a JAVAJARDIR where most ports should install their JAR files.  I was thinking the netty-tcnative port would place it's JAR file there.  Then for logstash we'd just removed the supplied JAR and symlink in the one from the netty-tcnative port.

The logstash port shouldn't put its JAR files in JAVAJARDIR since they are all specific to it rather than JARs that should be shared
Comment 6 Dan Langille freebsd_committer 2019-10-29 21:23:37 UTC
(In reply to Greg Lewis from comment #5)
I'm working on this Greg, but I'm confused. Based on https://netty.io/wiki/forked-tomcat-native.html#wiki-h2-7 the build process includes 'mvn clean install'.

I'm running into problems with that.   See https://reviews.freebsd.org/D22182
Comment 7 Greg Lewis freebsd_committer 2019-10-30 18:44:48 UTC
Hi Dan,

It looks like it is trying to download the src from Maven central and can't find it.  Ideally this should be done during the fetch step, but it looks like the fetch is instead pulling the binary distribution.  I'll try and take a look at this tomorrow and see if I can get a little further.
Comment 8 Greg Lewis freebsd_committer 2019-10-30 19:11:40 UTC
I pulled the distfile myself, and I see that the source in there.  I'm confused as to why Maven would also be attempting to download a separate source bundle as well.
Comment 9 Greg Lewis freebsd_committer 2019-11-01 03:25:43 UTC
I need to get into this deeper, but some other potential concerns:

* netty-tcnative looks like it will build native libraries against libressl, boringssl, and openssl.  You can't have all of those installed at once from ports because they conflict with each other (boringssl doesn't list the conflict, but it clearly has it).  So there are some gymnastics that would need to happen there to allow the full netty-tcnative build to work.

* The JAR in logstash isn't the full netty-tcnative JAR, it's the one linked against boringssl based on the name (netty-tcnative-boringssl-static-2.0.12.Final.jar).  That's unfortunate in that I'm guessing boringssl is the least likely of the three for someone to have installed (given the general state of the project and port).

One question I have here, without knowing the answer yet, is whether LogStash is doing anything that is specific to boringssl?  My guess would be no, since the tc-native code allows all three libraries to potentially be used as the native component.

If that is the case, then it might be possible to build a native library for FreeBSD using whatever SSL lib is set as the default or maybe ignore ports altogether and build it against the system installed version.  The dependency on apr might mean we have to use the version in ports though, I haven't checked into that.  Were that the case, it might make more sense to do it within the LogStash port and just build that library and add it into the JAR rather than attempt a general netty-tcnative port since that is going to have problems.

Is someone either able to test or willing to share configuration files and testing steps if I were to try and put something like that together?
Comment 10 Dan Langille freebsd_committer 2019-11-01 13:33:57 UTC
Greg: we (I am cooperating with Wallace on this project) are happy to test anything you can create.

So far our testing has consisted of trying to get logstash5 and logstash6 to run with SSL. Both work fine without. Both fail with.
Comment 11 Wallace 2019-11-04 14:46:13 UTC
Hi Greg, yes, we can test.
Comment 12 Greg Lewis freebsd_committer 2019-11-08 04:55:31 UTC
Created attachment 208961 [details]
Preliminary SSL patch

Pull down the appropriate version of netty-tcnative and build the native library for FreeBSD then insert it into the netty-tcnative jar that ships with logstash.

This needs some clean up before it could be committed, but the bigger question is whether it works or not.  Logstash starts for me but I don't have a set up to test the use of SSL.

Please give it a try and report results.
Comment 13 Greg Lewis freebsd_committer 2019-11-08 05:11:22 UTC
Created attachment 208962 [details]
Updated patch
Comment 14 Greg Lewis freebsd_committer 2019-11-08 05:11:54 UTC
Forgot the dependency on apr in the first patch
Comment 15 Dan Langille freebsd_committer 2019-11-08 12:42:36 UTC
(In reply to Greg Lewis from comment #14)
Working on this now. We will have results today. Thank you.
Comment 16 Dan Langille freebsd_committer 2019-11-08 14:19:17 UTC
I'm getting compile errors on head with both FreeBSD 11.3 and FreeBSD 12.0: 

--- sslutils.pico ---
/usr/local/bin/ccache cc -fpic -DPIC  -O2 -pipe  -I/usr/local/openjdk8/include -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -DHAVE_OPENSSL -I/usr/include/openssl `/usr/local/bin/apr-1-config --includes` -I/usr/local/openjdk8/include -I/usr/local/openjdk8/include/freebsd -D_LARGEFILE64_SOURCE `/usr/local/bin/apr-1-config --cflags` -fvisibility=hidden   -MD  -MF.depend.sslutils.pico -MTsslutils.pico -std=gnu99 -fstack-protector-strong    -Qunused-arguments  -c sslutils.c -o sslutils.pico
--- bb.pico ---
In file included from bb.c:32:
./tcn.h:36:10: fatal error: 'jni.h' file not found
#include <jni.h>
         ^~~~~~~


I tried adding this to the Makefile, but that makes no difference:

CFLAGS+=        "-I/usr/local/openjdk8/include"

That is where I found the required file:

root@120R-dvl:~ # find / -name jni.h
/usr/local/openjdk8/include/jni.h


Discussions on IRC speculated it might be -isystem is causing the problem.  The full poudriere testport output is at: https://gist.github.com/dlangille/5a4d586507738f5acb173688080cf7c0
Comment 17 Greg Lewis freebsd_committer 2019-11-10 06:31:55 UTC
It seems like -isystem is something that is set in a configuration file on the system somewhere?  Or maybe part of the ccache set up?

Either way it isn't used on my 11.3 system when I compile.  I'd suggest looking at /etc/make.conf or similar configuration files to see if it is set there and turning it off.  Either that or maybe try setting

NO_CCACHE=      yes

in the Makefile to see if it is related to the use of ccache.