This patch updates the logstash port to 1.4.2. This port is obsolete since a long time (more than one year), the reason being the existence of a FreeBSD-related bug in a library shipped with logstash (jnr-ffi). I've notified upstream and provided a solution, but in the last few months they gave no feedback. Hence, I decided to fork the offending library in a GitHub repository (https://github.com/emcrisostomo/jnr-ffi) and to try and keep it working on FreeBSD until upstream acknowledges the problem and fixes it. Since this issue is long standing and the port is not being updated since a long time, I'm claiming authorship and maintainership of the logstash port, unless of course the current maintainer provides a better solution. Attachments: * Subversion patch. * poudriere testport output. Cheers, -- Enrico
Created attachment 148664 [details] logstash 1.4.2 - svn diff
Created attachment 148665 [details] poudriere testport sysutils/logstash output
I;ve an interest in getting this committed. Awaiting feedback from the maintainer.
Hi, Like i said by mail previously, there is two issues : - jruby cannot handle correctly the new libc header - logstash need to update to latest w/complete rebrand of the content Fom what i can see the ‘hack’ here is about to pollute jruby port from logstash port. If i’m the only one that seems conceptually puzzled; fine let’s go. If not, PR the jruby port maintainer and doing some lobby toward should be a cleaner way to maintain all this. During all this happening, we could add some guidance into pkg-message in order to let know how we can have things working. Even a decollated script into « share » could be a good idea in order to be user-friendly. So, to my point of vue, let’s update to 1.4.2 this port and see what we can do a bit more cleaner. Lastly, let’s add my 2 cents regarding the port maintenance thing. If port maintenance had changed ownership every time people collaborate and provide a patch it would have been a bit messy =) p.s. : the patch you provide even remove the original port authors =/ More seriously, I have very few time and it could be better for community that some else maintain this port, so if you are up to, i’m not against. Regards, — Regis A. Despres
Hi Regis, hi all, To say the truth, and you know since the first time I talked to you about this logstash bug was January 22th, I'm puzzled that it's more than one year that people like me are providing workarounds and PR and nobody has done anything upstream. I don't want to pollute this PR with non-essential information, but here's a little summary: * I've opened an issue to JRuby (https://github.com/jruby/jruby/issues/1754) in June and acknowledge there's problem with jnr-ffi. * A PR has been opened to jnr-ffi at least 3 times (I'm not counting private contact attempts) and nobody has ever answered, not even an acknowledgment. Summarising: I'm puzzled because a port such as logstash is stuck since so much time and nobody upstream seems to care. Technically speaking, though, I'm *not* puzzled by this solution and here's why: * It's simply not true that this port pollutes the JRuby port. Logstash is a self-contained Java application that bundles JRuby as a vendor JAR in a directory of its own, so there's no dependency on JRuby, let alone polluting it. The port updates the JRuby JAR bundled with logstash, keeping the patch private to it. * As you can see in my fork, the solution is not a "hack": it just fixes the bug the proper way. I bet upstream would merge it as it is since it leverages the same technique they are using to open native libraries; it just extends an existing regular expression in order for the library loader to property detect the FreeBSD library script, to scan it and to find the native library using *exactly* the same code path jnr-ffi is already using. I certainly agree on two things with you: * The JRuby port may be affected by the same bug (it probably is) and this information may be of interest to the JRuby's port maintainer, in case he has to handle a similar problem. * We can add clear information to the port so that people is aware of what's going on. I already have two detailed blog posts about this problem, plus the JRuby and the jnr-ffi PR's, plus the port itself. Maybe I could add some documentation about the change. Lastly, let me clarify my position on the maintainer thing: I set the maintainer in the Makefile because that Makefile comes straight from a GitHub repository of mine. I left my name in it when I submitted because I volunteered in case you would not/could not keep on actively maintaining the logstash port, and since the patch implies using a fork of mine, volunteering just seemed to me a logical option. I've been in touch with you and I even sent you this PR privately so that you could give me your feedback. While I empathise that you may not like the PR being sent this way and that you may be very busy, I kindly ask you to empathise with me and with all the people that has been asking for solutions to this problem (even privately, after reading my blog post about it). Months pass and a port is obsolete because of the unresponsiveness of a Java project which seems to be interested close to nothing to what happens to FreeBSD users. If you add to this that a working solution is a few-liners (to a fork, which can track future upstream changes), then I see no reason for waiting for something that may not be happening anytime soon. Cheers, -- Enrico
> Like i said by mail previously, there is two issues : > > - jruby cannot handle correctly the new libc header Agreed: this is principally a bug in jruby's handling of libc.so in FreeBSD 10.x and above. However, the proposed update doesn't affect anything other than the copy of jruby bundled within logstash, so I'm satisfied on that front. > - logstash need to update to latest w/complete rebrand of the content Are you saying here that logstash-1.2.1 and logstash-1.4.2 have incompatible data formats? If so, that does need to be addressed -- presumably there is some sort of update process provided by upstream. An UPDATING entry would be indicated. > p.s. : the patch you provide even remove the original port authors =/ Indeed. The original authors of the port should still be credited. Enrico, can you update your patch to restore the original 'Created by' line in the Makefile, and comment on what, if anything, users of 1.2.1 will need to do to update an existing installation to 1.4.2 please? > More seriously, I have very few time and it could be better for community that > some else maintain this port, so if you are up to, i’m not against. Thank you very much for approving the update and maintainership handover, and thank you very much indeed for your past work in maintaining this port.
[snip] Are you saying here that logstash-1.2.1 and logstash-1.4.2 have incompatible data formats? If so, that does need to be addressed -- presumably there is some sort of update process provided by upstream. An UPDATING entry would be indicated. [snip] Nope, i was referring to the way that logstash ships ; standalone jar vs ruby code.
Created attachment 148694 [details] logstash 1.4.2 - svn diff Updated to credit the original authors.
>> Like i said by mail previously, there is two issues : >> >> - jruby cannot handle correctly the new libc header > > Agreed: this is principally a bug in jruby's handling of libc.so in > FreeBSD 10.x and above. However, the proposed update doesn't affect > anything other than the copy of jruby bundled within logstash, so I'm > satisfied on that front. Thanks >> - logstash need to update to latest w/complete rebrand of the content > > Are you saying here that logstash-1.2.1 and logstash-1.4.2 have > incompatible data formats? If so, that does need to be addressed -- > presumably there is some sort of update process provided by upstream. > An UPDATING entry would be indicated. No, that's not an issue. The default port configuration stores and reads data in and from the bundled ElasticSearch instance. Each Logstash release comes with the recommended ElasticSearch release and there should not be such problems. In fact, I've been running almost any Logstash release since the current port version (1.2.x) and never had issues starting an updated service over an existing set of data. >> p.s. : the patch you provide even remove the original port authors =/ > > Indeed. The original authors of the port should still be credited. > Enrico, can you update your patch to restore the original 'Created by' > line in the Makefile, and comment on what, if anything, users of 1.2.1 > will need to do to update an existing installation to 1.4.2 please? I did not know that, I beg your pardon Regis, I thought the scope of that line was the current Makefile. I have just updated the Makefile to restore the headline with the original authors and updated this PR.
Created attachment 148696 [details] logstash 1.4.2 - svn diff * Updated to 1.4.2. * Restored missing metadata in logstash.in RC script.
Created attachment 148697 [details] poudriere testport sysutils/logstash output
Here are the SVN changes, to highlight new and deleted files: % svn st M Makefile M distinfo A files/elasticsearch.yml D files/elasticsearch.yml.sample A files/logstash.conf M files/logstash.in M pkg-plist
A commit references this bug: Author: matthew Date: Mon Oct 27 10:01:42 UTC 2014 New revision: 371543 URL: https://svnweb.freebsd.org/changeset/ports/371543 Log: Update to version 1.4.2 - New maintainer: enrico.m.crisostomo@gmail.com - Work-around for jruby problems groking the scripted libc.so in FreeBSD 10.0 and above. - Upstream has switched from shipping one big .jar file to expanding everything in the filesystem. This port has a long pkg-plist. PR: 194610 Submitted by: enrico.m.crisostomo@gmail.com (New maintainer) Approved by: regis.despres@gmail.com (Outgoing maintainer) Changes: head/sysutils/logstash/Makefile head/sysutils/logstash/distinfo head/sysutils/logstash/files/logstash.in head/sysutils/logstash/pkg-plist
Committed, thanks!
A commit references this bug: Author: matthew Date: Mon Oct 27 11:02:17 UTC 2014 New revision: 371550 URL: https://svnweb.freebsd.org/changeset/ports/371550 Log: It always helps if you use the correct patch file PR: 194610 Submitted by: enrico.m.crisostomo@gmail.com (maintainer) Changes: head/sysutils/logstash/files/elasticsearch.yml head/sysutils/logstash/files/elasticsearch.yml.sample head/sysutils/logstash/files/logstash.conf head/sysutils/logstash/files/logstash.in
Created attachment 148699 [details] logstash RC patch to add missing metadata Thanks Matt. The attached logstash-rc-patch.diff is required to bring files/logstash.in in line and it should be applied on top of the current SVN version.
A commit references this bug: Author: matthew Date: Mon Oct 27 11:30:54 UTC 2014 New revision: 371555 URL: https://svnweb.freebsd.org/changeset/ports/371555 Log: Eventually I'll get this right... PR: 194610 Submitted by: enrico.m.crisostomo@gmail.com Changes: head/sysutils/logstash/files/logstash.in
Thanks for the patch and commit. Just want to mention, that i've tried to install logstash through 'pkg install logstash' and it was falling with this exact error. Then i made portsnap update and rebuild/reinstalled logstash from the source and it finally is working for a while. Strange thing is that in both cases 'pkg info' showed "logstash-1.4.2_1 Tool for managing events and logs"
(In reply to Ilmar from comment #18) > Thanks for the patch and commit. > Just want to mention, that i've tried to install logstash through 'pkg > install logstash' and it was falling with this exact error. Then i made > portsnap update and rebuild/reinstalled logstash from the source and it > finally is working for a while. > Strange thing is that in both cases 'pkg info' showed > "logstash-1.4.2_1 Tool for managing events and logs" I found why this was working from port, i've uncommented "stdout { debug => "true" }" line in logstash.conf. With this config it starts logstash and Kubana web interface is accessible, but i see warnings in logstash.log: {:timestamp=>"2014-12-06T17:07:39.983000+0400", :message=>"Unknown setting 'debug' for stdout", :level=>:error} {:timestamp=>"2014-12-06T17:07:40.000000+0400", :message=>"Error: Something is wrong with your configuration."} {:timestamp=>"2014-12-06T17:07:40.019000+0400", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."} After i've removed this line, elasticserch begin to fall again. What can i do to track down the problem and make it work as it should?
Hi! I'm still getting the same error (see below) as You mentioned in https://github.com/jruby/jruby/issues/1754. I'm either doing something fundamentally wrong or the problem still persists with the logstash in the ports tree. I have tried using logstash package from pkgng repository and built it using poudriere, both ways I'm getting the same error message. # uname -mrsv FreeBSD 10.1-RELEASE-p6 FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 # pkg info logstash logstash-1.4.2_1 Name : logstash Version : 1.4.2_1 Installed on : Mon Mar 2 14:52:35 EET 2015 Origin : sysutils/logstash Architecture : freebsd:10:x86:64 Prefix : /usr/local Categories : sysutils java Licenses : APACHE20 Maintainer : enrico.m.crisostomo@gmail.com WWW : http://logstash.net/ Comment : Tool for managing events and logs Annotations : repo_type : binary repository : FreeBSD NotImplementedError: stat.st_dev unsupported or native support failed to load dev_major at org/jruby/RubyFileStat.java:190 _discover_file at /usr/local/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/watch.rb:140 each at org/jruby/RubyArray.java:1613 _discover_file at /usr/local/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/watch.rb:122 watch at /usr/local/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/watch.rb:34 tail at /usr/local/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb:58 run at /usr/local/logstash/lib/logstash/inputs/file.rb:130 each at org/jruby/RubyArray.java:1613 run at /usr/local/logstash/lib/logstash/inputs/file.rb:130 inputworker at /usr/local/logstash/lib/logstash/pipeline.rb:163 start_input at /usr/local/logstash/lib/logstash/pipeline.rb:157
Nevermind. My own fault, I was testing new config and at some point started to launch logstash by hand with "-v" effectively "working around" the fix implemented in this PR and that's why I saw these errors. Everything works if I launch Logstash using startup scripts where appropriate arguments are used, that disable built in "ffi".