Created attachment 163046 [details] hbase 1.1.2 patch Take maintainership per private email exchange. This passes poudriere's testport. There are a few things made easier to standardize on JDK 1.8 which ports is trying to do anyways, but it does cut against upstream -bin builds for now.
Well, few things: 1) AFAIR we did not discuss maintainership change :) 2) Since you use JAVA_VERSION= 1.8 does it mean HBase dropped support for 1.7??
1) Fair enough, happy to pawn this off to you :) 2) This was a choice I made in porting; their official -bin.tar.gz are compiled with 1.7, but it makes things harder for us, specifically handling bugs/fallout in doc generation/jruby. AFAIK the desire is to move all ports to Java 1.8 since it is more friendly with clang.
(In reply to kevin.bowling from comment #2) Problem with Java 1.8 is that current version of hadoop (2.7.1) officially supports Java 1.7 only. There is a separated branch of hadoop where they do porting to version 1.8, but I do not know what stable version they target. And since hbase is coupled tightly with hadoop, I think it would be probably wise to require the same Java version for both of them.
Created attachment 163073 [details] hbase 1.1.2 patch with fixes
You're right. It struggled with my JDK1.7 Hadoop environment with native libs. 2015-11-12 23:00:03,447 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable If I set JAVA_VERSION= 1.7, it isn't enough because maven uses JDK1.8 by default. Any suggestions?
If I do 'make JAVA_HOME=/usr/local/openjdk7' everything works great, but that obviously wont work in pourdriere, etc.
Perfect issue self-triaging Kevin, well done.
Well, it seems you are right: as long as I build ports myself with Java 7 installed, everything is fine. Since default Java version is 1.8, package builder cluster will install JDK-1.8 as maven dependency and HBase (as well as Hadoop and other maven-based ports) will use Java-8.
Could we do something like append JAVA_HOME=${JAVA_HOME} in do-build's call to mvn, which gets set by USE_JAVA=1.7? It seems kind of hacks but may be acceptable?
I think this is a common issue with ports using non-default java version and maven. I asked Java team: https://lists.freebsd.org/pipermail/freebsd-java/2015-November/011313.html I think we should discuss a common solution there and use it in all relevant ports.
Anyway, if you agree that we should stick with JDK 1.7 for now, deel free to update the patch and I'll commit it, it would not hurt, the problem with package builder is here even before this update.
Yes let's stick with JDK7 for everything for now. Hadoop 2.8.0 should let us reevaluate (i.e. https://issues.apache.org/jira/browse/YARN-4294) and force things hadoop2 and this to JDK8. One other thing to discuss, I was sometimes seeing permgen exhaustion during build even with my last patch in the second mvn run.
Will you update your patch for Java-7?
Created attachment 163271 [details] hbase 1.1.2 patch with mandatory JDK 1.7 This has mandatory JDK 1.7 JAVA_HOME passed to mvn. It should also work around fat permgen with MAVEN_OPTS (I was incorrectly setting MAVEN_OPTIONS).
It looks better now, thanks! A couple of questions: why do you apply extra-patch-hbase-annotations-pom.xml in case of JDK-1.7? I think the problem with java-7 ports which use maven (which in turn pulls java-8) is common for all such ports. Shouldn't we use MVN_OPTS= JAVA_HOME=${JAVA_HOME} in all such ports to workaround this problem?
hbase-annotations-pom was an accidental inclusion for 1.7. Do you need a new patch or can you fix that on commit? The other ports (hadoop2, maven33) build with 1.8 but seem to emit 1.7 compatible bytecode in that they run for me with JDK 1.7. Maybe commit this as known working and we decide what to do next? -- In talking with a large FreeBSD hadoop user that doesn't use ports, I'm more inclined to suggest we update hadoop2, spark, hbase, snappy-java, etc all to 1.8 so JNI libs line up. We are in a gray zone with support, but so is FreeBSD support in general and the wins of JDK 1.8 offset the risks IMHO. The only things in Apache's JIRA related to JDK 1.8 seem to be around enabling lint/unit tests by default in Hadoop 2.8.0 to make it "officially supported" (we may need to pull back YARN-4294). * https://issues.apache.org/jira/browse/HADOOP-11090 * https://wiki.apache.org/hadoop/MovingToJdk7and8 * https://issues.apache.org/jira/browse/YARN-4294
Okay, I am trying to build 1.1.2 with your patch and I get the following error: [root@agata /usr/ports/databases/hbase/work/hbase-1.1.2]# JAVA_HOME=/usr/local/openjdk7 MAVEN_OPTS="-XX:MaxPermSize=256m" /usr/local/share/java/maven3/bin/mvn -X -Dmaven.repo.local=/usr/ports/databases/hbase/work/m2 install -DskipTests site assembly:single -Prelease,native <...> [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.asciidoctor:asciidoctor-maven-plugin:1.5.2.1:process-asciidoc (output-pdf) on project hbase: Execution output-pdf of goal org.asciidoctor:ascii doctor-maven-plugin:1.5.2.1:process-asciidoc failed: (NotImplementedError) fstat unimplemented unsupported or native support failed to load -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.asciidoctor:asciidoctor-maven-plugin:1.5.2.1:process-asciidoc (output-pdf) on project hbase: Ex ecution output-pdf of goal org.asciidoctor:asciidoctor-maven-plugin:1.5.2.1:process-asciidoc failed: (NotImplementedError) fstat unimplemented unsupported or native support faile d to load at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) at org.codehaus.classworlds.Launcher.main(Launcher.java:47) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution output-pdf of goal org.asciidoctor:asciidoctor-maven-plugin:1.5.2.1:process-asciidoc failed: (NotImplemente dError) fstat unimplemented unsupported or native support failed to load at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 20 more Caused by: org.jruby.exceptions.RaiseException: (NotImplementedError) fstat unimplemented unsupported or native support failed to load at org.jruby.RubyFile.size(org/jruby/RubyFile.java:1108) at RUBY.render_body(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/pdf-core-0.2.5/lib/pdf/core /document_state.rb:69) at RUBY.each(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/pdf-core-0.2.5/lib/pdf/core/object _store.rb:70) at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613) at RUBY.each(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/pdf-core-0.2.5/lib/pdf/core/object _store.rb:69) at RUBY.render_body(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/pdf-core-0.2.5/lib/pdf/core /document_state.rb:68) at RUBY.render_body(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/prawn-1.2.1/lib/prawn/docum ent/internals.rb:141) at RUBY.render(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/prawn-1.2.1/lib/prawn/document.r b:359) at RUBY.render_file(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/prawn-1.2.1/lib/prawn/docum ent.rb:376) at org.jruby.RubyIO.open(org/jruby/RubyIO.java:1181) at RUBY.render_file(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/prawn-1.2.1/lib/prawn/docum ent.rb:376) at RUBY.write(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj-pdf/1.5.0-alpha.6/asciidoctorj-pdf-1.5.0-alpha.6.jar!/gems/asciidoctor-pdf-1.5.0.alpha.6/lib /asciidoctor-pdf/converter.rb:1235) at RUBY.write(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj/1.5.2/asciidoctorj-1.5.2.jar!/gems/asciidoctor-1.5.2/lib/asciidoctor/document.rb:1051) at RUBY.convert(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj/1.5.2/asciidoctorj-1.5.2.jar!/gems/asciidoctor-1.5.2/lib/asciidoctor.rb:1504) at RUBY.convert_file(/usr/ports/databases/hbase/work/m2/org/asciidoctor/asciidoctorj/1.5.2/asciidoctorj-1.5.2.jar!/gems/asciidoctor-1.5.2/lib/asciidoctor.rb:1562) at RUBY.convertFile(<script>:68) at org.jruby.gen.InterfaceImpl2135749672.convertFile(org/jruby/gen/InterfaceImpl2135749672.gen:13) [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
I tried with and without ruby installed. Do we need some specific ruby modules?
Looks like the following chunk + <dependency> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + <version>9.0.3.0</version> + </dependency> is needed in Java-7 case too, not only for Java-8.
A commit references this bug: Author: demon Date: Thu Nov 19 15:21:16 UTC 2015 New revision: 401951 URL: https://svnweb.freebsd.org/changeset/ports/401951 Log: Update to version 1.1.2. PR: 204495 Submitted by: kevin.bowling@kev009.com Changes: head/databases/hbase/Makefile head/databases/hbase/distinfo head/databases/hbase/files/extra-patch-PoolMap.java head/databases/hbase/files/extra-patch-hbase-annotations-pom.xml head/databases/hbase/files/extra-patch-pom.xml head/databases/hbase/files/patch-bin-hbase head/databases/hbase/files/patch-hbase-daemon.sh head/databases/hbase/files/patch-hbase-env.sh head/databases/hbase/files/patch-pom.xml head/databases/hbase/pkg-plist
I committed your patch with slight modifications. I hope I did not broke things, if so please report.