Bug 215286 - [java/openjdk8] Java crashes/exits on i386 as soon as kernel options KVA_PAGES = 512 and KSTACK_PAGES = 4 set for ZFS
Summary: [java/openjdk8] Java crashes/exits on i386 as soon as kernel options KVA_PAGE...
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: i386 Any
: --- Affects Many People
Assignee: freebsd-java mailing list
Depends on:
Reported: 2016-12-13 19:51 UTC by Michael Osipov
Modified: 2017-05-07 15:47 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (java)

Log file (18.26 KB, text/plain)
2016-12-13 19:51 UTC, Michael Osipov
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Osipov 2016-12-13 19:51:42 UTC
Created attachment 177950 [details]
Log file


$ uname -a
FreeBSD bsd1home 11.0-STABLE FreeBSD 11.0-STABLE #0 r309819: Sun Dec 11 11:45:30 CET 2016     mosipov@bsd1home:/usr/obj/usr/src/sys/BSD1HOME  i386

With OpenJDK 8 from ports and pkg (tried both):
$ java -version
openjdk version "1.8.0_112"
OpenJDK Runtime Environment (build 1.8.0_112-b16)
OpenJDK Server VM (build 25.112-b16, mixed mode)

Maven 3.3.9 installed from ports:
$ mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T17:41:47+01:00)
Maven home: /usr/local/apache-maven-3.3.9
Java version: 1.8.0_112, vendor: Oracle Corporation
Java home: /usr/local/openjdk8/jre
Default locale: de_DE, platform encoding: UTF-8
OS name: "freebsd", version: "11.0-stable", arch: "i386", family: "unix"

Trying compile Maven (mvn clean package) from Git master 33c79c6e4662abb836df95a5d70c236f2e5cda3d aborts with:
# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0x28c551dc, pid=1679, tid=0x00018949
# JRE version: OpenJDK Runtime Environment (8.0_112-b16) (build 1.8.0_112-b16)
# Java VM: OpenJDK Server VM (25.112-b16 mixed mode bsd-x86 )
# Problematic frame:
# V  [libjvm.so+0x4551dc]  AsyncGetCallTrace+0x6cefc
# Core dump written. Default location: /usr/home/mosipov/Projekte/maven/java.core
# An error report file with more information is saved as:
# /usr/home/mosipov/Projekte/maven/hs_err_pid1679.log
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
Abort trap (Speicherabzug geschrieben)

Surprisingly, all is peachy with OpenJDK 7.

Log file attached.
Comment 1 Michael Osipov 2016-12-13 19:52:52 UTC
Find the core file in this directory: http://home.apache.org/~michaelo/freebsd/issue-215286/
Comment 2 Michael Osipov 2017-05-07 09:54:03 UTC
I was finally able to locate the cause of this. Tried on two different setups:

Machine 1:
Pentium 4 1.4 GHz and 2 GiB RAM.

> FreeBSD bsd1home 11.0-STABLE FreeBSD 11.0-STABLE #0 r317824: Sat May  6 22:35:35 CEST 2017     mosipov@bsd1home:/usr/obj/usr/src/sys/GENERIC  i386

> openjdk version "1.8.0_131"
> OpenJDK Runtime Environment (build 1.8.0_131-b11)
> OpenJDK Server VM (build 25.131-b11, mixed mode)

to make sure that the machine is not faulty, I have set up the same in VirtualBox, 32 bit with 2 GiB, machine 2:

> FreeBSD freebsd11-32bit 11.0-STABLE FreeBSD 11.0-STABLE #1 r317854: Fri May  5 23:55:01 CEST 2017     root@freebsd11-32bit:/usr/obj/usr/src/sys/GENERIC  i386

> openjdk version "1.8.0_131"
> OpenJDK Runtime Environment (build 1.8.0_131-b11)
> OpenJDK Server VM (build 25.131-b11, mixed mode)

Then run as root:
$ pkg install openjdk7 openjdk8 git maven33
$ git clone https://git-wip-us.apache.org/repos/asf/maven.git
$ cd maven
$ mvn clean package

It shall run flawlessly on Java 8

Now create a custom kernel config ZFS:
include GENERIC
ident BSD1HOME
options KVA_PAGES=512
options KSTACK_PAGES=4

Compile the kernel with that and rerun the mvn command, you'll see crashes like:

> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (malloc) failed to allocate 146808 bytes for Chunk::new
> # An error report file with more information is saved as:
> # /usr/home/mosipov/Projekte/maven/hs_err_pid934.log
> #
> # Compiler replay data is saved as:
> # /usr/home/mosipov/Projekte/maven/replay_pid934.log

or hard SIGSEGV with AsyncGetCallTrace+0x6cefc. Updated files in http://home.apache.org/~michaelo/freebsd/issue-215286/.

Surprisingly, OpenJDK 7 works flawlessly.

There must be some broken interaction between KVA_PAGES = 512/KSTACK_PAGES = 4 and Java 8.
Comment 3 Konstantin Belousov freebsd_committer 2017-05-07 10:48:04 UTC
The message from the JVM allocator is clear enough, isn't it ?

KVA_PAGES=512 option re-partitions the kernel/user virtual address space split into 2G/2G, while the default partition is 3G/1G.  Among the 2G space, at least 512M is reserved for the binary and sbrk, rest is fragmented by the loaded libraries and other mappings.

This probably lefts around 700-800M, in the best case, for the Java heap.  Since JDK 7 and 8 have different GCs, it is not surprising that things start to break.  I suspect that you use copying GC with JVM 8, which requires approximately twice as many address space as compacting collector, and 800Mb only allows for 400Mb max heap size.
Comment 4 Michael Osipov 2017-05-07 15:09:34 UTC
(In reply to Konstantin Belousov from comment #3)

Thanks for the analysis. I already assumed that due to Metaspace in Java 8 the process is starving for memory. In fact, I did top on Java 7 and 8 on that VM and the memory consumption of Java 8 is way higher, so is the runtime (doubles for the same operation).

This basically means that one cannot reconcile ZFS and Java 8 on i386?
Comment 5 Konstantin Belousov freebsd_committer 2017-05-07 15:44:28 UTC
If you need 32bit JVM, it happily run on amd64 kernel with compat32.  From what I know, ZFS on i386 is not robust for many reasons, only one of which is the limited KVA.
Comment 6 Michael Osipov 2017-05-07 15:47:51 UTC
(In reply to Konstantin Belousov from comment #5)

Running amd64 is not an option on this machine, this is a dated machine used for server services at home as well as sw development. What we can do at me is to improve docs on ZFS in the handbook and referencing this issue?