Bug 223806 - java/openjdk8: [PATCH] arm64 package builds do not allow enough time for build to complete
Summary: java/openjdk8: [PATCH] arm64 package builds do not allow enough time for buil...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: arm64 Any
: --- Affects Only Me
Assignee: Port Management Team
Keywords: patch
Depends on:
Blocks: 203349
  Show dependency treegraph
Reported: 2017-11-22 20:43 UTC by Ed Maste
Modified: 2018-06-19 05:36 UTC (History)
5 users (show)

See Also:

patch to increase verbosity of openjdk8 build (16.31 KB, patch)
2017-12-01 21:14 UTC, Zak
no flags Details | Diff
patch to increase verbosity of openjdk8 build (404 bytes, patch)
2017-12-01 21:26 UTC, Zak
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer 2017-11-22 20:43:52 UTC
From http://thunderx1.nyi.freebsd.org/data/head-arm64-default/p454157_s325785/logs/errors/openjdk8-8.144.1.log:

Compiling 9467 files for BUILD_JDK
====>> Killing runaway build after 7200 seconds with no output

I tested openjdk build on my SoftIron OverDrive 1000 and the package built successfully. It does have a relatively long period of no output in the "BUILD_JDK" stage.
Comment 1 Antoine Brodin freebsd_committer 2017-11-22 20:46:27 UTC
Isn't it possible to make the build more verbose?  2 hours without any output on physical machine is.. a bit strange.
Comment 2 Ed Maste freebsd_committer 2017-11-22 20:46:58 UTC
The individual build steps are taking exceptionally long on the ThunderX official builds though; from my build on the OD1000, on a 12-CURRENT kernel with WITNESS enabled,

## Finished langtools (build time 00:16:51)
## Finished hotspot (build time 00:03:57)
## Finished corba (build time 00:07:23)
## Finished jaxp (build time 00:10:57)
## Finished jaxws (build time 00:12:22)
## Finished jdk (build time 01:27:50)
## Finished demos (build time 00:00:59)
## Finished nashorn (build time 00:04:55)
## Finished images (build time 00:03:35)

Excerpted build times, from the aborted official build:

## Finished langtools (build time 01:45:27)
## Finished hotspot (build time 00:50:23)
## Finished corba (build time 00:58:04)
## Finished jaxp (build time 00:49:54)
## Finished jaxws (build time 01:00:41)
Comment 3 Ed Maste freebsd_committer 2017-11-23 04:16:12 UTC
I just completed a non-parallel openjdk8 build on a 96-core ThunderX system at packet.net (so it had 1 core building openjdk, and 95 idle cores). The build took about 3.5 hours, with component build times as follows:

## Finished langtools (build time 00:24:52)
## Finished hotspot (build time 00:02:43)
## Finished corba (build time 00:10:58)
## Finished jaxp (build time 00:16:26)
## Finished jaxws (build time 00:18:57)
## Finished jdk (build time 02:07:51)
## Finished demos (build time 00:00:34)
## Finished nashorn (build time 00:07:06)
## Finished images (build time 00:05:08)

This is using Packet's arm64 testing image:

FreeBSD freebsd-java-test 12.0-CURRENT FreeBSD 12.0-CURRENT #0: Sun Nov 19 00:36:40 UTC 2017     root@packet-build-arm64:/usr/obj/usr/src/arm64.aarch64/sys/PACKET  arm64

This image has WITNESS on (as did my OD1000 image).
Comment 4 Antoine Brodin freebsd_committer 2017-11-28 15:38:19 UTC
Even with a 4 hours timeout it fails:

Comment 5 Ed Maste freebsd_committer 2017-12-01 14:03:46 UTC
I was mistaken: timings I reported above for the od1000 and 96-core ThunderX were with the ports system's default of building on all CPUs. I a spot check of the build and saw only one process running, but presume I must have just been unlucky and happened to look during a stage in the build which was inherently non-parallel.

With MAKE_JOBS_NUMBER=2 in /etc/make.conf the build reported the following times:

----- Build times -------
Start 2017-12-01 02:07:47
End   2017-12-01 06:01:19
00:11:04 corba
00:02:39 demos
00:08:57 hotspot
00:05:11 images
00:16:28 jaxp
00:18:36 jaxws
02:18:41 jdk
00:24:47 langtools
00:07:08 nashorn
03:53:32 TOTAL

This is on an otherwise unused 96 core ThunderX at packet.net, running a recent FreeBSD 12-CURRENT GENERIC-NODEBUG kernel, with a local SSD.
Comment 6 Ed Maste freebsd_committer 2017-12-01 18:03:15 UTC
For comparison, with MAKE_JOBS_NUMBER=1:

Start 2017-12-01 12:41:42
End   2017-12-01 17:21:15
00:19:46 corba
00:05:07 demos
00:15:55 hotspot
00:06:04 images
00:16:30 jaxp
00:19:01 jaxws
02:45:02 jdk
00:24:52 langtools
00:07:16 nashorn
04:39:33 TOTAL
Comment 7 Zak 2017-12-01 21:14:08 UTC
Created attachment 188454 [details]
patch to increase verbosity of openjdk8 build
Comment 8 Zak 2017-12-01 21:17:26 UTC
Possible solution to increase verbosity of build; from the openJDK readme:

>Q: I want to see the output of the commands that make runs, like in the old build. How do I do that?
>A: You specify the LOG variable to make. There are several log levels:
>       warn — Default and very quiet.
>       info — Shows more progress information than warn.
>       debug — Echos all command lines and prints all macro calls for compilation definitions.
>       trace — Echos all $(shell) command lines as well.

appending LOG=debug to MAKE_ENV in the openJDK makefile seems to produce more output during the BUILD_JDK stage. see patch.
Comment 9 Zak 2017-12-01 21:26:09 UTC
Created attachment 188455 [details]
patch to increase verbosity of openjdk8 build
Comment 10 Jung-uk Kim freebsd_committer 2017-12-04 20:53:18 UTC
(In reply to Zak from comment #9)
If my memory serves, it won't fix the problem.  Basically, all you can see is "javac ..." after "Compiling 9467 files for BUILD_JDK" because just one command is taking too long.
Comment 11 Ed Maste freebsd_committer 2017-12-04 22:54:59 UTC
(In reply to Jung-uk Kim from comment #10)
I thought LOG=debug produced some output for each of the files in that stage.
Comment 12 Jung-uk Kim freebsd_committer 2017-12-04 23:15:56 UTC
(In reply to Ed Maste from comment #11)
I just tested the patch and I verified that the exact command you see on amd64 is:

(/usr/local/bootstrap-openjdk8/bin/java -Xms64M -Xmx1600M -XX:ThreadStackSize=1536 "-Xbootclasspath/p:/wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/langtools/dist/bootstrap/lib/javac.jar" -cp /wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/langtools/dist/bootstrap/lib/javac.jar com.sun.tools.javac.Main -bootclasspath /wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/jdk/classes -source 8 -target 8 -encoding ascii -XDignore.symbol.file=true -Xlint:-unchecked,-deprecation,-overrides,auxiliaryclass,classfile,dep-ann,divzero,empty,try,varargs -Werror   -implicit:none -sourcepath "/wrkdirs/usr/ports/java/openjdk8/work/openjdk/jdk/src/share/classes:/wrkdirs/usr/ports/java/openjdk8/work/openjdk/jdk/src/solaris/classes:/wrkdirs/usr/ports/java/openjdk8/work/openjdk/jdk/src/bsd/classes:/wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/jdk/gensrc:/wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/jdk/gensrc_no_srczip" -d /wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/jdk/classes -h /wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/jdk/gensrc_headers.BUILD_JDK.tmp @/wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/jdk/classes/_the.BUILD_JDK_batch.tmp && /bin/mv /wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/jdk/classes/_the.BUILD_JDK_batch.tmp /wrkdirs/usr/ports/java/openjdk8/work/openjdk/build/bsd-x86_64-normal-server-release/jdk/classes/_the.BUILD_JDK_batch)

This one liner compiles 9469 files.
Comment 13 Jung-uk Kim freebsd_committer 2017-12-04 23:30:55 UTC
If there are many cores, we can improve compile time with sjavac.  I tried it couple of years ago but I had to back it out because some people reported it didn't work for them.  See bug 205544.


I didn't try harder at the time, though.
Comment 14 Ed Maste freebsd_committer 2018-06-19 02:10:25 UTC
Can we try just increasing the build timeout for the time being?
Comment 15 Antoine Brodin freebsd_committer 2018-06-19 05:36:53 UTC
It seems this was fixed a few months ago.