Bug 222322 - parallel doc build
Summary: parallel doc build
Status: Closed FIXED
Alias: None
Product: Documentation
Classification: Unclassified
Component: Books & Articles (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Wolfram Schneider
URL:
Keywords:
Depends on: 222186 222327
Blocks:
  Show dependency tree / graph
 
Reported: 2017-09-14 09:52 UTC by Wolfram Schneider
Modified: 2017-09-21 10:08 UTC (History)
2 users (show)

See Also:

Attachments
patch (588 bytes, text/plain)
2017-09-14 09:52 UTC, Wolfram Schneider 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Wolfram Schneider freebsd_committer freebsd_triage 2017-09-14 09:52:21 UTC 
Created attachment 186370 [details]
patch

Hi,

the build in ./doc runs for 20-25minutes. Speeding it up with `make -j`' has no effect. It would be great if we can build the documentation faster.


Looking at the makefiles in doc/share/mk this doesn't seems to be an easy task.

Also, most sub-directories contains only a single input file which means `make -j' has no effect on a directory level. The workaround could be to run `make -j' for the sub-directories in a given directory.

Attached is a simple workaround to run the sub-directories in parallel. It works for the standard target ("all") only. I could run a build in 4 minutes now (down from 20 minutes). I do not recommend this for production, but for test & developing it makes a huge difference.


$ make p-all
Comment 1 Benedict Reuschling freebsd_committer freebsd_triage 2017-09-14 13:26:51 UTC 
Hi Wosch,

yes, the 20 minutes build time is a problem. We've tried to involve people more knowlegdeable about Make and friends to improve this, but were not successful yet or they had time constraints. Glad to see you managed to improve the build, but a long term solution would be to have this (in a stable version) in the regular doc build. Adding doceng@ so that they can look at integrating this into the doc toolchain.
Comment 2 Wolfram Schneider freebsd_committer freebsd_triage 2017-09-14 15:11:40 UTC 
I did the tests on a quad core machine. On an octa-core or dual hexa-core machine you may get a full build done in less than two minutes. However, it cannot be faster than the slowest sub-directory build which is en_US.ISO8859-1
Comment 3 Wolfram Schneider freebsd_committer freebsd_triage 2017-09-17 07:40:29 UTC 
Actually, the perl one-liners in the patch should be replaced with the standard tools sed/tr. Not a big deal, I will do this later.
Comment 4 commit-hook freebsd_committer freebsd_triage 2017-09-19 10:03:38 UTC 
A commit references this bug:

Author: wosch
Date: Tue Sep 19 10:03:26 UTC 2017
New revision: 50880
URL: https://svnweb.freebsd.org/changeset/doc/50880

Log:
  Implement workaround for a parallel doc build

  $ make p-all

  It works for the sub-directories in a given directory.

  On a standard quad core machine the build time goes down
  from ca. 20min to 5 minutes.

  PR: 222322

Changes:
  head/share/mk/doc.project.mk
Comment 5 commit-hook freebsd_committer freebsd_triage 2017-09-19 17:40:35 UTC 
A commit references this bug:

Author: wosch
Date: Tue Sep 19 17:40:06 UTC 2017
New revision: 50887
URL: https://svnweb.freebsd.org/changeset/doc/50887

Log:
  support parallel clean

  $ make p-clean

  PR: 222322

Changes:
  head/share/mk/doc.project.mk
Comment 6 commit-hook freebsd_committer freebsd_triage 2017-09-21 10:08:48 UTC 
A commit references this bug:

Author: wosch
Date: Thu Sep 21 10:08:01 UTC 2017
New revision: 50896
URL: https://svnweb.freebsd.org/changeset/doc/50896

Log:
  support parallel obj

  $ make p-obj

  PR: 222322

Changes:
  head/share/mk/doc.project.mk