Bug 227460 - devel/llvm60: build failure in poudriere: Exception occured
Summary: devel/llvm60: build failure in poudriere: Exception occured
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Brooks Davis
Depends on:
Reported: 2018-04-12 10:07 UTC by O. Hartmann
Modified: 2018-05-06 16:45 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (brooks)

poudriere log (146.32 KB, application/x-xz)
2018-05-06 06:14 UTC, Fukang Chen
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description O. Hartmann 2018-04-12 10:07:17 UTC
Recent build of devel/llvm60 on CURRENT host, poudriere CURRENT jail (1200061) fails due to the error shown below:

[4590/4596] cd /wrkdirs/usr/ports/devel/llvm60/work/.build/docs && /usr/local/bin/sphinx-build -b man -d /wrkdirs/usr/ports/devel/llvm60/work/.build/docs/_doctrees-llvm-dwarfdump-man -q /wrkdirs/usr/ports/devel/llvm60/work/llvm-6.0.0.src/docs /wrkdirs/usr/ports/devel/llvm60/work/.build/docs/man
FAILED: docs/CMakeFiles/docs-llvm-dwarfdump-man 
cd /wrkdirs/usr/ports/devel/llvm60/work/.build/docs && /usr/local/bin/sphinx-build -b man -d /wrkdirs/usr/ports/devel/llvm60/work/.build/docs/_doctrees-llvm-dwarfdump-man -q /wrkdirs/usr/ports/devel/llvm60/work/llvm-6.0.0.src/docs /wrkdirs/usr/ports/devel/llvm60/work/.build/docs/man

Exception occurred:
  File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/wrkdirs/usr/ports/devel/llvm60/work/.build/docs/man'
The full traceback has been saved in /tmp/sphinx-err-VHavaH.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
[4591/4596] cd /wrkdirs/usr/ports/devel/llvm60/work/.build/docs && /usr/local/bin/sphinx-build -b man -d /wrkdirs/usr/ports/devel/llvm60/work/.build/docs/_doctrees-dsymutil-man -q /wrkdirs/usr/ports/devel/llvm60/work/llvm-6.0.0.src/docs /wrkdirs/usr/ports/devel/llvm60/work/.build/docs/man
FAILED: docs/CMakeFiles/docs-dsymutil-man 
cd /wrkdirs/usr/ports/devel/llvm60/work/.build/docs && /usr/local/bin/sphinx-build -b man -d /wrkdirs/usr/ports/devel/llvm60/work/.build/docs/_doctrees-dsymutil-man -q /wrkdirs/usr/ports/devel/llvm60/work/llvm-6.0.0.src/docs /wrkdirs/usr/ports/devel/llvm60/work/.build/docs/man

Exception occurred:
  File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/wrkdirs/usr/ports/devel/llvm60/work/.build/docs/man'
Comment 1 Brooks Davis freebsd_committer 2018-04-13 21:24:27 UTC
Is this reproducible?
Comment 2 O. Hartmann 2018-04-14 14:30:43 UTC
(In reply to Brooks Davis from comment #1)

No, not on all (CURRENT and 11.1-RELENG) machines I use. The problem is alos NOT present when compiling the port the only true way, via make ;-)

I found out that even (on most recent CURRENT, host and poudriere jail) when the whole set of packages of our individual repository is deleted and recompiled due to a bump of the revision number of the jail, the problem persists. But the problem vanish into thin air when performing a "poudriere options ... -c devel/llvm60".

Sometimes I wish I would have a knob which could provide such a thing like cleaning up the ports configuration and restore it as it has been set.

I think this problem affects only a very small set of people, it might be random and seems to be a flaw in the bigger picture of how ports work.

I close the PR ...

Thanks for responding.
Comment 3 Brooks Davis freebsd_committer 2018-04-15 15:10:42 UTC
There have been races in the llcm document build infrastructure before.  I wouldn't surprise if someone reintroduced one.
Comment 4 Fukang Chen freebsd_committer 2018-05-06 06:14:45 UTC
Created attachment 193077 [details]
poudriere log

   113      if (${SPHINX_OUTPUT_MAN})
   114        add_sphinx_target(man llvm)
   115        add_sphinx_target(man llvm-dwarfdump)
   116        add_sphinx_target(man dsymutil)
   117      endif()

It generates three targes with the same output directory

build docs/CMakeFiles/docs-dsymutil-man: CUSTOM_COMMAND
  COMMAND = cd /usr/ports/devel/llvm60/work/.build/docs && /usr/local/bin/sphinx-build -b man -d /usr/ports/devel/llvm60/work/.build/docs/_doctrees-dsymutil-man -q /usr/ports/devel/llvm60/work/llvm-6.0.0.src/docs /usr/ports/devel/llvm60/work/.build/docs/man
build docs/CMakeFiles/docs-llvm-man: CUSTOM_COMMAND
  COMMAND = cd /usr/ports/devel/llvm60/work/.build/docs && /usr/local/bin/sphinx-build -b man -d /usr/ports/devel/llvm60/work/.build/docs/_doctrees-llvm-man -q /usr/ports/devel/llvm60/work/llvm-6.0.0.src/docs /usr/ports/devel/llvm60/work/.build/docs/man
build docs/CMakeFiles/docs-llvm-dwarfdump-man: CUSTOM_COMMAND
  COMMAND = cd /usr/ports/devel/llvm60/work/.build/docs && /usr/local/bin/sphinx-build -b man -d /usr/ports/devel/llvm60/work/.build/docs/_doctrees-llvm-dwarfdump-man -q /usr/ports/devel/llvm60/work/llvm-6.0.0.src/docs /usr/ports/devel/llvm60/work/.build/docs/man

all these sphinx-build processes check/create the output directory:
161       if not path.isdir(outdir):
162            logger.info('making output directory...')
163            os.makedirs(outdir)

a workaround maybe

Index: Makefile
--- Makefile	(revision 469090)
+++ Makefile	(working copy)
@@ -358,6 +358,9 @@
+	${MKDIR} ${WRKDIR}/.build/docs/man
 	${MV} ${WRKSRC_clang} ${PATCH_WRKSRC}/tools/clang
Comment 5 Brooks Davis freebsd_committer 2018-05-06 15:09:35 UTC
I believe this analysis is correct.  The workaround might work, but isn't right.  I've got a patch that I think does the right that that I'll commit shortly.
Comment 6 commit-hook freebsd_committer 2018-05-06 16:00:43 UTC
A commit references this bug:

Author: brooks
Date: Sun May  6 16:00:01 UTC 2018
New revision: 469235
URL: https://svnweb.freebsd.org/changeset/ports/469235

  Work around a race in sphinx.

  Create the target directory before calling sphinx to render
  content into it.  Sphinx suffers from a race where it crashes with an
  exception if it tries to make a directory and someone else does first.

  PR:		227460
  Reported by:	ohartmann@walstatt.org
  MFH:		2018Q2

Comment 7 Fukang Chen freebsd_committer 2018-05-06 16:28:08 UTC
Thanks brooks@ :)
Comment 8 Brooks Davis freebsd_committer 2018-05-06 16:45:47 UTC
Hopefully fixed in llvm60 and llvm-devel.  I've submitted the patch upstream as well https://reviews.llvm.org/D46503.