Bug 252352
| Summary: | biology/pooler needs MAKE_JOBS_UNSAFE | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Silas S. Brown <ssb22> | ||||||
| Component: | Individual Port(s) | Assignee: | Jason W. Bacon <jwb> | ||||||
| Status: | Closed FIXED | ||||||||
| Severity: | Affects Only Me | CC: | jwb | ||||||
| Priority: | --- | ||||||||
| Version: | Latest | ||||||||
| Hardware: | Any | ||||||||
| OS: | Any | ||||||||
| Attachments: |
|
||||||||
Maintainer informed via mail The only failures I'm seeing in pkg-fallout are on armv[67], e.g. https://docs.freebsd.org/cgi/getmsg.cgi?fetch=173847123+0+current/freebsd-pkg-fallout That's showing a failure to find omp.h. Did you get something different? Ah, that's different from the email I got from pkg-fallout, which showed a permissions error after it tried to write to 32-64.h a second time when the first process had already marked it read-only. The make rule chmods it read-write, updates it, then chmods it read-only again, and the log showed that two identical copies of this same make rule were being run in parallel, with obvious race-conditions that wouldn't be reproduced every time. Since 32-64.h only needs to be made once, I don't know how make ended up deciding to run it in duplicate on two separate cores, but it's not suitable for that (even if we didn't have the read-only thing, there might still be issues on some filesystems if two processes try to write the same file simultaneously). omp.h is included only if _OPENMP is defined. For single-core machines we could drop the -fopenmp flag and it should then compile without OpenMP libraries, but it would be nice if OpenMP can work on armv6/7 as this architecture is supposed to support SMP and the later Raspberry Pi machines are quad-core. I don't know why omp.h is missing on that architecture; maybe this isn't something we can fix yet. But it would still be nice to eliminate the chance of having two copies of 32-64.h generated at the same time. thanks. I see the problem since you sent the pkg-fallout log (off-list). I'm still not seeing it on the pkg-fallout web sites, which seems odd.
It's an issue of how make handles multiple targets in the same rule:
128.h 32.h 32-64.h: 64.h 64-128.h
chmod +w 128.h 32.h 32-64.h 2>/dev/null || true
sed -e s/64/128/g < 64.h > 128.h
sed -e s/64/32/g < 64.h > 32.h
sed -e s/64/32/g -e s/128/64/g < 64-128.h > 32-64.h
chmod -w 128.h 32.h 32-64.h
Both BSD make and gmake try to build them all in parallel by simply running the same build commands for each target, so switching to gmake won't help:
BSD make:
===> Building for pooler-1.75
--- 32.h ---
--- 32-64.h ---
--- 128.h ---
--- 32.h ---
chmod +w 128.h 32.h 32-64.h 2>/dev/null || true
--- 128.h ---
chmod +w 128.h 32.h 32-64.h 2>/dev/null || true
--- 32-64.h ---
chmod +w 128.h 32.h 32-64.h 2>/dev/null || true
sed -e s/64/128/g < 64.h > 128.h
--- 32.h ---
sed -e s/64/128/g < 64.h > 128.h
--- 128.h ---
sed -e s/64/128/g < 64.h > 128.h
--- 32-64.h ---
sed -e s/64/32/g < 64.h > 32.h
--- 128.h ---
sed -e s/64/32/g < 64.h > 32.h
--- 32.h ---
sed -e s/64/32/g < 64.h > 32.h
--- 32-64.h ---
sed -e s/64/32/g -e s/128/64/g < 64-128.h > 32-64.h
--- 32.h ---
sed -e s/64/32/g -e s/128/64/g < 64-128.h > 32-64.h
--- 128.h ---
sed -e s/64/32/g -e s/128/64/g < 64-128.h > 32-64.h
--- 32-64.h ---
chmod -w 128.h 32.h 32-64.h
--- 32.h ---
chmod -w 128.h 32.h 32-64.h
--- 128.h ---
chmod -w 128.h 32.h 32-64.h
--- pooler ---
gcc9 -O2 -pipe -fopenmp -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc9 -fno-strict-aliasing *.c -o pooler -lm
gmake:
===> Building for pooler-1.75
gmake[2]: Entering directory '/usr/ports/wip/pooler/work/PrimerPooler-1.75/pooler'
chmod +w 128.h 32.h 32-64.h 2>/dev/null || true
chmod +w 128.h 32.h 32-64.h 2>/dev/null || true
chmod +w 128.h 32.h 32-64.h 2>/dev/null || true
sed -e s/64/128/g < 64.h > 128.h
sed -e s/64/128/g < 64.h > 128.h
sed -e s/64/128/g < 64.h > 128.h
sed -e s/64/32/g < 64.h > 32.h
sed -e s/64/32/g < 64.h > 32.h
sed -e s/64/32/g < 64.h > 32.h
sed -e s/64/32/g -e s/128/64/g < 64-128.h > 32-64.h
sed -e s/64/32/g -e s/128/64/g < 64-128.h > 32-64.h
sed -e s/64/32/g -e s/128/64/g < 64-128.h > 32-64.h
chmod -w 128.h 32.h 32-64.h
chmod -w 128.h 32.h 32-64.h
chmod -w 128.h 32.h 32-64.h
gcc9 -O2 -pipe -fopenmp -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc9 -fno-strict-aliasing -fopenmp *.c -o pooler -lm
gmake[2]: Leaving directory '/usr/ports/wip/pooler/work/PrimerPooler-1.75/pooler'
Would you consider the following patch as a permanent upstream fix? This will eliminate the need to disable make jobs while ensuring that each command is run only once.
--- Makefile.orig 2021-01-07 19:49:26 UTC
+++ Makefile
@@ -61,12 +61,20 @@ clean:
# For the generated 128.h etc, we keep them read-only as
# an extra reminder that they're auto-generated (as well
# as the explanatory comment at the top)
-128.h 32.h 32-64.h: 64.h 64-128.h
- chmod +w 128.h 32.h 32-64.h 2>/dev/null || true
+128.h: 64.h 64-128.h
+ chmod +w 128.h 2>/dev/null || true
sed -e s/64/128/g < 64.h > 128.h
+ chmod -w 128.h
+
+32.h: 64.h 64-128.h
+ chmod +w 32.h 2>/dev/null || true
sed -e s/64/32/g < 64.h > 32.h
+ chmod -w 32.h
+
+32-64.h: 64.h 64-128.h
+ chmod +w 32-64.h 2>/dev/null || true
sed -e s/64/32/g -e s/128/64/g < 64-128.h > 32-64.h
- chmod -w 128.h 32.h 32-64.h
+ chmod -w 32-64.h
# -----------------------------------------------------
# Rules you won't need unless releasing new versions:
Created attachment 221387 [details]
Revised patch for biology/pooler
Thanks, I've done that upstream (with minor dependency cleanup) now tagged as v1.76 and attached a revised patch.
A commit references this bug: Author: jwb Date: Fri Jan 8 22:44:14 UTC 2021 New revision: 560817 URL: https://svnweb.freebsd.org/changeset/ports/560817 Log: biology/pooler: Upgrade to 1.76 Fixes race condition for make jobs in upstream Makefile PR: 252352 Submitted by: silas-freebsd@flatline.org.uk Changes: head/biology/pooler/Makefile head/biology/pooler/distinfo Thanks! |
Created attachment 221201 [details] Patch for biology/pooler/Makefile Hi, I got an email from pkg-fallout showing make attempted to create two copies of 32-64.h at the same time, which failed. Perhaps USES=gmake might avoid this, but I suppose the safest thing to set is MAKE_JOBS_UNSAFE=yes (diff attached).