pkgbase `make packages` emits many duplicate file/directory warnings of the form pkg: duplicate directory listing: /boot/defaults, ignoring pkg: duplicate file listing: /etc/libmap.conf, ignoring In total, 485 duplicate directory 172 duplicate file in some cases there are multiple lines in METALOG for a file or directory with different attributes - e.g., ./etc has 80 copies of ./etc type=dir uname=root gname=wheel mode=0755 tags=package=utilities and one ./etc type=dir uname=root gname=wheel mode=0755
When the METALOG is processed to plist files, the duplicates land in these 4 plists: 2 duplicates in clibs.plist: 2 @config(root,wheel,0644,) /etc/libmap.conf 2 duplicates in runtime.plist: 2 @dir(root,wheel,0755,) /var/spool 686 duplicates in utilities.plist, eg. ... 7 @dir(root,wheel,0755,) /usr/share/misc 8 @dir(root,wheel,0755,) /etc/defaults 10 @dir(root,wheel,0755,) /usr/share/snmp/mibs 13 @dir(root,wheel,0755,) /usr/share/snmp/defs 16 @dir(root,wheel,0755,) /boot 80 @dir(root,wheel,0755,) /etc 334 duplicates in utlilities-dev.plist, eg: ... 2 @(root,wheel,0444,) /usr/lib/clang/11.0.0/include/sanitizer/scudo_interface.h 2 @(root,wheel,0444,) /usr/lib/clang/11.0.0/include/sanitizer/tsan_interface.h 2 @(root,wheel,0444,) /usr/lib/clang/11.0.0/include/sanitizer/tsan_interface_atomic.h 2 @(root,wheel,0444,) /usr/lib/clang/11.0.0/include/sanitizer/ubsan_interface.h ... Maybe this narrows down, where these duplicates come from.
Note we now have a tool in the src tree for checking pkgbase METALOG for issues - tools/pkgbase/metalog_reader.lua
Thanks, i found the script, while looking for lua scripts in base. It helps analyzing the METALOG quiete a bit. It shows the duplicates and i am not sure how they happen. But, i see in the logs of my build run, that these steps are often repeated: ... --- installdirs-CONFSDIR --- installing DIRS CONFSDIR install -N /usr/src/etc -U -M /usr/obj/usr/src/amd64.amd64/worldstage//METALOG -D /usr/obj/usr/src/amd64.amd64/worldstage -T package=utilities -d -m 0755 -o root -g wheel /usr/obj/usr/src/amd64.amd64/worldstage/etc ... --- installdirs-CONFSDIR --- installing DIRS CONFSDIR install -N /usr/src/etc -U -M /usr/obj/usr/src/amd64.amd64/worldstage//METALOG -D /usr/obj/usr/src/amd64.amd64/worldstage -T package=utilities -d -m 0755 -o root -g wheel /usr/obj/usr/src/amd64.amd64/worldstage/etc ... --- installdirs-CONFETCDIR --- installing DIRS CONFETCDIR CONFETCEXECDIR install -N /usr/src/etc -U -M /usr/obj/usr/src/amd64.amd64/worldstage//METALOG -D /usr/obj/usr/src/amd64.amd64/worldstage -T package=utilities -d -m 0755 -o root -g wheel /usr/obj/usr/src/amd64.amd64/worldstage/etc ... I guess, these are responsible for the multiple /etc entries (warning: ./etc repeated with same meta: line 2091,2092,2097,2117,2138,2170,2174,2182,2186,2198,2204,2244,2249,2259,2283,2296,2324,2353,2364,2366,2368,2370,2377,2380,2384,2454,2457,2462,2468,2469,2475,2486,2494,2500,2502,2504,2506,2511,2513,2669,2709,2755,2793,2912,3049,5998,6007,6034,6041,6110,6380,7937,8077,8178,8931,14366,15617,16480,16933,17120,17238,18047,18106,18545,19268,19808,20655,20770,20938,21193,21272,21806,21817,23053,24592,27092,27545,29176)
The lua script also finds errors in regard to some hardlinks error: entries point to the same inode but have different meta: ./root/.profile,./.profile in line 2126,2127. off by "mode" which comes from the different mode settings between the "hardlinked" files: ./.profile type=file uname=root gname=wheel mode=0555 size=567 tags=package=runtime ./root/.profile type=file uname=root gname=wheel mode=0644 size=567 tags=package=runtime,config these: error: entries point to the same inode but have different meta: ./root/.cshrc,./.cshrc in line 2149,2153. off by "mode" -> ./.cshrc type=file uname=root gname=wheel mode=0555 size=1083 tags=package=runtime ./root/.cshrc type=file uname=root gname=wheel mode=0644 size=1083 tags=package=runtime,config error: entries point to the same inode but have different meta: ./etc/pam.d/ftp,./etc/pam.d/ftpd in line 2363,2361. off by "mode" -> ./etc/pam.d/ftp type=file uname=root gname=wheel mode=0555 size=528 tags=package=utilities ./etc/pam.d/ftpd type=file uname=root gname=wheel mode=0644 size=528 tags=package=utilities,config and a more hardlinks in the /usr/libexec/dwatch/ directory: error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/ip-receive,./usr/libexec/dwatch/ip-send,./usr/libexec/dwatch/ip in line 3486,3487,3414. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/openat,./usr/libexec/dwatch/open in line 3488,3429. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/proc-exit,./usr/libexec/dwatch/proc-exec-failure,./usr/libexec/dwatch/proc-exec,./usr/libexec/dwatch/proc-signal-discard,./usr/libexec/dwatch/proc-create,./usr/libexec/dwatch/proc-signal-send,./usr/libexec/dwatch/proc-exec-success,./usr/libexec/dwatch/proc-status,./usr/libexec/dwatch/proc-signal,./usr/libexec/dwatch/proc-signal-clear,./usr/libexec/dwatch/proc in line 3495,3492,3491,3498,3490,3499,3494,3500,3496,3497,3432. off by "mode" ... maybe there is an easy fix for this?
There was a bug in the way the inode lookup was done, see https://reviews.freebsd.org/D37412 for the fix. With that fixed when testing against recent main I get: % tools/pkgbase/metalog_reader.lua -c <path>/amd64.amd64/stage/METALOG | grep 'same inode' error: entries point to the same inode but have different meta: ./etc/pam.d/ftpd,./etc/pam.d/ftp in line 1997,1998. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/io-start,./usr/libexec/dwatch/io-done,./usr/libexec/dwatch/io in line 13138,13137,13118. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/vop_mknod,./usr/libexec/dwatch/vop_mkdir,./usr/libexec/dwatch/vop_create,./usr/libexec/dwatch/vop_remove,./usr/libexec/dwatch/vop_rmdir,./usr/libexec/dwatch/vop_lookup in line 13198,13197,13131,13199,13200,13196. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/recvfrom,./usr/libexec/dwatch/recvmsg,./usr/libexec/dwatch/sendto,./usr/libexec/dwatch/send,./usr/libexec/dwatch/sendrecv,./usr/libexec/dwatch/sendmsg,./usr/libexec/dwatch/recv in line 13172,13173,13176,13174,13126,13175,13171. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/udplite,./usr/libexec/dwatch/udplite-receive,./usr/libexec/dwatch/udplite-send in line 13130,13194,13195. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/chmod,./usr/libexec/dwatch/fchmodat,./usr/libexec/dwatch/lchmod in line 13116,13135,13136. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/open,./usr/libexec/dwatch/openat in line 13122,13141. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/tcp-io,./usr/libexec/dwatch/tcp-accept,./usr/libexec/dwatch/tcp-connect-established,./usr/libexec/dwatch/tcp-established,./usr/libexec/dwatch/tcp-receive,./usr/libexec/dwatch/tcp-connect,./usr/libexec/dwatch/tcp,./usr/libexec/dwatch/tcp-init,./usr/libexec/dwatch/tcp-state-change,./usr/libexec/dwatch/tcp-connect-request,./usr/libexec/dwatch/tcp-accept-established,./usr/libexec/dwatch/tcp-status,./usr/libexec/dwatch/tcp-accept-refused,./usr/libexec/dwatch/tcp-refused,./usr/libexec/dwatch/tcp-connect-refused,./usr/libexec/dwatch/tcp-send in line 13186,13177,13181,13184,13187,13180,13128,13185,13190,13183,13178,13191,13179,13188,13182,13189. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/udp,./usr/libexec/dwatch/udp-send,./usr/libexec/dwatch/udp-receive in line 13129,13193,13192. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/write,./usr/libexec/dwatch/rw,./usr/libexec/dwatch/read in line 13153,13124,13152. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/ip-receive,./usr/libexec/dwatch/ip,./usr/libexec/dwatch/ip-send in line 13139,13119,13140. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/sched-surrender,./usr/libexec/dwatch/sched-queue,./usr/libexec/dwatch/sched-sleep,./usr/libexec/dwatch/sched-change-pri,./usr/libexec/dwatch/sched-off-cpu,./usr/libexec/dwatch/sched-on-cpu,./usr/libexec/dwatch/sched-pri,./usr/libexec/dwatch/sched-enqueue,./usr/libexec/dwatch/sched-tick,./usr/libexec/dwatch/sched-load-change,./usr/libexec/dwatch/sched-wakeup,./usr/libexec/dwatch/sched-cpu,./usr/libexec/dwatch/sched-lend-pri,./usr/libexec/dwatch/sched-dequeue,./usr/libexec/dwatch/sched,./usr/libexec/dwatch/sched-exec,./usr/libexec/dwatch/sched-preempt,./usr/libexec/dwatch/sched-remain-cpu in line 13168,13165,13167,13154,13161,13162,13164,13157,13169,13160,13170,13155,13159,13156,13125,13158,13163,13166. off by "mode" error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/proc,./usr/libexec/dwatch/proc-exec-success,./usr/libexec/dwatch/proc-signal,./usr/libexec/dwatch/proc-signal-discard,./usr/libexec/dwatch/proc-create,./usr/libexec/dwatch/proc-signal-clear,./usr/libexec/dwatch/proc-signal-send,./usr/libexec/dwatch/proc-exec,./usr/libexec/dwatch/proc-status,./usr/libexec/dwatch/proc-exec-failure,./usr/libexec/dwatch/proc-exit in line 13123,13145,13147,13149,13142,13148,13150,13143,13151,13144,13146. off by "mode" error: entries point to the same inode but have different meta: ./.profile,./root/.profile in line 2203,2202. off by "mode" error: entries point to the same inode but have different meta: ./root/.cshrc,./.cshrc in line 2209,2211. off by "mode"
Looking at the first entry: error: entries point to the same inode but have different meta: ./etc/pam.d/ftpd,./etc/pam.d/ftp in line 1997,1998. off by "mode" This comes from lib/libpam/pam.d/Makefile: afterinstallconfig: ${INSTALL_LINK} ${TAG_ARGS} ${DESTDIR}${CONFDIR}/ftpd ${DESTDIR}${CONFDIR}/ftp which uses share/mk/bsd.own.mk: HRDLINK?= -l h -o ${_LINKOWN} -g ${_LINKGRP} -m ${_LINKMODE} ... INSTALL_LINK?= ${INSTALL} ${HRDLINK} results in METALOG: ./etc/pam.d/ftpd type=file uname=root gname=wheel mode=0644 size=359 tags=package=ftp,config ./etc/pam.d/ftp type=file uname=root gname=wheel mode=0555 size=359 tags=package=runtime The dwatch ones use e.g. `LINKS+= ${LIBEXECDIR}/dwatch/io ${LIBEXECDIR}/dwatch/io-done` rather than INSTALL_LINK explicitly. (It looks like I've just found again what Martin reported in comment #4.)
Some of these have been addressed by: https://cgit.FreeBSD.org/src/commit/?id=227f47ce0c48acdb466366b136b8c522bb8937e0 commit 227f47ce0c48acdb466366b136b8c522bb8937e0 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-11-17 00:12:18 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2022-11-17 00:15:20 +0000 libcompat: avoid installing include files twice Previously some headers were getting installed twice, once as expected and then a second time as part of the compat32 library stage. Makefile.libcompat sets -DLIBRARIES_ONLY for the install make invocation which causes bsd.lib.mk to skip headers. However some headers are handled via bsd.prog.mk, which does not use LIBRARIES_ONLY. Explicitly set MK_INCLUDES=no. Reviewed by: brooks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D37413
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=42af87b83bce90e5e2e3cf762917a6355376fd06 commit 42af87b83bce90e5e2e3cf762917a6355376fd06 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-11-17 00:28:17 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2022-11-17 00:30:23 +0000 man9: remove duplicate superio_find_dev link PR: 244596 Fixes: c812bea351021 ("add superio.4 and superio.9 ...") Sponsored by: The FreeBSD Foundation share/man/man9/Makefile | 1 - 1 file changed, 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=005609addaab2ec83f97054fa4c60b3981f84271 commit 005609addaab2ec83f97054fa4c60b3981f84271 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-11-17 02:29:51 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2022-11-17 15:08:36 +0000 man9: correct vm_page_wire.9 links PR: 244596 Reviewed by: markj Fixes: fee2a2fa3983 ("Change synchonization rules for ...") Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D37416 ObsoleteFiles.inc | 3 +++ share/man/man9/Makefile | 7 +++---- 2 files changed, 6 insertions(+), 4 deletions(-)
At this point the remaining issues are duplicated directories (e.g. ./etc appears 74 times in METALOG), and the handful of hard links with differing modes ./etc/pam.d/ftpd ./usr/libexec/dwatch/io (and many other dwatch cases) ./.profile ./.cshrc
https://reviews.freebsd.org/D37499 addresses the last instance of links with differing modes. After that only duplicate directories remain. Probably the path forward for those is just to have metalog_reader.lua ignore those (assuming metadata is identical).
Triage: assignment to committer (thanks) …
(In reply to Ed Maste from comment #11) That change is in review at https://reviews.freebsd.org/D39898 - print duplicate METALOG entry if it is not a directory or if in verbose mode.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2a3bd0870869819354b5ea65761d13e5e2987697 commit 2a3bd0870869819354b5ea65761d13e5e2987697 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-05-01 17:25:18 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-05-01 19:22:36 +0000 pkgbase: hide duplicate METALOG directory warnings under verbose Creating directories multiple times is an inherent side effect of the way installation is done. Hide warnings from duplicate directory entries (with identical metadata) under metalog_reader's verbose mode. Duplicate file entries are always reported. They currently generate warnings but will be switched to errors once the few instances currently in the tree are fixed. PR: 244596, 271178 Reviewed by: kevans Sponsored By: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39898 tools/pkgbase/metalog_reader.lua | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
At present there are three duplicate files (loader help files), and they are tracked in PR271178. Duplicate directory entries are now hidden under verbose mode. The full output from the CI metalog check is now: ===== su user -c "/usr/libexec/flua tools/pkgbase/metalog_reader.lua -c /usr/obj/$(pwd -P)/${TARGET}.${TARGET_ARCH}/worldstage/METALOG" error: ./boot/loader.help.bios file repeated with same meta: line 26844,26847,26852 error: ./boot/loader.help.efi file repeated with same meta: line 26868,26871,26875 error: ./boot/loader.help.userboot file repeated with same meta: line 26822,26826 ===== Once PR271178 is addressed we'll change the metalog checker to make these errors.
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f6488428308edeb1fb8fc8aa0eb4e2744db78748 commit f6488428308edeb1fb8fc8aa0eb4e2744db78748 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-05-01 17:25:18 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-05-04 16:45:50 +0000 pkgbase: hide duplicate METALOG directory warnings under verbose Creating directories multiple times is an inherent side effect of the way installation is done. Hide warnings from duplicate directory entries (with identical metadata) under metalog_reader's verbose mode. Duplicate file entries are always reported. They currently generate warnings but will be switched to errors once the few instances currently in the tree are fixed. PR: 244596, 271178 Reviewed by: kevans Sponsored By: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39898 (cherry picked from commit 2a3bd0870869819354b5ea65761d13e5e2987697) tools/pkgbase/metalog_reader.lua | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=0e04dd3b66c053422b90c387f7bbd82a0921bda0 commit 0e04dd3b66c053422b90c387f7bbd82a0921bda0 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-05-05 17:44:39 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-05-10 01:10:03 +0000 pkgbase: report error if files are installed multiple times Files installed by `make installworld` should be installed only once. PR: 244596 Sponsored by: The FreeBSD Foundation tools/pkgbase/metalog_reader.lua | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)