Bug 244596 - pkgbase: duplicate files and directories in `make packages`
Summary: pkgbase: duplicate files and directories in `make packages`
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-04 16:17 UTC by Ed Maste
Modified: 2020-10-11 10:40 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer 2020-03-04 16:17:18 UTC
pkgbase `make packages` emits many duplicate file/directory warnings of the form

pkg: duplicate directory listing: /boot/defaults, ignoring
pkg: duplicate file listing: /etc/libmap.conf, ignoring

In total,
485 duplicate directory
172 duplicate file

in some cases there are multiple lines in METALOG for a file or directory with different attributes - e.g., ./etc has 80 copies of

./etc type=dir uname=root gname=wheel mode=0755 tags=package=utilities

and one

./etc type=dir uname=root gname=wheel mode=0755
Comment 1 Martin 2020-09-15 18:32:39 UTC
When the METALOG is processed to plist files, the duplicates land in these 4 plists:

2 duplicates in clibs.plist:
2	@config(root,wheel,0644,)	/etc/libmap.conf



2 duplicates in runtime.plist:
2	@dir(root,wheel,0755,)	/var/spool



686 duplicates in utilities.plist, eg.
...
7	@dir(root,wheel,0755,)	/usr/share/misc
8	@dir(root,wheel,0755,)	/etc/defaults
10	@dir(root,wheel,0755,)	/usr/share/snmp/mibs
13	@dir(root,wheel,0755,)	/usr/share/snmp/defs
16	@dir(root,wheel,0755,)	/boot
80	@dir(root,wheel,0755,)	/etc



334 duplicates in utlilities-dev.plist, eg:
...
2	@(root,wheel,0444,)	/usr/lib/clang/11.0.0/include/sanitizer/scudo_interface.h
2	@(root,wheel,0444,)	/usr/lib/clang/11.0.0/include/sanitizer/tsan_interface.h
2	@(root,wheel,0444,)	/usr/lib/clang/11.0.0/include/sanitizer/tsan_interface_atomic.h
2	@(root,wheel,0444,)	/usr/lib/clang/11.0.0/include/sanitizer/ubsan_interface.h
...


Maybe this narrows down, where these duplicates come from.
Comment 2 Ed Maste freebsd_committer 2020-10-05 15:24:28 UTC
Note we now have a tool in the src tree for checking pkgbase METALOG for issues - tools/pkgbase/metalog_reader.lua
Comment 3 Martin 2020-10-11 10:40:18 UTC
Thanks, i found the script, while looking for lua scripts in base. It helps analyzing the METALOG quiete a bit.

It shows the duplicates and i am not sure how they happen. But, i see in the logs of my build run, that these steps  are often repeated:
...
--- installdirs-CONFSDIR ---
installing DIRS CONFSDIR
install -N /usr/src/etc -U -M /usr/obj/usr/src/amd64.amd64/worldstage//METALOG -D /usr/obj/usr/src/amd64.amd64/worldstage -T package=utilities -d -m 0755 -o root  -g wheel  /usr/obj/usr/src/amd64.amd64/worldstage/etc
...
--- installdirs-CONFSDIR ---
installing DIRS CONFSDIR
install -N /usr/src/etc -U -M /usr/obj/usr/src/amd64.amd64/worldstage//METALOG -D /usr/obj/usr/src/amd64.amd64/worldstage -T package=utilities -d -m 0755 -o root  -g wheel  /usr/obj/usr/src/amd64.amd64/worldstage/etc
...
--- installdirs-CONFETCDIR ---
installing DIRS CONFETCDIR CONFETCEXECDIR
install -N /usr/src/etc -U -M /usr/obj/usr/src/amd64.amd64/worldstage//METALOG -D /usr/obj/usr/src/amd64.amd64/worldstage -T package=utilities -d -m 0755 -o root  -g wheel  /usr/obj/usr/src/amd64.amd64/worldstage/etc
...

I guess, these are responsible for the multiple /etc entries
(warning: ./etc repeated with same meta: line 2091,2092,2097,2117,2138,2170,2174,2182,2186,2198,2204,2244,2249,2259,2283,2296,2324,2353,2364,2366,2368,2370,2377,2380,2384,2454,2457,2462,2468,2469,2475,2486,2494,2500,2502,2504,2506,2511,2513,2669,2709,2755,2793,2912,3049,5998,6007,6034,6041,6110,6380,7937,8077,8178,8931,14366,15617,16480,16933,17120,17238,18047,18106,18545,19268,19808,20655,20770,20938,21193,21272,21806,21817,23053,24592,27092,27545,29176)
Comment 4 Martin 2020-10-11 10:40:31 UTC
The lua script also finds errors in regard to some hardlinks

error: entries point to the same inode but have different meta: ./root/.profile,./.profile in line 2126,2127. off by "mode"

which comes from the different mode settings between the "hardlinked" files:

./.profile       type=file  uname=root  gname=wheel  mode=0555  size=567  tags=package=runtime
./root/.profile  type=file  uname=root  gname=wheel  mode=0644  size=567  tags=package=runtime,config

these:
error: entries point to the same inode but have different meta: ./root/.cshrc,./.cshrc in line 2149,2153. off by "mode"
->
./.cshrc       type=file  uname=root  gname=wheel  mode=0555  size=1083  tags=package=runtime
./root/.cshrc  type=file  uname=root  gname=wheel  mode=0644  size=1083  tags=package=runtime,config

error: entries point to the same inode but have different meta: ./etc/pam.d/ftp,./etc/pam.d/ftpd in line 2363,2361. off by "mode"
->
./etc/pam.d/ftp   type=file  uname=root  gname=wheel  mode=0555  size=528  tags=package=utilities
./etc/pam.d/ftpd  type=file  uname=root  gname=wheel  mode=0644  size=528  tags=package=utilities,config

and a more hardlinks in the /usr/libexec/dwatch/ directory:
error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/ip-receive,./usr/libexec/dwatch/ip-send,./usr/libexec/dwatch/ip in line 3486,3487,3414. off by "mode"
error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/openat,./usr/libexec/dwatch/open in line 3488,3429. off by "mode"
error: entries point to the same inode but have different meta: ./usr/libexec/dwatch/proc-exit,./usr/libexec/dwatch/proc-exec-failure,./usr/libexec/dwatch/proc-exec,./usr/libexec/dwatch/proc-signal-discard,./usr/libexec/dwatch/proc-create,./usr/libexec/dwatch/proc-signal-send,./usr/libexec/dwatch/proc-exec-success,./usr/libexec/dwatch/proc-status,./usr/libexec/dwatch/proc-signal,./usr/libexec/dwatch/proc-signal-clear,./usr/libexec/dwatch/proc in line 3495,3492,3491,3498,3490,3499,3494,3500,3496,3497,3432. off by "mode"
...

maybe there is an easy fix for this?