ports make install doesn't install (some or all?) binaries to /usr/local, if /usr/local is symbolic link. It doesn't report anything, simply some files are not get installed. In my case this happened during upgrade of bind99 and binutils. This resulted in missing /usr/local/sbin/named and all binutils: pkg check -s binutils-2.27_4,1: missing file /usr/local/bin/ld.bfd binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/ar binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/as binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/ld binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/ld.bfd binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/ld.gold binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/nm binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/objcopy binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/objdump binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/ranlib binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/readelf binutils-2.27_4,1: missing file /usr/local/x86_64-portbld-freebsd10.3/bin/strip Could it be related with https://svnweb.freebsd.org/changeset/ports/425793 ?
I do not see how it could be related to that commit. Nothing uses COPYTREE_* macros to copy files in PREFIX or LOCALBASE.
I'm also experiencing this bug. Everything was working fine for me until I upgraded to FreeBSD 11 and decided to recompile packages. For me following packages get affected: bind99 git libzip llvm36 perl5.24 zsh It seems that the files that are not getting installed are hardlinks, perhaps it is due to staging place being on a different mount than /usr/local? I found a workaround, which involves issuing make package to generate binary package and then installing it.
After I posted a note (https://docs.freebsd.org/cgi/getmsg.cgi?fetch=146131+0+/usr/local/www/mailindex/archive/2016/freebsd-ports/20161106.freebsd-ports) to freebsd-ports@ describing my experience, someone contacted me and directed my attention to this PR. As with the original reporter, /usr/local is a symlink in /usr to a different file system on my affected systems.
Created attachment 177213 [details] Shell script to do a reality check on ports/packages vs. installed files
So.. bugzilla didn't include the comment I had written.... :-! Anyway: I wrote the shell script (from previous "comment") in order to do a reality check, after this morning's port updates showed that lang/gcc was also affected by this issue. Here's its output from one system: lgld-dhw(11.0-S)[31] /tmp/port_install_chk devel/llvm3\* lang/gcc math/R port_install_chk: Identified 51 missing file(s) for devel/llvm38: /usr/local/bin/argdumper38 /usr/local/bin/clang++38 /usr/local/bin/clang-apply-replacements38 /usr/local/bin/clang-check38 /usr/local/bin/clang-cpp38 /usr/local/bin/clang-format38 /usr/local/bin/clang-modernize38 /usr/local/bin/clang-query38 /usr/local/bin/clang-rename38 /usr/local/bin/clang-tblgen38 /usr/local/bin/clang-tidy38 /usr/local/bin/clang38 /usr/local/bin/llc38 /usr/local/bin/lld38 /usr/local/bin/lldb-mi38 /usr/local/bin/lldb-server38 /usr/local/bin/lldb38 /usr/local/bin/lli38 /usr/local/bin/llvm-ar38 /usr/local/bin/llvm-as38 /usr/local/bin/llvm-bcanalyzer38 /usr/local/bin/llvm-config38 /usr/local/bin/llvm-cov38 /usr/local/bin/llvm-diff38 /usr/local/bin/llvm-dis38 /usr/local/bin/llvm-dwarfdump38 /usr/local/bin/llvm-extract38 /usr/local/bin/llvm-link38 /usr/local/bin/llvm-lit38 /usr/local/bin/llvm-mc38 /usr/local/bin/llvm-mcmarkup38 /usr/local/bin/llvm-nm38 /usr/local/bin/llvm-objdump38 /usr/local/bin/llvm-profdata38 /usr/local/bin/llvm-ranlib38 /usr/local/bin/llvm-readobj38 /usr/local/bin/llvm-rtdyld38 /usr/local/bin/llvm-size38 /usr/local/bin/llvm-split38 /usr/local/bin/llvm-stress38 /usr/local/bin/llvm-symbolizer38 /usr/local/bin/llvm-tblgen38 /usr/local/bin/macho-dump38 /usr/local/bin/modularize38 /usr/local/bin/opt38 /usr/local/bin/scan-build38 /usr/local/bin/scan-view38 /usr/local/llvm38/bin/FileCheck /usr/local/llvm38/bin/clang-cpp /usr/local/llvm38/bin/lit /usr/local/llvm38/bin/llvm-lit port_install_chk: Identified 12 missing file(s) for lang/gcc: /usr/local/bin/c++49 /usr/local/bin/g++ /usr/local/bin/g++49 /usr/local/bin/gcc /usr/local/bin/gcc-ar49 /usr/local/bin/gcc-nm49 /usr/local/bin/gcc-ranlib49 /usr/local/bin/gcc49 /usr/local/bin/gfortran /usr/local/bin/gfortran49 /usr/local/bin/x86_64-portbld-freebsd11.0-g++49 /usr/local/bin/x86_64-portbld-freebsd11.0-gcc-4.9.4 lgld-dhw(11.0-S)[32] echo $? 63 lgld-dhw(11.0-S)[33] (I apologize for the name; naming things isn't one of my strong points.) As mentioned earlier, a circumvention is to make the package, then "pkg add" the package.
Created attachment 177214 [details] Updated shell script for reality-checking installed files I forgot that some folks put token-separator characters in filenames; this version at least copes with embedded spaces.
Created attachment 177218 [details] Cleaned-up version of script Final(?) version of a shell script to check each specified installed port/package and squawk about any files determined to be missing. Invoke with -h to get a "usage" message that should explain things.
Based on the contents of a typescript file on one affected machine, I believe that I first encountered evidence of this problem between 25 September - 02 October.
I have been able to easily reproduce and I have a regression test ready, I'm working on a fix. Too me so far it looks like a bug in libarchve and the way it walk through the filesystem and die with a "Operation Not Permited" if one directory is a symlink to a directory (only it case it wants to create a hardlink) and a bug in pkg which misses an error checking in that case so does not report anything. I'm working on a fix.
Same bug as bug 213255.
(In reply to Tijl Coosemans from comment #10) This does not appear to be the same bug. I applied your patch and now I get: john@pumpkin:~ % rm -rf dir2 john@pumpkin:~ % mkdir dir2 john@pumpkin:~ % ln -s dir2 dir john@pumpkin:~ % touch dir/file john@pumpkin:~ % ln dir/file dir/file2 john@pumpkin:~ % tar cf dir.tar dir/file dir/file2 john@pumpkin:~ % rm dir/* john@pumpkin:~ % tar xf dir.tar -P john@pumpkin:~ % ls -l dir/ total 1 -rw-r--r-- 2 john john 0 25 Nov 23:42 file -rw-r--r-- 2 john john 0 25 Nov 23:42 file2 john@pumpkin:~ % [works as expected] This is on 11.0-STABLE #0 r308922 & ports r427130 however I still have problems with installing (some, not all) ports such as portupgrade: root@pumpkin:/root# which portupgrade /usr/local/sbin/portupgrade root@pumpkin:/root# pkg delete portupgrade Updating database digests format: 100% Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 1 packages (of 0 packages in the universe): Installed packages to be REMOVED: portupgrade-2.4.14,2 Number of packages to be removed: 1 Proceed with deinstalling packages? [y/N]: y [1/1] Deinstalling portupgrade-2.4.14,2... [1/1] Deleting files for portupgrade-2.4.14,2: 100% root@pumpkin:/root# which portupgrade portupgrade: Command not found. root@pumpkin:/root# cd /usr/ports/ports-mgmt/portupgrade root@pumpkin:/usr/ports/ports-mgmt/portupgrade# make ===> License BSD3CLAUSE accepted by the user [...] bin/mkdir -p /storage/usr/ports/ports-mgmt/portupgrade/work/stage/usr/local/share/zsh/site-functions install -o root -g wheel -m 444 _pkgtools /storage/usr/ports/ports-mgmt/portupgrade/work/stage/usr/local/share/zsh/site-functions/ /bin/mkdir -p /storage/usr/ports/ports-mgmt/portupgrade/work/stage/usr/local/share/doc/pkgtools install -c -o root -g wheel -m 444 NEWS.md README.md /storage/usr/ports/ports-mgmt/portupgrade/work/stage/usr/local/share/doc/pkgtools /bin/mkdir -p /storage/usr/ports/ports-mgmt/portupgrade/work/stage/usr/local/lib/compat/pkg ====> Compressing man pages (compress-man) ===> Installing ldconfig configuration file root@pumpkin:/usr/ports/ports-mgmt/portupgrade# make install ===> Installing for portupgrade-2.4.14,2 ===> portupgrade-2.4.14,2 depends on file: /usr/local/bin/ruby22 - found ===> portupgrade-2.4.14,2 depends on file: /usr/local/lib/ruby/site_ruby/2.2/amd64-freebsd11/bdb.so - found ===> Checking if portupgrade already installed ===> Registering installation for portupgrade-2.4.14,2 Installing portupgrade-2.4.14,2... root@pumpkin:/usr/ports/ports-mgmt/portupgrade# root@pumpkin:/usr/ports/ports-mgmt/portupgrade# which portupgrade portupgrade: Command not found. Instead, I have to install it like this: make clean package deinstall && pkg add work/pkg/* && make clean Installing portupgrade-2.4.14,2... Extracting portupgrade-2.4.14,2: 100% ===> Cleaning for portupgrade-2.4.14,2 root@pumpkin:/usr/ports/ports-mgmt/portupgrade# which portupgrade /usr/local/sbin/portupgrade root@pumpkin:/usr/ports/ports-mgmt/portupgrade#
A note for everyone: please use attachments instead of comments for large bodies of text as it makes issues (particularly complex ones) much easier to follow and understand.
I've set this issue depends on bug 213255 (and not duplicate of it) anticipating that there will be port changes separate from the base change in the other issue.
(In reply to tech-lists from comment #11) pkg links to libarchive statically, so you must rebuild it after rebuilding libarchive.
(In reply to Tijl Coosemans from comment #14) it works! many thanks
(In reply to Kubilay Kocak from comment #12) I disagree I, as one who need to follow this bug prefer to see an ordered discussion with details like it is in this thread rather than having to open an attachement for just a bunch of lines (of course not the same for a large bunch on lines).
Created attachment 177544 [details] Fixed version of script Fixed the sed expression.... :-/
Created attachment 177546 [details] Script to check for ports/packages that are incompletely installed One last(?) fix
We have an apparent fix. Starting with a freshly-built: FreeBSD g1-252.catwhisker.org 11.0-STABLE FreeBSD 11.0-STABLE #176 r309315M/309324:1100506: Wed Nov 30 04:19:19 PST 2016 root@g1-252.catwhisker.org:/common/S1/obj/usr/src/sys/CANARY amd64 I then did "svn merge -c 309300 ^/head /usr/src" ("cherry-pick merge" of update to src/contrib/libarchive), rebuilt stable/11, rebooted/smoke-tested. That done, rebuilt/installed ports-mgmt/pkg (e.g. "portmaster ports-mgmt/pkg"). Then, using the attached script: g1-252(11.0-S)[6] port_install_chk devel/binutils port_install_chk: Identified 12 missing file(s) for devel/binutils: /usr/local/bin/ld.bfd /usr/local/x86_64-portbld-freebsd11.0/bin/ar /usr/local/x86_64-portbld-freebsd11.0/bin/as /usr/local/x86_64-portbld-freebsd11.0/bin/ld /usr/local/x86_64-portbld-freebsd11.0/bin/ld.bfd /usr/local/x86_64-portbld-freebsd11.0/bin/ld.gold /usr/local/x86_64-portbld-freebsd11.0/bin/nm /usr/local/x86_64-portbld-freebsd11.0/bin/objcopy /usr/local/x86_64-portbld-freebsd11.0/bin/objdump /usr/local/x86_64-portbld-freebsd11.0/bin/ranlib /usr/local/x86_64-portbld-freebsd11.0/bin/readelf /usr/local/x86_64-portbld-freebsd11.0/bin/strip g1-252(11.0-S)[7] sudo portmaster !$ ... Installing binutils-2.27_5,1... ===>>> Re-installation of binutils-2.27_5,1 complete g1-252(11.0-S)[8] port_install_chk devel/binutils g1-252(11.0-S)[9] echo $? 0 g1-252(11.0-S)[10]
Hmmm... :-( Despite the apparent good news, there seems to remain an edge case that still fails -- and the circumvention also fails. g1-252(11.0-S)[18] port_install_chk -d 2 \* port_install_chk: Identified 1 missing file(s) for print/ghostscript9-agpl-base: /usr/local/share/ghostscript/9.16/Resource/CIDFont/fonts port_install_chk: Identified 7 missing file(s) for print/gsfonts: /usr/local/share/ghostscript/fonts/GothicBBB-Medium /usr/local/share/ghostscript/fonts/GothicBBB-Medium.gs7 /usr/local/share/ghostscript/fonts/MSung-Light /usr/local/share/ghostscript/fonts/Ryumin-Light /usr/local/share/ghostscript/fonts/Ryumin-Light.gs7 /usr/local/share/ghostscript/fonts/STHeiti-Regular /usr/local/share/ghostscript/fonts/STSong-Light port_install_chk: Identified 1 missing file(s) for net/linux-c6-openldap: /compat/linux/etc/openldap port_install_chk: Identified 1 missing file(s) for devel/linux-c6-qt47: /compat/linux/usr/lib/qt47/bin/qdbus port_install_chk: Identified 2 missing file(s) for emulators/linux_base-c6: /compat/linux/usr/lib/lsb/install_initd /compat/linux/usr/lib/lsb/remove_initd port_install_chk: Identified 1 missing file(s) for sysutils/polkit: /usr/local/etc/polkit-1/rules.d/50-default.rules g1-252(11.0-S)[19] The above seem to tend to be symlinks themselves -- e.g.: g1-252(11.0-S)[20] ls -lT /compat/linux/etc/openldap lrwxr-xr-x 1 root wheel 23 Nov 30 09:52:13 2016 /compat/linux/etc/openldap -> /usr/local/etc/openldap g1-252(11.0-S)[21] ls -lT /usr/local/etc/openldap ls: /usr/local/etc/openldap: No such file or directory g1-252(11.0-S)[22] pkg info --list-files net/linux-c6-openldap | grep etc /compat/linux/etc/openldap g1-252(11.0-S)[23]
should be 100% fixed in pkg 1.9.99.5 (pkg-devel) what ever version of libarchive is available in base. Once confirmed I will release 1.9.4 with that fix and plenty of others.
(In reply to Baptiste Daroussin from comment #21) Sorry for a slight offtopic, but I switched to pkg-devel to try it, but now I can't switch back, because schema changed: /usr/ports/ports-mgmt/pkg# make install ===> Installing for pkg-1.9.3 ===> Checking if pkg already installed pkg-static: warning: database version 34 is newer than libpkg(3) version 33, but still compatible pkg-static: sqlite error while executing INSERT OR ROLLBACK INTO pkg_search(id, name, origin) VALUES (?1, ?2 || '-' || ?3, ?4); in file pkgdb.c:1544: no such table: pkg_search *** Error code 74 Stop. make[1]: stopped in /usr/ports/ports-mgmt/pkg *** Error code 1 Stop. make: stopped in /usr/ports/ports-mgmt/pkg Exit 1 How can I switch back to the stable version?
BTW, as David pointed out, there's another issue with symlinks: The new pkg-devel still has it and actually after installation is in an unusable state: # ls -l libpkg.so* lrwxr-xr-x 1 root wheel 6 Dec 4 18:10 libpkg.so -> @????? lrwxr-xr-x 1 root wheel 6 Dec 4 18:10 libpkg.so.4 -> @????? -rwxr-xr-x 1 root wheel 5160043 Dec 4 18:10 libpkg.so.4.0.0 After I installed it, I fixed symlinks by hand, and reinstalled pkg-devel again but pkg-devel broke them again.
so reinstall pkg 1.9.3: pkg-static shell > CREATE VIRTUAL TABLE pkg_search USING fts4(id, name, origin); > pragma user_version=33; About the changes so you have exactly the same issue as David, which somehow I cannot reproduce... On what filesystem are you running?
Can you try what would become pkg 1.9.4: http://bapt.etoilebsd.net/pkg-1.9.4.diff
OK; I applied the diff, then built & installed the shiny new pkg-1.9.4. :-) A test ("portmaster devel/git") showed that there were no files missing from the install; that's definite progress: thank you! A further test with print/gsfonts -- first via "portmaster print/gsfonts", then "cd /usr/ports/print/gsfonts && make clean package deinstall && pkg add work/pkg/*" -- still showed: port_install_chk: Identified 7 missing file(s) for print/gsfonts: /usr/local/share/ghostscript/fonts/GothicBBB-Medium /usr/local/share/ghostscript/fonts/GothicBBB-Medium.gs7 /usr/local/share/ghostscript/fonts/MSung-Light /usr/local/share/ghostscript/fonts/Ryumin-Light /usr/local/share/ghostscript/fonts/Ryumin-Light.gs7 /usr/local/share/ghostscript/fonts/STHeiti-Regular /usr/local/share/ghostscript/fonts/STSong-Light g1-252(11.0-S)[11] I believe that each of these is intended to be a symlink. This was running: FreeBSD g1-252.catwhisker.org 11.0-STABLE FreeBSD 11.0-STABLE #182 r309548M/309548:1100506: Mon Dec 5 04:06:06 PST 2016 root@g1-252.catwhisker.org:/common/S1/obj/usr/src/sys/CANARY amd64
The last bug print/gsfont is totally unrelated the port is broken as shown by the Q/A scripts: Warning: Symlink '/usr/local/share/ghostscript/fonts/GothicBBB-Medium.gs7' pointing to '/usr/local/share/fonts/std.ja_JP/GothicBBB-Medium.gs7' which does not exist in the stage directory or in localbase Warning: Symlink '/usr/local/share/ghostscript/fonts/GothicBBB-Medium' pointing to '/usr/local/share/fonts/std.ja_JP/GothicBBB-Medium' which does not exist in the stage directory or in localbase Warning: Symlink '/usr/local/share/ghostscript/fonts/STHeiti-Regular' pointing to '/usr/local/share/fonts/std.zh_CN/STHeiti-Regular' which does not exist in the stage directory or in localbase Warning: Symlink '/usr/local/share/ghostscript/fonts/MSung-Light' pointing to '/usr/local/share/fonts/std.zh_CN/MSung-Light' which does not exist in the stage directory or in localbase Warning: Symlink '/usr/local/share/ghostscript/fonts/Ryumin-Light.gs7' pointing to '/usr/local/share/fonts/std.ja_JP/Ryumin-Light.gs7' which does not exist in the stage directory or in localbase Warning: Symlink '/usr/local/share/ghostscript/fonts/Ryumin-Light' pointing to '/usr/local/share/fonts/std.ja_JP/Ryumin-Light' which does not exist in the stage directory or in localbase Warning: Symlink '/usr/local/share/ghostscript/fonts/STSong-Light' pointing to '/usr/local/share/fonts/std.zh_CN/STSong-Light' which does not exist in the stage directory or in localbase
Ah, OK; thank you for the clarification. I'll test with some of the other ports I have installed that had shown similar symptoms previously; they were: port_install_chk: Identified 1 missing file(s) for print/ghostscript9-agpl-base: port_install_chk: Identified 7 missing file(s) for print/gsfonts: port_install_chk: Identified 6 missing file(s) for www/libxul: port_install_chk: Identified 1 missing file(s) for net/linux-c6-openldap: port_install_chk: Identified 1 missing file(s) for devel/linux-c6-qt47: port_install_chk: Identified 2 missing file(s) for emulators/linux_base-c6: port_install_chk: Identified 1 missing file(s) for sysutils/polkit:
(In reply to Baptiste Daroussin from comment #27) The link targets are provided by chinese/font-std and japanese/font-std.
A commit references this bug: Author: bapt Date: Mon Dec 5 14:02:54 UTC 2016 New revision: 427895 URL: https://svnweb.freebsd.org/changeset/ports/427895 Log: Add missing runtime dependencies PR: 214381 MFH: 2016Q4 Changes: head/print/gsfonts/Makefile
(In reply to david from comment #28) I have fixed print/gsfonts. For others it would be preferable to open a PR for each. I can confirm that it is the same issue for them.
(In reply to Baptiste Daroussin from comment #31) I don't think we should force everybody to install japanese and chinese fonts. The broken links are fine. They only come up as errors because the script attached to this bug uses [ ! -e file ] which dereferences links, so it reports every broken link as missing even if the link itself is present.
(In reply to Baptiste Daroussin from comment #24) Thanks for the information how to revert pkg. Issue with symlinks is worse than hardlinks, because virtually every package uses symlinks. So this is how my layout looks like: /usr is on UFS2 /usr/local is a symlink to a ZFS filesystem /var is on ZFS mounted directly portsworkdir is configured to be in /var/tmp/portsworkdir
Created attachment 177768 [details] Script to check for ports/packages that are incompletely installed (In reply to Tijl Coosemans from comment #32) OK; this version of the script does not whine about a file unless it fails both -e and -L. I'm not completely convinced that this is ideal, but it may be the best we can do given the available information. It may whine if the file exists, but the invoker lacks permission to determine that (e.g., in the case of ${LOCALBASE}/etc/polkit-1/rules.d/50-default.rules when the invoker has eUID != 0), so that is also something worth noting (as Baptiste Daroussin pointed out in an out-of-band conversation). Invoking this version of the script as root with the '\*' argument, my systems report "clean" as of this writing.
A commit references this bug: Author: bapt Date: Thu Dec 8 09:27:05 UTC 2016 New revision: 428110 URL: https://svnweb.freebsd.org/changeset/ports/428110 Log: Update to 1.9.4 - Stop dropping privileges for fetching, it causes more issues than it solves - Fix segfault when ABI is defined by empty - Fix reporting of deprecated packages - Provide instructions on how to unset vital flages - Fix libfetch for NetBSD - Fix @config potential double free - Always warn about the deprecated @dirrm/@dirrmtry - Always warn about the deprecated @exec - Rework pkg register to share more code with pkg install * It makes it more robust to libarchive regressions [1] PR: 214381 [1], 215029 [2] Reported by: lampa@fit.vutbr.cz [1] Exp-run: antoine [2] Changes: head/ports-mgmt/pkg/Makefile head/ports-mgmt/pkg/distinfo
(In reply to Tijl Coosemans from comment #32) I agree 100%. I compare the number of ports I have installed before and after running portupgrade, and I discovered this change. I suggest making this a configuration option for the port (defaulting to off, please) instead of installing a number of fonts that are un-needed by the majority of users.
(In reply to Terry Kennedy from comment #36) These symlinks were installed without dependency *intentionally* to let users who need Japanese and/or Chinese fonts to install actual font files. They are used just to normalize the font file location because it is difficult to find what font name is used and where it supposed to be. I think forcing to install big Japanese and Chinese fonts is not acceptable for most of people. I want to revert the change. Any objections?
(In reply to Hiroki Sato from comment #37) To be clear: I mean print/gsfonts only.
(In reply to Hiroki Sato from comment #38) No objection here. I apologize for catalyzing the change, as it was my lack of understanding that caused me to write the script that did the checks as it did. (I've changed that behavior of the script since.)
A commit references this bug: Author: hrs Date: Mon Dec 12 17:49:38 UTC 2016 New revision: 428425 URL: https://svnweb.freebsd.org/changeset/ports/428425 Log: Revert r427895. The symlinks to Japanese and Chinese fonts are provided to normalize the font names into standardized ones, and to make the users be able to install the actual font files into the location pointed by them. japanese/font-std and chinese/font-std will install ones which can be redistributed freely, and one can install commercial font files instead. To avoid installing big Japanese and Chinese font files for people who do not need them, this port does not have explicit runtime dependency on japanese/font-std or chinese/font-std. PR: 214381 Changes: head/print/gsfonts/Makefile
(In reply to david from comment #39) Thank you. I committed the change.