One problem with portupgrade is that if port A lists port D as an extract, patch, or fetch dependency, and port B lists port D as a build or run dependency, and if portupgrade decides to build port A before port B, then the following happens: portupgrade builds port A, which has the side effect of building and installing port D because D is listed as a dependency in the Makefile for A. portupgrade then installs port A. portupgrade then builds port D. portupgrade then attempts to install port D, which fails because D is already installed. portupgrade then skips port B because its dependency D failed. Portupgrade can be cumbersome to use if ports are only updated unfrequently. Sometimes the instructions in ports/UPDATING will suggest running "portupgrade -fr someport" to force the rebuilding of all ports that depend on someport. The problem is that some of the other ports that these ports depend on will still be out of date. The -a option can't be included on the command line to fix that problem because *all* ports are rebuilt when both the -a and -f flags are specified, and the -a flag disables the -r flag. The problem gets worse if multiple ports need to have all of their dependent ports rebuilt. Fix: The following patch to lib/pkgtools/portinfo.rb causes the extract, patch, and fetch dependencies to be added to INDEX.db when it is built, and allows portupgrade to see these extra dependencies so that the ports are all built in the proper order. The following patch to bin/portupgrade does several things: Allows the -a and -r options to be used together. Does not apply the -f option to all ports if the -a option is specified. Adds a --force_all (or -ff) option to force all ports to be rebuilt if that is desired. Computes the upgrade_tasks list in a more efficient manner. These changes allow "portupgrade -afr port1 port2" to be used to rebuild all out of date ports as well port1, port2, and all ports that depend on these two ports to be built in the proper dependency order without duplicate rebuilds. There is also a patch for the man page to document the user visible changes.
Responsible Changed From-To: freebsd-ports-bugs->bdrewery Over to maintainer (via the GNATS Auto Assign Tool)
I don't think it makes sense to need -r with -a, -a already includes everything. I think the real problem is that *-R* should be used as well. You didn't mention it anywhere, but it is what would cause anything a port depends on to be included in the upgrade list. This would fix the ordering issues. -R uses get_all_depends which already includes all of the depends: > def get_all_depends(origin, parents_list = nil) > if $use_packages_only > depends_vars = %w{LIB_DEPENDS RUN_DEPENDS} > else > depends_vars = %w{FETCH_DEPENDS EXTRACT_DEPENDS PATCH_DEPENDS > BUILD_DEPENDS LIB_DEPENDS RUN_DEPENDS} > end portmaster basically does -R by default and I have been considering doing the same with portupgrade.
On 25 Mar, Bryan Drewery wrote: > I don't think it makes sense to need -r with -a, -a already includes > everything. The problem is that the instructions in UPDATING say to use -fr to force the rebuilding of all the dependent ports, but I don't want to rebuild *all* ports by specifying -fa. To use a specific example from UPDATING, with this change I can do: portupgrade -arf archivers/libarchive graphics/poppler will rebuild all out of date ports as well as force the rebuilding of the two specified ports and any ports that depend on them. > I think the real problem is that *-R* should be used as well. You didn't > mention it anywhere, but it is what would cause anything a port depends > on to be included in the upgrade list. This would fix the ordering issues. > > -R uses get_all_depends which already includes all of the depends: > >> def get_all_depends(origin, parents_list = nil) >> if $use_packages_only >> depends_vars = %w{LIB_DEPENDS RUN_DEPENDS} >> else >> depends_vars = %w{FETCH_DEPENDS EXTRACT_DEPENDS PATCH_DEPENDS >> BUILD_DEPENDS LIB_DEPENDS RUN_DEPENDS} >> end > > portmaster basically does -R by default and I have been considering > doing the same with portupgrade. I think that using -R sort of helps avoid the issue of having ports being rebuilt while their dependencies are out of date, but using -a instead solves the same problem. If -rf someport is specified and the -R behavior is added, it is probably undesirable to force rebuilding of the dependencies of someport. Even after portupgrade is run this way, it would still need to be run again with -a to upgrade the remaining ports. portmaster also seems to implement the force action when -r is used. My patches also don't totally fix the duplicate install problem. Origin changes can still trip things up, but this is a fairly rare problem.
Is this PR still relevant?
(In reply to Carlo Strub from comment #4) > Is this PR still relevant? Yes, though the original patch has rotted. I've been using a different patch for a while, but have recently run into other issues with it. The importance of this to me has declined a lot recently because I've pretty much given up on upgrading ports in place and switched to poudriere.
Thanks for your answer. So, that means we should probably close it if also your newer patch is problematic? How shall we resolve this PR now?
I think this is worth keeping open for now. Portupgrade is wrong in many ways when it comes to handling dependencies right. It passes off too much of the control to ports resulting in situations such as this and https://github.com/freebsd/portupgrade/issues/30 and making fixing https://github.com/freebsd/portupgrade/issues/58 difficult. As I noted before it is also quite wrong to not recursively upgrade by default as often it is expected by ports that a dependencies are updated first.
A commit references this bug: Author: bdrewery Date: Thu May 14 22:03:34 UTC 2015 New revision: 386365 URL: https://svnweb.freebsd.org/changeset/ports/386365 Log: Update to 2.4.13-6-g72b4c34 Changes: * Fix ordering of build based on FETCH/PATCH/EXTRACT dependencies. PR: 177365 Submitted by: truckman@ Changes: head/ports-mgmt/portupgrade-devel/Makefile head/ports-mgmt/portupgrade-devel/distinfo
Note that I only committed a portion of the patch to include fetch/patch/extract dependencies in portsdb and PortInfo.all_depends()
A commit references this bug: Author: bdrewery Date: Wed May 27 15:17:57 UTC 2015 New revision: 387621 URL: https://svnweb.freebsd.org/changeset/ports/387621 Log: Update to 2.4.14 Changes: * Fix ordering of build based on FETCH/PATCH/EXTRACT dependencies. [1] * Support security vulnerabilities as a build failure reason. [2] * portupgrde -p: Fix new dependencies not getting installed. (#58) [3] * Don't install missing dependencies which are already satisfied (#62). Note that this purposely causes these missing dependencies to not show in the job (-n) and final results output which is a compromise to at least avoid the wrong packages being installed. The logging issue is in #30. PR: 177365 [1] Submitted by: truckman@ [1] PR: 192232 [2] Submitted by: Yuri <yuri@rawbw.com> [2] PR: 184672 [3] Changes: head/ports-mgmt/portupgrade/Makefile head/ports-mgmt/portupgrade/distinfo
Note that my commit is only a partial commit of the patch in this PR. It is not fully satisfied.
Created attachment 157531 [details] updated portupgrade enhancement patch Regenerate portupgrade-devel patch after the partial commit of the previous patch in r386365. This patch eases the pain when upgrading ports infrequently, when UPDATING contains instructions to run portupgrade -fr on multiple ports. It changes the interaction between the -a -f and -r flags so that running portupgrade -afr port1 port2 ... upgrades all out of date ports as well as reinstalling all ports that depend on port1, port2, etc., all in the proper dependency order. It is possible to force all ports to be reinstalled by using -aff, see the updated man page. Note: I stopped using portupgrade some time ago and switched to building binary packages with poudriere and keeping up to date with pkg upgrade. One of the reasons that I switched is that I was trying to using portupgrade to build packages using the -p option, and I ran into the bug in PR 184672. The problem that I was seeing was that portupgrade wasn't building and installing new dependencies for the ports that it was upgrading. Instead, the new dependencies were getting built by the logic in the ports Makefiles that detected that the new dependencies were missing, but it appeared that they were not getting installed because of the interaction between PACKAGE_BUILDING being set and the its interaction with staging. My guess that the reason that portupgrade wasn't installing (and also building packages for) the new dependencies was that it was using pkgdb (which has the previous set of dependencies) instead of portsdb (which has the new set of dependencies). On the other hand, I think the -r flag needs to use pkgdb. I haven't looked at the fix for PR 184672, so I can't say whether or not this is totally fixed, but I'm pretty sure that the problem will no longer be fatal. Even though I haven't used portupgrade with this patch in the recent past, I'm attaching the most recent version here in case someone might find it useful. For extra credit, a way of combining this with one or more origin changes would be really cool (consider portupgrade -a -fro newport1 oldport1 -fr port2).
Thanks for updating the patch. Likewise I have not used Portupgrade in maybe 2 years now. I only use Poudriere and most of my development time goes into it as well.
PR is open more than 4 years. Is this still relevant?
(In reply to w.schwarzenfeld from comment #14) > PR is open more than 4 years. Is this still relevant? Yes
Keyword: patch or patch-ready – in lieu of summary line prefix: [patch] * bulk change for the keyword * summary lines may be edited manually (not in bulk). Keyword descriptions and search interface: <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>