Bug 177365 - [patch] ports-mgmt/portupgrade-devel enhancements
Summary: [patch] ports-mgmt/portupgrade-devel enhancements
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Only Me
Assignee: Bryan Drewery
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-03-25 08:10 UTC by Don Lewis
Modified: 2018-01-12 01:20 UTC (History)
2 users (show)

See Also:


Attachments
file.diff (4.94 KB, patch)
2013-03-25 08:10 UTC, Don Lewis
no flags Details | Diff
updated portupgrade enhancement patch (5.51 KB, patch)
2015-06-08 00:39 UTC, Don Lewis
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Don Lewis freebsd_committer 2013-03-25 08:10:00 UTC
	One problem with portupgrade is that if port A lists port D as
	an extract, patch, or fetch dependency, and port B lists port D
	as a build or run dependency, and if portupgrade decides to build
	port A before port B, then the following happens:

		portupgrade builds port A, which has the side effect of
		building and installing port D because D is listed as
		a dependency in the Makefile for A.

		portupgrade then installs port A.

		portupgrade then builds port D.

		portupgrade then attempts to install port D, which fails
		because D is already installed.

		portupgrade then skips port B because its dependency D
		failed.

	
	Portupgrade can be cumbersome to use if ports are only updated
	unfrequently.  Sometimes the instructions in ports/UPDATING
	will suggest running "portupgrade -fr someport" to force the
	rebuilding of all ports that depend on someport.  The problem is
	that some of the other ports that these ports depend on will
	still be out of date.  The -a option can't be included on the
	command line to fix that problem because *all* ports are rebuilt
	when both the -a and -f flags are specified, and the -a flag disables
	the -r flag.  The problem gets worse if multiple ports need to have
	all of their dependent ports rebuilt.

Fix: The following patch to lib/pkgtools/portinfo.rb causes the
	extract, patch, and fetch dependencies to be added to INDEX.db when
	it is built, and allows portupgrade to see these extra dependencies
	so that the ports are all built in the proper order.

	The following patch to bin/portupgrade does several things:

		Allows the -a and -r options to be used together.

		Does not apply the -f option to all ports if the -a
		option is specified.

		Adds a --force_all (or -ff) option to force all ports to
		be rebuilt if that is desired.

		Computes the upgrade_tasks list in a more efficient
		manner.

	These changes allow "portupgrade -afr port1 port2" to be used
	to rebuild all out of date ports as well port1, port2, and
	all ports that depend on these two ports to be built in the
	proper dependency order without duplicate rebuilds.

	There is also a patch for the man page to document the user
	visible changes.
Comment 1 Edwin Groothuis freebsd_committer 2013-03-25 08:10:08 UTC
Responsible Changed
From-To: freebsd-ports-bugs->bdrewery

Over to maintainer (via the GNATS Auto Assign Tool)
Comment 2 Bryan Drewery freebsd_committer 2013-03-25 10:55:21 UTC
I don't think it makes sense to need -r with -a, -a already includes
everything.

I think the real problem is that *-R* should be used as well. You didn't
mention it anywhere, but it is what would cause anything a port depends
on to be included in the upgrade list. This would fix the ordering issues.

-R uses get_all_depends which already includes all of the depends:

> def get_all_depends(origin, parents_list = nil)
>   if $use_packages_only
>     depends_vars = %w{LIB_DEPENDS RUN_DEPENDS}
>   else
>   depends_vars = %w{FETCH_DEPENDS EXTRACT_DEPENDS PATCH_DEPENDS
>                       BUILD_DEPENDS LIB_DEPENDS RUN_DEPENDS}
>   end


portmaster basically does -R by default and I have been considering
doing the same with portupgrade.





Comment 3 Don Lewis freebsd_committer 2013-03-25 16:40:19 UTC
On 25 Mar, Bryan Drewery wrote:
> I don't think it makes sense to need -r with -a, -a already includes
> everything.

The problem is that the instructions in UPDATING say to use -fr to force
the rebuilding of all the dependent ports, but I don't want to rebuild
*all* ports by specifying -fa.

To use a specific example from UPDATING, with this change I can do:
	portupgrade -arf archivers/libarchive graphics/poppler
will rebuild all out of date ports as well as force the rebuilding of
the two specified ports and any ports that depend on them.

> I think the real problem is that *-R* should be used as well. You didn't
> mention it anywhere, but it is what would cause anything a port depends
> on to be included in the upgrade list. This would fix the ordering issues.
> 
> -R uses get_all_depends which already includes all of the depends:
> 
>> def get_all_depends(origin, parents_list = nil)
>>   if $use_packages_only
>>     depends_vars = %w{LIB_DEPENDS RUN_DEPENDS}
>>   else
>>   depends_vars = %w{FETCH_DEPENDS EXTRACT_DEPENDS PATCH_DEPENDS
>>                       BUILD_DEPENDS LIB_DEPENDS RUN_DEPENDS}
>>   end
> 
> portmaster basically does -R by default and I have been considering
> doing the same with portupgrade.

I think that using -R sort of helps avoid the issue of having ports
being rebuilt while their dependencies are out of date, but using -a
instead solves the same problem.  If -rf someport is specified and the
-R behavior is added, it is probably undesirable to force rebuilding of
the dependencies of someport.  Even after portupgrade is run this way,
it would still need to be run again with -a to upgrade the remaining
ports.

portmaster also seems to implement the force action when -r is used.


My patches also don't totally fix the duplicate install problem.  Origin
changes can still trip things up, but this is a fairly rare problem.
Comment 4 Carlo Strub freebsd_committer 2014-09-11 20:13:54 UTC
Is this PR still relevant?
Comment 5 Don Lewis freebsd_committer 2014-09-11 22:56:53 UTC
(In reply to Carlo Strub from comment #4)
> Is this PR still relevant?

Yes, though the original patch has rotted.  I've been using a different patch for a while, but have recently run into other issues with it.

The importance of this to me has declined a lot recently because I've pretty much given up on upgrading ports in place and switched to poudriere.
Comment 6 Carlo Strub freebsd_committer 2014-09-12 04:38:38 UTC
Thanks for your answer. So, that means we should probably close it if also your newer patch is problematic? How shall we resolve this PR now?
Comment 7 Bryan Drewery freebsd_committer 2014-10-06 22:05:26 UTC
I think this is worth keeping open for now.

Portupgrade is wrong in many ways when it comes to handling dependencies right. It passes off too much of the control to ports resulting in situations such as this and https://github.com/freebsd/portupgrade/issues/30 and making fixing https://github.com/freebsd/portupgrade/issues/58 difficult. As I noted before it is also quite wrong to not recursively upgrade by default as often it is expected by ports that a dependencies are updated first.
Comment 8 commit-hook freebsd_committer 2015-05-14 22:04:31 UTC
A commit references this bug:

Author: bdrewery
Date: Thu May 14 22:03:34 UTC 2015
New revision: 386365
URL: https://svnweb.freebsd.org/changeset/ports/386365

Log:
  Update to 2.4.13-6-g72b4c34

  Changes:
    * Fix ordering of build based on FETCH/PATCH/EXTRACT dependencies.

  PR:		177365
  Submitted by:	truckman@

Changes:
  head/ports-mgmt/portupgrade-devel/Makefile
  head/ports-mgmt/portupgrade-devel/distinfo
Comment 9 Bryan Drewery freebsd_committer 2015-05-14 22:05:09 UTC
Note that I only committed a portion of the patch to include fetch/patch/extract dependencies in portsdb and PortInfo.all_depends()
Comment 10 commit-hook freebsd_committer 2015-05-27 15:18:32 UTC
A commit references this bug:

Author: bdrewery
Date: Wed May 27 15:17:57 UTC 2015
New revision: 387621
URL: https://svnweb.freebsd.org/changeset/ports/387621

Log:
  Update to 2.4.14

  Changes:
    * Fix ordering of build based on FETCH/PATCH/EXTRACT dependencies. [1]
    * Support security vulnerabilities as a build failure reason. [2]
    * portupgrde -p: Fix new dependencies not getting installed. (#58) [3]
    * Don't install missing dependencies which are already satisfied (#62).
      Note that this purposely causes these missing dependencies to not show
      in the job (-n) and final results output which is a compromise to at least
      avoid the wrong packages being installed. The logging issue is in #30.

  PR:		177365 [1]
  Submitted by:	truckman@ [1]
  PR:		192232 [2]
  Submitted by:	Yuri <yuri@rawbw.com> [2]
  PR:		184672 [3]

Changes:
  head/ports-mgmt/portupgrade/Makefile
  head/ports-mgmt/portupgrade/distinfo
Comment 11 Bryan Drewery freebsd_committer 2015-05-27 15:21:28 UTC
Note that my commit is only a partial commit of the patch in this PR. It is not fully satisfied.
Comment 12 Don Lewis freebsd_committer 2015-06-08 00:39:09 UTC
Created attachment 157531 [details]
updated portupgrade enhancement patch

Regenerate portupgrade-devel patch after the partial commit of the previous patch in r386365.

This patch eases the pain when upgrading ports infrequently, when UPDATING contains instructions to run portupgrade -fr on multiple ports.  It changes the interaction between the -a -f and -r flags so that running portupgrade -afr port1 port2 ... upgrades all out of date ports as well as reinstalling all ports that depend on port1, port2, etc., all in the proper dependency order.  It is possible to force all ports to be reinstalled by using -aff, see the updated man page.

Note: I stopped using portupgrade some time ago and switched to building binary packages with poudriere and keeping up to date with pkg upgrade.

One of the reasons that I switched is that I was trying to using portupgrade to build packages using the -p option, and I ran into the bug in PR 184672.  The problem that I was seeing was that portupgrade wasn't building and installing new dependencies for the ports that it was upgrading.  Instead, the new dependencies were getting built by the logic in the ports Makefiles that detected that the new dependencies were missing, but it appeared that they were not getting installed because of the interaction between PACKAGE_BUILDING being set and the its interaction with staging.  My guess that the reason that portupgrade wasn't installing (and also building packages for) the new dependencies was that it was using pkgdb (which has the previous set of dependencies) instead of portsdb (which has the new set of dependencies).  On the other hand, I think the -r flag needs to use pkgdb.  I haven't looked at the fix for  PR 184672, so I can't say whether or not this is totally fixed, but I'm pretty sure that the problem will no longer be fatal.

Even though I haven't used portupgrade with this patch in the recent past, I'm attaching the most recent version here in case someone might find it useful.  For extra credit, a way of combining this with one or more origin changes would be really cool (consider portupgrade -a -fro newport1 oldport1 -fr port2).
Comment 13 Bryan Drewery freebsd_committer 2015-06-08 14:45:00 UTC
Thanks for updating the patch. Likewise I have not used Portupgrade in maybe 2 years now. I only use Poudriere and most of my development time goes into it as well.
Comment 14 w.schwarzenfeld freebsd_triage 2018-01-12 01:09:10 UTC
PR is open more than 4 years.  Is this still relevant?
Comment 15 Bryan Drewery freebsd_committer 2018-01-12 01:20:23 UTC
(In reply to w.schwarzenfeld from comment #14)
> PR is open more than 4 years.  Is this still relevant?

Yes