Bug 210393 - ports-mgmt/poudriere: Unnecessary compilation
Summary: ports-mgmt/poudriere: Unnecessary compilation
Status: Closed Works As Intended
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Bryan Drewery
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2016-06-19 20:13 UTC by leeb
Modified: 2016-06-20 18:37 UTC (History)
5 users (show)

See Also:
vlad-fbsd: maintainer-feedback? (bdrewery)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description leeb 2016-06-19 20:13:35 UTC
Port A was v1.0 now is v1.1
Port B lib-depends only Port A.

Poudriere recompiles Port A because it's version has bumped.
Poudriere recompiles Port B because it depends on Port A.

Client fetching new packages gets the new Port A, but Port B has not changed.
Therefore I see no need to recompile Port B.

Here's a real-world example
---------------------------
Poudriere
=========
[00:01:15] ====>> Starting jail 101amd64-head-RL_WS
[00:01:15] ====>> Logs: /usr/local/poudriere/data/logs/bulk/101amd64-head-RL_WS/2016-06-19_04h03m58s
[00:01:15] ====>> Loading MOVED
[00:01:20] ====>> Calculating ports order and dependencies
[00:02:39] ====>> Sanity checking the repository
[00:02:39] ====>> Checking packages for incremental rebuild needed
[00:02:42] ====>> Deleting faad2-2.7_5,1.txz: new version: 2.7_6,1
echo: write error on stdout
[00:02:58] ====>> Deleting libxine-1.2.6_7.txz: missing dependency: faad2-2.7_5,1
[00:02:59] ====>> Deleting vlc-2.2.4,4.txz: missing dependency: faad2-2.7_5,1
[00:03:00] ====>> Deleting eaglemode-0.89.1_1.txz: missing dependency: libxine-1.2.6_7
[00:03:02] ====>> Deleting stale symlinks
[00:03:02] ====>> Deleting empty directories
[00:03:02] ====>> Cleaning the build queue
[00:03:09] ====>> Recording filesystem state for prepkg... done
[00:03:18] ====>> Building 4 packages using 4 builders
[00:03:18] ====>> Starting/Cloning builders
101amd64-head-RL_WS-job-02: removed
101amd64-head-RL_WS-job-04: removed
101amd64-head-RL_WS-job-01: removed
101amd64-head-RL_WS-job-03: removed
[00:04:06] ====>> Hit CTRL+t at any time to see build progress and stats
[00:04:06] ====>> [01][00:00:00] Starting build of audio/faad
[00:05:00] ====>> [01][00:00:54] Finished build of audio/faad: Success
[00:05:00] ====>> [01][00:00:00] Starting build of multimedia/libxine
[00:05:00] ====>> [02][00:00:00] Starting build of multimedia/vlc
[00:07:40] ====>> [01][00:02:40] Finished build of multimedia/libxine: Success
[00:07:41] ====>> [01][00:00:00] Starting build of x11/eaglemode
[00:09:40] ====>> [01][00:01:59] Finished build of x11/eaglemode: Success
[00:11:55] ====>> [02][00:06:55] Finished build of multimedia/vlc: Success
[00:11:56] ====>> Stopping 4 builders

Client
======
Installed packages to be UPGRADED:
	xfce4-session: 4.12.1_2 -> 4.12.1_3
	python34: 3.4.4_2 -> 3.4.4_3
	python27: 2.7.11_2 -> 2.7.11_3
	libgpg-error: 1.22 -> 1.23
	libgcrypt: 1.7.0 -> 1.7.1
	gettext-runtime: 0.19.7 -> 0.19.8.1
	faad2: 2.7_5,1 -> 2.7_6,1

Installed packages to be REINSTALLED:
	pinentry-0.9.7 (options changed)

You will see that faad2 was compiled and vlc has it as a library dependency.
faad2 was installed on the client machine, but vlc was unchanged.
Comment 1 VK 2016-06-19 21:05:05 UTC
Thanks for the report. I'd reassign this under ports-mgmt/poudriere, but you're actually asking about pkg not reinstalling VLC even though Poudriere rebuilt it due to a dependency upgrade, correct?n
Comment 2 Kimmo Paasiala 2016-06-19 21:08:00 UTC
It's not possible to know in advance that port B doesn't need recompilation. Any change in port A might cause a change in port B requiring a rebuild of B and it's not possible by programmatic means to judge that in advance. The only way is to rebuild B with the updated port A and then see if port B actually changed. This also applies recursively so if you now have a port C that depends on B you must recompile port C as well.

This is not a pkg issue, it's a feature of the ports system and package builders such as poudriere.
Comment 3 leeb 2016-06-19 21:28:58 UTC
(In reply to Vladimir Krstulja from comment #1)
Sorry, this should be poudriere, not pkg.  My bad.
Comment 4 leeb 2016-06-19 21:42:55 UTC
(In reply to kpaasial from comment #2)
First, no, this is not a pkg issue, it's a poudriere issue, I put this in the wrong place, sorry.

Second, I disagree:
Point 1: Port B actually was recompiled, but as it's version number never changed, the client never got it.

Point 2: Port A is a run-time library, dynamically loaded.  I fail to see how changes to it would require recompilation of it's consumers, provided it's API didn't change.  If the API did change, then Port B would require changes as well, changing it's version number in the process.  Can you please provide an example of how a dyn-loaded library can cause a recompile in it's consumer.
Comment 5 VK 2016-06-19 22:09:43 UTC
Reassigning the issue to Poudriere.
Comment 6 Walter Schwarzenfeld 2016-06-20 13:54:57 UTC
It is rather more a discussion for pipermail, than for a PR.
I also never understand, why there is no effective way, to check if a library is still the same or had changed.
Now we have the situation, that portmaster recompiles not enough, poudriere too much and synth far too much.
(Don't missunderstand, I think poudreie is a very good tool). I like FreeBSD, but the only thing that's really missing ist an effective update-tool.
Comment 7 Bryan Drewery freebsd_committer freebsd_triage 2016-06-20 18:37:37 UTC
This is by design and won't change.

The lack of using an ABI compliance checker means we always
need to rebuild a port if its dependencies change.  There is generally no way
to know what impact that dependency will have on the port.  This originally
was added in because of missed shlib bump portrevision bumps, but has been
extended because shlibs are not the only reason to need a rebuild.  A PHP upgrade
means all pecl packages need to be rebuilt as well.  Pkg not chasing this is documented at https://github.com/freebsd/pkg/issues/585.  The solution is provides/requires which the ports framework does not yet have.