Bug 200391 - ports-mgmt/poudriere: Clear out unneeded packets after build (was: Missing packets after bulk build)
Summary: ports-mgmt/poudriere: Clear out unneeded packets after build (was: Missing pa...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Bryan Drewery
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-05-22 14:16 UTC by elofu17
Modified: 2018-02-06 10:23 UTC (History)
1 user (show)

See Also:
bugzilla: maintainer-feedback? (bdrewery)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description elofu17 2015-05-22 14:16:43 UTC
I think I found a bug.
Poudriere just discarded my lynx package when doing a bulk build.


Step by step:

# poudriere ports -u
It updated a couple of ports, lynx was one of them.
(note: mutt was *not* updated)

# grep mutt ports_to_build.txt
mail/mutt

So, in my list of ports to build, I have mail/mutt.

I rebuild my ports...
# poudriere bulk -f ports_to_build.txt -j 93amd64
[00:00:00] ====>> Creating the reference jail... done
[00:01:11] ====>> Mounting system devices for 93amd64-default
[00:01:11] ====>> Mounting ports/packages/distfiles
[00:01:11] ====>> Stashing existing package repository
[00:01:12] ====>> Mounting packages from: /usr/local/poudriere/data/packages/93amd64-default
[00:01:12] ====>> Copying /var/db/ports from: /usr/local/share/freebsd-ports/poudriere.d/93amd64-options
[00:01:13] ====>> Appending to make.conf: /usr/local/share/freebsd-ports/poudriere.d/make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/93amd64-default/ref/etc/resolv.conf
[00:01:13] ====>> Starting jail 93amd64-default
[00:01:13] ====>> Logs: /usr/local/poudriere/data/logs/bulk/93amd64-default/2015-05-22_15h05m55s
[00:01:13] ====>> Loading MOVED
[00:01:13] ====>> Calculating ports order and dependencies
[00:01:13] ====>> MOVED: www/squid33 renamed to www/squid
[00:01:13] ====>> MOVED: lang/lua renamed to lang/lua51
[00:01:27] ====>> Sanity checking the repository
[00:01:27] ====>> Checking packages for incremental rebuild needed
[00:01:28] ====>> Deleting argus-sasl-3.0.8.txz: new version: 3.0.8.1
[00:01:32] ====>> Deleting easy-rsa-2.2.0.m.txz: new version: 2.2.2
[00:01:35] ====>> Deleting libgpg-error-1.19.txz: new version: 1.19_1
[00:01:36] ====>> Deleting libtasn1-4.5.txz: new version: 4.5_1
[00:01:37] ====>> Deleting lynx-2.8.8.2_1,1.txz: new version: 2.8.8.2_2,1
  Here, it deletes lynx.
[00:01:45] ====>> Deleting p5-Net-HTTP-6.07_1.txz: new version: 6.09
[00:01:54] ====>> Deleting rubygem-facter-2.4.3.txz: new version: 2.4.4
[00:01:57] ====>> Deleting gnutls-3.2.21_1.txz: missing dependency: libtasn1-4.5
[00:01:57] ====>> Deleting libassuan-2.2.0.txz: missing dependency: libgpg-error-1.19
[00:01:57] ====>> Deleting libgcrypt-1.6.3.txz: missing dependency: libgpg-error-1.19
[00:01:57] ====>> Deleting libksba-1.3.3.txz: missing dependency: libgpg-error-1.19
[00:01:57] ====>> Deleting libxslt-1.1.28_6.txz: missing dependency: libgcrypt-1.6.3
[00:01:58] ====>> Deleting openvpn-2.3.6_4.txz: missing dependency: easy-rsa-2.2.0.m
[00:01:58] ====>> Deleting p11-kit-0.23.1.txz: missing dependency: libtasn1-4.5
[00:01:59] ====>> Deleting p5-libwww-6.13.txz: missing dependency: p5-Net-HTTP-6.07_1
[00:01:59] ====>> Deleting pinentry-tty-0.9.2.txz: missing dependency: libgcrypt-1.6.3
[00:02:00] ====>> Deleting puppet-3.7.5.txz: missing dependency: rubygem-facter-2.4.3
[00:02:00] ====>> Deleting xmlto-0.0.26_2.txz: missing dependency: libxslt-1.1.28_6
[00:02:00] ====>> Deleting emacs-nox11-24.5,3.txz: missing dependency: gnutls-3.2.21_1
[00:02:00] ====>> Deleting gnupg-2.1.4_1.txz: missing dependency: gnutls-3.2.21_1
[00:02:01] ====>> Deleting munin-node-2.0.25_4.txz: missing dependency: p5-libwww-6.13
[00:02:01] ====>> Deleting pinentry-0.9.2.txz: missing dependency: pinentry-tty-0.9.2
[00:02:02] ====>> Deleting stale symlinks
[00:02:02] ====>> Deleting empty directories
[00:02:02] ====>> Cleaning the build queue
[00:02:02] ====>> Recording filesystem state for prepkg... done
[00:02:02] ====>> Building 19 packages using 3 builders
[00:02:02] ====>> Starting/Cloning builders
[00:04:05] ====>> Hit CTRL+t at any time to see build progress and stats
[00:04:05] ====>> [01][00:00:00] Starting build of security/libgpg-error
[00:04:05] ====>> [02][00:00:00] Starting build of security/libtasn1
[00:04:05] ====>> [03][00:00:00] Starting build of net/p5-Net-HTTP
[00:04:40] ====>> [03][00:00:35] Finished build of net/p5-Net-HTTP: Success
[00:04:40] ====>> [03][00:00:00] Starting build of security/easy-rsa
[00:04:45] ====>> [01][00:00:40] Finished build of security/libgpg-error: Success
[00:04:48] ====>> [01][00:00:00] Starting build of security/libgcrypt
[00:04:53] ====>> [02][00:00:48] Finished build of security/libtasn1: Success
[00:04:55] ====>> [02][00:00:00] Starting build of security/libksba
[00:04:57] ====>> [03][00:00:17] Finished build of security/easy-rsa: Success
[00:04:58] ====>> [03][00:00:00] Starting build of security/p11-kit
[00:05:18] ====>> [02][00:00:23] Finished build of security/libksba: Success
[00:05:19] ====>> [02][00:00:00] Starting build of security/libassuan
[00:05:26] ====>> [01][00:00:38] Finished build of security/libgcrypt: Success
[00:05:26] ====>> [01][00:00:00] Starting build of security/pinentry-tty
[00:05:28] ====>> [02][00:00:09] Finished build of security/libassuan: Success
[00:05:28] ====>> [02][00:00:00] Starting build of sysutils/rubygem-facter
[00:05:48] ====>> [01][00:00:22] Finished build of security/pinentry-tty: Success
[00:05:54] ====>> [01][00:00:00] Starting build of security/pinentry
[00:06:01] ====>> [01][00:00:07] Finished build of security/pinentry: Success
[00:06:01] ====>> [01][00:00:00] Starting build of www/p5-libwww
[00:06:03] ====>> [03][00:01:05] Finished build of security/p11-kit: Success
[00:06:04] ====>> [03][00:00:00] Starting build of security/gnutls
[00:06:08] ====>> [02][00:00:40] Finished build of sysutils/rubygem-facter: Success
[00:06:10] ====>> [02][00:00:00] Starting build of net-mgmt/argus3
[00:06:22] ====>> [01][00:00:21] Finished build of www/p5-libwww: Success
[00:06:23] ====>> [01][00:00:00] Starting build of sysutils/munin-node
[00:06:51] ====>> [02][00:00:41] Finished build of net-mgmt/argus3: Success
[00:06:52] ====>> [02][00:00:00] Starting build of security/openvpn
[00:06:54] ====>> [01][00:00:31] Finished build of sysutils/munin-node: Success
[00:06:55] ====>> [01][00:00:00] Starting build of sysutils/puppet
[00:07:22] ====>> [01][00:00:27] Finished build of sysutils/puppet: Success
[00:07:26] ====>> [02][00:00:34] Finished build of security/openvpn: Success
[00:08:08] ====>> [03][00:02:04] Finished build of security/gnutls: Success
[00:08:10] ====>> [01][00:00:00] Starting build of editors/emacs-nox11
[00:08:10] ====>> [02][00:00:00] Starting build of security/gnupg
[00:09:09] ====>> [02][00:00:59] Finished build of security/gnupg: Success
[00:11:09] ====>> [01][00:02:59] Finished build of editors/emacs-nox11: Success
[00:11:10] ====>> Stopping 3 builders
[00:11:19] ====>> Creating pkgng repository
Creating repository in /tmp/packages: 100%
Packing files for repository: 100%
[00:11:35] ====>> Committing packages to repository
[00:11:35] ====>> Removing old packages
[00:11:35] ====>> Built ports: net/p5-Net-HTTP security/libgpg-error security/libtasn1 security/easy-rsa security/libksba security/libgcrypt security/libassuan security/pinentry-tty security/pinentry security/p11-kit sysutils/rubygem-facter www/p5-libwww net-mgmt/argus3 sysutils/munin-node sysutils/puppet security/openvpn security/gnutls security/gnupg editors/emacs-nox11
[93amd64-default] [2015-05-22_15h05m55s] [committing:] Queued: 19 Built: 19 Failed: 0  Skipped: 0  Ignored: 0  Tobuild: 0   Time: 00:10:22
[00:11:35] ====>> Logs: /usr/local/poudriere/data/logs/bulk/93amd64-default/2015-05-22_15h05m55s
[00:11:35] ====>> Cleaning up
[00:11:35] ====>> Umounting file systems

Done.
It never built the new lynx port though, so now my pkg repo is missing lynx. :-/


I have a 10.1 build-jail as well. The same error happened there.


Just to verify the deps, I ran 'poudriere bulk -vv -f ports_to_build.txt -j 93amd64'.
[00:01:19] ====>> Debug: mail/mutt depends on converters/libiconv
[00:01:19] ====>> Debug: mail/mutt depends on databases/db5
[00:01:19] ====>> Debug: mail/mutt depends on devel/autoconf
[00:01:19] ====>> Debug: mail/mutt depends on devel/automake
[00:01:19] ====>> Debug: mail/mutt depends on devel/gettext-runtime
[00:01:19] ====>> Debug: mail/mutt depends on devel/gettext-tools
[00:01:19] ====>> Debug: mail/mutt depends on dns/libidn
[00:01:19] ====>> Debug: mail/mutt depends on misc/mime-support
[00:01:20] ====>> Debug: mail/mutt depends on ports-mgmt/pkg
[00:01:20] ====>> Debug: mail/mutt depends on security/cyrus-sasl2
[00:01:20] ====>> Debug: mail/mutt depends on textproc/docbook-xsl
[00:01:20] ====>> Debug: mail/mutt depends on textproc/libxslt
[00:01:20] ====>> Debug: mail/mutt depends on textproc/urlview
[00:01:20] ====>> Debug: mail/mutt depends on www/lynx

Yepp, mutt still depends on lynx.


So, is this a bug? Shouldn't lynx have been built?

/Elof
Comment 1 Bryan Drewery freebsd_committer 2015-05-22 15:10:47 UTC
No, lynx is a BUILD_DEPENDS only in mutt. It's not a poudriere bug.

./files/patch-doc-Makefile.am:+ -LC_ALL=C lynx -dump -nolist -with_backspaces -display_charset=us-ascii manual.html > $@

It's using it for the actual build. Just ww lynx to your list of ports if you want it.
Comment 2 elofu17 2015-05-23 19:54:59 UTC
Unfortunetly I know too little about BUILD_DEPENDS to fully understand.

I don't explicitly want Lynx, so I don't want to add it to my list of ports to build.
Mutt however is in my list.


Mutt depend on Lynx, so it has always been built on my build machine.
My poudriere run yesterday removed Lynx but then never built a new one.

[00:01:37] ====>> Deleting lynx-2.8.8.2_1,1.txz: new version: 2.8.8.2_2,1


I'm having a hard time understanding how this behavior can be ok.
It makes me not trust Poudriere if various ports suddenly can vanish from my destination directory just because the port was updated.
I would expect Poudriere to update it, not remove it.
Comment 3 Bryan Drewery freebsd_committer 2015-05-23 20:08:42 UTC
A build dependency means it is only needed to build mutt. It means that
installing mutt _will not_ implicitly install lynx as well. It only
does that if lynx is a RUN dependency. Given the only package you asked
for it to build here was mutt and mutt is built there is no cause for concern.

In your bulk output mutt was not rebuilt, so while it depends on lynx to build, because mutt was not rebuilt Poudriere did not queue a build of lynx. It noted that it will need lynx to build mutt, but mutt wasn't built. So
no lynx built.

If you want lynx then add it to your list. It is by sheer accident that lynx
was ever provided in your repository.

The bug I do see here is that Poudriere should never have provided lynx in your repository since it was not asked for by you and was not a RUN dependency of anything else you asked for. This is not really worth solving though as it is overcomplicated and risky for little gain.
Comment 4 elofu17 2015-05-23 23:16:51 UTC
Thanks for a thorough explaination!

Ok, then I guess that Mutt or some other port in my list did at some point require Lynx as a Run dependency, so the lynx package got built.

Oh, well, no need to dig further.

Since my Mutt users won't need Lynx, it doesn't matter that this package is now gone from my repo.
Comment 5 elofu17 2015-05-25 12:34:40 UTC
Hmmm, no, this was not a RUN dependency.

I just ran 'poudriere bulk -c -f ports_to_build.txt -j 93amd64' with '-c' to clear all packages and then rebuild it all. Now the Lynx package reappeared in the destination directory!

So this is not a bug, but for me, this is undesireable.

1. I do not want to put packages on my repo server that aren't really needed (like Lynx). I want it clean.
2. I don't want packages on my repo server to sometimes exist and sometimes vanish (as e.g. the Lynx package just did). This is just confusing (and my sanity checking script on the repo server goes bananas).
3. I don't want to change to *adding* newly built packages to my repo and let the old ones remain, instead of replacing everything on the server with the latest build. No, I want to have a clean repo with only the latest build. No old, unpatched, packages should ever exist in my repo.

Currently, the package Lynx is (sometimes) existing in my repo. If some of my users search for it, they will find it and can install it. (note: Lynx is just an example, it could be any other port)
This is bad!
I didn't intend for Lynx to be available, so it shouldn't be.
(if I some day remove Mutt from my list of ports to build, Lynx will never be updated, and hence the clients that did install it will be using unsupported software that will never be updated.)



Suggestion:
After a bulk build, I think that poudriere should remove all packages from the destination dir that are not depended on for running (RUN_DEPENDS).
That way the destination dir, and in turn the repo server, will never contain unneeded/unwanted packages.
The problem with flapping packages is then solved.
The problem that poudriere creates more packages than I actually need or want is solved.
Bonus: There will be less packages to copy to the repo server.



In my opinion, removing the unneeded packages should be the default.
If users would like to keep packages built solely for the BUILD_DEPENDS, add an option for it. Something like this:

bulk -b   By default poudriere removes all unnecessary packages from the destination directory, keeping only the specified ports and their RUN_DEPENDS. Adding -b will keep all built packages (BUILD_DEPENDS) even though the specified ports don't need them for running, just for building.


...or something simillar.

I suspect that removing unneeded packages can not be the default behavior though, so maybe you can instead add an option for purging them?

bulk -d   After building everything, delete all unnecessary packages (BUILD_DEPENDS) from the destination directory, leaving only the specified ports and their RUN_DEPENDS.

...or something simillar.

You can probably come up with the best solution to the problem.
Comment 6 elofu17 2015-06-04 10:06:21 UTC
Any comments to this?

I think it is wrong that poudriere deletes packages like this just because a newer version exist.

Or rather, the "problem" is that poudriere currently leave the BUILD_DEPENDS packages in the destination dir even though those packages aren't really needed.
These "garbage" packages are the problem and should be removed by default, leaving a clean dir with only what is actually needed to run.

Therefore I would like an option to 'bulk' that removes all un-needed packages after the build is complete.

(
note:
Even running 'poudriere pkgclean -y -f ports.txt -j foobar' after the bulk build won't clean all un-needed packages, I think. And even if it did, I still would like the 'bulk' option so that the purging can be done while the jail is still running.
Executing 'poudriere pkgclean' afterwards takes a lot of time starting and stopping all the jails.
)

/Elof
Comment 7 Bryan Drewery freebsd_committer 2015-06-04 17:46:52 UTC
(In reply to elofu17 from comment #6)
> Any comments to this?
> 
> I think it is wrong that poudriere deletes packages like this just because a
> newer version exist.
> 
> Or rather, the "problem" is that poudriere currently leave the BUILD_DEPENDS
> packages in the destination dir even though those packages aren't really
> needed.
> These "garbage" packages are the problem and should be removed by default,
> leaving a clean dir with only what is actually needed to run.

I do somewhat agree. I thought I mentioned it before but maybe not. It would
be a lot of extra effort on Poudriere's part to clean this up and will
surprise a lot of people.

> 
> Therefore I would like an option to 'bulk' that removes all un-needed
> packages after the build is complete.
> 

I do want to make bulk remove packages which are not needed at all, including
build depends. I have similar concerns though that it will surprise people.

There's currently a pkg issue because Poudriere does not remove some older
unneeded perl packages.
Comment 8 elofu17 2015-06-04 23:45:25 UTC
...but if you add a new option to 'bulk', that wouldn't surprise people.

Like: poudriere bulk -k -f ports_to_build.txt -j 93amd64

Example for the man-page:

-k   Remove all un-needed packages from the destination dir after the build is complete, leaving only what is really needed in your package repository. Extra packages that were built merely for BUILD_DEPENDS will be removed. Important: This option will remove any and all packages that are not explicitly specified or not depended upon for running (RUN_DEPENDS), so make sure you have listed ALL your top ports (via -f or on commandline). Example: If you usually build 100 ports but only specify one port, this option will purke the other 99 ports and all their depending packages. Probably not what you intended.
Comment 9 elofu17 2015-06-29 13:47:50 UTC
Shorter version:

-k   Remove all un-needed packages from the destination dir after the build. Packages that were automatically built merely for BUILD_DEPENDS will be purged, keeping only what is actually needed. Note: This option will remove any and all packages that are not explicitly specified (via -f or on commandline) and are not depended upon for running (RUN_DEPENDS).



Any thoughts?
Comment 10 elofu17 2015-11-23 16:20:10 UTC
Hi Bryan.

Any thoughts of how to fix this?

The final goal is to stop "polluting" the pkg repository with unwanted packages. Only a bare minimum of packages should be present.
Comment 11 Walter Schwarzenfeld freebsd_triage 2018-01-12 23:40:25 UTC
Is this still relevant?
Comment 12 Bryan Drewery freebsd_committer 2018-01-24 19:44:57 UTC
I think TRIM_ORPHANED_BUILD_DEPS=no (defaults to yes) and use of pkgclean would avoid the issues reported here.
Comment 13 elofu17 2018-02-06 10:20:30 UTC
Poudriere nowdays generates fewer packets.
However, it doesn't feel like all unneeded packets are purged.

I don't really know how to verify this, but here's an attempt:


A spot test:
I start from scratch and build 108 specified ports:
  poudriere bulk -c -v -f my-packages.txt -j 103amd64
A total of 412 packets are compiled.

One of the 412 packets are 'bison'. I select that one for analysis.
  /usr/local/poudriere/data/packages/103amd64-default/All/bison-3.0.4,1.txz

I grep for 'bison' in the verbose log, to see where it is used:
  databases/mysql56-client depends on devel/bison
  databases/mysql56-server depends on devel/bison
  devel/bison depends on devel/gettext-runtime
  devel/bison depends on devel/gettext-tools
  devel/bison depends on devel/m4
  devel/bison depends on lang/perl5.24
  devel/bison depends on ports-mgmt/pkg
  devel/bison depends on print/indexinfo
  net-mgmt/argus3 depends on devel/bison
  net-mgmt/argus3-clients depends on devel/bison
  net/daq depends on devel/bison
  net/libpcap depends on devel/bison
  shells/bash depends on devel/bison
  sysutils/monit depends on devel/bison
  textproc/flex depends on devel/bison

I browse to https://www.freshports.org/ and check if mysql56-client, mysql56-server, argus3, argus3-clients, daq, libpcap, bash, monit or flex has a RUNNING dependency to bison.
None of them do!
They all only have a BUILD dependency to 'bison'.
So... Based on this quick spot test it seems the problem still exist.

If no packet has a RUNNING dependency to bison I think it should be automatically cleaned after the build.

Perhaps this ticket should be re-opened?
Comment 14 elofu17 2018-02-06 10:22:46 UTC
Reopening.