Bug 209183 - Uses/bdb.mk conversion appears broken
Summary: Uses/bdb.mk conversion appears broken
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Port Management Team
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-05-01 19:18 UTC by Peter Wemm
Modified: 2017-01-16 15:02 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Wemm freebsd_committer freebsd_triage 2016-05-01 19:18:52 UTC
The conversion of bsd.databases.mk to Uses/bdb.mk doesn't work as expected.

Even with a global override, the USE_BDB compatibility hook in bsd.port.mk is selecting the wrong version.

eg:
WITH_BDB_VER=           48
.if ${.CURDIR:M*/databases/db5*} != ""
IGNORE= Damage control
.endif
.if ${.CURDIR:M*/databases/db6*} != ""
IGNORE= Damage control
.endif

.. leads to this:

[00:00:09] ====>> Checking packages for incremental rebuild needed
[00:00:20] ====>> Deleting py27-bsddb-2.7.11_5.txz: missing dependency: db5-5.3.28_3
[..]
[00:00:22] ====>> Hit CTRL+t at any time to see build progress and stats
[00:00:22] ====>> [01][00:00:00] Starting build of databases/db5
[00:00:23] ====>> [01][00:00:01] Finished build of databases/db5: Ignored: Damage control
[00:00:23] ====>> [01][00:00:01] Skipping build of databases/py-bsddb: Dependent port databases/db5 ignored
[..]

py-bsddb/Makefile has "USE_BDB=43+" (like many others) and until r414018, it used to work.

It does seem to probe the installed system and uses whatever it found that is installed, but that doesn't help a poudriere run.  In that case it seems to ignore any user preference of a global default and uses db5 instead.
Comment 1 Peter Wemm freebsd_committer freebsd_triage 2016-05-01 19:19:10 UTC
PS: shouldn't this be using DEFAULT_VERSIONS?
Comment 2 Peter Wemm freebsd_committer freebsd_triage 2016-05-01 19:23:32 UTC
From Uses/bdb.mk
# WITH_BDB_VER  - User defined global variable to set Berkeley DB version.

.. which I have set and is being ignored.
Comment 3 Peter Wemm freebsd_committer freebsd_triage 2016-05-01 19:51:37 UTC
With no bdb already installed:

ports/databases/py-bsddb# make -V WITH_BDB_VER
48
ports/databases/py-bsddb# make debug-bdb      
--INPUTS----------------------------------------------------
PY27_BSDDB_WITH_BDB_VER: 
WITH_BDB_VER: 48
BDB_BUILD_DEPENDS: 
bdb_ARGS (original): 43+
WITH_BDB_HIGHEST (original): 
--PROCESSING------------------------------------------------
supported versions:  48 5 6
invalid versions: 
installed versions: 
eligible versions: 48 5
bdb_ARGS (effective): 48+
WITH_BDB_HIGHEST (override): yes
--OUTPUTS---------------------------------------------------
BDB_VER=5
BDB_INCLUDE_DIR=/usr/local/include/db5
BDB_LIB_NAME=db-5.3
BDB_LIB_CXX_NAME=db_cxx-5.3
BDB_LIB_DIR=/usr/local/lib/db5
BUILD_DEPENDS=
LIB_DEPENDS=
------------------------------------------------------------

It appears that if there's no pre-installed version, it ignores the user's choice and instead internally sets _WITH_BDB_HIGHEST.  This is a serious POLA violation.
Comment 4 Peter Wemm freebsd_committer freebsd_triage 2016-05-01 20:01:45 UTC
The old code had:

-# Override USE_BDB with global WITH_BDB_VER
-.if defined(WITH_BDB_VER)
-. if ${WITH_BDB_VER} != 1
-USE_BDB=       ${WITH_BDB_VER}
-. endif
-.endif
-
-# Override USE_BDB with maintainer's WANT_BDB_VER
-.if defined(WANT_BDB_VER)
-USE_BDB=       ${WANT_BDB_VER}
-.endif

The new code has:

+# Override _bdb_ARGS with global WITH_BDB_VER if the maintainer did not
+# ask for a more specific version.
+.if defined(WITH_BDB_VER)
+. if ${WITH_BDB_VER} != 1 && ${_bdb_ARGS} == yes
+_bdb_ARGS=     ${WITH_BDB_VER}
+. endif
+.endif

This is a loss of functionality. If the maintainer says "43+", then why does that have to mean "ignore the user's request for 48 and use db5 unconditionally" instead?  At the very least it should exclude plus versions from this rule.  "43+" is not "more specific" than "48" at all.
Comment 5 Mathieu Arnold freebsd_committer 2016-05-02 12:03:09 UTC
Yes, it should be moved to DEFAULT_VERSIONS, I'll try to see what I can do.
Comment 6 Mathieu Arnold freebsd_committer 2016-05-02 12:29:31 UTC
There was a reason for that change, maybe I collapsed too many variables.

Can you try 

https://github.com/mat813/freebsd-ports/compare/origin/trunk...bdb
https://github.com/mat813/freebsd-ports/compare/origin/trunk...bdb.patch

And let me know if it solves the problem ?
Comment 7 commit-hook freebsd_committer 2016-05-02 13:17:02 UTC
A commit references this bug:

Author: mat
Date: Mon May  2 13:16:52 UTC 2016
New revision: 414444
URL: https://svnweb.freebsd.org/changeset/ports/414444

Log:
  Fix WITH_BDB_VER support, and switch to DEFAULT_VERSIONS.

  PR:		209183
  Reported by:	peter
  Sponsored by:	Absolight

Changes:
  head/Mk/Uses/bdb.mk
  head/Mk/bsd.default-versions.mk
Comment 8 Peter Wemm freebsd_committer freebsd_triage 2016-07-24 08:10:15 UTC
Unfortunately, this is still broken.

This code:

# 3b. if there is no usable version installed, check defaults
.if empty(_INST_BDB_VER)
_DFLT_BDB_VER:=${_DB_DEFAULTS}
# make sure we use a reasonable version for package builds
_WITH_BDB_HIGHEST=yes
. for i in ${_DFLT_BDB_VER}
.  if empty(_SUPP_BDB_VER:M${i})
_DFLT_BDB_VER:= ${_DFLT_BDB_VER:N${i}}
.  endif
. endfor
_ELIGIBLE_BDB_VER:=${_DFLT_BDB_VER}

.. completely ignores the one from DEFAULT_VERSIONS and uses its hard-coded version list:

With DEFAULT_VERSIONS=bdb=48 in make.conf, then:

databases/py-bsddb # make debug-bdb
--INPUTS----------------------------------------------------
PY27_BSDDB_WITH_BDB_VER: 
BDB_DEFAULT: 48
^^^^
BDB_BUILD_DEPENDS: 
bdb_ARGS (original): 43+
WITH_BDB_HIGHEST (original): 
--PROCESSING------------------------------------------------
supported versions:  48 5 6
invalid versions: 
installed versions: 
eligible versions: 48 5
bdb_ARGS (effective): 48+
WITH_BDB_HIGHEST (override): yes
^^^ What?
--OUTPUTS---------------------------------------------------
IGNORE=
BDB_VER=5
BDB_INCLUDE_DIR=/usr/local/include/db5
BDB_LIB_NAME=db-5.3
BDB_LIB_CXX_NAME=db_cxx-5.3
BDB_LIB_DIR=/usr/local/lib/db5
BUILD_DEPENDS=
LIB_DEPENDS=
------------------------------------------------------------

The code, as written, is if you don't have a version of bdb installed in /usr/local already before you start, it completely ignores what you requested and depends on bdb-5 instead.

# WITH_BDB_HIGHEST
#                       - Use the highest installed version of Berkeley DB.

This is not what is happening.
Comment 9 Peter Wemm freebsd_committer freebsd_triage 2016-07-24 08:54:53 UTC
Looking a little closer..

If I look over the tree, I see a million
USE_BDB=40+
etc.

The Uses/bdb.mk code converts all these to "48+" which it then interprets as "Use the latest version in _DB_PORTS, not the DEFAULT_VERSIONS one"

One possible fix:
sweep the ports tree and get rid of any USE_BDB that's older than 5.  ie:  USE_BDB=yes  or USES+= bdb.  This seems to work and should cover most of the tree.

The real fix would be to rework the selection logic to actually respect the user's request like the rest of the ports tree does for DEFAULT_VERSIONS.

A quick and dirty change to the logic is to leverage the "already installed" check.  This appears to place nicely with the Version+ logic.

Uses/bdb.mk:
@@ -117,6 +116,8 @@
 _INST_BDB_VER+=${bdb}
 . endif
 .endfor
+_INST_BDB_VER:=${_INST_BDB_VER:N${BDB_DEFAULT}}
+_INST_BDB_VER+=${BDB_DEFAULT}
 
 # 2. parse supported versions:
 # 2a. build list from _bdb_ARGS

Then it looks like this:
databases/py-bsddb # make DEFAULT_VERSIONS=bdb=48 debug-bdb | & grep '^BDB_VER'
BDB_VER=48
databases/py-bsddb # make DEFAULT_VERSIONS=bdb=5 debug-bdb | & grep '^BDB_VER'
BDB_VER=5
databases/py-bsddb # make DEFAULT_VERSIONS=bdb=48 USE_BDB=5 debug-bdb | & grep '^BDB_VER'
BDB_VER=5
databases/py-bsddb # make DEFAULT_VERSIONS=bdb=48 USE_BDB=yes debug-bdb | & grep '^BDB_VER'
BDB_VER=48

databases/py-bsddb # make DEFAULT_VERSIONS=bdb=5 USE_BDB=yes debug-bdb | & grep '^BDB_VER'BDB_VER=5
databases/py-bsddb # make USE_BDB=yes debug-bdb | & grep '^BDB_VER'
BDB_VER=5

databases/py-bsddb # make DEFAULT_VERSIONS=bdb=48 USE_BDB=5+ debug-bdb | & grep '^BDB_VER'
BDB_VER=5
databases/py-bsddb # make DEFAULT_VERSIONS=bdb=48 USE_BDB=6 WITH_BDB6_PERMITTED=yes debug-bdb | & grep '^BDB_VER'
BDB_VER=6

This trivial change gives the least astonishing outcome that I can imagine.

Pretending that the default version is already installed avoids duplicating and messing with an awful lot of ugly make logic.  After all, it will be installed at the end of the run.
Comment 10 Mathieu Arnold freebsd_committer 2016-08-08 13:46:10 UTC
Even if I'm the one who touched most of that aweful code most lately, I won't say I understand it.
I'm wondering if the code was not supposed to not let you have 4.8 as the default, so that everything but the *coin ports use 5 or more.
Comment 11 commit-hook freebsd_committer 2016-08-08 13:47:57 UTC
A commit references this bug:

Author: mat
Date: Mon Aug  8 13:46:57 UTC 2016
New revision: 419843
URL: https://svnweb.freebsd.org/changeset/ports/419843

Log:
  USE_BDB cleanup.

  - USE_BDB=4x+ -> USES=bdb.
  - USE_BDB=yes -> USES=bdb.
  - USE_BDB=xx  -> USES=bdb:xx.

  Other modernisations when I see them.

  PR:		209183
  Sponsored by:	Absolight

Changes:
  head/archivers/rpm4/Makefile
  head/audio/jack/Makefile
  head/audio/moc/Makefile
  head/chinese/libpinyin/Makefile
  head/chinese/libtabe/Makefile
  head/chinese/p5-Lingua-ZH-TaBE/Makefile
  head/comms/trustedqsl/Makefile
  head/comms/xastir/Makefile
  head/databases/dbtool/Makefile
  head/databases/evolution-data-server/Makefile
  head/databases/fortytwo-bdb/Makefile
  head/databases/libgda4/Makefile
  head/databases/libgda5/Makefile
  head/databases/memcacheq/Makefile
  head/databases/p5-BDB/Makefile
  head/databases/p5-BerkeleyDB/Makefile
  head/databases/py-bsddb/Makefile
  head/databases/rdfdb/Makefile
  head/databases/ruby-bdb/Makefile
  head/devel/ice/Makefile
  head/devel/librcc/Makefile
  head/devel/subversion/Makefile.common
  head/devel/subversion18/Makefile.common
  head/dns/dnshistory/Makefile
  head/dns/fastresolve/Makefile
  head/editors/nvi-devel/Makefile
  head/editors/poedit/Makefile
  head/graphics/fortytwo/Makefile
  head/japanese/mutt-devel/Makefile
  head/lang/gnu-cobol/Makefile
  head/lang/php55/Makefile.ext
  head/lang/php56/Makefile.ext
  head/lang/php70/Makefile.ext
  head/mail/avenger/Makefile
  head/mail/bmf/Makefile
  head/mail/bogofilter/Makefile
  head/mail/dk-milter/Makefile
  head/mail/dovecot/Makefile
  head/mail/drac/Makefile
  head/mail/evolution-ews/Makefile
  head/mail/exim/Makefile
  head/mail/greyfix/Makefile
  head/mail/isync/Makefile
  head/mail/meta1/Makefile
  head/mail/opendkim/Makefile
  head/mail/perdition/Makefile
  head/mail/postfix/Makefile
  head/mail/postfix-current/Makefile
  head/mail/postfix211/Makefile
  head/mail/prayer/Makefile
  head/mail/sendmail/Makefile
  head/mail/spamprobe/Makefile
  head/mail/spmfilter/Makefile
  head/mail/vacation/Makefile
  head/misc/hotkeys/Makefile
  head/net/netatalk/Makefile
  head/net/openldap24-server/Makefile
  head/net-im/jabberd/Makefile
  head/net-p2p/bitcoin/Makefile
  head/net-p2p/digitalcoin/Makefile
  head/net-p2p/dogecoin/Makefile
  head/net-p2p/jigdo/Makefile
  head/net-p2p/litecoin/Makefile
  head/net-p2p/namecoin/Makefile
  head/net-p2p/twister/Makefile
  head/net-p2p/zetacoin/Makefile
  head/news/inn/Makefile
  head/print/panda/Makefile
  head/security/pks/Makefile
  head/security/sks/Makefile
  head/sysutils/apt/Makefile
  head/sysutils/cfengine22/Makefile
  head/sysutils/cfengine32/Makefile
  head/textproc/p5-RDFStore/Makefile
  head/textproc/redland/Makefile
  head/www/c-icap/Makefile
  head/www/crawl/Makefile
  head/www/squidguard/Makefile
  head/www/webalizer/Makefile
  head/www/xshttpd-devel/Makefile
Comment 12 Mathieu Arnold freebsd_committer 2016-09-08 14:26:48 UTC
Peter, is this still a problem, or did r419843 fix it ?
Comment 13 Peter Wemm freebsd_committer freebsd_triage 2016-10-03 22:05:35 UTC
I'm waiting for builds to run to confirm, but I believe r419843 didn't fix the original problem.
Comment 14 Mathieu Arnold freebsd_committer 2017-01-16 15:02:48 UTC
ping