Bug 225104

Summary: ports-mgmt/pkg: "pkg update" fails to update package database saying that kernel too old for some package
Product: Ports & Packages Reporter: Pavel Gubin <oit>
Component: Individual Port(s)Assignee: freebsd-pkg (Nobody) <pkg>
Status: Closed FIXED    
Severity: Affects Some People CC: Trond.Endrestol, bdrewery, brd, dch, feld, madpilot, mike.jochen, mizhka, mjoras, nbe, royger, shawn.webb, silvein+freebsd, yfisk
Priority: ---    
Version: Latest   
Hardware: amd64   
OS: Any   

Description Pavel Gubin 2018-01-12 12:06:29 UTC
When I recently try to update package database, pkg fails to perform this, saying that "package timidity++-tcltk" requires newer kernel. (See below)

I can understand if pkg will refuse to "install"/"upgrade"/"add" package which needs newer kernel, but seems to me that disallow updating whole database because of some package I will never install - is obviously a bug.

----------------------
# uname -a
FreeBSD hammer.2lazy.ru 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #2 r327103: Sat Dec 23 15:17:41 MSK 2017     pg@qub.local:/usr/obj/usr/src/sys/GENERIC  amd64
# pkg info -I pkg
pkg-1.10.4                     Package manager
# pkg update -f
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB   3.1MB/s    00:02    
Processing entries:   0%
pkg: Newer FreeBSD version for package timidity++-tcltk:
- package: 1101001
- running kernel: 1101000
pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:11:amd64
Processing entries: 100%
Unable to update repository FreeBSD
Error updating repositories!
#
Comment 1 Mike Jochen 2018-01-13 18:00:55 UTC
Same here in FreeBSD 11.1 with pkg 1.10.4:

root@Omni:/ # pkg update -f
Updating FreeBSD repository catalogue...
[Omni] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[Omni] Fetching packagesite.txz: 100%    6 MiB   3.1MB/s    00:02    
Processing entries:   0%
pkg: Newer FreeBSD version for package timidity++-tcltk:
- package: 1101001
- running kernel: 1100122
pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:11:amd64
Processing entries: 100%
Unable to update repository FreeBSD
Error updating repositories!
Comment 2 Baptiste Daroussin freebsd_committer freebsd_triage 2018-01-15 12:12:48 UTC
What is uname -UK telling you? (both of you?)
Comment 3 Nils Beyer 2018-01-15 12:45:03 UTC
(In reply to Baptiste Daroussin from comment #2)
--------------------------------------------------------------------------
#pkg update^M
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.10.4...
pkg-static: Newer FreeBSD version for package pkg:
- package: 1200055
- running kernel: 1200054

Failed to install the following 1 package(s): /tmp//pkg.txz.hxlOzx

#uname -UK^M
1200056 1200056
--------------------------------------------------------------------------
Comment 4 Mike Jochen 2018-01-15 13:48:32 UTC
root@Omni:/ # uname -UK
1101505 1100122
Comment 5 Pavel Gubin 2018-01-16 06:54:14 UTC
(In reply to Baptiste Daroussin from comment #2)
btw, it also says wrong kernel version, different from uname's 

# pkg update 
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB   3.1MB/s    00:02    
Processing entries:   0%
pkg: Newer FreeBSD version for package py27-sorl-thumbnail:
- package: 1101001
- running kernel: 1101000
pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:11:amd64
Processing entries: 100%
Unable to update repository FreeBSD
Error updating repositories!
# uname -UK
1101001 1101001
Comment 6 Guido Falsi freebsd_committer freebsd_triage 2018-01-16 06:58:59 UTC
(In reply to Baptiste Daroussin from comment #2)
> What is uname -UK telling you? (both of you?)

I'm seeing the same problem with packaged base on head.

I built the packages using "make packages" as usual and did not change anything else from the setup which has been working fine for more than a year.

my uname:

# uname -UK
1200054 1200054
Comment 7 Guido Falsi freebsd_committer freebsd_triage 2018-01-16 07:01:00 UTC
(In reply to Guido Falsi from comment #6)
> (In reply to Baptiste Daroussin from comment #2)
> > What is uname -UK telling you? (both of you?)
> 
> I'm seeing the same problem with packaged base on head.
> 
> I built the packages using "make packages" as usual and did not change
> anything else from the setup which has been working fine for more than a
> year.
> 
> my uname:
> 
> # uname -UK
> 1200054 1200054

Also, this is the full error I get:

# pkg update -f
Updating mpnet-base repository catalogue...
Fetching meta.txz: 100%    820 B   0.8kB/s    00:01    
Fetching packagesite.txz: 100%   40 KiB  40.9kB/s    00:01    
Processing entries:   0%
pkg: Newer FreeBSD version for package FreeBSD-zfs:
- package: 1200056
- running kernel: 1200054
pkg: repository mpnet-base contains packages for wrong OS version: FreeBSD:12:amd64
Processing entries: 100%
Unable to update repository mpnet-base
Comment 8 Yngve Fiskum 2018-01-21 10:17:31 UTC
FreeNAS with Plex jail

root@Plex:/ # pkg update -f                                                     
Updating FreeBSD repository catalogue...                                        
[Plex] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01                      
[Plex] Fetching packagesite.txz: 100%    6 MiB   6.2MB/s    00:01               
Processing entries:   0%                                                        
pkg: Newer FreeBSD version for package gnome-desktop:                           
- package: 1101001                                                              
- running kernel: 1100122                                                       
pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:11:amd64
Processing entries: 100%                                                        
Unable to update repository FreeBSD                                             
Error updating repositories!                                                    
root@Plex:/ # uname -UK                                                         
1101505 1100122
Comment 9 Nils Beyer 2018-01-22 09:28:30 UTC
Okay, after digging through the pkg source code, I found a possible workaround. Add following line to "/usr/local/etc/pkg.conf":
-------------------------------------------------------------------------
OSVERSION = 1200056;
-------------------------------------------------------------------------

for the 12 users.


Respective:
-------------------------------------------------------------------------
OSVERSION = 1101001
-------------------------------------------------------------------------

for the 11 users.


After that, the repository updates should be working again...
Comment 10 Guido Falsi freebsd_committer freebsd_triage 2018-01-22 10:28:41 UTC
(In reply to Nils Beyer from comment #9)
> Okay, after digging through the pkg source code, I found a possible
> workaround. Add following line to "/usr/local/etc/pkg.conf":
> -------------------------------------------------------------------------
> OSVERSION = 1200056;
> -------------------------------------------------------------------------
> 
> for the 12 users.
> 
> 
> Respective:
> -------------------------------------------------------------------------
> OSVERSION = 1101001
> -------------------------------------------------------------------------
> 
> for the 11 users.
> 
> 
> After that, the repository updates should be working again...

Can also be passed through the command line:

pkg -o OSVERSION=1200056 upgrade

Which reduces the risk of forgetting the line in the configuration file.
Comment 11 Trond Endrestøl 2018-02-08 16:19:29 UTC
(In reply to Nils Beyer from comment #9)
The advice in comment #9 is needed when using synth to upgrade the "LiveSystem".
Comment 12 Nils Beyer 2018-02-13 15:06:56 UTC
pkg verifies "osversion" using the data in the ELF ".note.tag" section of the file specified by the environment variable "ABI_FILE". If "ABI_FILE" is not defined it chooses "/bin/sh" as the reference file:
-----------------------------------------------------------------------------
#objdump --section=.note.tag -s /bin/sh

/bin/sh:     file format elf64-x86-64-freebsd

Contents of section .note.tag:
 400218 08000000 04000000 01000000 46726565  ............Free
 400228 42534400 b84f1200 08000000 04000000  BSD..O..........
 400238 02000000 46726565 42534400 00000000  ....FreeBSD.....

#echo "ibase=16; 00124FB8" | bc
1200056

#uname -KU
1200058 1200058
-----------------------------------------------------------------------------

Because I'm doing incremental builds (using -DNO_CLEAN) of world/kernel, the version information in my "/bin/sh" and all other binaries is not increased because "host-osreldate.h" in ${OBJTOP} is not increased for some reason. "getosreldate()" returns the correct value though...
Comment 13 Roger Pau Monné freebsd_committer freebsd_triage 2018-02-14 09:39:43 UTC
I certainly preferred previous pkg behaviour, which would let you install anything (and it might fail to load because of missing/wrong shared libraries). Is there any option to restore previous behaviour? If not, can we please get one?

Thanks, Roger.
Comment 14 Shawn Webb 2018-02-14 13:52:57 UTC
I myself have been bit with this on both vanilla FreeBSD and HardenedBSD. We at HardenedBSD have at least two or three users getting hit with this on a weekly basis.
Comment 15 Nils Beyer 2018-02-14 14:26:34 UTC
(In reply to Roger Pau Monné from comment #13)

well, I can understand that some packages - especially kernel modules packages - need a minimum kernel version for functioning properly.

But why "timidity++-tcltk" or "qt5-x11extras" (in my yesterday case) are needing a minimum kernel version, that's a mystery.

And relying on data in ELF sections in binaries where the version of the building host and not the version of the currently running host is saved, instead of using getosreldate() or sysctls is another question IMHO...
Comment 16 Roger Pau Monné freebsd_committer freebsd_triage 2018-02-14 14:29:44 UTC
(In reply to Nils Beyer from comment #15)
Also keep in mind that's there's no base library ABI stability in HEAD, so it's not just kernel modules, but everything linked to any base library is only guaranteed to work against the version it has been build against. Everything else is just luck.
Comment 17 Nils Beyer 2018-02-14 14:38:47 UTC
(In reply to Roger Pau Monné from comment #16)

true; so, there really have to be two checks:

    current kernel version  vs.  package kernel version requirement

AND

    current base system version  vs.  package base system version requirement
Comment 18 Trond Endrestøl 2018-02-14 17:37:36 UTC
(In reply to Trond.Endrestol from comment #11)
The right place is the LiveSystem-environment file or equivalent.

(In reply to Nils Beyer from comment #12)
Regarding comment #12, I first thought it prudent to delete the binaries from /usr/obj/$ARCH/usr/src/bin/sh prior to running make buildworld, until the matter is fully resolved. I.e. delete sh, sh.debug, and sh.full. I was proved wrong. Clearing /usr/obj and rebuilding world led to a sh binary with the right osversion, see below. Which file(s) should we forcefully remove to ensure the right value of osversion for sh?

# objdump --section=.note.tag -s sh
sh:     file format elf64-x86-64-freebsd
Contents of section .note.tag:
 400218 08000000 04000000 01000000 46726565  ............Free
 400228 42534400 c5ce1000 08000000 04000000  BSD.............
 400238 02000000 46726565 42534400 00000000  ....FreeBSD.....

C5CE1000 (LE) => 0010CEC5 (BE) = 1101509
Comment 19 Nils Beyer 2018-02-14 17:55:11 UTC
(In reply to Trond.Endrestol from comment #18)

As far as I understand; the responsible file would be:

   ${OBJTOP}/host-osreldate.h

But that doesn't matter because the "osversion" tag within the ELF files is always lagging behind.

Example: you're running osversion=1101508 and building a new world/kernel with osversion=1101509. All of your ELF files are tagged with what is in

    /usr/include/osreldate.h

at the moment when you build world. So, after finishing building and installing world you have all ELF files tagged with 1101508 although you're running 1101509. Furthermore, "host-osreldate.h" is never updated (here on my machine using NO_CLEAN), even if I build the same world with the up-to-date installed world; it just says "`host-osreldate.h' is up to date." which is not true.

Perhaps it would be better for "libpkg" to rely on what "sysctl" or "getosreldate()" are saying.

But that's just my simple-minded user's point of view...
Comment 20 Nils Beyer 2018-02-14 18:00:56 UTC
(In reply to Roger Pau Monné from comment #13)

my solution to get the old behaviour back would be to add following line to "/usr/local/etc/pkg.conf":
--------------------------------------------------------------------------
OSVERSION = 9999999;
--------------------------------------------------------------------------
Comment 21 Trond Endrestøl 2018-02-14 19:45:44 UTC
(In reply to Nils Beyer from comment #19)
I see the same behaviour on stable/11 and there's no ${OBJTOP}/host-osreldate.h in sight, only:

# locate osreldate.h
/usr/include/osreldate.h
/usr/obj/usr/src/include/osreldate.h
/usr/obj/usr/src/lib32/usr/include/osreldate.h
/usr/obj/usr/src/tmp/usr/include/osreldate.h
/usr/obj/usr/src/world32/usr/src/include/osreldate.h
Comment 22 Nils Beyer 2018-02-14 20:14:35 UTC
(In reply to Trond.Endrestol from comment #21)

you're right; I don't have that file on my older system, too. So this file does not matter.

I'm out off ideas here, sorry; you'll have to clean "/usr/obj" before building world or have to use the "OSVERSION" workaround in "/usr/local/etc/pkg.conf"...
Comment 23 Trond Endrestøl 2018-02-14 20:29:43 UTC
(In reply to Nils Beyer from comment #22)
Relying on /usr/include/osreldate.h during make buildworld must be due to a short circuit somewhere. Maybe I'm wrong. Using /usr/obj/usr/src/include/osreldate.h might be problematic for the preprocessor when handling the other header files.

(In reply to Nils Beyer from comment #20)
I like your idea of setting OSVERSION to its maximum value in pkg.conf (and LiveSystem-environment for users of synth).
Comment 24 Bryan Drewery freebsd_committer freebsd_triage 2018-02-14 20:34:39 UTC
(In reply to Roger Pau Monné from comment #13)
> I certainly preferred previous pkg behaviour, which would let you install
> anything (and it might fail to load because of missing/wrong shared
> libraries). Is there any option to restore previous behaviour? If not, can
> we please get one?
> 
> Thanks, Roger.

The previous behavior was called "russian roulette". Every time you run
"pkg upgrade" you risk nuking the usability of all packages depending on
what forward-incompatible change they were built against.  Some recent
examples are linking against libdl and ino64.

The workaround people are using should not be used - the proper thing with
pkg on CURRENT has *always been* that you should upgrade the system before
upgrading packages.  It is a different process than building ports directly
on your system.  Building locally ensures ABI compatibility every time, but
with pkg you are getting packages from a future CURRENT system.  FreeBSD
does not have forward-compatibility in general, only backwards.
Comment 25 Bryan Drewery freebsd_committer freebsd_triage 2018-02-14 20:35:17 UTC
(In reply to Nils Beyer from comment #19)
> (In reply to Trond.Endrestol from comment #18)
> 
> As far as I understand; the responsible file would be:
> 
>    ${OBJTOP}/host-osreldate.h

This file should be used for absolutely nothing.  It is a private build
metadata file.
Comment 26 Nils Beyer 2018-02-14 20:53:12 UTC
(In reply to Bryan Drewery from comment #24)

> [...] the proper thing with pkg on CURRENT has *always been* that you should upgrade the system before upgrading packages.

that's what I have done (and always have done); please see my comment #3 - that doesn't explain why I'm getting while bootstrapping "pkg":
------------------------------------------------------------------------
pkg-static: Newer FreeBSD version for package pkg:
- package: 1200055
- running kernel: 1200054
------------------------------------------------------------------------

although "uname -UK" tells otherwise:
------------------------------------------------------------------------
#uname -UK
1200056 1200056
------------------------------------------------------------------------


I think the string "running kernel" in pkg's output is misleading as the "libpkg" logic doesn't check the kernel version but the tagged version in "/bin/sh"...
Comment 27 commit-hook freebsd_committer freebsd_triage 2018-02-15 00:14:58 UTC
A commit references this bug:

Author: brd
Date: Thu Feb 15 00:14:50 UTC 2018
New revision: 461849
URL: https://svnweb.freebsd.org/changeset/ports/461849

Log:
  Update pkg to 1.10.5

  PR:		225104
  Reviewed by:	bdrewery

Changes:
  head/ports-mgmt/pkg/Makefile
  head/ports-mgmt/pkg/distinfo
Comment 28 Roger Pau Monné freebsd_committer freebsd_triage 2018-02-15 09:29:16 UTC
(In reply to Bryan Drewery from comment #24)
I agree, but I still want to know what's the recommended way to circumvent this limitation. Is adding "OSVERSION = 9999999;" to pkg.conf the right way?

I know things might blow up, but worst case I will just update base if required.
Comment 29 Brad Davis freebsd_committer freebsd_triage 2018-02-15 15:01:42 UTC
(In reply to Roger Pau Monné from comment #28)
Hi Roger,

I would recommend using:

pkg -o OSVERSION=whatever install/upgrade/etc

until 1.10.5 is on the mirrors.