I have a couple of remote systems that I maintain with ssh, with no console access. The ipfw incompatibility would make 14.3->15.0 a disaster if I hadn't already seen chatter on the lists about it. Doesn't this deserve an erratum, or a warning from freebsd-update itself before installing the new kernel? Even knowing I'm left with unpleasant alternatives. I jiggered something in a local rc file to run the second freebsd-update install and then reboot, trusting that the update was successful with no interaction needed. That worked, with the amusing glitch that many installed files ended up with date 12/31/69 because on my little rpi4b systems the clock is unset until ntpd can do it. btw, why does freebsd-update not set the date/time of the files it installs to what the repo has? The alternative is to manually extract ipfw and its needed libraries, and then hope that ipfw works to install my intended firewall rules which will let me get in after the new kernel boots. I haven't dared to do that. Do you have advice on which to try? Will the pkg setup in 16 not be subject to the same sort of issues?
It should be here https://www.freebsd.org/releases/15.0R/relnotes/, no?
Certainly belongs in relnotes, but since this issue has the potential to essentially brick systems unless or until console access can be arranged, something more prominent seems to me to be appropriate. I can understand and agree with the fbsd approach that requires user interaction to complete a major upgrade. But as a macOS user also I greatly appreciate the ability to just click ok and walk away, and come back to an upgraded system. Is there some scenario where an admin would want/need to do anything after the first reboot, before running freebsd-update install again to install the world and then reboot once more? As a safety net, that first boot could be done as a boot-once, with automatic fallback to the previous kernel if for some reason the new kernel fails to boot successfully. There will always be some failure mode where that doesn't work, but that might decrease the incidence of disaster. Oh, and just to state the obvious, turning on default-to-accept in sysctl.conf, even temporarily, is not acceptable on an Internet-connected system that's not behind a firewall. I'm not up to speed on pkgbase. Does that solve this issue?
> I'm not up to speed on pkgbase. Does that solve this issue? I will stick to source updates and freebsd-update for the time being.
^triage: cc recent committers
IMO, this is a kernel bug that it doesn't work with old ipfw binaries. That is an unacceptable level of ABI breakage. _Maybe_ you could break the ABI if it's been multiple stable branches, but upgrading from stable/N to stable/N+1 has to work following the usual process of rebooting into the kernel first.
(In reply to John Baldwin from comment #6) Do we know where the breakage comes from?
IPFW is foot-shooting by design. Every single person who ever used this firewall learned that adding the entry net.inet.ip.fw.default_to_accept="1" to loader.conf is inevitable. The ABI was changed in commit 4a77657cbc011ea657ccb079fff6b58b295eccb0 (ipfw: migrate ipfw to 32-bit size rule number), and the commit message is clear in this regard. Among other things, we can read: "This changes ABI due to the changed opcodes...". If someone skipped adding this to the documentation, then the documentation has to be updated. Regarding upgrades and double reboots, this can probably be changed, or at least no longer be an official recommendation. Recommending the use of ZFS and boot environments for major version upgrades by upgrading both kernel and userland in one step in a newly created boot environment is the safest way to upgrade in 2026. If people have to use UFS, let them follow the old, official guide, but we should also allow taking some risk by upgrading in one step - sometimes it works wonkily, but usually it does work for major version upgrades too.
doing `service ipfw disable` and `service jail disable` before the reboot did the trick for us where all services except ssh run in jails, so we would live without host firewall until the second `freebsd-update install` run. Running `freebsd-update install` twice and then reboot and then do the `pkg upgrade -fy` step should also work but would leave you with 15-kernel _and_ userland, which might be harder to recover from than just booting `kernel.old`. But yes, I totally expected ipfw not to break. No idea if this is fixable.
I filed a differential (https://reviews.freebsd.org/D54763) with an alternative way for almost seamless upgrade. It requires an additional intermediate upgrade to the most recent stable/14 though. Feel free to comment and test.
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=704ec5e68c44f08d83f3b0daa315c6143338863f commit 704ec5e68c44f08d83f3b0daa315c6143338863f Author: Boris Lytochkin <lytboris@gmail.com> AuthorDate: 2026-02-10 11:50:20 +0000 Commit: Andrey V. Elsukov <ae@FreeBSD.org> CommitDate: 2026-02-10 11:50:20 +0000 ipfw: add ability to run ipfw(8) binary with 15.0+ kernel module After D46183 the KBI was changed and this made the upgrade procedure to 15.0+ version a bit difficult, because the old binary can not load firewall rules when the new kernel is loaded. This commit adds the sbin/ipfw15 binary that uses new KBI, and then original sbin/ipfw can detect new KBI and run the new binary instead. PR: 291562 Reviewed by: jhb, glebius Fixes: 4a77657cbc01 MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D54763 sbin/Makefile | 1 + sbin/ipfw/main.c | 26 + sbin/ipfw15/Makefile (new) | 23 + sbin/ipfw15/Makefile.depend (new) | 19 + sbin/ipfw15/altq.c (new) | 152 + sbin/ipfw15/dummynet.c (new) | 2016 +++++++ sbin/ipfw15/include15/alias15.h (new) | 259 + .../ipfw15/include15/netinet/ip_dummynet15.h (new) | 284 + sbin/ipfw15/include15/netinet/ip_fw15.h (new) | 1172 ++++ .../include15/netinet6/ip_fw_nat64_15.h (new) | 212 + .../include15/netinet6/ip_fw_nptv6_15.h (new) | 52 + sbin/ipfw15/ip_fw15.h (new) | 1172 ++++ sbin/ipfw15/ipfw.8 (new) | 5094 ++++++++++++++++ sbin/ipfw15/ipfw2.c (new) | 6129 ++++++++++++++++++++ sbin/ipfw15/ipfw2.h (new) | 470 ++ sbin/ipfw15/ipv6.c (new) | 519 ++ sbin/ipfw15/main.c (new) | 716 +++ sbin/ipfw15/nat.c (new) | 1196 ++++ sbin/ipfw15/nat64clat.c (new) | 536 ++ sbin/ipfw15/nat64lsn.c (new) | 901 +++ sbin/ipfw15/nat64stl.c (new) | 552 ++ sbin/ipfw15/nptv6.c (new) | 452 ++ sbin/ipfw15/tables.c (new) | 2096 +++++++ 23 files changed, 24049 insertions(+)
would it also be possible to add this to 13? I can imagine many people upgrading straight from 13 to 15 as that will be EoL in April.
> would it also be possible to add this to 13? We can commit it to stable/13 > I can imagine many people upgrading straight from 13 to 15 Do you expect them to upgrade via sources? I doubt there will be any patch releases for RELENG-13.
A commit in branch releng/14.4 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=969e2b406835839be6188f9dc3b1600b6ca21ed8 commit 969e2b406835839be6188f9dc3b1600b6ca21ed8 Author: Boris Lytochkin <lytboris@gmail.com> AuthorDate: 2026-02-10 11:50:20 +0000 Commit: Colin Percival <cperciva@FreeBSD.org> CommitDate: 2026-02-12 19:28:43 +0000 ipfw: add ability to run ipfw(8) binary with 15.0+ kernel module After D46183 the KBI was changed and this made the upgrade procedure to 15.0+ version a bit difficult, because the old binary can not load firewall rules when the new kernel is loaded. This commit adds the sbin/ipfw15 binary that uses new KBI, and then original sbin/ipfw can detect new KBI and run the new binary instead. Approved by: re (cperciva) PR: 291562 Reviewed by: jhb, glebius Fixes: 4a77657cbc01 MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D54763 (cherry picked from commit 704ec5e68c44f08d83f3b0daa315c6143338863f) sbin/Makefile | 1 + sbin/ipfw/main.c | 26 + sbin/ipfw15/Makefile (new) | 23 + sbin/ipfw15/Makefile.depend (new) | 19 + sbin/ipfw15/altq.c (new) | 152 + sbin/ipfw15/dummynet.c (new) | 2016 +++++++ sbin/ipfw15/include15/alias15.h (new) | 259 + .../ipfw15/include15/netinet/ip_dummynet15.h (new) | 284 + sbin/ipfw15/include15/netinet/ip_fw15.h (new) | 1172 ++++ .../include15/netinet6/ip_fw_nat64_15.h (new) | 212 + .../include15/netinet6/ip_fw_nptv6_15.h (new) | 52 + sbin/ipfw15/ip_fw15.h (new) | 1172 ++++ sbin/ipfw15/ipfw.8 (new) | 5094 ++++++++++++++++ sbin/ipfw15/ipfw2.c (new) | 6129 ++++++++++++++++++++ sbin/ipfw15/ipfw2.h (new) | 470 ++ sbin/ipfw15/ipv6.c (new) | 519 ++ sbin/ipfw15/main.c (new) | 716 +++ sbin/ipfw15/nat.c (new) | 1196 ++++ sbin/ipfw15/nat64clat.c (new) | 536 ++ sbin/ipfw15/nat64lsn.c (new) | 901 +++ sbin/ipfw15/nat64stl.c (new) | 552 ++ sbin/ipfw15/nptv6.c (new) | 452 ++ sbin/ipfw15/tables.c (new) | 2096 +++++++ 23 files changed, 24049 insertions(+)
(In reply to lytboris from comment #13) I would expect them to upgrade via freebsd-update. We normally skip a major version, so we run multiple machines on 13.x and multiple machines on 14. Most of the 14.x boxes were upgraded from 12.x and we plan to upgrade the 13.x machines to 15.x I would guess that many other people do something similar.
(In reply to FiLiS from comment #15) That would require to: 1) Release a patch revision for 13.5, say 13.5-X 2) Upgrade all your installations to 13.5-X 3) Continue upgrade to 15.0 While this plan is 100% OK for me, I have no idea how to spawn a 13.5-X patch release. Can you be my guide for this?
(In reply to lytboris from comment #16) I think if you develop a patch and merge it stable/13, then an EN for 13.5 is doable. To do an EN, you would need to fill out the EN template from https://www.freebsd.org/security/errata-template.txt and send that along with the hash of the commit to stable/13 to secteam@ to request that they include it as an EN for 13.5 in the next batch of advisories for 13.5. secteam@ can refine the template, they mostly just want a good starting point that describes the problem, other workarounds if you don't apply the patch, etc.
Beside critical breaking of upgrade process it also, will break any vnet=1 jails that are using ipfw, and I am pretty sure that there are quite a lot of jails frozen on historical versions of FreeBSD userland which are still in use. And this is the real pity that due to the change all of them broken. there be a lot of reason to justify existance of historical jails on non-recent versions of FreeBSD, which are safily executed before, and now some someone should dive into and inject here fixed binary of ipfw, which is not a trivial task for historical versions. what if there are many such jails? Is there are a way to return back backward compatibility on ipfw ABI ?
Even updated 14.4 jail with vnet+ipfw - does not work: # cbsd jrestart ns3 Stopping jail: ns3, parallel timeout=5 ... add net ::0.0.0.0: gateway ::1 ipfw: setsockopt(IP_FW_XDEL): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument ipfw: getsockopt(IP_FW_XADD): Invalid argument Firewall rules loaded. ``` in jail: ``` ns3# freebsd-version 14.4-RELEASE-p2 ns3# uname -UK 1404000 1404000 ns3# strings /sbin/ipfw | grep ipfw15 /sbin/ipfw15 ns3# freebsd-update IDS | fgrep -v /etc/ | fgrep -v /root/ src component not installed, skipped Looking up update.FreeBSD.org mirrors... 3 mirrors found. Fetching metadata signature for 14.4-RELEASE from update2.freebsd.org... done. Fetching metadata index... done. Inspecting system... done. ns3# ``` in host: ``` # uname -UK 1500068 1500068 # freebsd-version 15.0-RELEASE-p6 ``` ```
more information: explicit set of OSVERSION to 1500025 fixes fallback ``` ns3# env OSVERSION=1500025 ipfw show WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15. 65535 0 0 count ip from any to any not // orphaned dynamic states counter 65535 6117 6054521 allow ip from any to any ```
(In reply to vova from comment #20) I ran into that too. I think it's due to defining jail.osreldate. I don't use cbsd but it does set that too. The host's /rescue/ipfw works from inside the jails too if upgrading to 14.4/stable is impractical.
(In reply to Bryan Drewery from comment #21) my suggestion for solving that - to introduce compatibility layer it appeared not that "big" - see https://reviews.freebsd.org/D56660 although it introduce some "more" kernel code, but hopfuly it may be kept in module or under ifdef