Bug 240442 - Upgrade from 11Stable to 12Stable - dying jail requires host reboot
Summary: Upgrade from 11Stable to 12Stable - dying jail requires host reboot
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.0-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-jail (Nobody)
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2019-09-09 12:25 UTC by Dewayne
Modified: 2022-02-05 04:28 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dewayne 2019-09-09 12:25:54 UTC
Environment: amd64 on Xeon board. Long running 11.3Stable updated to 12Stable

~# uname -a
FreeBSD hathor 12.0-STABLE FreeBSD 12.0-STABLE #0 r351834M: Sat Sep  7 17:20:37 AEST 2019     root@hathor:/usr/obj/prod/120101/D/K8/smallblocks/src/amd64.amd64/sys/hqdev-amd64-smp-vga  amd64

Prior to the upgrade to 12Stable we built and tested 11.3Stable the day prior, everything was ok; only a few additions to the kernel config were applied (eg MAC_NTPD), otherwise unchanged.

Summary:
System is regularly updated from svnlite source (every 4-6 weeks) throughout the 11. series.  The system was updated to 12Stable over the weekend.

The system comprises multiple jailed environments including two i386 and a couple of amd64 jails.
The i386 jails have a common base. Everything was working, until, the upgrade:
Start jails - ok.  
Stop jails - ok.
Restart jails - not ok.  Further testing all (i386 and amd64)'s are caught in a dying state.  Host reboot required.

Details
The test machine is quite dated and we used
exec.poststop="umount -aF /etc/fstab.$name; umount $path/dev";
which, subsequent to reading Jamie's r349180 (that test is later) the problem remains

We'll stop and start the jail, the jail.conf was slightly modified between tests.

# jail -vr b3
b3: run command in jail: /bin/sh /etc/rc.shutdown
b3: sent SIGTERM to: 1026
b3: removed
b3: run command: /bin/sh -c umount -aF /etc/fstab.b3; umount /usr/jails/b3/dev
jail: b3: mount.devfs: /usr/jails/b3/dev: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/pd/pd1: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/var/squid: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/var/mailctl: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/var/maildir: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/fifo: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/tmp: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/usr/ports: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/distfiles: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/packages: not a mount point
jail: b3: mount.fstab: /usr/jails/b3/bj: not a mount point

Rerun of jail removal after deleting the following historical workaround from jail.conf
exec.poststop="umount -aF /etc/fstab.$name; umount $path/dev";

~# jail -vr b3
b3: run command in jail: /bin/sh /etc/rc.shutdown
b3: sent SIGTERM to: 26278
b3: removed
b3: run command: /sbin/umount /usr/jails/b3/dev
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/pd/pd1
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/var/squid
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/var/mailctl
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/var/maildir
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/fifo
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/tmp
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/usr/ports
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/distfiles
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/packages
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/bj

Now to (attempt to) restart the jail
# jail -vc b3
b3: run command: /sbin/mount -t nullfs -o rw /usr/jails/bj1 /usr/jails/b3/bj
b3: run command: /sbin/mount -t nullfs -o ro /pd9/packages /usr/jails/b3/packages
b3: run command: /sbin/mount -t nullfs -o ro /pd9/usr/distfiles /usr/jails/b3/distfiles
b3: run command: /sbin/mount -t nullfs -o ro /usr/ports /usr/jails/b3/usr/ports
b3: run command: /sbin/mount -t nullfs -o rw,noatime /tmp/b3 /usr/jails/b3/tmp
b3: run command: /sbin/mount -t nullfs -o rw,noatime /m/b3/fifo /usr/jails/b3/fifo
b3: run command: /sbin/mount -t nullfs -o rw,noatime /pd6/mail/maildir /usr/jails/b3/var/maildir
b3: run command: /sbin/mount -t nullfs -o rw /pd6/mail/mailctl /usr/jails/b3/var/mailctl
b3: run command: /sbin/mount -t nullfs -o rw,noatime /pd6/squid /usr/jails/b3/var/squid
b3: run command: /sbin/mount -t nullfs -o rw,noatime /pd1 /usr/jails/b3/pd/pd1
b3: run command: /sbin/mount -t devfs -oruleset=6 . /usr/jails/b3/dev
b3: jail_set(JAIL_CREATE) name=b3 host.hostname=b3.hs path=/usr/jails/b3 children.max=0 securelevel=3 enforce_statfs=1 jid=3 persist ip4.addr=10.0.7.91,10.0.5.91,127.0.5.91 devfs_ruleset=6 sysvmsg=new sysvsem=new sysvshm=new allow.raw_sockets allow.chflags host.hostuuid=00000000-0000-0000-0003-000000000003 host.hostid=3: File exists
jail: b3: jail 3 already exists
b3: run command: /sbin/umount /usr/jails/b3/dev
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/pd/pd1
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/var/squid
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/var/mailctl
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/var/maildir
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/fifo
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/tmp
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/usr/ports
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/distfiles
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/packages
b3: run command: /sbin/umount -ft nullfs /usr/jails/b3/bj
~#

Can anyone reproduce by upgrading from 11Stable to 12Stable from source?
Comment 1 beebeetles 2022-02-05 04:28:41 UTC
I'm experiencing something similar on 13.0-RELEASE: My jails are stuck at the dying stage, and will always show up in `jls -d`. Does not require rebooting in my case though.