Bug 253593

Summary: Process hangs if nullfs mounted cwd moved out of chroot
Product: Base System Reporter: Gregor Koscak <elogin41>
Component: kernAssignee: Konstantin Belousov <kib>
Status: Closed FIXED    
Severity: Affects Some People CC: chris, kib, markj, pho, sulli00777
Priority: ---    
Version: 12.2-STABLE   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
screenshot of the VM console
none
repros bug 253593 none

Description Gregor Koscak 2021-02-17 18:24:56 UTC
Overview:
Moving a current working directory of a chrooted/jailed process, which is accessed via nullfs mount, to the outside of chroot/jail will hang the process in R state with CPU maxed out. The process cannot be killed. Whether mount is rw or ro makes no difference.

Steps to reproduce:
Setup directories
/test/mounted/subdir
/test/outside
/test/jail  <--- minimum chroot environment, or static shell

With nullfs mount:
host# mount
/test/inside on /test/jail/mnt (nullfs, local, soft-updates)

Procedure:
[terminal1]
host# chroot /test/jail /bin/sh
jail# cd /mnt/subdir
[terminal2]
host# mv /test/mounted/subdir /test/outside
! at this point we have:
! /test/mounted
! /test/outside/subdir
[terminal1]
jail# pwd
/mnt/subdir
jail# cd .

Actual result:
Shell process hangs w/ CPU utilization maxed and cannot be killed (R state). Nullfs mount cannot be unmounted, results in D state.
Return to normal operation requires reboot.

Expected result:
Graceful failure.

Tested on:
12.2-RELEASE-p1, 12.2-RELEASE-p3, physical and virtual (virtualbox).
14.0-CURRENT will panic the kernel.
Comment 1 Konstantin Belousov freebsd_committer freebsd_triage 2021-02-17 19:08:12 UTC
Show the panic, at least.
Comment 2 Gregor Koscak 2021-02-17 19:23:19 UTC
Created attachment 222534 [details]
screenshot of the VM console
Comment 3 Konstantin Belousov freebsd_committer freebsd_triage 2021-02-26 15:42:51 UTC
I suspect there are too many typos in what you wrote in the bug description.
Also, from the panic message, it should matter that some thing in your setup
is the mount point.

Provide the self-contained script (may be two scripts) using tmpfs/nullfs and perhaps rescue/sh
for simplicity, that allows others to reproduce.
Comment 4 Gregor Koscak 2021-02-26 16:37:59 UTC
(In reply to Konstantin Belousov from comment #3)
Ah yes, there's one mistake in copypaste.
This:
/test/inside on /test/jail/mnt (nullfs, local, soft-updates)

Shoud read:
/test/mounted on /test/jail/mnt (nullfs, local, soft-updates)

I hope that clears it up.
Comment 5 Konstantin Belousov freebsd_committer freebsd_triage 2021-02-26 17:04:40 UTC
No, it does not.  Please provide scripts.
Comment 6 Patrick Sullivan 2021-03-31 16:26:49 UTC
Created attachment 223733 [details]
repros bug 253593

Pho had recommended that I use this bug to get started assisting with FreeBSD testing so I wrote a script that repros the panic on CURRENT. It takes an arg pointing to the locally installed source to build/place the jail and requires tmux for the jail console. It hits the same panic that Gregor had posted a picture of earlier.
Comment 7 Peter Holm freebsd_committer freebsd_triage 2021-03-31 23:59:16 UTC
Yes, I was able to reproduce it:
https://people.freebsd.org/~pho/stress/log/log0087.txt
Comment 8 Patrick Sullivan 2021-04-01 18:57:02 UTC
spoke with Peter earlier and realized that the script I posted was way too heavy. This is enough to cause the panic here:
mkdir -p /test/jail/mnt /test/mounted/subdir && cp /rescue/(sh|sleep) /test/jail && mount_nullfs /test/mounted /test/jail/mnt &&  chroot /test/jail /sh -c ' cd /mnt/subdir; /sleep 10; cd .' & sleep 1; mv /test/mounted/subdir /test/outside ; sleep 10; umount /test/jail/mnt ; rm -rf /test

It's a one liner you can copy and paste in to repro the issue.
Comment 9 commit-hook freebsd_committer freebsd_triage 2021-04-02 12:41:31 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=76b1b5ce6d81f66b09be8a20aecd064b65fd6b50

commit 76b1b5ce6d81f66b09be8a20aecd064b65fd6b50
Author:     Konstantin Belousov <kib@FreeBSD.org>
AuthorDate: 2021-04-01 17:42:14 +0000
Commit:     Konstantin Belousov <kib@FreeBSD.org>
CommitDate: 2021-04-02 12:40:25 +0000

    nullfs: protect against user creating inconsistent state

    The VFS conventions is that VOP_LOOKUP() methods do not need to handle
    ISDOTDOT lookups for VV_ROOT vnodes (since they cannot, after all).  Nullfs
    bypasses VOP_LOOKUP() to lower filesystem, and there, due to user actions,
    it is possible to get into situation where
    - upper vnode does not have VV_ROOT set
    - lower vnode is root
    - ISDOTDOT is requested
    User just needs to nullfs-mount non-root of some filesystem, and then move
    some directory under mount, out of mount, using lower filesystem.

    In this case, nullfs cannot do much, but we still should and can ensure
    internal kernel structures are consistent.  Avoid ISDOTDOT lookup forwarding
    when VV_ROOT is set on lower dvp, return somewhat arbitrary ENOENT.

    PR:     253593
    Reported by:    Gregor Koscak <elogin41@gmail.com>
    Test by:        Patrick Sullivan <sulli00777@gmail.com>
    Tested by:      pho
    Sponsored by:   The FreeBSD Foundation
    MFC after:      1 week

 sys/fs/nullfs/null_vnops.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
Comment 10 Patrick Sullivan 2021-04-02 18:52:30 UTC
[eshnunna]|<->|[~]Z#uname -a
FreeBSD eshnunna 14.0-CURRENT FreeBSD 14.0-CURRENT #2 pat-n245800-7e199c730ccd: Fri Apr  2 02:28:32 PDT 2021     root@eshnunna:/usr/obj/root/stuff/repos/freebsd-src/amd64.amd64/sys/GENERIC  amd64
[eshnunna]|<->|[~]Z#mkdir -p /test/jail/mnt /test/mounted/subdir && cp /rescue/(sh|sleep) /test/jail && mount_nullfs /test/mounted /test/jail/mnt &&  chroot /test/jail /sh -c ' cd /mnt/subdir; /sleep 10; cd .' & sleep 1; mv /test/mounted/subdir /test/outside ; sleep 10; umount /test/jail/mnt ; rm -rf /test
[1] 971
cd: warning: failed to get name of current directory
[1]  + done       chroot /test/jail /sh -c ' cd /mnt/subdir; /sleep 10; cd .'

Looks good to me. We don't see a panic anymore.
Comment 11 commit-hook freebsd_committer freebsd_triage 2021-04-09 04:57:55 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ee389afecf85571936957b183921165893c680a6

commit ee389afecf85571936957b183921165893c680a6
Author:     Konstantin Belousov <kib@FreeBSD.org>
AuthorDate: 2021-04-01 17:42:14 +0000
Commit:     Konstantin Belousov <kib@FreeBSD.org>
CommitDate: 2021-04-09 04:52:05 +0000

    nullfs: protect against user creating inconsistent state

    PR:     253593

    (cherry picked from commit 76b1b5ce6d81f66b09be8a20aecd064b65fd6b50)

 sys/fs/nullfs/null_vnops.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
Comment 12 commit-hook freebsd_committer freebsd_triage 2021-04-09 05:08:58 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ecc571a858586e7b62cae3794c0f6250a91a483b

commit ecc571a858586e7b62cae3794c0f6250a91a483b
Author:     Konstantin Belousov <kib@FreeBSD.org>
AuthorDate: 2021-04-01 17:42:14 +0000
Commit:     Konstantin Belousov <kib@FreeBSD.org>
CommitDate: 2021-04-09 04:57:37 +0000

    nullfs: protect against user creating inconsistent state

    PR:     253593

    (cherry picked from commit 76b1b5ce6d81f66b09be8a20aecd064b65fd6b50)

 sys/fs/nullfs/null_vnops.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)