Bug 238883 - Fatal double fault using unionfs to build www/node port (repeatable)
Summary: Fatal double fault using unionfs to build www/node port (repeatable)
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
Keywords: panic
Depends on:
Reported: 2019-06-29 20:03 UTC by chadf
Modified: 2021-12-14 21:27 UTC (History)
2 users (show)

See Also:

/var/crash/core.txt.4 file (95.27 KB, text/plain)
2019-06-29 20:03 UTC, chadf
no flags Details
/var/crash/core.txt.5 file (98.52 KB, text/plain)
2019-06-29 23:35 UTC, chadf
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description chadf 2019-06-29 20:03:09 UTC
Created attachment 205420 [details]
/var/crash/core.txt.4 file

Repeatably able to panic kernel while building www/node ports while using nullfs and unionfs layered mounts for /usr/ports directory.

To trigger, setup mounts as:

/export/ports on /usr/ports (nullfs, local, read-only)
<above>:/usr/src/local.ports on /usr/ports (unionfs, local)

The /export/ports directory contains a master copy of the ports tree (ufs or zfs doesn't matter - haven't tried nfs).

Then build www/node port:

root@krash:~ # cd /usr/ports/www/node
root@krash:/usr/ports/www/node # make
===>  Building for node-12.4.0

Wait for panic (may take awhile into build). After panic/reboot, restarting build panics again, almost immediately.

I was able to recreate crash in a fresh virtualbox install. The .vdi file for that VM (in a quick to re-panic state) is available upon request (7.3GB uncompressed).

/var/crash/core.txt file attached. vmcore file available (12MB compressed, unattached due to size).
Comment 1 chadf 2019-06-29 23:32:58 UTC
Replaced nullfs with symlink and still panics. Removing nullfs
Comment 2 chadf 2019-06-29 23:34:11 UTC
Replaced nullfs with symlink and still panics. Removed nullfs reference from summary line and adding core.txt.5 attachment.
Comment 3 chadf 2019-06-29 23:35:09 UTC
Created attachment 205426 [details]
/var/crash/core.txt.5 file
Comment 4 Mark Linimon freebsd_committer freebsd_triage 2019-06-30 01:09:01 UTC
The panic and any associated backtrace will be of more use than the core file.
Comment 5 commit-hook freebsd_committer 2021-06-29 13:02:04 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=372691a7ae1878ecdf707195b0854750f07bf44e

commit 372691a7ae1878ecdf707195b0854750f07bf44e
Author:     Jason A. Harmening <jah@FreeBSD.org>
AuthorDate: 2021-06-12 19:45:18 +0000
Commit:     Jason A. Harmening <jah@FreeBSD.org>
CommitDate: 2021-06-29 13:02:01 +0000

    unionfs: release parent vnodes in deferred context

    Each unionfs node holds a reference to its parent directory vnode.
    A single open file reference can therefore end up keeping an
    arbitrarily deep vnode hierarchy in place.  When that reference is
    released, the resulting VOP_RECLAIM call chain can then exhaust the
    kernel stack.

    This is easily reproducible by running the unionfs.sh stress2 test.
    Fix it by deferring recursive unionfs vnode release to taskqueue

    PR: 238883
    Reviewed By:    kib (earlier version), markj
    Differential Revision: https://reviews.freebsd.org/D30748

 sys/fs/unionfs/union.h      |  6 ++++-
 sys/fs/unionfs/union_subr.c | 55 ++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 55 insertions(+), 6 deletions(-)
Comment 6 Ed Maste freebsd_committer 2021-12-14 21:26:54 UTC
Is this issue now resolved (modulo any MFCs)?