Bug 236466 - [sendfile]: panic when VOP_READ returns an error
Summary: [sendfile]: panic when VOP_READ returns an error
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: Alan Somers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-03-11 15:57 UTC by Alan Somers
Modified: 2019-08-07 00:40 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alan Somers freebsd_committer 2019-03-11 15:57:32 UTC
sendfile will panic if it can't read the pages it wants to send.  This panic is easily reproducible using FUSE.  I've seen the error when the fuse filesystem returns either EOPNOTSUP or EIO for its read, but presumably it can happen with any sort of read error.

FreeBSD fbsd12.lauralan.noip.me 13.0-CURRENT FreeBSD 13.0-CURRENT #3 r344740:344786M: Tue Mar  5 10:37:58 MST 2019     somers@fbsd12.lauralan.noip.me:/usr/obj/usr/home/somers/freebsd/base/projects/fuse2/amd64.amd64/sys/GENERIC  amd64

panic: sendfile_swapin: pager fail obj 0xfffff800343f6300 page 0xfffff800b9a93430

cpuid = 3
time = 1552317254
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0031fc15d0
vpanic() at vpanic+0x1b4/frame 0xfffffe0031fc1630
panic() at panic+0x43/frame 0xfffffe0031fc1690
vn_sendfile() at vn_sendfile+0x1619/frame 0xfffffe0031fc1900
sendfile() at sendfile+0x12b/frame 0xfffffe0031fc1990
amd64_syscall() at amd64_syscall+0x276/frame 0xfffffe0031fc1ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0031fc1ab0
--- syscall (393, FreeBSD ELF64, sys_sendfile), rip = 0x8005077da, rsp = 0x7fffffffe6e8, rbp = 0x7fffffffe840 ---
KDB: enter: panic

I'll commit a reproduction case shortly.
Comment 1 Conrad Meyer freebsd_committer 2019-03-11 18:08:09 UTC
Interestingly, only one other location in tree asserts VM_PAGER_OK, and that's when we're swapping a *kernel* stack back into memory (vm_thread_swapin).  Panic probably makes sense for that case — and putting swap on a network filesystem (or FUSE) probably doesn't make sense anyway.  If your swap disk EIOs the kernel stack, at least that thread is hosed.  You might be able to zombie it and limp the system along (Linux approach) but maybe not, either.

Obviously, sendfile(2) is not an appropriate place to assert on the success of vm_pager_get_pages.
Comment 2 commit-hook freebsd_committer 2019-03-11 18:29:04 UTC
A commit references this bug:

Author: asomers
Date: Mon Mar 11 18:28:22 UTC 2019
New revision: 345017
URL: https://svnweb.freebsd.org/changeset/base/345017

Log:
  fuse(4): add tests for FUSE_READ

  PR:		236379
  PR:		236466
  PR:		236472
  Sponsored by:	The FreeBSD Foundation

Changes:
  projects/fuse2/tests/sys/fs/fuse/Makefile
  projects/fuse2/tests/sys/fs/fuse/mockfs.cc
  projects/fuse2/tests/sys/fs/fuse/mockfs.hh
  projects/fuse2/tests/sys/fs/fuse/read.cc
  projects/fuse2/tests/sys/fs/fuse/utils.cc
  projects/fuse2/tests/sys/fs/fuse/utils.hh
  projects/fuse2/tests/sys/fs/fuse/write.cc
Comment 3 Alan Somers freebsd_committer 2019-03-11 18:31:30 UTC
Steps to reproduce on HEAD:
1) sudo pkg install googletest googlemock
2) svn checkout https://svn.freebsd.org/base/projects/fuse2
3) cd /path/to/base/projects/fuse2/tests/sys/fs/fuse
4) sudo mkdir -p /usr/tests/sys/fs/fuse
5) make -j8 && sudo make install
6) cd /usr/tests/sys/fs/fuse
7) ./read --gtest_also_run_disabled_tests --gtest_filter=Read.DISABLED_sendfile_eio
Comment 4 Conrad Meyer freebsd_committer 2019-03-11 19:21:17 UTC
You could probably also reproduce fairly easily with a small test case involving md(4) and gnop(4).
Comment 5 commit-hook freebsd_committer 2019-07-19 18:03:39 UTC
A commit references this bug:

Author: asomers
Date: Fri Jul 19 18:03:30 UTC 2019
New revision: 350144
URL: https://svnweb.freebsd.org/changeset/base/350144

Log:
  sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error

  PR:		236466
  Sponsored by:	The FreeBSD Foundation

Changes:
  projects/fuse2/sys/kern/kern_sendfile.c
  projects/fuse2/tests/sys/fs/fusefs/read.cc
Comment 6 commit-hook freebsd_committer 2019-07-29 20:51:14 UTC
A commit references this bug:

Author: asomers
Date: Mon Jul 29 20:50:26 UTC 2019
New revision: 350426
URL: https://svnweb.freebsd.org/changeset/base/350426

Log:
  sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error

  This is a partial merge of 350144 from projects/fuse2

  PR:		236466
  Reviewed by:	markj
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D21095

Changes:
_U  head/
  head/sys/kern/kern_sendfile.c
Comment 7 commit-hook freebsd_committer 2019-08-07 00:40:08 UTC
A commit references this bug:

Author: asomers
Date: Wed Aug  7 00:38:30 UTC 2019
New revision: 350665
URL: https://svnweb.freebsd.org/changeset/base/350665

Log:
  fusefs: merge from projects/fuse2

  This commit imports the new fusefs driver. It raises the protocol level
  from 7.8 to 7.23, fixes many bugs, adds a test suite for the driver, and
  adds many new features. New features include:

  * Optional kernel-side permissions checks (-o default_permissions)
  * Implement VOP_MKNOD, VOP_BMAP, and VOP_ADVLOCK
  * Allow interrupting FUSE operations
  * Support named pipes and unix-domain sockets in fusefs file systems
  * Forward UTIME_NOW during utimensat(2) to the daemon
  * kqueue support for /dev/fuse
  * Allow updating mounts with "mount -u"
  * Allow exporting fusefs file systems over NFS
  * Server-initiated invalidation of the name cache or data cache
  * Respect RLIMIT_FSIZE
  * Try to support servers as old as protocol 7.4

  Performance enhancements include:

  * Implement FUSE's FOPEN_KEEP_CACHE and FUSE_ASYNC_READ flags
  * Cache file attributes
  * Cache lookup entries, both positive and negative
  * Server-selectable cache modes: writethrough, writeback, or uncached
  * Write clustering
  * Readahead
  * Use counter(9) for statistical reporting

  PR:		199934 216391 233783 234581 235773 235774 235775
  PR:		236226 236231 236236 236291 236329 236381 236405
  PR:		236327 236466 236472 236473 236474 236530 236557
  PR:		236560 236844 237052 237181 237588 238565
  Reviewed by:	bcr (man pages)
  Reviewed by:	cem, ngie, rpokala, glebius, kib, bde, emaste (post-commit
  		review on project branch)
  MFC after:	3 weeks
  Relnotes:	yes
  Sponsored by:	The FreeBSD Foundation
  Pull Request:	https://reviews.freebsd.org/D21110

Changes:
_U  head/
  head/MAINTAINERS
  head/UPDATING
  head/etc/mtree/BSD.tests.dist
  head/sbin/mount_fusefs/mount_fusefs.8
  head/sbin/mount_fusefs/mount_fusefs.c
  head/share/man/man5/fusefs.5
  head/sys/fs/fuse/fuse.h
  head/sys/fs/fuse/fuse_device.c
  head/sys/fs/fuse/fuse_file.c
  head/sys/fs/fuse/fuse_file.h
  head/sys/fs/fuse/fuse_internal.c
  head/sys/fs/fuse/fuse_internal.h
  head/sys/fs/fuse/fuse_io.c
  head/sys/fs/fuse/fuse_io.h
  head/sys/fs/fuse/fuse_ipc.c
  head/sys/fs/fuse/fuse_ipc.h
  head/sys/fs/fuse/fuse_kernel.h
  head/sys/fs/fuse/fuse_main.c
  head/sys/fs/fuse/fuse_node.c
  head/sys/fs/fuse/fuse_node.h
  head/sys/fs/fuse/fuse_param.h
  head/sys/fs/fuse/fuse_vfsops.c
  head/sys/fs/fuse/fuse_vnops.c
  head/sys/sys/param.h
  head/tests/sys/fs/Makefile
  head/tests/sys/fs/fusefs/