There is a race condition that must be worked around by everyone using kqueue(s) to monitor files for changes. The race condition is. 1. open file and get fd 2. create kqueue to listen for events like (rename, delete, etc.) 3. go to sleep and wait for changes The problem is that if the file is deleted or renamed in between the call to open and before the kqueue is setup to monitor the file, you will not receive any notification In order to work around this people do: 1. create/open file 2. check stat and path of file 3. create kqueue to listen for events 4. check stat and path of file again and make sure nothing changed 5. go to sleep and wait for changes. It would be useful if there was a kqueue API for opening a file that would eliminate the race condition between opening a file and setting up monitoring an API that combines the parameters of open and kqueue so that the file descriptor returned and the kqueue monitoring setup will be atomic. Maybe a 'kopen' api or something like that. This would raise awareness of the race condition and provide a standard way of navigating around it that can be stabalized and utilized throughout code to provide support for a much more robust platform.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f1f230439fa48581f40a57f095627f667a9713c3 commit f1f230439fa48581f40a57f095627f667a9713c3 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2025-07-03 20:07:45 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2025-07-04 14:42:33 +0000 vfs: Initial revision of inotify Add an implementation of inotify_init(), inotify_add_watch(), inotify_rm_watch(), source-compatible with Linux. This provides functionality similar to kevent(2)'s EVFILT_VNODE, i.e., it lets applications monitor filesystem files for accesses. Compared to inotify, however, EVFILT_VNODE has the limitation of requiring the application to open the file to be monitored. This means that activity on a newly created file cannot be monitored reliably, and that a file descriptor per file in the hierarchy is required. inotify on the other hand allows a directory and its entries to be monitored at once. It introduces a new file descriptor type to which "watches" can be attached; a watch is a pseudo-file descriptor associated with a file or directory and a set of events to watch for. When a watched vnode is accessed, a description of the event is queued to the inotify descriptor, readable with read(2). Events for files in a watched directory include the file name. A watched vnode has its usecount bumped, so name cache entries originating from a watched directory are not evicted. Name cache entries are used to populate inotify events for files with a link in a watched directory. In particular, if a file is accessed with, say, read(2), an IN_ACCESS event will be generated for any watched hard link of the file. The inotify_add_watch_at() variant is included so that this functionality is available in capability mode; plain inotify_add_watch() is disallowed in capability mode. When a file in a nullfs mount is watched, the watch is attached to the lower vnode, such that accesses via either layer generate inotify events. Many thanks to Gleb Popov for testing this patch and finding lots of bugs. PR: 258010, 215011 Reviewed by: kib Tested by: arrowd MFC after: 3 months Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D50315 share/man/man4/rights.4 | 10 +- sys/bsm/audit_kevents.h | 1 + sys/conf/files | 1 + sys/fs/nullfs/null_subr.c | 4 + sys/fs/nullfs/null_vnops.c | 29 +- sys/kern/kern_resource.c | 21 + sys/kern/subr_capability.c | 4 + sys/kern/sys_generic.c | 35 +- sys/kern/syscalls.master | 15 + sys/kern/vfs_cache.c | 59 +++ sys/kern/vfs_default.c | 17 + sys/kern/vfs_inotify.c (new) | 1008 ++++++++++++++++++++++++++++++++++++++++++ sys/kern/vfs_subr.c | 7 +- sys/kern/vfs_vnops.c | 3 +- sys/kern/vnode_if.src | 21 + sys/sys/caprights.h | 2 + sys/sys/capsicum.h | 8 +- sys/sys/exterr_cat.h | 1 + sys/sys/file.h | 1 + sys/sys/inotify.h (new) | 146 ++++++ sys/sys/resourcevar.h | 4 + sys/sys/specialfd.h | 5 + sys/sys/user.h | 5 + sys/sys/vnode.h | 12 +- sys/tools/vnode_if.awk | 1 + 25 files changed, 1405 insertions(+), 15 deletions(-)
After the above commit (everything rebuilt), I'm getting panics immediately upon starting both xfce and cinnamon: Sun Jul 13 08:30:38 EDT 2025 FreeBSD lehman 15.0-CURRENT FreeBSD 15.0-CURRENT #2 main-n278710-f5dba673345e: Sat Jul 12 19:18:59 EDT 2025 root@lehman:/usr/obj/usr/src/amd64.amd64/sys/ODYSSEY amd64 panic: Assertion *eofflag != 0 failed at /usr/src/sys/kern/vfs_vnops.c:4058 Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Reading symbols from /boot/kernel/zfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug... Reading symbols from /boot/modules/i915kms.ko... (No debugging symbols found in /boot/modules/i915kms.ko) Reading symbols from /boot/modules/drm.ko... (No debugging symbols found in /boot/modules/drm.ko) Reading symbols from /boot/kernel/iic.ko... Reading symbols from /usr/lib/debug//boot/kernel/iic.ko.debug... Reading symbols from /boot/kernel/linuxkpi_video.ko... Reading symbols from /usr/lib/debug//boot/kernel/linuxkpi_video.ko.debug... Reading symbols from /boot/modules/dmabuf.ko... (No debugging symbols found in /boot/modules/dmabuf.ko) Reading symbols from /boot/kernel/lindebugfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/lindebugfs.ko.debug... Reading symbols from /boot/modules/ttm.ko... (No debugging symbols found in /boot/modules/ttm.ko) Reading symbols from /boot/kernel/if_iwlwifi.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_iwlwifi.ko.debug... Reading symbols from /boot/kernel/if_iwm.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_iwm.ko.debug... Reading symbols from /boot/kernel/cuse.ko... Reading symbols from /usr/lib/debug//boot/kernel/cuse.ko.debug... Reading symbols from /boot/kernel/uplcom.ko... Reading symbols from /usr/lib/debug//boot/kernel/uplcom.ko.debug... Reading symbols from /boot/kernel/nlsysevent.ko... Reading symbols from /usr/lib/debug//boot/kernel/nlsysevent.ko.debug... __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 57 __asm("movq %%gs:%c1,%0" : "=r" (td) (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 td = <optimized out> #1 doadump (textdump=textdump@entry=1) at /usr/src/sys/kern/kern_shutdown.c:399 error = 0 coredump = <optimized out> #2 0xffffffff80b98983 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:519 once = 0 __pc = 0x0 #3 0xffffffff80b98ea7 in vpanic ( fmt=0xffffffff8131e855 "Assertion %s failed at %s:%d", ap=ap@entry=0xfffffe00c4797a50) at /usr/src/sys/kern/kern_shutdown.c:974 buf = "Assertion *eofflag != 0 failed at /usr/src/sys/kern/vfs_vnops.c:4058", '\000' <repeats 187 times> __pc = 0x0 __pc = 0x0 __pc = 0x0 other_cpus = {__bits = {13, 0 <repeats 15 times>}} td = 0xfffff800375c1780 bootopt = <unavailable> newpanic = <optimized out> #4 0xffffffff80b98cd3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:887 ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0xfffffe00c4797a80, reg_save_area = 0xfffffe00c4797a20}} #5 0xffffffff80cbaebc in vn_dir_next_dirent (vp=0xfffff80104d3ddc0, td=<optimized out>, dirbuf=0xfffffe00c45c9000 "\001", dirbuflen=<optimized out>, dpp=0xfffffe00c4797c58, len=0xfffffe00c4797c48, off=0xfffffe00c4797c40, eofflag=0xfffffe00c4797c60) at /usr/src/sys/kern/vfs_vnops.c:4058 uio = {uio_iov = 0xfffffe00c4797ae8, uio_iovcnt = 1, uio_offset = 128, uio_resid = 35840, uio_segflg = UIO_SYSSPACE, uio_rw = UIO_READ, uio_td = 0xfffff800375c1780} iov = {iov_base = 0xfffffe00c45c9000, iov_len = 35840} dp = 0x0 reclen = <optimized out> error = <optimized out> success = <optimized out> #6 0xffffffff80c89ba5 in vn_inotify_add_watch (vp=0xfffff80104d3ddc0, vp@entry=<error reading variable: value is not available>, sc=0xfffff80003115600, sc@entry=<error reading variable: value is not available>, mask=16789454, mask@entry=<error reading variable: value is not available>, wdp=0xfffffe00c4797dd4, wdp@entry=<error reading variable: value is not available>, td=0xfffff800375c1780, td@entry=<error reading variable: value is not available>) at /usr/src/sys/kern/vfs_inotify.c:747 nd = {ni_dirp = 0xfffffe00c45c9058 "efi", ni_segflg = UIO_SYSSPACE, ni_rightsneeded = 0xffffffff814f2010 <cap_no_rights>, ni_startdir = 0x0, ni_rootdir = 0xfffff8000938a898, ni_topdir = 0x0, ni_dirfd = -100, ni_lcf = 0, ni_filecaps = {fc_rights = { cr_rights = {0, 0}}, fc_ioctls = 0x0, fc_nioctls = -1, fc_fcntls = 0}, ni_vp = 0xfffff80104d3dc08, ni_dvp = 0xfffff80104d3ddc0, ni_resflags = 0, ni_debugflags = 7, ni_loopcnt = 0, ni_pathlen = 1, ni_next = 0xfffff80074c8a403 "", ni_cnd = {cn_flags = 8925478912, cn_cred = 0xfffff80003100a00, cn_nameiop = LOOKUP, cn_lkflags = 2097152, cn_pnbuf = 0xfffff80074c8a400 "efi", cn_nameptr = 0xfffff80074c8a400 "efi", cn_namelen = 3}, ni_cap_tracker = {tqh_first = 0x0, tqh_last = 0xfffffe00c4797c08}, ni_rbeneath_dpp = 0x0, ni_nctrack_mnt = 0x0, ni_dvp_seqc = 493, ni_vp_seqc = 0} dp = 0xfffffe00c45c9040 off = 128 len = 0 buflen = 35840 buf = 0xfffffe00c45c9000 "\001" eof = 0 error = 0 watch = <optimized out> watch1 = <optimized out> wd = <optimized out> #7 0xffffffff812321c9 in VOP_INOTIFY_ADD_WATCH_APV (vop=<optimized out>, a=a@entry=0xfffffe00c4797da0) at vnode_if.c:4410 rc = <optimized out> #8 0xffffffff80c8a50a in VOP_INOTIFY_ADD_WATCH (vp=0xfffff80104d3ddc0, vp@entry=<error reading variable: value is not available>, sc=0xfffff80003115600, sc@entry=<error reading variable: value is not available>, mask=16789454, mask@entry=<error reading variable: value is not available>, wdp=0xfffffe00c4797dd4, td=0xfffff800375c1780, td@entry=<error reading variable: value is not available>) at ./vnode_if.h:2552 a = {a_gen = { a_desc = 0xffffffff81b86ad0 <vop_inotify_add_watch_desc>}, a_vp = 0xfffff80104d3ddc0, a_sc = 0xfffff80003115600, a_mask = 16789454, a_wdp = 0xfffffe00c4797dd4, a_td = 0xfffff800375c1780} #9 kern_inotify_add_watch (fd=<unavailable>, fd@entry=<error reading variable: value is not available>, dfd=<unavailable>, dfd@entry=<error reading variable: value is not available>, path=<unavailable>, path@entry=<error reading variable: value is not available>, mask=16789454, mask@entry=<error reading variable: value is not available>, td=0xfffff800375c1780, td@entry=<error reading variable: value is not available>) at /usr/src/sys/kern/vfs_inotify.c:935 nd = { ni_dirp = 0xc45bf800090 <error: Cannot access memory at address 0xc45bf800090>, ni_segflg = UIO_USERSPACE, ni_rightsneeded = 0xffffffff814f2010 <cap_no_rights>, ni_startdir = 0x0, ni_rootdir = 0xfffff8000938a898, ni_topdir = 0x0, ni_dirfd = -100, ni_lcf = 0, ni_filecaps = {fc_rights = { cr_rights = {0, 0}}, fc_ioctls = 0x0, fc_nioctls = -1, fc_fcntls = 0}, ni_vp = 0xfffff80104d3ddc0, ni_dvp = 0xfffff80003179370, ni_resflags = 1, ni_debugflags = 3, ni_loopcnt = 0, ni_pathlen = 1, ni_next = 0xfffff80074c8a409 "\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336", <incomplete sequence \336>..., ni_cnd = {cn_flags = 8925741380, cn_cred = 0xfffff80003100a00, cn_nameiop = LOOKUP, cn_lkflags = 2097152, cn_pnbuf = 0x0, cn_nameptr = 0xfffff80074c8a406 "\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300"..., cn_namelen = 3}, ni_cap_tracker = {tqh_first = 0x0, tqh_last = 0xfffffe00c4797d70}, ni_rbeneath_dpp = 0x0, ni_nctrack_mnt = 0x0, ni_dvp_seqc = 0, ni_vp_seqc = 0} wd = 2 fp = 0xfffff8003757d230 vp = 0xfffff80104d3ddc0 error = <optimized out> sc = 0xfffff80003115600 count = <optimized out> #10 0xffffffff8116998e in syscallenter (td=0xfffff800375c1780) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:193 se = 0xffffffff81ac83f0 <sysent+18976> p = 0xfffffe00831515c0 sa = 0xfffff800375c1b98 error = <optimized out> sy_thr_static = true traced = <optimized out> _audit_entered = <optimized out> #11 amd64_syscall (td=0xfffff800375c1780, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1215 ksi = {ksi_link = {tqe_next = 0xfffff800375c1780, tqe_prev = 0xffffffff8316d400}, ksi_info = { si_signo = -2127967715, si_errno = -1, si_code = 928782208, si_pid = -2048, si_uid = 2199311360, si_status = -1, si_addr = 0xffffffff00000000, si_value = {sival_int = 0, sival_ptr = 0x60100000000, sigval_int = 0, sigval_ptr = 0x60100000000}, _reason = {_fault = { _trapno = 1537}, _timer = {_timerid = 1537, _overrun = 0}, _mesgq = {_mqd = 1537}, _poll = {_band = 1537}, _capsicum = { _syscall = 1537}, __spare__ = {__spare1__ = 1537, __spare2__ = {0, 0, -1082124768, 3141, 928782208, -2048, 1}}}}, ksi_flags = -998670704, ksi_sigq = 0x46} #12 <signal handler called> No locals. #13 0x000000082c4d29aa in ?? () No symbol table info available. Backtrace stopped: Cannot access memory at address 0x831746818 (kgdb) /usr/libexec/kgdb/acttrace.py:8: Error in sourced command file: Undefined command: "import". Try "help". (kgdb) Undefined command: "acttrace". Try "help". (kgdb) There is also a flood of "fstat: znode_t size mismatch, data could be wrong".
(In reply to Charlie Li from comment #2) I hadn't seen this comment earlier, but this is most likely fixed by commit 198d2ed2d4c6c921928a9175f825d0d8345e5a46.
The solution for the original problem is, "use inotify."
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=296d7f95aab8d3c948fd6a421eb49513aa48435b commit 296d7f95aab8d3c948fd6a421eb49513aa48435b Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2025-07-03 20:07:45 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2026-08-05 15:15:14 +0000 vfs: Initial revision of inotify Add an implementation of inotify_init(), inotify_add_watch(), inotify_rm_watch(), source-compatible with Linux. This provides functionality similar to kevent(2)'s EVFILT_VNODE, i.e., it lets applications monitor filesystem files for accesses. Compared to inotify, however, EVFILT_VNODE has the limitation of requiring the application to open the file to be monitored. This means that activity on a newly created file cannot be monitored reliably, and that a file descriptor per file in the hierarchy is required. inotify on the other hand allows a directory and its entries to be monitored at once. It introduces a new file descriptor type to which "watches" can be attached; a watch is a pseudo-file descriptor associated with a file or directory and a set of events to watch for. When a watched vnode is accessed, a description of the event is queued to the inotify descriptor, readable with read(2). Events for files in a watched directory include the file name. A watched vnode has its usecount bumped, so name cache entries originating from a watched directory are not evicted. Name cache entries are used to populate inotify events for files with a link in a watched directory. In particular, if a file is accessed with, say, read(2), an IN_ACCESS event will be generated for any watched hard link of the file. The inotify_add_watch_at() variant is included so that this functionality is available in capability mode; plain inotify_add_watch() is disallowed in capability mode. When a file in a nullfs mount is watched, the watch is attached to the lower vnode, such that accesses via either layer generate inotify events. Many thanks to Gleb Popov for testing this patch and finding lots of bugs. PR: 258010, 215011 Reviewed by: kib Tested by: arrowd MFC after: 3 months Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D50315 (cherry picked from commit f1f230439fa48581f40a57f095627f667a9713c3) share/man/man4/rights.4 | 10 +- sys/bsm/audit_kevents.h | 1 + sys/conf/files | 1 + sys/fs/nullfs/null_subr.c | 4 + sys/fs/nullfs/null_vnops.c | 29 +- sys/kern/kern_resource.c | 21 + sys/kern/subr_capability.c | 4 + sys/kern/sys_generic.c | 33 +- sys/kern/syscalls.master | 15 + sys/kern/vfs_cache.c | 59 +++ sys/kern/vfs_default.c | 17 + sys/kern/vfs_inotify.c (new) | 1004 ++++++++++++++++++++++++++++++++++++++++++ sys/kern/vfs_subr.c | 7 +- sys/kern/vfs_vnops.c | 3 +- sys/kern/vnode_if.src | 21 + sys/sys/caprights.h | 2 + sys/sys/capsicum.h | 8 +- sys/sys/file.h | 1 + sys/sys/inotify.h (new) | 146 ++++++ sys/sys/resourcevar.h | 4 + sys/sys/specialfd.h | 5 + sys/sys/user.h | 5 + sys/sys/vnode.h | 12 +- sys/tools/vnode_if.awk | 1 + 24 files changed, 1399 insertions(+), 14 deletions(-)