Bug 18012

I've been seeing a rash of "free vnode isn't" panics lately.  Some
machines were panicing several times a day.  Along with this we saw
occasional "object inconsistent state: RPC: %d, RC: %d" messages.

I was able to replicate the problem by running multiple (~8 on a
pentium 200 system with 32 MB of RAM) copies of each of the attached
simple shell scripts (with all output redirected to /dev/null).  It
would often panic within 10-20 minutes.


I tracked the problem down to a race between getnewvnode() recycling a
vnode and vhold().  I found that vhold() was calling vbusy() for a vnode
with the VDOOMED flag set.

This is bad since getnewvnode() removes the vnode from the free list
before setting this flag so vbusy() is calling TAILQ_REMOVE for a vnode
that is not on the free list.  This can easily result in corruption of
the free list pointers causing future getnewvnode() calls to find active
vnodes that it thinks are on the free list.

I added a panic in vbusy() if VDOOMED is set and this hit quite often
during my tests.  Typically the call chain looked something like:

ffs_truncate
ffs_indirtrunc
getblk
bgetvp
vhold
vbusy
panic

With ffs_truncate often being called due to rename(2) or unlink(2).


I managed to solve the problem here by adding a VOP_ISLOCKED(vp) check
to getnewvnode() and skipping such vnodes instead of trying to recycle
them.  From my searches of the mailing lists it appears I'm not the
first one to think of this but apparently this isn't guaranteed to work
for all files system types.  I just know it works for the FFS problems I
was seeing.


At a minimum I'd highly recommend that someone commit a panic to vbusy()
for vnodes with VDOOMED set since letting it continue if that flag is
set can and does result in the corruption of the vnode_free_list.  I'd
also recommend the addition of the VOP_ISLOCKED() check to getnewvnode()
even if it doesn't work for all file system types it will help in some
(most?) cases.  A patch for CURRENT is attached.

How-To-Repeat: 
Running simultaneous multiple copies of the following simple scripts:

#!/bin/sh
cd /var/log
while true; do
	mkdir copy; cp * copy
	mkdir copy/2; mv copy/* copy/2
	rm -rf copy
done

#!/bin/sh
FS="/tmp /var /usr"
while true; do
	for f in $FS; do
		find -x $f -type f -print | xargs -n 4 head -1
	done
done

#!/bin/sh
while true; do
	ps -aux; ps -ajx
done