Summary: | [zfs] zfs is losing the snapshot directory, | ||
---|---|---|---|
Product: | Base System | Reporter: | bmeyer |
Component: | kern | Assignee: | freebsd-fs (Nobody) <fs> |
Status: | Closed Unable to Reproduce | ||
Severity: | Affects Only Me | CC: | aaron, andrew, bmeyer, jwdevel, karl, smh, wiml |
Priority: | Normal | ||
Version: | 8.2-STABLE | ||
Hardware: | Any | ||
OS: | Any |
Description
bmeyer
2011-05-03 00:40:07 UTC
Responsible Changed From-To: freebsd-bugs->freebsd-fs reclassify. I have had this occur on amd64 with 8.2-RELEASE. I also came to the conclusion that it takes a reboot to fix the issue. Is there any activity on this bug? Thanks, Nathan On 05/07/2011, at 11:56 PM, Nathan Bowyer wrote: > I have had this occur on amd64 with 8.2-RELEASE. I also came to the = conclusion that it takes a reboot to fix the issue. > Is there any activity on this bug? Not sure to be honest. I'm going actually in the process of building a = new testing environment for a updated mail server and will probably be = able to give a more definitive answer a bit later (the build will be = done later this week and then I'll be letting my test scrips run mad on = it for a week or so). =20 The fact that you have had the same problems with AMD means that the = problem isn't SPARC specific. I am guessing that it is probably a = commit that was made between to the zfs or solaris code between Mon Feb = 28 16:07:24 EST 2011 and Sun Apr 24 21:28:12 EST 2011 (those are two = systems I have that are very similarly configured but the earlier one = doesn't have the problem whereas the later one does). =20 /BGM= Same problem here: FreeBSD mybox 8.1-RELEASE-p5 FreeBSD 8.1-RELEASE-p5 #0: Tue Sep 27 = 16:49:00 UTC 2011 = root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 ls of some .zfs directories returns the "Bad file descriptor" error. zfs list still shows every snapshots, and daily snapshot = creation/rotation works as expected. I tried to unmount the faulty zfs volume, but the system froze, I had to = reboot the server.= Is the faulty dataset shared over NFS? If so, is it the new NFS server (default in 9.0 I think) or the old (default in 8.2)? My ZFS over NFS on 8-STABLE has slightly similar problems on the NFS client side, but only when viewed by a Linux client. A simple "ls -l" on the .zfs/snapshot directory on a Linux client hangs the dataset. And these lines show up in /var/log/messages when reloading the NFS server: Apr 12 10:08:38 [hostname] mountd[54479]: can't delete exports for /tank/[...]/.zfs/snapshot/replication-20120411112000: Invalid argument And an "ls" without "-l" on a Linux client shows that the ones mentioned in messages are files and the rest are directories (Linux shells generally color them blue and append a / for directories). I can only guess that they may be related. No sharing, nothing fancy, only ZFS and snapshot on a server. Well, this bug is really really annoying. I would reclassify it as = critical: my server is hosted in a data center, and when I want/need to = reboot it, it just crashes and stays frozen. I've to ask my hosting = provider to hard reboot the server. I wonder if I need to use `reboot -qn` until the bug is corrected, or = should I ditch ZFS?= I would like to note that we are also experiencing this issue. We are = just using the basic core ZFS features only - snapshots and zfs = send/receive. There does not seem to be any set pattern as to when this = issue occurs (the machine below had the issue appear after 47 days) and = it doesn't affect all filesystems at once. It would be good if we could get some traction on this issue as it = appears to be a systemic issue, across all releases and architectures, = reported by multiple people over the last 2-3 years. As snapshots are a = basic core feature of ZFS, I would really like to see this bug moved to = a higher priority. # ls -l /mailstore/domains/.zfs ls: snapshot: Bad file descriptor total 2 dr-xr-xr-x 2 root wheel 2 Jul 19 15:46 shares # uname -a FreeBSD 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 02:52:29 = UTC 2012 = root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 # uptime 2:45AM up 47 days, 4:30, 1 user, load averages: 0.29, 0.29, 0.21 # zfs list NAME USED AVAIL REFER MOUNTPOINT mailstore 466G 3.11T 34K /mailstore mailstore/domains 464G 3.11T 247G /mailstore/domains mailstore/queue 3.31M 100G 102K /mailstore/queue mailstore/var 1.30G 3.11T 182M /mailstore/var system 15.4G 258G 32K none system/root 15.2G 258G 7.08G legacy # zpool status pool: mailstore state: ONLINE scan: scrub repaired 0 in 0h11m with 0 errors on Sun Aug 19 03:13:05 = 2012 config: NAME STATE READ WRITE CKSUM mailstore ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 label/disk8 ONLINE 0 0 0 label/disk16 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 label/disk9 ONLINE 0 0 0 label/disk17 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 label/disk10 ONLINE 0 0 0 label/disk18 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 label/disk11 ONLINE 0 0 0 label/disk19 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 label/disk12 ONLINE 0 0 0 label/disk20 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 label/disk13 ONLINE 0 0 0 label/disk21 ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 label/disk14 ONLINE 0 0 0 label/disk22 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 label/disk15 ONLINE 0 0 0 label/disk23 ONLINE 0 0 0 logs mirror-8 ONLINE 0 0 0 label/disk3 ONLINE 0 0 0 label/disk4 ONLINE 0 0 0 cache label/disk5 ONLINE 0 0 0 spares label/disk6 AVAIL =20 label/disk7 AVAIL =20 errors: No known data errors pool: system state: ONLINE scan: scrub repaired 0 in 0h4m with 0 errors on Mon Aug 20 03:05:52 = 2012 config: NAME STATE READ WRITE CKSUM system ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 spares gpt/disk2 AVAIL =20 errors: No known data errors # zpool get all mailstore NAME PROPERTY VALUE SOURCE mailstore size 3.62T - mailstore capacity 12% - mailstore altroot - default mailstore health ONLINE - mailstore guid 5477043980362510628 default mailstore version 28 default mailstore bootfs - default mailstore delegation on default mailstore autoreplace off default mailstore cachefile - default mailstore failmode wait default mailstore listsnapshots off default mailstore autoexpand off default mailstore dedupditto 0 default mailstore dedupratio 1.00x - mailstore free 3.17T - mailstore allocated 466G - mailstore readonly off =20 # zfs get all mailstore/domains NAME PROPERTY VALUE SOURCE mailstore/domains type filesystem - mailstore/domains creation Thu Jul 19 15:46 2012 - mailstore/domains used 463G - mailstore/domains available 3.11T - mailstore/domains referenced 247G - mailstore/domains compressratio 1.00x - mailstore/domains mounted yes - mailstore/domains quota none default mailstore/domains reservation none default mailstore/domains recordsize 128K default mailstore/domains mountpoint /mailstore/domains default mailstore/domains sharenfs off default mailstore/domains checksum on default mailstore/domains compression off default mailstore/domains atime off local mailstore/domains devices on default mailstore/domains exec on default mailstore/domains setuid off = inherited from mailstore mailstore/domains readonly off default mailstore/domains jailed off default mailstore/domains snapdir hidden default mailstore/domains aclmode discard default mailstore/domains aclinherit restricted default mailstore/domains canmount on default mailstore/domains xattr off = temporary mailstore/domains copies 1 default mailstore/domains version 5 - mailstore/domains utf8only off - mailstore/domains normalization none - mailstore/domains casesensitivity sensitive - mailstore/domains vscan off default mailstore/domains nbmand off default mailstore/domains sharesmb off default mailstore/domains refquota none default mailstore/domains refreservation none default mailstore/domains primarycache all default mailstore/domains secondarycache all default mailstore/domains usedbysnapshots 217G - mailstore/domains usedbydataset 247G - mailstore/domains usedbychildren 0 - mailstore/domains usedbyrefreservation 0 - mailstore/domains logbias latency default mailstore/domains dedup off default mailstore/domains mlslabel - mailstore/domains sync standard default mailstore/domains refcompressratio 1.00x - # zpool history History for 'mailstore': 2012-07-19.15:46:37 zpool create -f mailstore mirror label/disk8 = label/disk16 mirror label/disk9 label/disk17 mirror label/disk10 = label/disk18 mirror label/disk11 label/disk19 mirror label/disk12 = label/disk20 mirror label/disk13 label/disk21 mirror label/disk14 = label/disk22 mirror label/disk15 label/disk23 log mirror label/disk3 = label/disk4 cache label/disk5 spare label/disk6 label/disk7 2012-07-28.10:30:00 zfs snapshot -r mailstore@autosnap-2012-07-28_10.30 2012-07-28.10:30:05 zfs destroy -r mailstore@autosnap-2012-07-28_08.30 2012-07-28.10:45:01 zfs snapshot -r mailstore@autosnap-2012-07-28_10.45 2012-07-28.10:45:06 zfs destroy -r mailstore@autosnap-2012-07-28_08.45 2012-07-28.11:00:01 zfs snapshot -r mailstore@autosnap-2012-07-28_11.00 2012-07-28.11:00:06 zfs destroy -r mailstore@autosnap-2012-07-27_11.00 2012-07-28.11:15:00 zfs snapshot -r mailstore@autosnap-2012-07-28_11.15 2012-07-28.11:15:05 zfs destroy -r mailstore@autosnap-2012-07-28_09.15 2012-07-28.11:30:01 zfs snapshot -r mailstore@autosnap-2012-07-28_11.30 ... 2012-09-06.02:00:01 zfs snapshot -r mailstore@autosnap-2012-09-06_02.00 2012-09-06.02:00:06 zfs destroy -r mailstore@autosnap-2012-09-05_02.00 2012-09-06.02:15:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.15 2012-09-06.02:15:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.15 2012-09-06.02:30:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.30 2012-09-06.02:30:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.30 2012-09-06.02:45:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.45 2012-09-06.02:45:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.45 # truss ls -l /mailstore/domains/.zfs mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D = 34366279680 (0x80063d000) issetugid(0x80063e015,0x800632d7e,0x80084e7f0,0x80084e7c0,0xb297,0x0) =3D = 0 (0x0) open("/etc/libmap.conf",O_RDONLY,0666) ERR#2 'No such file or = directory' open("/var/run/ld-elf.so.hints",O_RDONLY,057) =3D 3 (0x3) read(3,"Ehnt\^A\0\0\0\M^@\0\0\0u\0\0\0\0"...,128) =3D 128 (0x80) lseek(3,0x80,SEEK_SET) =3D 128 (0x80) read(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,117) =3D 117 (0x75) close(3) =3D 0 (0x0) access("/lib/libutil.so.9",0) =3D 0 (0x0) open("/lib/libutil.so.9",O_RDONLY,041017540) =3D 3 (0x3) fstat(3,{ mode=3D-r--r--r-- ,inode=3D10999,size=3D70168,blksize=3D70656 = }) =3D 0 (0x0) pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D= 4096 (0x1000) mmap(0x0,2166784,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D = 34368450560 (0x80084f000) = mmap(0x80084f000,61440,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCOR= E,3,0x0) =3D 34368450560 (0x80084f000) = mmap(0x800a5e000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xf000)= =3D 34370609152 (0x800a5e000) = mmap(0x800a5f000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,= -1,0x0) =3D 34370613248 (0x800a5f000) close(3) =3D 0 (0x0) access("/lib/libncurses.so.8",0) =3D 0 (0x0) open("/lib/libncurses.so.8",O_RDONLY,041017540) =3D 3 (0x3) fstat(3,{ mode=3D-r--r--r-- ,inode=3D10996,size=3D321040,blksize=3D131072 = }) =3D 0 (0x0) pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D= 4096 (0x1000) mmap(0x0,2412544,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D = 34370617344 (0x800a60000) = mmap(0x800a60000,294912,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCO= RE,3,0x0) =3D 34370617344 (0x800a60000) = mmap(0x800ca8000,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x4800= 0) =3D 34373009408 (0x800ca8000) close(3) =3D 0 (0x0) access("/lib/libc.so.7",0) =3D 0 (0x0) open("/lib/libc.so.7",O_RDONLY,041017540) =3D 3 (0x3) fstat(3,{ mode=3D-r--r--r-- ,inode=3D10991,size=3D1315160,blksize=3D131072= }) =3D 0 (0x0) pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D= 4096 (0x1000) mmap(0x0,3432448,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D = 34373029888 (0x800cad000) = mmap(0x800cad000,1179648,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOC= ORE,3,0x0) =3D 34373029888 (0x800cad000) = mmap(0x800fcd000,45056,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x1200= 00) =3D 34376306688 (0x800fcd000) = mmap(0x800fd8000,110592,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANO= N,-1,0x0) =3D 34376351744 (0x800fd8000) close(3) =3D 0 (0x0) munmap(0x800644000,4096) =3D 0 (0x0) mmap(0x0,40960,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D = 34366308352 (0x800644000) munmap(0x80064a000,16384) =3D 0 (0x0) mmap(0x0,102400,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D = 34366332928 (0x80064a000) = sysarch(0x81,0x7fffffffd310,0x800643188,0x0,0xffffffffff672680,0x808080808= 0808080) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) ioctl(1,TIOCGETA,0xffffd410) =3D 0 (0x0) ioctl(1,TIOCGWINSZ,0xffffd470) =3D 0 (0x0) getuid() =3D 0 (0x0) readlink("/etc/malloc.conf",0x7fffffffcef0,1024) ERR#2 'No such file or = directory' = issetugid(0x800da9bc1,0x7fffffffcef0,0xffffffffffffffff,0x0,0x2,0x7fffffff= d18f) =3D 0 (0x0) break(0x800000) =3D 0 (0x0) mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D = 34376462336 (0x800ff3000) mmap(0x8013f3000,53248,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = =3D 34380656640 (0x8013f3000) munmap(0x800ff3000,53248) =3D 0 (0x0) lstat("/mailstore/domains/.zfs",{ mode=3Ddr-xr-xr-x = ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 (0x0) open(".",O_RDONLY,00) =3D 3 (0x3) fchdir(0x3,0x0,0x20,0x20,0x1,0x0) =3D 0 (0x0) stat("/mailstore/domains/.zfs",{ mode=3Ddr-xr-xr-x = ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 (0x0) open("/mailstore/domains/.zfs",O_NONBLOCK|0x20000,0100401200) =3D 4 = (0x4) fstat(4,{ mode=3Ddr-xr-xr-x ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 = (0x0) fcntl(4,F_SETFD,FD_CLOEXEC) =3D 0 (0x0) fstatfs(0x4,0x7fffffffd0a0,0x0,0x26,0x606fe0,0x7fffffffc630) =3D 0 (0x0) fstat(4,{ mode=3Ddr-xr-xr-x ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 = (0x0) fchdir(0x4,0x7fffffffd2b0,0x4,0x0,0x606fe0,0x7fffffffc610) =3D 0 (0x0) = getdirentries(0x4,0x801027000,0x1000,0x801025068,0x606fe0,0x7fffffffc610) = =3D 60 (0x3c) lstat("snapshot",0x801020450) ERR#9 'Bad file = descriptor' lstat("shares",{ mode=3Ddr-xr-xr-x ,inode=3D7,size=3D2,blksize=3D4096 }) = =3D 0 (0x0) getdirentries(0x4,0x801027000,0x1000,0x801025068,0xffffffffffff95d4,0x0) = =3D 0 (0x0) lseek(4,0x0,SEEK_SET) =3D 0 (0x0) madvise(0x801069000,0x1000,0x5,0x68,0x7fffffffca10,0x7fffffffca10) =3D 0 = (0x0) madvise(0x801026000,0x2000,0x5,0x25,0x7fffffffca10,0x7fffffffca10) =3D 0 = (0x0) madvise(0x801025000,0x1000,0x5,0x24,0x7fffffffca70,0x7fffffffca10) =3D 0 = (0x0) close(4) =3D 0 (0x0) fchdir(0x3,0x0,0x0,0x6,0x801000000,0x7fffffffca80) =3D 0 (0x0) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) open("/etc/nsswitch.conf",O_RDONLY,0666) =3D 4 (0x4) ioctl(4,TIOCGETA,0xffffcaf0) ERR#25 'Inappropriate = ioctl for device' fstat(4,{ mode=3D-rw-r--r-- ,inode=3D11094,size=3D323,blksize=3D4096 }) = =3D 0 (0x0) read(4,"#\n# nsswitch.conf(5) - name ser"...,4096) =3D 323 (0x143) read(4,0x801045000,4096) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) access("/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/compat/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/dovecot/nss_compat.so.1",0) ERR#2 'No such file = or directory' access("/usr/local/lib/mysql/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/libexec/openldap/nss_compat.so.1",0) ERR#2 'No such = file or directory' access("/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) access("/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/compat/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/dovecot/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/mysql/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/libexec/openldap/nss_nis.so.1",0) ERR#2 'No such file = or directory' access("/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) access("/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/compat/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/dovecot/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/mysql/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/libexec/openldap/nss_files.so.1",0) ERR#2 'No such = file or directory' access("/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) access("/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/compat/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/dovecot/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/mysql/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/libexec/openldap/nss_dns.so.1",0) ERR#2 'No such file = or directory' access("/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) ioctl(4,TIOCGETA,0xffffcb00) ERR#25 'Inappropriate = ioctl for device' close(4) =3D 0 (0x0) madvise(0x80108a000,0x1000,0x5,0x89,0x7fffffffc2e0,0xffffffff) =3D 0 = (0x0) madvise(0x801044000,0x2000,0x5,0x43,0x7fffffffc2e0,0xffffffff) =3D 0 = (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) geteuid() =3D 0 (0x0) open("/etc/spwd.db",O_RDONLY,00) =3D 4 (0x4) fcntl(4,F_SETFD,FD_CLOEXEC) =3D 0 (0x0) fstat(4,{ mode=3D-rw------- ,inode=3D54894,size=3D40960,blksize=3D40960 = }) =3D 0 (0x0) read(4,"\0\^F\^Ua\0\0\0\^B\0\0\^D\M-R\0"...,260) =3D 260 (0x104) pread(0x4,0x801090000,0x1000,0x6000,0x1,0x0) =3D 4096 (0x1000) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) pread(0x4,0x801091000,0x1000,0x4000,0x1,0x0) =3D 4096 (0x1000) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) open("/etc/group",O_RDONLY,0666) =3D 5 (0x5) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) fstat(5,{ mode=3D-rw-r--r-- ,inode=3D11080,size=3D570,blksize=3D4096 }) = =3D 0 (0x0) lseek(5,0x0,SEEK_CUR) =3D 0 (0x0) lseek(5,0x0,SEEK_SET) =3D 0 (0x0) read(5,"# $FreeBSD: release/9.0.0/etc/gr"...,4096) =3D 570 (0x23a) close(5) =3D 0 (0x0) stat("/usr/share/nls/C/libc.cat",0x7fffffffc920) ERR#2 'No such file or = directory' stat("/usr/share/nls/libc/C",0x7fffffffc920) ERR#2 'No such file or = directory' stat("/usr/local/share/nls/C/libc.cat",0x7fffffffc920) ERR#2 'No such = file or directory' stat("/usr/local/share/nls/libc/C",0x7fffffffc920) ERR#2 'No such file = or directory' ls: write(2,"ls: ",4) =3D 4 (0x4) snapshot: Bad file descriptorwrite(2,"snapshot: Bad file descriptor",29) = =3D 29 (0x1d) write(2,"\n",1) =3D 1 (0x1) fstat(1,{ mode=3Dcrw--w---- ,inode=3D185,size=3D0,blksize=3D4096 }) =3D = 0 (0x0) ioctl(1,TIOCGETA,0xffffbff0) =3D 0 (0x0) total 2 write(1,"total 2\n",8) =3D 8 (0x8) lpathconf(0x7fffffffc9e0,0x40,0x0,0x0,0x7ff7fefdc3f0,0x7fffffffc43f) = ERR#22 'Invalid argument' = lpathconf(0x7fffffffc9e0,0x3b,0xffffffffffffffff,0x16,0x7ff7fefdc3f0,0x7ff= fffffc43f) ERR#22 'Invalid argument' clock_gettime(13,{1346858032.000000000 }) =3D 0 (0x0) access("/etc/localtime",4) =3D 0 (0x0) open("/etc/localtime",O_RDONLY,0100000700) =3D 5 (0x5) fstat(5,{ mode=3D-r--r--r-- ,inode=3D16452,size=3D2434,blksize=3D4096 }) = =3D 0 (0x0) read(5,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) =3D 2434 (0x982) close(5) =3D 0 (0x0) issetugid(0x800db1fd1,0x8010d6000,0x0,0xa7,0x4e,0xc) =3D 0 (0x0) open("/usr/share/zoneinfo/posixrules",O_RDONLY,056) =3D 5 (0x5) fstat(5,{ mode=3D-r--r--r-- ,inode=3D545,size=3D3519,blksize=3D4096 }) =3D= 0 (0x0) read(5,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) =3D 3519 (0xdbf) close(5) =3D 0 (0x0) madvise(0x80111e000,0xb000,0x5,0x11d,0x7fffffffb880,0x607050) =3D 0 = (0x0) madvise(0x8010d6000,0x5000,0x5,0xd5,0x7fffffffba70,0xffffffff) =3D 0 = (0x0) madvise(0x8010d2000,0x1000,0x5,0xd1,0x7fffffffba70,0xffffffff) =3D 0 = (0x0) madvise(0x801075000,0xb000,0x5,0x74,0x7fffffffba70,0xffffffff) =3D 0 = (0x0) dr-xr-xr-x 2 root wheel 2 Jul 19 15:46 shares write(1,"dr-xr-xr-x 2 root wheel 2 Jul"...,49) =3D 49 (0x31) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) process exit, rval =3D 1 Regards, Nikolai Schupbach As I am not experiencing this problem on my systems, it would be great to create a way to reproduce it. E.g. a script that creates/deletes 1000 snapshots, mounts/unmounts them, etc. -- Martin Matuska FreeBSD committer http://blog.vx.sk I can confirm that this still occurs, this time on 8.3-RELEASE on amd64. Exactly as described by the previous submitters - nothing special going on on our system (no NFS, etc.) - just plain old snapshots being created and removed and ... eventually some .zfs/snapshot directories just disappear. Two things: 1) In our experience, enough retrying of ls/find/etc. of the missing snapshot directory will eventually lock the system up 2) non-critical severity is incorrect - snapshots are a critical feature and this is a critical bug. Thanks. This is happening for us too in the 9.1-RELEASE. Just noticed yesterday. We don't have a ton of snapshots, just a week's worth of daily ones. About 40GB of data. We've experienced the same problem on 8.2-stable. Seems to be related to upgrade to 8.3-RELEASE not finished (we cannot shutdown the machine until replacement is ready). So the world and kernel upgrade is installed but system has not been rebooted. Your aware 8.3 is now EOL right? Although I have never gotten a system freeze or panic, I have seen this issue of not being able to access the .zfs/snapshot/xyz mount. For instance: $ ls /myvolume/.zfs ls: snapshot: Bad file descriptor shares For me, I have had success manually mounting the snapshot. Adding this advice in case others like me hit this issue, as the "manual mount" option does not seem to be well-documented - no reference to ZFS in mount(8) or nmount(2), for instance. I had to look at some ZFS-on-Linux sites to discover it (: $ mount -t zfs dataset@snapshot /mountpoint (replace with args appropriate for your system, of course) System info: FreeBSD FreeBSD 9.1-RELEASE-p17 #0 r268574 amd64 ZFS 5, pool version 28 9.1 is unsupported as well. This is an old problem that I have seen repeatedly with that OS revision, and beware -- attempting to send that snapshot is likely to lead to an immediate kernel panic. I recommend updating the system as I've not seen that issue arise on 11.0 or 11.1. |