---------------------------------------------------------------------- Command Log Output: helios.mesoft.com.au : Tue May 3 08:15:30 EST 2011 : /tmp -> 1013# hostname helios.mesoft.com.au helios.mesoft.com.au : Tue May 3 08:15:32 EST 2011 : /tmp -> 1014# uname -a FreeBSD helios.mesoft.com.au 8.2-STABLE FreeBSD 8.2-STABLE #1: Sun Apr 24 21:28:12 EST 2011 root@helios.mesoft.com.au:/usr/obj/usr/src/sys/GENERIC sparc64 helios.mesoft.com.au : Tue May 3 08:15:34 EST 2011 : /tmp -> 1015# gmirror status Name Status Components mirror/gm0 COMPLETE da0 da1 helios.mesoft.com.au : Tue May 3 08:15:38 EST 2011 : /tmp -> 1016# zpool status -v pool: pool0 state: ONLINE scrub: scrub completed after 0h2m with 0 errors on Tue May 3 07:15:24 2011 config: NAME STATE READ WRITE CKSUM pool0 ONLINE 0 0 0 mirror/gm0h ONLINE 0 0 0 errors: No known data errors helios.mesoft.com.au : Tue May 3 08:15:42 EST 2011 : /tmp -> 1017# zfs list NAME USED AVAIL REFER MOUNTPOINT pool0 5.00G 263G 24K /pool0 pool0/databases 128M 263G 128M /pool0/databases pool0/home 45.2M 263G 45.2M /pool0/home pool0/mail 4.83G 263G 4.83G /pool0/mail helios.mesoft.com.au : Tue May 3 08:15:49 EST 2011 : /tmp -> 1018# zfs list -t snapshot no datasets available helios.mesoft.com.au : Tue May 3 08:15:53 EST 2011 : /tmp -> 1019# zfs snapshot pool0/mail@test helios.mesoft.com.au : Tue May 3 08:16:07 EST 2011 : /tmp -> 1020# zfs snapshot pool0/home@test helios.mesoft.com.au : Tue May 3 08:16:19 EST 2011 : /tmp -> 1021# zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT pool0/home@test 0 - 45.2M - pool0/mail@test 59K - 4.83G - helios.mesoft.com.au : Tue May 3 08:16:22 EST 2011 : /tmp -> 1022# ls -l /pool0/mail/.zfs total 0 dr-xr-xr-x 2 root wheel 2 Jun 16 2010 snapshot helios.mesoft.com.au : Tue May 3 08:16:32 EST 2011 : /tmp -> 1023# ls -l /pool0/home/.zfs ls: snapshot: Bad file descriptor total 0 helios.mesoft.com.au : Tue May 3 08:17:08 EST 2011 : /tmp -> 1024# zfs destroy pool0/mail@test helios.mesoft.com.au : Tue May 3 08:17:52 EST 2011 : /tmp -> 1025# zfs destroy pool0/home@test helios.mesoft.com.au : Tue May 3 08:18:05 EST 2011 : /tmp -> 1026# zfs list -t snapshot no datasets available helios.mesoft.com.au : Tue May 3 08:22:13 EST 2011 : /tmp -> 1027# zfs upgrade This system is currently running ZFS filesystem version 4. All filesystems are formatted with the current version. helios.mesoft.com.au : Tue May 3 08:22:18 EST 2011 : /tmp -> 1028# zpool upgrade This system is currently running ZFS pool version 15. All pools are formatted using this version. helios.mesoft.com.au : Tue May 3 08:22:47 EST 2011 : /tmp -> 1029# zdb pool0 version=15 name='pool0' state=0 txg=908395 pool_guid=2223392563195472406 hostid=1798529942 hostname='helios.mesoft.com.au' vdev_tree type='root' id=0 guid=2223392563195472406 children[0] type='disk' id=0 guid=8062368332970890863 path='/dev/mirror/gm0h' whole_disk=0 metaslab_array=23 metaslab_shift=31 ashift=9 asize=293168742400 is_log=0 DTL=40 helios.mesoft.com.au : Tue May 3 08:24:05 EST 2011 : /tmp -> 1030# zpool scrub pool0 helios.mesoft.com.au : Tue May 3 08:31:22 EST 2011 : /tmp -> 1031# zpool status -v pool: pool0 state: ONLINE scrub: scrub completed after 0h2m with 0 errors on Tue May 3 08:26:35 2011 config: NAME STATE READ WRITE CKSUM pool0 ONLINE 0 0 0 mirror/gm0h ONLINE 0 0 0 errors: No known data errors ---------------------------------------------------------------------- DMESG, CONFIGs, etc. /var/run/dmesg.boot: Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #1: Sun Apr 24 21:28:12 EST 2011 root@helios.mesoft.com.au:/usr/obj/usr/src/sys/GENERIC sparc64 real memory = 4294967296 (4096 MB) avail memory = 4165672960 (3972 MB) cpu0: Sun Microsystems UltraSparc-IIe Processor (648.00 MHz CPU) ispfw: registered firmware <isp_1000> ispfw: registered firmware <isp_1040> ispfw: registered firmware <isp_1040_it> ispfw: registered firmware <isp_1080> ispfw: registered firmware <isp_1080_it> ispfw: registered firmware <isp_12160> ispfw: registered firmware <isp_12160_it> ispfw: registered firmware <isp_2100> ispfw: registered firmware <isp_2200> ispfw: registered firmware <isp_2300> ispfw: registered firmware <isp_2322> ispfw: registered firmware <isp_2400> ispfw: registered firmware <isp_2400_multi> ispfw: registered firmware <isp_2500> ispfw: registered firmware <isp_2500_multi> kbd0 at kbdmux0 nexus0: <Open Firmware Nexus device> pcib0: <U2P UPA-PCI bridge> mem 0x1fe00000000-0x1fe0000ffff,0x1fe01000000-0x1fe010000ff irq 2032,2030,2031,2021 on nexus0 pcib0: Sabre, impl 0, version 0, IGN 0x1f, bus A, 66MHz pcib0: DVMA map: 0xc0000000 to 0xc3ffffff 8192 entries pcib0: [FILTER] pcib0: [FILTER] pcib0: [GIANT-LOCKED] pcib0: [ITHREAD] pcib0: [FILTER] pci0: <OFW PCI bus> on pcib0 pcib1: <APB PCI-PCI bridge> at device 1.1 on pci0 pci1: <OFW PCI bus> on pcib1 ebus0: <PCI-EBus3 bridge> mem 0xf0000000-0xf0ffffff,0xf1000000-0xf17fffff at device 12.0 on pci1 ebus0: <idprom>: incomplete pcib2: <APB PCI-PCI bridge> at device 1.0 on pci0 pci2: <OFW PCI bus> on pcib2 ebus0: <flashprom> addr 0x1000000000-0x10000fffff (no driver attached) eeprom0: <EEPROM/clock> addr 0x1400000000-0x1400001fff on ebus0 eeprom0: model mk48t59 ebus0: <SUNW,lomh> addr 0x1400200000-0x1400200003 irq 42 (no driver attached) pci1: <old, non-VGA display device> at device 3.0 (no driver attached) isab0: <PCI-ISA bridge> at device 7.0 on pci1 isa0: <ISA bus> on isab0 gem0: <Sun ERI 10/100 Ethernet> mem 0xe0400000-0xe041ffff at device 12.1 on pci1 miibus0: <MII bus> on gem0 bmtphy0: <BCM5221 10/100baseTX PHY> PHY 1 on miibus0 bmtphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow gem0: 2kB RX FIFO, 2kB TX FIFO gem0: Ethernet address: 00:03:ba:68:45:35 gem0: [ITHREAD] ohci0: <Sun PCIO-2 USB controller> mem 0xe2000000-0xe2007fff at device 12.3 on pci1 ohci0: [ITHREAD] usbus0: <Sun PCIO-2 USB controller> on ohci0 atapci0: <AcerLabs M5229 UDMA66 controller> port 0x400-0x407,0x418-0x41b,0x410-0x417,0x408-0x40b,0x420-0x42f at device 13.0 on pci1 atapci0: [ITHREAD] atapci0: using PIO transfers above 137GB as workaround for 48bit DMA access bug, expect reduced performance ata2: <ATA channel 0> on atapci0 ata2: [ITHREAD] ata3: <ATA channel 1> on atapci0 ata3: [ITHREAD] gem1: <Sun ERI 10/100 Ethernet> mem 0xe0440000-0xe045ffff at device 5.1 on pci1 miibus1: <MII bus> on gem1 bmtphy1: <BCM5221 10/100baseTX PHY> PHY 1 on miibus1 bmtphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow gem1: 2kB RX FIFO, 2kB TX FIFO gem1: Ethernet address: 00:03:ba:68:45:35 gem1: [ITHREAD] ohci1: <Sun PCIO-2 USB controller> mem 0xe5000000-0xe5007fff at device 5.3 on pci1 ohci1: [ITHREAD] usbus1: <Sun PCIO-2 USB controller> on ohci1 sym0: <896> port 0xc00000-0xc000ff mem 0x2000-0x23ff,0x4000-0x5fff at device 8.0 on pci2 sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking sym0: [ITHREAD] sym1: <896> port 0xc00100-0xc001ff mem 0x6000-0x63ff,0x8000-0x9fff at device 8.1 on pci2 sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking sym1: [ITHREAD] re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> port 0xc00200-0xc002ff mem 0xa000-0xa0ff at device 5.0 on pci2 re0: Chip rev. 0x04000000 re0: MAC rev. 0x00000000 miibus2: <MII bus> on re0 rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus2 rgephy0: 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 00:08:54:35:a4:97 re0: [FILTER] nexus0: <syscons> type unknown (no driver attached) uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 43 on isa0 uart0: [FILTER] uart0: console (9600,n,8,1) uart1: <16550 or compatible> at port 0x2e8-0x2ef irq 43 on isa0 uart1: [FILTER] ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 4 ZFS storage pool version 15 Timecounter "tick" frequency 648000000 Hz quality 1000 Timecounters tick every 10.000 msec ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to deny, logging disabled load_dn_sched dn_sched FIFO loaded load_dn_sched dn_sched QFQ loaded load_dn_sched dn_sched RR loaded load_dn_sched dn_sched WF2Q+ loaded load_dn_sched dn_sched PRIO loaded usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 ugen0.1: <SUN> at usbus0 uhub0: <SUN OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen1.1: <SUN> at usbus1 uhub1: <SUN OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 acd0: DVDROM <DV-28E-B/1.2B> at ata2-master UDMA33 uhub0: 4 ports with 4 removable, self powered uhub1: 4 ports with 4 removable, self powered da0 at sym0 bus 0 scbus0 target 0 lun 0 da0: <ACARD SCSIDE BRIDGE320 10EP> Fixed Direct Access SCSI-3 device da0: 80.000MB/s transfers (40.000MHz, offset 28, 16bit) da0: Command Queueing enabled da0: 305245MB (625142448 512 byte sectors: 255H 63S/T 38913C) da1 at sym0 bus 0 scbus0 target 1 lun 0 da1: <ACARD SCSIDE BRIDGE320 10EP> Fixed Direct Access SCSI-3 device da1: 80.000MB/s transfers (40.000MHz, offset 28, 16bit) da1: Command Queueing enabled da1: 305245MB (625142448 512 byte sectors: 255H 63S/T 38913C) GEOM_MIRROR: Device mirror/gm0 launched (1/2). GEOM_MIRROR: Device gm0: rebuilding provider da1. GEOM: mirror/gm0: adding VTOC8 information. GEOM_JOURNAL: Journal 4175325748: mirror/gm0e contains data. GEOM_JOURNAL: Journal 4175325748: mirror/gm0e contains journal. GEOM_JOURNAL: Journal mirror/gm0e clean. GEOM_JOURNAL: Journal 2946333449: mirror/gm0g contains data. GEOM_JOURNAL: Journal 2946333449: mirror/gm0g contains journal. GEOM_JOURNAL: Journal mirror/gm0g clean. Trying to mount root from ufs:/dev/mirror/gm0a /boot/loader.conf : ipfw_load="YES" dummynet_load="YES" geom_mirror_load="YES" geom_journal_load="YES" opensolaris_load="YES" zfs_load="YES" vm.kmem_size="1024M" vm.kmem_size_max="1024M" vfs.zfs.arc_max="128M" vfs.zfs.vdev.cache.size="16M" kern.hz="100" /etc/sysctl.conf : # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith Exp $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Uncomment this to prevent users from seeing information about processes that # are being run under another UID. #security.bsd.see_other_uids=0 kern.ipc.shmmax=268435456 net.inet.ip.fw.verbose=1 Disk Slicing : sunlabel da0 # /dev/da0: text: FreeBSD298G cyl 38911 alt 2 hd 255 sec 63 bytes/sector: 512 sectors/cylinder: 16065 sectors/unit: 625105215 8 partitions: # # Size is in sectors. # Offset is in cylinders. # size offset # ---------- ---------- a: 1060290 0 b: 4209030 66 c: 625105215 0 d: 2104515 328 e: 8401995 459 f: 1060290 982 g: 35664300 1048 h: 572604795 3268 sunlabel da1 # /dev/da1: text: FreeBSD298G cyl 38911 alt 2 hd 255 sec 63 bytes/sector: 512 sectors/cylinder: 16065 sectors/unit: 625105215 8 partitions: # # Size is in sectors. # Offset is in cylinders. # size offset # ---------- ---------- a: 1060290 0 b: 4209030 66 c: 625105215 0 d: 2104515 328 e: 8401995 459 f: 1060290 982 g: 35664300 1048 h: 572604795 3268 Fix: Only known workaround at this time is to periodically reboot the box - usually nightly. How-To-Repeat: Allow system to run for any length of time.
Responsible Changed From-To: freebsd-bugs->freebsd-fs reclassify.
I have had this occur on amd64 with 8.2-RELEASE. I also came to the conclusion that it takes a reboot to fix the issue. Is there any activity on this bug? Thanks, Nathan
On 05/07/2011, at 11:56 PM, Nathan Bowyer wrote: > I have had this occur on amd64 with 8.2-RELEASE. I also came to the = conclusion that it takes a reboot to fix the issue. > Is there any activity on this bug? Not sure to be honest. I'm going actually in the process of building a = new testing environment for a updated mail server and will probably be = able to give a more definitive answer a bit later (the build will be = done later this week and then I'll be letting my test scrips run mad on = it for a week or so). =20 The fact that you have had the same problems with AMD means that the = problem isn't SPARC specific. I am guessing that it is probably a = commit that was made between to the zfs or solaris code between Mon Feb = 28 16:07:24 EST 2011 and Sun Apr 24 21:28:12 EST 2011 (those are two = systems I have that are very similarly configured but the earlier one = doesn't have the problem whereas the later one does). =20 /BGM=
Same problem here: FreeBSD mybox 8.1-RELEASE-p5 FreeBSD 8.1-RELEASE-p5 #0: Tue Sep 27 = 16:49:00 UTC 2011 = root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 ls of some .zfs directories returns the "Bad file descriptor" error. zfs list still shows every snapshots, and daily snapshot = creation/rotation works as expected. I tried to unmount the faulty zfs volume, but the system froze, I had to = reboot the server.=
Is the faulty dataset shared over NFS? If so, is it the new NFS server (default in 9.0 I think) or the old (default in 8.2)? My ZFS over NFS on 8-STABLE has slightly similar problems on the NFS client side, but only when viewed by a Linux client. A simple "ls -l" on the .zfs/snapshot directory on a Linux client hangs the dataset. And these lines show up in /var/log/messages when reloading the NFS server: Apr 12 10:08:38 [hostname] mountd[54479]: can't delete exports for /tank/[...]/.zfs/snapshot/replication-20120411112000: Invalid argument And an "ls" without "-l" on a Linux client shows that the ones mentioned in messages are files and the rest are directories (Linux shells generally color them blue and append a / for directories). I can only guess that they may be related.
No sharing, nothing fancy, only ZFS and snapshot on a server.
Well, this bug is really really annoying. I would reclassify it as = critical: my server is hosted in a data center, and when I want/need to = reboot it, it just crashes and stays frozen. I've to ask my hosting = provider to hard reboot the server. I wonder if I need to use `reboot -qn` until the bug is corrected, or = should I ditch ZFS?=
I would like to note that we are also experiencing this issue. We are = just using the basic core ZFS features only - snapshots and zfs = send/receive. There does not seem to be any set pattern as to when this = issue occurs (the machine below had the issue appear after 47 days) and = it doesn't affect all filesystems at once. It would be good if we could get some traction on this issue as it = appears to be a systemic issue, across all releases and architectures, = reported by multiple people over the last 2-3 years. As snapshots are a = basic core feature of ZFS, I would really like to see this bug moved to = a higher priority. # ls -l /mailstore/domains/.zfs ls: snapshot: Bad file descriptor total 2 dr-xr-xr-x 2 root wheel 2 Jul 19 15:46 shares # uname -a FreeBSD 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 02:52:29 = UTC 2012 = root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 # uptime 2:45AM up 47 days, 4:30, 1 user, load averages: 0.29, 0.29, 0.21 # zfs list NAME USED AVAIL REFER MOUNTPOINT mailstore 466G 3.11T 34K /mailstore mailstore/domains 464G 3.11T 247G /mailstore/domains mailstore/queue 3.31M 100G 102K /mailstore/queue mailstore/var 1.30G 3.11T 182M /mailstore/var system 15.4G 258G 32K none system/root 15.2G 258G 7.08G legacy # zpool status pool: mailstore state: ONLINE scan: scrub repaired 0 in 0h11m with 0 errors on Sun Aug 19 03:13:05 = 2012 config: NAME STATE READ WRITE CKSUM mailstore ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 label/disk8 ONLINE 0 0 0 label/disk16 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 label/disk9 ONLINE 0 0 0 label/disk17 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 label/disk10 ONLINE 0 0 0 label/disk18 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 label/disk11 ONLINE 0 0 0 label/disk19 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 label/disk12 ONLINE 0 0 0 label/disk20 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 label/disk13 ONLINE 0 0 0 label/disk21 ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 label/disk14 ONLINE 0 0 0 label/disk22 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 label/disk15 ONLINE 0 0 0 label/disk23 ONLINE 0 0 0 logs mirror-8 ONLINE 0 0 0 label/disk3 ONLINE 0 0 0 label/disk4 ONLINE 0 0 0 cache label/disk5 ONLINE 0 0 0 spares label/disk6 AVAIL =20 label/disk7 AVAIL =20 errors: No known data errors pool: system state: ONLINE scan: scrub repaired 0 in 0h4m with 0 errors on Mon Aug 20 03:05:52 = 2012 config: NAME STATE READ WRITE CKSUM system ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 spares gpt/disk2 AVAIL =20 errors: No known data errors # zpool get all mailstore NAME PROPERTY VALUE SOURCE mailstore size 3.62T - mailstore capacity 12% - mailstore altroot - default mailstore health ONLINE - mailstore guid 5477043980362510628 default mailstore version 28 default mailstore bootfs - default mailstore delegation on default mailstore autoreplace off default mailstore cachefile - default mailstore failmode wait default mailstore listsnapshots off default mailstore autoexpand off default mailstore dedupditto 0 default mailstore dedupratio 1.00x - mailstore free 3.17T - mailstore allocated 466G - mailstore readonly off =20 # zfs get all mailstore/domains NAME PROPERTY VALUE SOURCE mailstore/domains type filesystem - mailstore/domains creation Thu Jul 19 15:46 2012 - mailstore/domains used 463G - mailstore/domains available 3.11T - mailstore/domains referenced 247G - mailstore/domains compressratio 1.00x - mailstore/domains mounted yes - mailstore/domains quota none default mailstore/domains reservation none default mailstore/domains recordsize 128K default mailstore/domains mountpoint /mailstore/domains default mailstore/domains sharenfs off default mailstore/domains checksum on default mailstore/domains compression off default mailstore/domains atime off local mailstore/domains devices on default mailstore/domains exec on default mailstore/domains setuid off = inherited from mailstore mailstore/domains readonly off default mailstore/domains jailed off default mailstore/domains snapdir hidden default mailstore/domains aclmode discard default mailstore/domains aclinherit restricted default mailstore/domains canmount on default mailstore/domains xattr off = temporary mailstore/domains copies 1 default mailstore/domains version 5 - mailstore/domains utf8only off - mailstore/domains normalization none - mailstore/domains casesensitivity sensitive - mailstore/domains vscan off default mailstore/domains nbmand off default mailstore/domains sharesmb off default mailstore/domains refquota none default mailstore/domains refreservation none default mailstore/domains primarycache all default mailstore/domains secondarycache all default mailstore/domains usedbysnapshots 217G - mailstore/domains usedbydataset 247G - mailstore/domains usedbychildren 0 - mailstore/domains usedbyrefreservation 0 - mailstore/domains logbias latency default mailstore/domains dedup off default mailstore/domains mlslabel - mailstore/domains sync standard default mailstore/domains refcompressratio 1.00x - # zpool history History for 'mailstore': 2012-07-19.15:46:37 zpool create -f mailstore mirror label/disk8 = label/disk16 mirror label/disk9 label/disk17 mirror label/disk10 = label/disk18 mirror label/disk11 label/disk19 mirror label/disk12 = label/disk20 mirror label/disk13 label/disk21 mirror label/disk14 = label/disk22 mirror label/disk15 label/disk23 log mirror label/disk3 = label/disk4 cache label/disk5 spare label/disk6 label/disk7 2012-07-28.10:30:00 zfs snapshot -r mailstore@autosnap-2012-07-28_10.30 2012-07-28.10:30:05 zfs destroy -r mailstore@autosnap-2012-07-28_08.30 2012-07-28.10:45:01 zfs snapshot -r mailstore@autosnap-2012-07-28_10.45 2012-07-28.10:45:06 zfs destroy -r mailstore@autosnap-2012-07-28_08.45 2012-07-28.11:00:01 zfs snapshot -r mailstore@autosnap-2012-07-28_11.00 2012-07-28.11:00:06 zfs destroy -r mailstore@autosnap-2012-07-27_11.00 2012-07-28.11:15:00 zfs snapshot -r mailstore@autosnap-2012-07-28_11.15 2012-07-28.11:15:05 zfs destroy -r mailstore@autosnap-2012-07-28_09.15 2012-07-28.11:30:01 zfs snapshot -r mailstore@autosnap-2012-07-28_11.30 ... 2012-09-06.02:00:01 zfs snapshot -r mailstore@autosnap-2012-09-06_02.00 2012-09-06.02:00:06 zfs destroy -r mailstore@autosnap-2012-09-05_02.00 2012-09-06.02:15:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.15 2012-09-06.02:15:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.15 2012-09-06.02:30:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.30 2012-09-06.02:30:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.30 2012-09-06.02:45:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.45 2012-09-06.02:45:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.45 # truss ls -l /mailstore/domains/.zfs mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D = 34366279680 (0x80063d000) issetugid(0x80063e015,0x800632d7e,0x80084e7f0,0x80084e7c0,0xb297,0x0) =3D = 0 (0x0) open("/etc/libmap.conf",O_RDONLY,0666) ERR#2 'No such file or = directory' open("/var/run/ld-elf.so.hints",O_RDONLY,057) =3D 3 (0x3) read(3,"Ehnt\^A\0\0\0\M^@\0\0\0u\0\0\0\0"...,128) =3D 128 (0x80) lseek(3,0x80,SEEK_SET) =3D 128 (0x80) read(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,117) =3D 117 (0x75) close(3) =3D 0 (0x0) access("/lib/libutil.so.9",0) =3D 0 (0x0) open("/lib/libutil.so.9",O_RDONLY,041017540) =3D 3 (0x3) fstat(3,{ mode=3D-r--r--r-- ,inode=3D10999,size=3D70168,blksize=3D70656 = }) =3D 0 (0x0) pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D= 4096 (0x1000) mmap(0x0,2166784,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D = 34368450560 (0x80084f000) = mmap(0x80084f000,61440,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCOR= E,3,0x0) =3D 34368450560 (0x80084f000) = mmap(0x800a5e000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xf000)= =3D 34370609152 (0x800a5e000) = mmap(0x800a5f000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,= -1,0x0) =3D 34370613248 (0x800a5f000) close(3) =3D 0 (0x0) access("/lib/libncurses.so.8",0) =3D 0 (0x0) open("/lib/libncurses.so.8",O_RDONLY,041017540) =3D 3 (0x3) fstat(3,{ mode=3D-r--r--r-- ,inode=3D10996,size=3D321040,blksize=3D131072 = }) =3D 0 (0x0) pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D= 4096 (0x1000) mmap(0x0,2412544,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D = 34370617344 (0x800a60000) = mmap(0x800a60000,294912,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCO= RE,3,0x0) =3D 34370617344 (0x800a60000) = mmap(0x800ca8000,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x4800= 0) =3D 34373009408 (0x800ca8000) close(3) =3D 0 (0x0) access("/lib/libc.so.7",0) =3D 0 (0x0) open("/lib/libc.so.7",O_RDONLY,041017540) =3D 3 (0x3) fstat(3,{ mode=3D-r--r--r-- ,inode=3D10991,size=3D1315160,blksize=3D131072= }) =3D 0 (0x0) pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D= 4096 (0x1000) mmap(0x0,3432448,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D = 34373029888 (0x800cad000) = mmap(0x800cad000,1179648,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOC= ORE,3,0x0) =3D 34373029888 (0x800cad000) = mmap(0x800fcd000,45056,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x1200= 00) =3D 34376306688 (0x800fcd000) = mmap(0x800fd8000,110592,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANO= N,-1,0x0) =3D 34376351744 (0x800fd8000) close(3) =3D 0 (0x0) munmap(0x800644000,4096) =3D 0 (0x0) mmap(0x0,40960,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D = 34366308352 (0x800644000) munmap(0x80064a000,16384) =3D 0 (0x0) mmap(0x0,102400,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D = 34366332928 (0x80064a000) = sysarch(0x81,0x7fffffffd310,0x800643188,0x0,0xffffffffff672680,0x808080808= 0808080) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) ioctl(1,TIOCGETA,0xffffd410) =3D 0 (0x0) ioctl(1,TIOCGWINSZ,0xffffd470) =3D 0 (0x0) getuid() =3D 0 (0x0) readlink("/etc/malloc.conf",0x7fffffffcef0,1024) ERR#2 'No such file or = directory' = issetugid(0x800da9bc1,0x7fffffffcef0,0xffffffffffffffff,0x0,0x2,0x7fffffff= d18f) =3D 0 (0x0) break(0x800000) =3D 0 (0x0) mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D = 34376462336 (0x800ff3000) mmap(0x8013f3000,53248,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = =3D 34380656640 (0x8013f3000) munmap(0x800ff3000,53248) =3D 0 (0x0) lstat("/mailstore/domains/.zfs",{ mode=3Ddr-xr-xr-x = ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 (0x0) open(".",O_RDONLY,00) =3D 3 (0x3) fchdir(0x3,0x0,0x20,0x20,0x1,0x0) =3D 0 (0x0) stat("/mailstore/domains/.zfs",{ mode=3Ddr-xr-xr-x = ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 (0x0) open("/mailstore/domains/.zfs",O_NONBLOCK|0x20000,0100401200) =3D 4 = (0x4) fstat(4,{ mode=3Ddr-xr-xr-x ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 = (0x0) fcntl(4,F_SETFD,FD_CLOEXEC) =3D 0 (0x0) fstatfs(0x4,0x7fffffffd0a0,0x0,0x26,0x606fe0,0x7fffffffc630) =3D 0 (0x0) fstat(4,{ mode=3Ddr-xr-xr-x ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 = (0x0) fchdir(0x4,0x7fffffffd2b0,0x4,0x0,0x606fe0,0x7fffffffc610) =3D 0 (0x0) = getdirentries(0x4,0x801027000,0x1000,0x801025068,0x606fe0,0x7fffffffc610) = =3D 60 (0x3c) lstat("snapshot",0x801020450) ERR#9 'Bad file = descriptor' lstat("shares",{ mode=3Ddr-xr-xr-x ,inode=3D7,size=3D2,blksize=3D4096 }) = =3D 0 (0x0) getdirentries(0x4,0x801027000,0x1000,0x801025068,0xffffffffffff95d4,0x0) = =3D 0 (0x0) lseek(4,0x0,SEEK_SET) =3D 0 (0x0) madvise(0x801069000,0x1000,0x5,0x68,0x7fffffffca10,0x7fffffffca10) =3D 0 = (0x0) madvise(0x801026000,0x2000,0x5,0x25,0x7fffffffca10,0x7fffffffca10) =3D 0 = (0x0) madvise(0x801025000,0x1000,0x5,0x24,0x7fffffffca70,0x7fffffffca10) =3D 0 = (0x0) close(4) =3D 0 (0x0) fchdir(0x3,0x0,0x0,0x6,0x801000000,0x7fffffffca80) =3D 0 (0x0) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) open("/etc/nsswitch.conf",O_RDONLY,0666) =3D 4 (0x4) ioctl(4,TIOCGETA,0xffffcaf0) ERR#25 'Inappropriate = ioctl for device' fstat(4,{ mode=3D-rw-r--r-- ,inode=3D11094,size=3D323,blksize=3D4096 }) = =3D 0 (0x0) read(4,"#\n# nsswitch.conf(5) - name ser"...,4096) =3D 323 (0x143) read(4,0x801045000,4096) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) access("/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/compat/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/dovecot/nss_compat.so.1",0) ERR#2 'No such file = or directory' access("/usr/local/lib/mysql/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/libexec/openldap/nss_compat.so.1",0) ERR#2 'No such = file or directory' access("/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_compat.so.1",0) ERR#2 'No such file or = directory' sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) access("/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/compat/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/dovecot/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/mysql/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/libexec/openldap/nss_nis.so.1",0) ERR#2 'No such file = or directory' access("/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_nis.so.1",0) ERR#2 'No such file or = directory' sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) access("/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/compat/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/dovecot/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/mysql/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/libexec/openldap/nss_files.so.1",0) ERR#2 'No such = file or directory' access("/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_files.so.1",0) ERR#2 'No such file or = directory' sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) access("/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/compat/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/dovecot/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/lib/mysql/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/local/libexec/openldap/nss_dns.so.1",0) ERR#2 'No such file = or directory' access("/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' access("/usr/lib/nss_dns.so.1",0) ERR#2 'No such file or = directory' sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) ioctl(4,TIOCGETA,0xffffcb00) ERR#25 'Inappropriate = ioctl for device' close(4) =3D 0 (0x0) madvise(0x80108a000,0x1000,0x5,0x89,0x7fffffffc2e0,0xffffffff) =3D 0 = (0x0) madvise(0x801044000,0x2000,0x5,0x43,0x7fffffffc2e0,0xffffffff) =3D 0 = (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) geteuid() =3D 0 (0x0) open("/etc/spwd.db",O_RDONLY,00) =3D 4 (0x4) fcntl(4,F_SETFD,FD_CLOEXEC) =3D 0 (0x0) fstat(4,{ mode=3D-rw------- ,inode=3D54894,size=3D40960,blksize=3D40960 = }) =3D 0 (0x0) read(4,"\0\^F\^Ua\0\0\0\^B\0\0\^D\M-R\0"...,260) =3D 260 (0x104) pread(0x4,0x801090000,0x1000,0x6000,0x1,0x0) =3D 4096 (0x1000) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) pread(0x4,0x801091000,0x1000,0x4000,0x1,0x0) =3D 4096 (0x1000) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) open("/etc/group",O_RDONLY,0666) =3D 5 (0x5) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- = ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0) fstat(5,{ mode=3D-rw-r--r-- ,inode=3D11080,size=3D570,blksize=3D4096 }) = =3D 0 (0x0) lseek(5,0x0,SEEK_CUR) =3D 0 (0x0) lseek(5,0x0,SEEK_SET) =3D 0 (0x0) read(5,"# $FreeBSD: release/9.0.0/etc/gr"...,4096) =3D 570 (0x23a) close(5) =3D 0 (0x0) stat("/usr/share/nls/C/libc.cat",0x7fffffffc920) ERR#2 'No such file or = directory' stat("/usr/share/nls/libc/C",0x7fffffffc920) ERR#2 'No such file or = directory' stat("/usr/local/share/nls/C/libc.cat",0x7fffffffc920) ERR#2 'No such = file or directory' stat("/usr/local/share/nls/libc/C",0x7fffffffc920) ERR#2 'No such file = or directory' ls: write(2,"ls: ",4) =3D 4 (0x4) snapshot: Bad file descriptorwrite(2,"snapshot: Bad file descriptor",29) = =3D 29 (0x1d) write(2,"\n",1) =3D 1 (0x1) fstat(1,{ mode=3Dcrw--w---- ,inode=3D185,size=3D0,blksize=3D4096 }) =3D = 0 (0x0) ioctl(1,TIOCGETA,0xffffbff0) =3D 0 (0x0) total 2 write(1,"total 2\n",8) =3D 8 (0x8) lpathconf(0x7fffffffc9e0,0x40,0x0,0x0,0x7ff7fefdc3f0,0x7fffffffc43f) = ERR#22 'Invalid argument' = lpathconf(0x7fffffffc9e0,0x3b,0xffffffffffffffff,0x16,0x7ff7fefdc3f0,0x7ff= fffffc43f) ERR#22 'Invalid argument' clock_gettime(13,{1346858032.000000000 }) =3D 0 (0x0) access("/etc/localtime",4) =3D 0 (0x0) open("/etc/localtime",O_RDONLY,0100000700) =3D 5 (0x5) fstat(5,{ mode=3D-r--r--r-- ,inode=3D16452,size=3D2434,blksize=3D4096 }) = =3D 0 (0x0) read(5,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) =3D 2434 (0x982) close(5) =3D 0 (0x0) issetugid(0x800db1fd1,0x8010d6000,0x0,0xa7,0x4e,0xc) =3D 0 (0x0) open("/usr/share/zoneinfo/posixrules",O_RDONLY,056) =3D 5 (0x5) fstat(5,{ mode=3D-r--r--r-- ,inode=3D545,size=3D3519,blksize=3D4096 }) =3D= 0 (0x0) read(5,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) =3D 3519 (0xdbf) close(5) =3D 0 (0x0) madvise(0x80111e000,0xb000,0x5,0x11d,0x7fffffffb880,0x607050) =3D 0 = (0x0) madvise(0x8010d6000,0x5000,0x5,0xd5,0x7fffffffba70,0xffffffff) =3D 0 = (0x0) madvise(0x8010d2000,0x1000,0x5,0xd1,0x7fffffffba70,0xffffffff) =3D 0 = (0x0) madvise(0x801075000,0xb000,0x5,0x74,0x7fffffffba70,0xffffffff) =3D 0 = (0x0) dr-xr-xr-x 2 root wheel 2 Jul 19 15:46 shares write(1,"dr-xr-xr-x 2 root wheel 2 Jul"...,49) =3D 49 (0x31) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) = sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER= M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG= XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) =3D 0 (0x0) process exit, rval =3D 1 Regards, Nikolai Schupbach
As I am not experiencing this problem on my systems, it would be great to create a way to reproduce it. E.g. a script that creates/deletes 1000 snapshots, mounts/unmounts them, etc. -- Martin Matuska FreeBSD committer http://blog.vx.sk
I can confirm that this still occurs, this time on 8.3-RELEASE on amd64. Exactly as described by the previous submitters - nothing special going on on our system (no NFS, etc.) - just plain old snapshots being created and removed and ... eventually some .zfs/snapshot directories just disappear. Two things: 1) In our experience, enough retrying of ls/find/etc. of the missing snapshot directory will eventually lock the system up 2) non-critical severity is incorrect - snapshots are a critical feature and this is a critical bug. Thanks.
This is happening for us too in the 9.1-RELEASE. Just noticed yesterday. We don't have a ton of snapshots, just a week's worth of daily ones. About 40GB of data.
We've experienced the same problem on 8.2-stable. Seems to be related to upgrade to 8.3-RELEASE not finished (we cannot shutdown the machine until replacement is ready). So the world and kernel upgrade is installed but system has not been rebooted.
Your aware 8.3 is now EOL right?
Although I have never gotten a system freeze or panic, I have seen this issue of not being able to access the .zfs/snapshot/xyz mount. For instance: $ ls /myvolume/.zfs ls: snapshot: Bad file descriptor shares For me, I have had success manually mounting the snapshot. Adding this advice in case others like me hit this issue, as the "manual mount" option does not seem to be well-documented - no reference to ZFS in mount(8) or nmount(2), for instance. I had to look at some ZFS-on-Linux sites to discover it (: $ mount -t zfs dataset@snapshot /mountpoint (replace with args appropriate for your system, of course) System info: FreeBSD FreeBSD 9.1-RELEASE-p17 #0 r268574 amd64 ZFS 5, pool version 28
9.1 is unsupported as well.
This is an old problem that I have seen repeatedly with that OS revision, and beware -- attempting to send that snapshot is likely to lead to an immediate kernel panic. I recommend updating the system as I've not seen that issue arise on 11.0 or 11.1.