Crashes on below rev stays up on above rev borg.lerctr.org dumped core - see /var/crash/vmcore.2 Sat Jan 25 15:25:30 CST 2014 FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #3 r261166: Sat Jan 25 14:55:24 CST 2014 root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 panic: solaris assert: l->l_phys->l_hdr.lh_pad1 == 0 (0xffff000000000000 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c, line: 479 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #3 r261166: Sat Jan 25 14:55:24 CST 2014 root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 info: [drm] Initialized drm 1.1.0 20060810 CPU: Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (2327.55-MHz K8-class CPU) Origin="GenuineIntel" Id=0x10676 Family=0x6 Model=0x17 Stepping=6 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 65641644032 (62600 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <PTLTD APIC > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard random: <Software, Yarrow> initialized kbd1 at kbdmux0 cryptosoft0: <software crypto> on motherboard acpi0: <PTLTD RSDT> on motherboard acpi0: Power Button (fixed) unknown: I/O range not supported cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 cpu4: <ACPI CPU> on acpi0 cpu5: <ACPI CPU> on acpi0 cpu6: <ACPI CPU> on acpi0 cpu7: <ACPI CPU> on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1 pci2: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3 pci4: <ACPI PCI bus> on pcib4 pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3 pci5: <ACPI PCI bus> on pcib5 pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2 pci6: <ACPI PCI bus> on pcib6 em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2000-0x201f mem 0xd9220000-0xd923ffff,0xd9200000-0xd921ffff irq 18 at device 0.0 on pci6 em0: Using an MSI interrupt em0: Ethernet address: 00:30:48:f2:29:9c em1: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2020-0x203f mem 0xd9260000-0xd927ffff,0xd9240000-0xd925ffff irq 19 at device 0.1 on pci6 em1: Using an MSI interrupt em1: Ethernet address: 00:30:48:f2:29:9d pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1 pci7: <ACPI PCI bus> on pcib7 pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0 pci8: <ACPI PCI bus> on pcib8 vgapci0: <VGA-compatible display> port 0x3000-0x307f mem 0xd8000000-0xd8ffffff,0xc0000000-0xc7ffffff,0xc8000000-0xc9ffffff irq 16 at device 0.0 on pci8 nvidia0: <GeForce GT 610> on vgapci0 vgapci0: child nvidia0 requested pci_enable_io vgapci0: child nvidia0 requested pci_enable_io hdac0: <NVIDIA GF119 HDA Controller> mem 0xd9000000-0xd9003fff irq 17 at device 0.1 on pci8 pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0 pci9: <ACPI PCI bus> on pcib9 pci0: <base peripheral> at device 8.0 (no driver attached) pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0 pci10: <ACPI PCI bus> on pcib10 pcib11: <PCI-PCI bridge> irq 16 at device 0.0 on pci10 pci11: <PCI bus> on pcib11 pcm0: <Envy24HT audio (Generic)> port 0x4080-0x409f,0x4000-0x407f irq 16 at device 0.0 on pci11 pcm0: system configuration SubVendorID: 0x1412, SubDeviceID: 0x2403 XIN2 Clock Source: 24.576MHz(96kHz*256) MPU-401 UART(s) #: not implemented ADC #: 1 and SPDIF receiver connected DAC #: 4 Multi-track converter type: AC'97(SDATA_OUT:packed) S/PDIF(IN/OUT): 1/1 ID# 0x00 GPIO(mask/dir/state): 0xff/0xff/0xff uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0 usbus0 on uhci0 uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0 usbus1 on uhci1 uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0 usbus2 on uhci2 ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd9600400-0xd96007ff irq 17 at device 29.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci12: <ACPI PCI bus> on pcib12 vgapci1: <VGA-compatible display> port 0x5000-0x50ff mem 0xd0000000-0xd7ffffff,0xd9300000-0xd930ffff irq 18 at device 1.0 on pci12 drmn1: <ATI ES1000 RN50> on vgapci1 info: [drm] RADEON_IS_PCI info: [drm] initializing kernel modesetting (RV100 0x1002:0x515E 0x15D9:0x8080). info: [drm] register mmio base: 0xD9300000 info: [drm] register mmio size: 65536 info: [drm] radeon_atrm_get_bios: ===> Try ATRM... info: [drm] radeon_atrm_get_bios: pci_find_class() found: 0:8:0:0, vendor=10de, device=104a info: [drm] radeon_atrm_get_bios: Get ACPI device handle info: [drm] radeon_acpi_vfct_bios: ===> Try VFCT... info: [drm] radeon_acpi_vfct_bios: Get "VFCT" ACPI table info: [drm] radeon_acpi_vfct_bios: Failed to get "VFCT" table: AE_NOT_FOUND info: [drm] igp_read_bios_from_vram: ===> Try IGP's VRAM... info: [drm] igp_read_bios_from_vram: VRAM base address: 0xd0000000 info: [drm] igp_read_bios_from_vram: Map address: 0xfffff800d0000000 (262144 bytes) info: [drm] igp_read_bios_from_vram: Incorrect BIOS signature: 0x0000 info: [drm] radeon_read_bios: ===> Try PCI Expansion ROM... info: [drm] radeon_read_bios: Map address: 0xfffff800000c0000 (131072 bytes) drmn1: info: VRAM: 128M 0x00000000D0000000 - 0x00000000D7FFFFFF (16M used) drmn1: info: GTT: 512M 0x00000000B0000000 - 0x00000000CFFFFFFF info: [drm] Supports vblank timestamp caching Rev 1 (10.10.2010). info: [drm] Driver supports precise vblank timestamp query. info: [drm] radeon: irq initialized. info: [drm] Detected VRAM RAM=128M, BAR=128M info: [drm] RAM width 64bits SDR [TTM] Zone kernel: Available graphics memory: 33006090 kiB [TTM] Zone dma32: Available graphics memory: 2097152 kiB [TTM] Initializing pool allocator info: [drm] radeon: 16M of VRAM memory ready info: [drm] radeon: 512M of GTT memory ready. info: [drm] GART: num cpu pages 131072, num gpu pages 131072 info: [drm] PCI GART of 512M enabled (table at 0x0000000011200000). drmn1: info: WB disabled drmn1: info: fence driver on ring 0 use gpu addr 0x00000000b0000000 and cpu addr 0x0xfffff8001083c000 info: [drm] Loading R100 Microcode info: [drm] radeon: ring at 0x00000000B0001000 info: [drm] ring test succeeded in 1 usecs info: [drm] ib test succeeded in 0 usecs info: [drm] radeon_device_init: Taking over the fictitious range 0xd0000000-0xd4000000 iicbus0: <Philips I2C bus> on iicbb0 addr 0xff iic0: <I2C generic I/O> on iicbus0 iicbus1: <Philips I2C bus> on iicbb1 addr 0xff iic1: <I2C generic I/O> on iicbus1 iicbus2: <Philips I2C bus> on iicbb2 addr 0xff iic2: <I2C generic I/O> on iicbus2 iicbus3: <Philips I2C bus> on iicbb3 addr 0xff iic3: <I2C generic I/O> on iicbus3 info: [drm] Radeon Display Connectors info: [drm] Connector 0: info: [drm] VGA-1 info: [drm] DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60 info: [drm] Encoders: info: [drm] CRT1: INTERNAL_DAC1 info: [drm] Connector 1: info: [drm] DVI-I-1 info: [drm] HPD2 info: [drm] DDC: 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c info: [drm] Encoders: info: [drm] CRT2: INTERNAL_DAC2 info: [drm] DFP2: INTERNAL_DVO1 composite sync not supported composite sync not supported info: [drm] fb mappable at 0xD0040000 info: [drm] vram apper at 0xD0000000 info: [drm] size 2076672 info: [drm] fb depth is 8 info: [drm] pitch is 1920 fbd1 on drmn1 vt_allocate: Replace existing VT driver. info: [drm] Initialized radeon 2.29.0 20080528 vgapci1: Boot video device isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0 ata0: <ATA channel> at channel 0 on atapci0 ahci0: <Intel ESB2 AHCI SATA controller> port 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f mem 0xd9600800-0xd9600bff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x1100-0x111f irq 19 at device 31.3 on pci0 smbus0: <System Management Bus> on ichsmb0 acpi_button0: <Power Button> on acpi0 ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0 ipmi0: KCS mode found at io 0xca2 on acpi atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: <Parallel port bus> on ppc0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 ichwd0 on isa0 orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0 coretemp0: <CPU On-Die Thermal Sensors> on cpu0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 p4tcc0: <CPU Frequency Thermal Control> on cpu0 coretemp1: <CPU On-Die Thermal Sensors> on cpu1 est1: <Enhanced SpeedStep Frequency Control> on cpu1 p4tcc1: <CPU Frequency Thermal Control> on cpu1 coretemp2: <CPU On-Die Thermal Sensors> on cpu2 est2: <Enhanced SpeedStep Frequency Control> on cpu2 p4tcc2: <CPU Frequency Thermal Control> on cpu2 coretemp3: <CPU On-Die Thermal Sensors> on cpu3 est3: <Enhanced SpeedStep Frequency Control> on cpu3 p4tcc3: <CPU Frequency Thermal Control> on cpu3 coretemp4: <CPU On-Die Thermal Sensors> on cpu4 est4: <Enhanced SpeedStep Frequency Control> on cpu4 p4tcc4: <CPU Frequency Thermal Control> on cpu4 coretemp5: <CPU On-Die Thermal Sensors> on cpu5 est5: <Enhanced SpeedStep Frequency Control> on cpu5 p4tcc5: <CPU Frequency Thermal Control> on cpu5 coretemp6: <CPU On-Die Thermal Sensors> on cpu6 est6: <Enhanced SpeedStep Frequency Control> on cpu6 p4tcc6: <CPU Frequency Thermal Control> on cpu6 coretemp7: <CPU On-Die Thermal Sensors> on cpu7 est7: <Enhanced SpeedStep Frequency Control> on cpu7 p4tcc7: <CPU Frequency Thermal Control> on cpu7 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec vboxdrv: fAsync=0 offMin=0x387 offMax=0x50f hdacc0: <NVIDIA (0x001c) HDA CODEC> at cad 0 on hdac0 hdaa0: <NVIDIA (0x001c) Audio Function Group> at nid 1 on hdacc0 pcm1: <NVIDIA (0x001c) (HDMI/DP 8ch)> at nid 4 on hdaa0 pcm2: <NVIDIA (0x001c) (HDMI/DP 8ch)> at nid 5 on hdaa0 random: unblocking device. usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 ugen3.1: <Intel> at usbus3 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 ugen2.1: <Intel> at usbus2 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 ugen1.1: <Intel> at usbus1 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 ugen0.1: <Intel> at usbus0 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ipmi0: IPMI device rev. 1, firmware rev. 1.64, version 2.0 ata0: DMA limited to UDMA33, controller found non-ATA66 cable ipmi0: Number of channels 8 ipmi0: Attached watchdog uhub2: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device ada0: Serial Number 5YDA1ZL4 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada0: quirks=0x1<4K> ada0: Previously was known as ad4 ada1 at ahcich1 bus 0 scbus2 target 0 lun 0 ada1: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device ada1: Serial Number 5YD6FPLG ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada1: quirks=0x1<4K> ada1: Previously was known as ad6 ada2 at ahcich2 bus 0 scbus3 target 0 lun 0 ada2: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device ada2: Serial Number 5YDA3PC5 ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2: quirks=0x1<4K> ada2: Previously was known as ad8 ada3 at ahcich3 bus 0 scbus4 target 0 lun 0 ada3: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device ada3: Serial Number 5YD9Y0P4 ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3: quirks=0x1<4K> ada3: Previously was known as ad10 ada4 at ahcich4 bus 0 scbus5 target 0 lun 0 ada4: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device ada4: Serial Number 5YDA1R5W ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada4: quirks=0x1<4K> ada4: Previously was known as ad12 ada5 at ahcich5 bus 0 scbus6 target 0 lun 0 ada5: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device ada5: Serial Number 5YD5RBS8 ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada5: quirks=0x1<4K> ada5: Previously was known as ad14 cd0 at ata0 bus 0 scbus0 target 0 lun 0 cd0: <Memorex DVD+-RAM 510L v1 MWS7> Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present Netvsc initializing... done! SMP: AP CPU #7 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #5 Launched! Root mount waiting for: usbus3 uhub0: 6 ports with 6 removable, self powered Root mount waiting for: usbus3 Root mount waiting for: usbus3 ugen3.2: <Peppercon AG> at usbus3 ukbd0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3 kbd2 at ukbd0 Trying to mount root from zfs:zroot/ROOT/default []... ugen0.2: <American Power Conversion> at usbus0 ugen1.2: <Prolific Technology Inc.> at usbus1 <118>Setting hostuuid: 53d19f64-d663-a017-8922-0030488e9ff3. <118>Setting hostid: 0xf53a926e. <118>Entropy harvesting: interrupts ethernet point_to_point swi. <118>Starting file system checks: <118>Mounting local file systems:. panic: solaris assert: l->l_phys->l_hdr.lh_pad1 == 0 (0xffff000000000000 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c, line: 479 cpuid = 7 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe100c55d350 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe100c55d400 vpanic() at vpanic+0x126/frame 0xfffffe100c55d440 panic() at panic+0x43/frame 0xfffffe100c55d4a0 assfail3() at assfail3+0x2f/frame 0xfffffe100c55d4c0 zap_get_leaf_byblk() at zap_get_leaf_byblk+0x2ea/frame 0xfffffe100c55d510 zap_deref_leaf() at zap_deref_leaf+0xd1/frame 0xfffffe100c55d560 fzap_length() at fzap_length+0x31/frame 0xfffffe100c55d5d0 zap_length_uint64() at zap_length_uint64+0xbf/frame 0xfffffe100c55d630 ddt_zap_lookup() at ddt_zap_lookup+0x3d/frame 0xfffffe100c55d790 ddt_lookup() at ddt_lookup+0x17e/frame 0xfffffe100c55d960 zio_ddt_write() at zio_ddt_write+0xf2/frame 0xfffffe100c55dad0 zio_execute() at zio_execute+0x1eb/frame 0xfffffe100c55db30 taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame 0xfffffe100c55db80 taskqueue_thread_loop() at taskqueue_thread_loop+0x9b/frame 0xfffffe100c55dbb0 fork_exit() at fork_exit+0x84/frame 0xfffffe100c55dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe100c55dbf0 --- trap 0, rip = 0, rsp = 0xfffffe100c55dcb0, rbp = 0 --- KDB: enter: panic Uptime: 23s Dumping 2330 out of 64465 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/linux.ko.symbols...done. Loaded symbols for /boot/kernel/linux.ko.symbols Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. Loaded symbols for /boot/kernel/if_lagg.ko.symbols Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. Loaded symbols for /boot/kernel/snd_spicds.ko.symbols Reading symbols from /boot/kernel/coretemp.ko.symbols...done. Loaded symbols for /boot/kernel/coretemp.ko.symbols Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. Loaded symbols for /boot/kernel/ichsmb.ko.symbols Reading symbols from /boot/kernel/smbus.ko.symbols...done. Loaded symbols for /boot/kernel/smbus.ko.symbols Reading symbols from /boot/kernel/ichwd.ko.symbols...done. Loaded symbols for /boot/kernel/ichwd.ko.symbols Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. Loaded symbols for /boot/kernel/cpuctl.ko.symbols Reading symbols from /boot/kernel/crypto.ko.symbols...done. Loaded symbols for /boot/kernel/crypto.ko.symbols Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. Loaded symbols for /boot/kernel/cryptodev.ko.symbols Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. Loaded symbols for /boot/kernel/dtraceall.ko.symbols Reading symbols from /boot/kernel/profile.ko.symbols...done. Loaded symbols for /boot/kernel/profile.ko.symbols Reading symbols from /boot/kernel/cyclic.ko.symbols...done. Loaded symbols for /boot/kernel/cyclic.ko.symbols Reading symbols from /boot/kernel/dtrace.ko.symbols...done. Loaded symbols for /boot/kernel/dtrace.ko.symbols Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols Reading symbols from /boot/kernel/systrace.ko.symbols...done. Loaded symbols for /boot/kernel/systrace.ko.symbols Reading symbols from /boot/kernel/sdt.ko.symbols...done. Loaded symbols for /boot/kernel/sdt.ko.symbols Reading symbols from /boot/kernel/lockstat.ko.symbols...done. Loaded symbols for /boot/kernel/lockstat.ko.symbols Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. Loaded symbols for /boot/kernel/fasttrap.ko.symbols Reading symbols from /boot/kernel/fbt.ko.symbols...done. Loaded symbols for /boot/kernel/fbt.ko.symbols Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. Loaded symbols for /boot/kernel/dtnfscl.ko.symbols Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. Loaded symbols for /boot/kernel/dtmalloc.ko.symbols Reading symbols from /boot/modules/vboxdrv.ko...done. Loaded symbols for /boot/modules/vboxdrv.ko Reading symbols from /boot/modules/nvidia.ko...done. Loaded symbols for /boot/modules/nvidia.ko Reading symbols from /boot/kernel/ipmi.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi.ko.symbols Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. Loaded symbols for /boot/kernel/radeonkms.ko.symbols Reading symbols from /boot/kernel/iicbb.ko.symbols...done. Loaded symbols for /boot/kernel/iicbb.ko.symbols Reading symbols from /boot/kernel/iicbus.ko.symbols...done. Loaded symbols for /boot/kernel/iicbus.ko.symbols Reading symbols from /boot/kernel/iic.ko.symbols...done. Loaded symbols for /boot/kernel/iic.ko.symbols Reading symbols from /boot/kernel/drm2.ko.symbols...done. Loaded symbols for /boot/kernel/drm2.ko.symbols Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done. Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. Loaded symbols for /boot/kernel/fdescfs.ko.symbols Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. Loaded symbols for /boot/kernel/linprocfs.ko.symbols #0 doadump (textdump=1) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=1) at pcpu.h:219 #1 0xffffffff809b9c67 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff809ba175 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff809ba1c3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:688 #4 0xffffffff80327abf in assfail3 (a=<value optimized out>, lv=<value optimized out>, op=<value optimized out>, rv=<value optimized out>, f=<value optimized out>, l=<value optimized out>) at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91 #5 0xffffffff803bac1a in zap_get_leaf_byblk (zap=<value optimized out>, blkid=4858, tx=0x0, lt=RW_READER, lp=0xfffffe100c55d5a0) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:479 #6 0xffffffff803b84f1 in zap_deref_leaf (zap=0xfffff800220bf680, h=1610088556361285632, tx=0x0, lt=RW_READER, lp=0xfffffe100c55d5a0) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:585 #7 0xffffffff803b9481 in fzap_length (zn=0xfffff80028e95600, integer_size=0xfffffe100c55d658, num_integers=0xfffffe100c55d650) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:897 #8 0xffffffff803bf5bf in zap_length_uint64 (os=<value optimized out>, zapobj=<value optimized out>, key=<value optimized out>, key_numints=<value optimized out>, integer_size=0xfffffe100c55d658, num_integers=0xfffffe100c55d650) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:934 #9 0xffffffff80352e7d in ddt_zap_lookup (os=0xfffff80020793c00, object=155, dde=0xfffff80028e96600) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/ddt_zap.c:64 #10 0xffffffff8035146e in ddt_lookup (ddt=0xfffffe000d515000, bp=<value optimized out>, add=<value optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/ddt.c:178 #11 0xffffffff803fb242 in zio_ddt_write (zio=0xfffff800288a0730) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2242 #12 0xffffffff803f893b in zio_execute (zio=0xfffff800288a0730) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1355 #13 0xffffffff80a01970 in taskqueue_run_locked (queue=0xfffff80020127000) at /usr/src/sys/kern/subr_taskqueue.c:355 #14 0xffffffff80a022cb in taskqueue_thread_loop (arg=<value optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:576 #15 0xffffffff80989324 in fork_exit ( callout=0xffffffff80a02230 <taskqueue_thread_loop>, arg=0xfffff800200fd4e0, frame=0xfffffe100c55dc00) at /usr/src/sys/kern/kern_fork.c:977 #16 0xffffffff80d95b0e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:605 #17 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) ------------------------------------------------------------------------ ps -axlww UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 -8 0 0 0 - DLs - 0:00.10 [kernel] 0 1 0 0 24 0 9368 0 wait DLs - 0:00.00 [init] 0 2 0 0 -16 0 0 0 ftcl DL - 0:00.00 [ftcleanup] 0 3 0 0 -16 0 0 0 crypto_w DL - 0:00.00 [crypto] 0 4 0 0 -16 0 0 0 crypto_r DL - 0:00.00 [crypto returns] 0 5 0 0 -16 0 0 0 ccb_scan DL - 0:00.00 [cam] 0 6 0 0 -16 0 0 0 - DL - 0:00.00 [fdc0] 0 7 0 0 -8 0 0 0 zio->io_ DL - 0:00.00 [zfskern] 0 8 0 0 -16 0 0 0 waiting_ DL - 0:00.00 [sctp_iterator] 0 9 0 0 -16 0 0 0 ipmireq DL - 0:00.00 [ipmi0: kcs] 0 10 0 0 -16 0 0 0 audit_wo DL - 0:00.00 [audit] 0 11 0 0 155 0 0 0 - RL - 0:00.00 [idle] 0 12 0 0 -84 0 0 0 - WL - 0:00.00 [intr] 0 13 0 0 -8 0 0 0 - DL - 0:00.00 [geom] 0 14 0 0 -16 0 0 0 - DL - 0:00.00 [rand_harvestq] 0 15 0 0 -68 0 0 0 - DL - 0:00.00 [usb] 0 16 0 0 -20 0 0 0 VBoxIS DL - 0:00.00 [TIMER] 0 17 0 0 -16 0 0 0 psleep DL - 0:00.00 [pagedaemon] 0 18 0 0 -16 0 0 0 psleep DL - 0:00.00 [vmdaemon] 0 19 0 0 155 0 0 0 pgzero DL - 0:00.00 [pagezero] 0 20 0 0 -16 0 0 0 psleep DL - 0:00.00 [bufdaemon] 0 21 0 0 16 0 0 0 syncer DL - 0:00.00 [syncer] 0 22 0 0 -16 0 0 0 vlruwt DL - 0:00.00 [vnlru] 0 23 0 0 -16 0 0 0 sdflush DL - 0:00.00 [softdepflush] 0 24 1 0 52 0 16928 0 wait Ds+ - 0:00.01 [sh] 0 72 24 0 52 0 16928 0 wait D+ - 0:00.00 [sh] 0 75 72 0 52 0 40000 0 tx->tx_s D+ - 0:00.00 [zfs] ------------------------------------------------------------------------ vmstat -s 158237 cpu context switches 22164 device interrupts 800 software interrupts 5019 traps 5843 system calls 23 kernel threads created 37 fork() calls 15 vfork() calls 0 rfork() calls 0 swap pager pageins 0 swap pager pages paged in 0 swap pager pageouts 0 swap pager pages paged out 136 vnode pager pageins 911 vnode pager pages paged in 0 vnode pager pageouts 0 vnode pager pages paged out 0 page daemon wakeups 0 pages examined by the page daemon 0 pages reactivated 1771 copy-on-write faults 15 copy-on-write optimized faults 2232 zero fill pages zeroed 0 zero fill pages prezeroed 0 intransit blocking page faults 4649 total VM faults taken 113 page faults requiring I/O 0 pages affected by kernel thread creation 1334 pages affected by fork() 525 pages affected by vfork() 0 pages affected by rfork() 0 pages cached 55169 pages freed 0 pages freed by daemon 0 pages freed by exiting processes 465 pages active 860 pages inactive 0 pages in VM cache 190921 pages wired down 15883166 pages free 4096 bytes per page 1886 total name lookups cache hits (80% pos + 4% neg) system 0% per-directory deletions 0%, falsehits 0%, toolong 0% ------------------------------------------------------------------------ vmstat -m Type InUse MemUse HighUse Requests Size(s) cdev 8 2K - 8 256 ppbusdev 2 1K - 2 256 filedesc 27 54K - 76 2048 kdtrace 323 76K - 515 64,256 kenv 86 11K - 114 16,32,64,128 proc-args 3 1K - 31 32,64,128 hhook 2 1K - 2 256 ithread 125 22K - 125 32,128,256 KTRACE 100 13K - 100 128 entropy 1026 65K - 1026 32,64 acpiintr 1 1K - 1 64 linker 403 101K - 533 16,32,64,128,256,512,1024,2048,4096 lockf 2 1K - 2 64,128 loginclass 2 1K - 2 64 cache 1 1K - 1 32 devbuf 17314 34391K - 18687 16,32,64,128,256,512,1024,2048,4096 temp 57 18K - 538 16,32,64,128,256,512,1024,2048 ip6ndp 3 1K - 3 64 ata_pci 1 1K - 1 64 acpica 1824 187K - 63680 16,32,64,128,256,512,1024,2048 kbdmux 7 18K - 7 16,512,1024,2048 module 521 66K - 521 128 mtx_pool 2 16K - 2 LED 4 1K - 4 16,128 CAM XPT 58 4K - 308 16,32,64,128,512,1024,2048 osd 5 1K - 11 16,32,64,128 pmchooks 1 1K - 1 128 CAM DEV 15 30K - 24 2048 acpitask 1 8K - 1 hdaa 4 5K - 4 512,1024,2048 pgrp 2 1K - 2 128 session 2 1K - 2 128 proc 2 256K - 2 subproc 100 145K - 149 512,4096 cred 40 7K - 240 64,256 plimit 2 1K - 2 256 uidinfo 2 33K - 2 128 hdac 1 1K - 1 512 hdacc 1 1K - 1 32 vtbuf 24 5712K - 24 4096 acpisem 22 3K - 22 128 vt 11 3K - 11 256 sysctl 0 0K - 43 16,32,64 sysctloid 7607 378K - 7779 16,32,64,128 sysctltmp 0 0K - 6 32,64 tidhash 1 256K - 1 callout 9 3208K - 9 umtx 666 84K - 666 128 p1003.1b 1 1K - 1 16 SWAP 12 19681K - 12 64 bus 2257 362K - 8440 16,32,64,128,256,512,1024 bus-sc 157 310K - 5479 16,32,64,128,256,512,1024,2048,4096 devstat 32 65K - 32 32,4096 eventhandler 104 9K - 104 64,128 DEVFS3 230 58K - 270 256 kobj 349 1396K - 1111 4096 DEVFS1 202 101K - 238 512 Per-cpu 1 1K - 1 32 DEVFS 41 1K - 42 16,128 rman 360 41K - 754 16,32,128 sbuf 0 0K - 2295 16,32,64,128,256 sglist 3 1K - 3 32 taskqueue 115 17K - 159 16,32,64,128,256 terminal 11 3K - 11 256 Unitno 28 2K - 120 32,64 vmem 2 160K - 3 ioctlops 0 0K - 158 16,32,64 iov 1 1K - 27 64,256,512 msg 4 30K - 4 2048,4096 sem 4 106K - 4 2048,4096 shm 1 20K - 1 tty 14 14K - 14 1024 shmfd 1 8K - 1 pcb 12 8341K - 12 16,128,1024,2048 vfscache 1 16384K - 1 vfs_hash 1 8192K - 1 vnodes 1 1K - 1 256 mount 194 7K - 402 16,32,64,128,256 vnodemarker 0 0K - 8 512 BPF 3 1K - 3 128 ifnet 4 7K - 4 128,2048 ifaddr 34 10K - 34 32,64,256,512,2048,4096 clone 8 1K - 8 128 arpcom 2 1K - 2 16 lltable 6 3K - 6 512 routetbl 5 2K - 5 256,512 igmp 3 1K - 3 256 sctp_vrf 1 1K - 1 64 hostcache 1 28K - 1 syncache 1 64K - 1 mld 3 1K - 3 128 rpc 2 1K - 2 512 audit_evclass 187 6K - 228 32 ufs_quota 1 8192K - 1 vm_pgdata 7 8193K - 7 128 UMAHash 19 10K - 19 512 pfs_nodes 73 19K - 73 256 pfs_vncache 2 1K - 2 64 scsi_cd 0 0K - 11 16 GEOM 404 67K - 3278 16,32,64,128,256,512,1024,2048 memdesc 1 4K - 1 4096 feeder 20 2K - 24 32,128 acpidev 36 3K - 36 64 atkbddev 2 1K - 2 64 CAM CCB 1 2K - 22943 2048 mixer 3 12K - 3 4096 raid_data 0 0K - 480 32,128,256 CAM path 22 1K - 110 32 solaris 326492 166906K - 672607 16,32,64,128,256,512,1024,2048,4096 UART 6 5K - 6 16,1024 kstat_data 6 1K - 6 64 CAM periph 16 4K - 40 16,32,64,128,256 md_nvidia_data 0 0K - 78 512 pci_link 16 2K - 16 32,64,128 md_sii_data 0 0K - 78 512 acpi_perf 8 1K - 8 64 CAM queue 23 8K - 57 16,32,512 apmdev 1 1K - 1 128 madt_table 0 0K - 1 4096 CAM dev queue 8 1K - 8 64 CAM SIM 8 2K - 8 256 ddb_capture 1 48K - 1 io_apic 2 4K - 2 2048 MCA 8 1K - 8 128 msi 4 1K - 4 128 nexusdev 4 1K - 4 16 isadev 5 1K - 5 128 USB 46 59K - 52 16,32,64,128,256,512,1024,2048,4096 USBdev 35 4K - 35 32,64,128,256 linux 17 2K - 17 64 spicds 7 1K - 7 128 envy24ht 15 195K - 15 64,2048 cpuctl 1 1K - 1 64 crypto 1 1K - 1 512 cyclic 32 3K - 32 16,64,128 SDT 32 2K - 32 16,64 fbt 60661 7839K - 60661 128 iprtheap 17 54K - 17 32,64,128,256 nvidia 194 864K - 195 16,32,64,128,256,512,1024,2048,4096 ipmi 0 0K - 11 128 gem_name 59 10K - 62 32,4096 drm_global 2 1K - 2 128,256 drm_vblank 7 1K - 7 16,64 drm_dma 2 1K - 2 32 drm_sarea 1 1K - 1 16 drm_driver 32 2203K - 36 16,32,64,128,256,512,1024,4096 drm_ctxbitmap 1 4K - 1 4096 drm_sman 13 2K - 13 128 drm_hashtab 1 4096K - 1 drm_kms 90 18K - 108 16,32,64,128,256,2048 ttm_pd 6 9K - 6 16,2048 ttm_rman 2 1K - 2 256 ttm_zone 2 1K - 2 64 ttm_poolmgr 1 1K - 1 512 fdesc_mount 1 1K - 1 16 ------------------------------------------------------------------------ vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 267, 3, 267, 0, 0 UMA Zones: 1664, 0, 267, 1, 267, 0, 0 UMA Slabs: 112, 0, 12325, 30, 14833, 0, 0 UMA RCntSlabs: 120, 0, 0, 0, 0, 0, 0 UMA Hash: 256, 0, 58, 32, 77, 0, 0 4 Bucket: 32, 0, 1296, 1184, 9186, 0, 0 6 Bucket: 48, 0, 310, 1018, 1646, 0, 0 8 Bucket: 64, 0, 103, 1199, 2674, 0, 0 12 Bucket: 96, 0, 64, 1084, 1327, 0, 0 16 Bucket: 128, 0, 100, 1171, 1676, 17, 0 32 Bucket: 256, 0, 240, 630, 4097, 49, 0 64 Bucket: 512, 0, 238, 287, 1922, 49, 0 128 Bucket: 1024, 0, 1190, 102, 6750, 0, 0 vmem btag: 56, 0, 14856, 409, 14954, 215, 0 VM OBJECT: 256, 0, 583, 407, 1328, 0, 0 RADIX NODE: 144, 0, 8773, 272, 13246, 0, 0 MAP: 240, 0, 3, 61, 3, 0, 0 KMAP ENTRY: 128, 0, 8, 271, 8, 0, 0 MAP ENTRY: 128, 0, 103, 1044, 1717, 0, 0 VMSPACE: 448, 0, 4, 172, 54, 0, 0 fakepg: 104, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 405, 0, 405, 0, 0 16: 16, 0, 6, 243, 6, 0, 0 16: 16, 0, 2762, 724, 3015, 0, 0 16: 16, 0, 154129, 749, 173890, 0, 0 16: 16, 0, 54, 693, 28214, 0, 0 16: 16, 0, 62, 436, 63, 0, 0 16: 16, 0, 690, 804, 5386, 0, 0 16: 16, 0, 30, 219, 30, 0, 0 16: 16, 0, 1, 746, 18, 0, 0 32: 32, 0, 30, 962, 116, 0, 0 32: 32, 0, 3092, 876, 3286, 0, 0 32: 32, 0, 74071, 577, 140388, 0, 0 32: 32, 0, 66, 430, 4216, 0, 0 32: 32, 0, 18, 1098, 98, 0, 0 32: 32, 0, 833, 903, 2868, 0, 0 32: 32, 0, 94, 526, 186, 0, 0 32: 32, 0, 8, 612, 164, 0, 0 64: 64, 0, 5, 181, 6, 0, 0 64: 64, 0, 516, 786, 586, 0, 0 64: 64, 0, 20714, 1172, 73124, 0, 0 64: 64, 0, 767, 659, 1637, 0, 0 64: 64, 0, 80, 106, 80, 0, 0 64: 64, 0, 9171, 749, 9688, 0, 0 64: 64, 0, 131, 303, 131, 0, 0 64: 64, 0, 1038, 884, 1071, 0, 0 128: 128, 0, 34, 865, 95, 0, 0 128: 128, 0, 1899, 364, 1908, 0, 0 128: 128, 0, 121894, 13359, 190707, 0, 0 128: 128, 0, 1077, 318, 27008, 0, 0 128: 128, 0, 112, 911, 192, 0, 0 128: 128, 0, 1877, 882, 4086, 0, 0 128: 128, 0, 29, 126, 30, 0, 0 128: 128, 0, 524, 499, 526, 0, 0 256: 256, 0, 2, 73, 2, 0, 0 256: 256, 0, 301, 254, 444, 0, 0 256: 256, 0, 7514, 4696, 54916, 0, 0 256: 256, 0, 64, 131, 740, 0, 0 256: 256, 0, 28, 287, 348, 0, 0 256: 256, 0, 903, 447, 1957, 0, 0 256: 256, 0, 31, 44, 31, 0, 0 256: 256, 0, 4, 251, 18, 0, 0 512: 512, 0, 2, 33, 9, 0, 0 512: 512, 0, 88, 206, 244, 0, 0 512: 512, 0, 347, 500, 78882, 0, 0 512: 512, 0, 16, 47, 35, 0, 0 512: 512, 0, 3, 32, 3, 0, 0 512: 512, 0, 327, 198, 2807, 0, 0 512: 512, 0, 47, 16, 47, 0, 0 512: 512, 0, 1, 62, 2, 0, 0 1024: 1024, 0, 0, 92, 99, 0, 0 1024: 1024, 0, 9, 19, 9, 0, 0 1024: 1024, 0, 8351, 125, 23312, 0, 0 1024: 1024, 0, 5, 23, 2074, 0, 0 1024: 1024, 0, 16, 12, 16, 0, 0 1024: 1024, 0, 230, 82, 1273, 0, 0 1024: 1024, 0, 2, 14, 2, 0, 0 1024: 1024, 0, 0, 0, 0, 0, 0 2048: 2048, 0, 2, 4, 9, 0, 0 2048: 2048, 0, 6, 4, 6, 0, 0 2048: 2048, 0, 11, 25, 39, 0, 0 2048: 2048, 0, 8, 58, 22955, 0, 0 2048: 2048, 0, 21, 9, 30, 0, 0 2048: 2048, 0, 48, 38, 466, 0, 0 2048: 2048, 0, 5, 1, 5, 0, 0 2048: 2048, 0, 0, 0, 0, 0, 0 4096: 4096, 0, 0, 0, 0, 0, 0 4096: 4096, 0, 5, 0, 5, 0, 0 4096: 4096, 0, 298, 2, 497, 0, 0 4096: 4096, 0, 10, 0, 12, 0, 0 4096: 4096, 0, 16, 0, 17, 0, 0 4096: 4096, 0, 25, 0, 415, 0, 0 4096: 4096, 0, 15, 0, 15, 0, 0 4096: 4096, 0, 349, 9, 1111, 0, 0 uint64 pcpu: 8, 0, 1398, 138, 1398, 0, 0 SLEEPQUEUE: 88, 0, 334, 565, 334, 0, 0 Files: 80, 0, 8, 1021, 395, 0, 0 rl_entry: 40, 0, 6, 588, 6, 0, 0 TURNSTILE: 136, 0, 334, 226, 334, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1208, 0, 26, 46, 75, 0, 0 THREAD: 1168, 0, 295, 38, 438, 0, 0 cpuset: 72, 0, 282, 488, 425, 0, 0 cyclic_id_cache: 64, 0, 0, 0, 0, 0, 0 audit_record: 1248, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 25720710, 0, 10, 0, 0, 0 mbuf: 256, 25720710, 1, 134, 1, 0, 0 mbuf_cluster: 2048, 4018860, 0, 0, 0, 0, 0 mbuf_jumbo_page: 4096, 2009429, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 1786158, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 1339616, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 sendfile_sync: 128, 0, 0, 0, 0, 0, 0 dtrace_state_cache: 4096, 0, 0, 0, 0, 0, 0 g_bio: 248, 0, 3, 669, 68266, 0, 0 DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0 ttyinq: 160, 0, 15, 57, 15, 0, 0 ttyoutq: 256, 0, 8, 67, 8, 0, 0 ata_request: 336, 0, 0, 88, 42, 0, 0 vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0, 0 nv_stack_t: 12288, 0, 2, 1, 6, 0, 0 FPU_save_area: 512, 0, 0, 0, 0, 0, 0 taskq_zone: 48, 0, 0, 664, 47, 0, 0 taskq_zone: 48, 0, 0, 0, 0, 0, 0 VNODE: 472, 0, 4245, 139, 4246, 0, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0, 0 BUF TRIE: 144, 0, 0, 105948, 0, 0, 0 NAMEI: 1024, 0, 0, 104, 887, 0, 0 S VFS Cache: 108, 0, 247, 908, 285, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 0, 0, 0, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 NCLNODE: 528, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 0, 0, 0, 0, 0 reference_cache: 40, 0, 0, 0, 0, 0, 0 reference_history_cache: 8, 0, 0, 0, 0, 0, 0 range_seg_cache: 64, 0, 15246, 81288, 176756, 0, 0 zio_cache: 920, 0, 2874, 3294, 146114, 0, 0 zio_link_cache: 48, 0, 2867, 4686, 145725, 0, 0 zio_buf_512: 512, 0, 4917, 221, 5752, 0, 0 zio_data_buf_512: 512, 0, 362, 247, 801, 0, 0 zio_buf_1024: 1024, 0, 95, 113, 269, 0, 0 zio_data_buf_1024: 1024, 0, 421, 99, 521, 0, 0 zio_buf_1536: 1536, 0, 35, 127, 335, 0, 0 zio_data_buf_1536: 1536, 0, 717, 85, 845, 0, 0 zio_buf_2048: 2048, 0, 45, 91, 262, 0, 0 zio_data_buf_2048: 2048, 0, 295, 51, 408, 0, 0 zio_buf_2560: 2560, 0, 11, 70, 131, 0, 0 zio_data_buf_2560: 2560, 0, 202, 32, 291, 0, 0 zio_buf_3072: 3072, 0, 4, 53, 79, 0, 0 zio_data_buf_3072: 3072, 0, 151, 29, 225, 0, 0 zio_buf_3584: 3584, 0, 5, 28, 49, 0, 0 zio_data_buf_3584: 3584, 0, 106, 11, 153, 0, 0 zio_buf_4096: 4096, 0, 3767, 1199, 45373, 0, 0 zio_data_buf_4096: 4096, 0, 97, 116, 2911, 0, 0 zio_buf_5120: 5120, 0, 2, 55, 89, 0, 0 zio_data_buf_5120: 5120, 0, 185, 21, 272, 0, 0 zio_buf_6144: 6144, 0, 5, 35, 62, 0, 0 zio_data_buf_6144: 6144, 0, 150, 13, 206, 0, 0 zio_buf_7168: 7168, 0, 2, 37, 51, 0, 0 zio_data_buf_7168: 7168, 0, 110, 10, 164, 0, 0 zio_buf_8192: 8192, 0, 2, 286, 1766, 0, 0 zio_data_buf_8192: 8192, 0, 102, 9, 192, 0, 0 zio_buf_10240: 10240, 0, 1, 49, 59, 0, 0 zio_data_buf_10240: 10240, 0, 145, 10, 215, 0, 0 zio_buf_12288: 12288, 0, 1, 176, 1011, 0, 0 zio_data_buf_12288: 12288, 0, 102, 10, 149, 0, 0 zio_buf_14336: 14336, 0, 0, 24, 27, 0, 0 zio_data_buf_14336: 14336, 0, 67, 12, 100, 0, 0 zio_buf_16384: 16384, 0, 718, 40, 2238, 0, 0 zio_data_buf_16384: 16384, 0, 57, 8, 86, 0, 0 zio_buf_20480: 20480, 0, 0, 57, 571, 0, 0 zio_data_buf_20480: 20480, 0, 59, 7, 102, 0, 0 zio_buf_24576: 24576, 0, 0, 40, 565, 0, 0 zio_data_buf_24576: 24576, 0, 44, 6, 67, 0, 0 zio_buf_28672: 28672, 0, 0, 131, 1560, 0, 0 zio_data_buf_28672: 28672, 0, 40, 13, 60, 0, 0 zio_buf_32768: 32768, 0, 0, 27, 290, 0, 0 zio_data_buf_32768: 32768, 0, 31, 5, 43, 0, 0 zio_buf_36864: 36864, 0, 1, 16, 244, 0, 0 zio_data_buf_36864: 36864, 0, 32, 3, 40, 0, 0 zio_buf_40960: 40960, 0, 0, 14, 139, 0, 0 zio_data_buf_40960: 40960, 0, 18, 1, 19, 0, 0 zio_buf_45056: 45056, 0, 0, 13, 99, 0, 0 zio_data_buf_45056: 45056, 0, 21, 3, 25, 0, 0 zio_buf_49152: 49152, 0, 0, 18, 89, 0, 0 zio_data_buf_49152: 49152, 0, 16, 3, 22, 0, 0 zio_buf_53248: 53248, 0, 1, 17, 148, 0, 0 zio_data_buf_53248: 53248, 0, 22, 3, 26, 0, 0 zio_buf_57344: 57344, 0, 0, 11, 73, 0, 0 zio_data_buf_57344: 57344, 0, 14, 1, 15, 0, 0 zio_buf_61440: 61440, 0, 0, 12, 56, 0, 0 zio_data_buf_61440: 61440, 0, 10, 3, 13, 0, 0 zio_buf_65536: 65536, 0, 0, 11, 46, 0, 0 zio_data_buf_65536: 65536, 0, 4, 1, 5, 0, 0 zio_buf_69632: 69632, 0, 0, 8, 27, 0, 0 zio_data_buf_69632: 69632, 0, 8, 1, 9, 0, 0 zio_buf_73728: 73728, 0, 0, 9, 27, 0, 0 zio_data_buf_73728: 73728, 0, 5, 0, 5, 0, 0 zio_buf_77824: 77824, 0, 0, 9, 41, 0, 0 zio_data_buf_77824: 77824, 0, 6, 1, 7, 0, 0 zio_buf_81920: 81920, 0, 0, 13, 84, 0, 0 zio_data_buf_81920: 81920, 0, 9, 1, 10, 0, 0 zio_buf_86016: 86016, 0, 0, 8, 25, 0, 0 zio_data_buf_86016: 86016, 0, 5, 3, 8, 0, 0 zio_buf_90112: 90112, 0, 0, 10, 33, 0, 0 zio_data_buf_90112: 90112, 0, 3, 1, 4, 0, 0 zio_buf_94208: 94208, 0, 0, 6, 15, 0, 0 zio_data_buf_94208: 94208, 0, 4, 1, 5, 0, 0 zio_buf_98304: 98304, 0, 1, 5, 20, 0, 0 zio_data_buf_98304: 98304, 0, 9, 0, 9, 0, 0 zio_buf_102400: 102400, 0, 0, 7, 13, 0, 0 zio_data_buf_102400: 102400, 0, 2, 1, 3, 0, 0 zio_buf_106496: 106496, 0, 0, 16, 99, 0, 0 zio_data_buf_106496: 106496, 0, 12, 2, 14, 0, 0 zio_buf_110592: 110592, 0, 0, 12, 72, 0, 0 zio_data_buf_110592: 110592, 0, 6, 1, 7, 0, 0 zio_buf_114688: 114688, 0, 0, 22, 79, 0, 0 zio_data_buf_114688: 114688, 0, 6, 0, 6, 0, 0 zio_buf_118784: 118784, 0, 0, 8, 22, 0, 0 zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0 zio_buf_122880: 122880, 0, 0, 9, 28, 0, 0 zio_data_buf_122880: 122880, 0, 6, 0, 6, 0, 0 zio_buf_126976: 126976, 0, 0, 10, 22, 0, 0 zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0 zio_buf_131072: 131072, 0, 160, 79, 862, 0, 0 zio_data_buf_131072: 131072, 0, 383, 8, 556, 0, 0 lz4_ctx: 16384, 0, 0, 0, 0, 0, 0 sa_cache: 80, 0, 4199, 848, 4199, 0, 0 dnode_t: 1096, 0, 4649, 31, 5833, 0, 0 dmu_buf_impl_t: 336, 0, 9678, 68, 14441, 0, 0 arc_buf_hdr_t: 328, 0, 10162, 134, 11502, 0, 0 arc_buf_t: 72, 0, 9396, 614, 12289, 0, 0 zil_lwb_cache: 192, 0, 0, 0, 0, 0, 0 zfs_znode_cache: 368, 0, 4199, 151, 4199, 0, 0 pipe: 744, 0, 0, 55, 13, 0, 0 Mountpoints: 816, 0, 17, 23, 17, 0, 0 procdesc: 128, 0, 0, 0, 0, 0, 0 ksiginfo: 112, 0, 24, 1026, 24, 0, 0 itimer: 352, 0, 0, 0, 0, 0, 0 KNOTE: 128, 0, 0, 0, 0, 0, 0 socket: 696, 2062880, 0, 0, 0, 0, 0 ipq: 56, 125599, 0, 0, 0, 0, 0 udp_inpcb: 392, 2062880, 0, 0, 0, 0, 0 udpcb: 16, 2062965, 0, 0, 0, 0, 0 tcp_inpcb: 392, 2062880, 0, 0, 0, 0, 0 tcpcb: 1024, 2062880, 0, 0, 0, 0, 0 tcptw: 88, 27810, 0, 0, 0, 0, 0 syncache: 160, 15360, 0, 0, 0, 0, 0 hostcache: 136, 15370, 0, 0, 0, 0, 0 tcpreass: 40, 251262, 0, 0, 0, 0, 0 sackhole: 32, 0, 0, 0, 0, 0, 0 sctp_ep: 1408, 2062880, 0, 0, 0, 0, 0 sctp_asoc: 2352, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80012, 0, 0, 0, 0, 0 sctp_raddr: 728, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400026, 0, 0, 0, 0, 0 sctp_readq: 104, 400026, 0, 0, 0, 0, 0 sctp_stream_msg_out: 104, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400059, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 ripcb: 392, 2062880, 0, 0, 0, 0, 0 unpcb: 240, 2062880, 0, 0, 0, 0, 0 rtentry: 200, 0, 0, 0, 0, 0, 0 selfd: 56, 0, 0, 0, 0, 0, 0 SWAPMETA: 288, 8037731, 0, 0, 0, 0, 0 ------------------------------------------------------------------------ vmstat -i interrupt total rate irq6: fdc0 22 0 irq14: ata0 61 0 irq17: uhci0 ehci0 54 0 irq18: uhci2+ 306 2 irq19: uhci1 ahci0+ 21712 146 cpu0:timer 3524 23 irq259: hdac0 9 0 cpu7:timer 2042 13 cpu1:timer 1450 9 cpu3:timer 3609 24 cpu2:timer 1414 9 cpu6:timer 1949 13 cpu4:timer 1408 9 cpu5:timer 1575 10 Total 39135 264 ------------------------------------------------------------------------ pstat -T 8/2062880 files 0M/147455M swap space ------------------------------------------------------------------------ pstat -s Device 512-blocks Used Avail Capacity /dev/#C:0xc0 50331392 0 50331392 0% /dev/ad14p2 50331392 0 50331392 0% /dev/usb/3.2.1 50331392 0 50331392 0% /dev/#C:0xd0 50331392 0 50331392 0% /dev/#C:0xd8 50331392 0 50331392 0% /dev/#C:0xe0 50331392 0 50331392 0% Total 301988352 0 301988352 0% ------------------------------------------------------------------------ iostat tty ada0 ada1 ada2 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 9 15.20 161 2.39 14.99 169 2.47 15.24 167 2.49 0 0 4 1 96 ------------------------------------------------------------------------ ipcs -a Message Queues: T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME Shared Memory: T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Semaphores: T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME ------------------------------------------------------------------------ ipcs -T msginfo: msgmax: 16384 (max characters in a message) msgmni: 40 (# of message queues) msgmnb: 2048 (max characters in a message queue) msgtql: 40 (max # of messages in system) msgssz: 8 (size of a message segment) msgseg: 2048 (# of message segments in system) shminfo: shmmax: 536870912 (max shared memory segment size) shmmin: 1 (min shared memory segment size) shmmni: 192 (max number of shared memory identifiers) shmseg: 128 (max shared memory segments per process) shmall: 131072 (max amount of shared memory in pages) seminfo: semmni: 50 (# of semaphore identifiers) semmns: 340 (# of semaphores in system) semmnu: 150 (# of undo structures in system) semmsl: 340 (max # of semaphores per id) semopm: 100 (max # of operations per semop call) semume: 50 (max # of undo entries per process) semusz: 632 (size in bytes of undo structure) semvmx: 32767 (semaphore maximum value) semaem: 16384 (adjust on exit max value) ------------------------------------------------------------------------ nfsstat Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 2 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 1 1 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 4 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 0 0 0 0 0 0 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 0 0 0 0 0 0 0 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 0 0 0 ------------------------------------------------------------------------ netstat -s tcp: 0 packets sent 0 data packets (0 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 0 ack-only packets (0 delayed) 0 URG only packets 0 window probe packets 0 window update packets 0 control packets 0 packets received 0 acks (for 0 bytes) 0 duplicate acks 0 acks for unsent data 0 packets (0 bytes) received in-sequence 0 completely duplicate packets (0 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 0 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 0 connection requests 0 connection accepts 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 0 connections established (including accepts) 0 connections closed (including 0 drops) 0 connections updated cached RTT on close 0 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 0 embryonic connections dropped 0 segments updated rtt (of 0 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 0 correct ACK header predictions 0 correct data packet header predictions 0 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 0 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 0 cookies sent 0 cookies received 0 hostcache entries added 0 bucket overflow 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window udp: 0 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 0 with no checksum 0 dropped due to no socket 0 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 0 delivered 0 datagrams output 0 times multicast source filter matched ip: 0 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 0 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for
Responsible Changed From-To: freebsd-bugs->freebsd-fs Over to maintainer(s).
I've wound up commenting out a bunch of these ASSERTS, and haven't seen any negative consequences, HOWEVER, I'd like someone to let me know how I could look at the FS(s)/POOL and see if there is a real issue.
I'm STILL seeing these, and have commented out: =================================================================== --- zap.c (revision 270765) +++ zap.c (working copy) @@ -476,7 +476,9 @@ * chain. There should be no chained leafs (as we have removed * support for them). */ +#if 0 /*LER: to see what else blows up */ ASSERT0(l->l_phys->l_hdr.lh_pad1); +#endif /* * There should be more hash entries than there can be @@ -531,7 +533,9 @@ ASSERT3U(l->l_blkid, ==, blkid); ASSERT3P(l->l_dbuf, ==, db); ASSERT3P(l->l_phys, ==, l->l_dbuf->db_data); +#if 0 /* LER */ ASSERT3U(l->l_phys->l_hdr.lh_block_type, ==, ZBT_LEAF); +#endif ASSERT3U(l->l_phys->l_hdr.lh_magic, ==, ZAP_LEAF_MAGIC); *lp = l; borg.lerctr.org /usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs $ Can I PLEASE get someone to look at these, and tell me what I need to do to fix the on-disk image?
Created attachment 146572 [details] Latest panic core.txt. I **DO** have the vmcore available
If you look at the whole comment for one of the assertions you disabled: /* * lhr_pad was previously used for the next leaf in the leaf * chain. There should be no chained leafs (as we have removed * support for them). */ ASSERT0(l->l_phys->l_hdr.lh_pad1); Or looking at the annotation: 168404 pjd /* 168404 pjd * lhr_pad was previously used for the next leaf in the leaf 168404 pjd * chain. There should be no chained leafs (as we have removed 168404 pjd * support for them). 168404 pjd */ 240415 mm ASSERT0(l->l_phys->l_hdr.lh_pad1); One of those is the initial commit: ------------------------------------------------------------------------ r168404 | pjd | 2007-04-05 18:09:06 -0700 (Thu, 05 Apr 2007) | 11 lines Please welcome ZFS - The last word in file systems. ZFS file system was ported from OpenSolaris operating system. The code in under CDDL license. ------------------------------------------------------------------------ The second is: ------------------------------------------------------------------------ r240415 | mm | 2012-09-12 11:05:43 -0700 (Wed, 12 Sep 2012) | 21 lines Merge recent zfs vendor changes, sync code and adjust userland DEBUG. ------------------------------------------------------------------------ The change is just a syntax one: - ASSERT3U(l->l_phys->l_hdr.lh_pad1, ==, 0); + ASSERT0(l->l_phys->l_hdr.lh_pad1); So.. ever since ZFS existed in FreeBSD, that field should be zero, on disk. You are tripping those asserts because it isn't the case on your machine. This suggests to me that you've either got a very old file system that pre-dates freebsd, or you've got some evil on-disk corruption. I suspect the latter. If this was my machine, I'd be contemplating a backup/restore. I see from the disk sizes you may have a lot of data, but that's what I would be doing. I also see the stack traces come from the zfs de-dupe code. There are a great number of people who have a very dim view of that code, even the authors apologized for it. If it were my machine, I would be turning off dedup immediately, if not sooner.
This pool has been around a LONG time -- I'm wondering where I can borrow a Terabyte of disk to backup what I care about and rebuild it one more time... (this pool started on FreeBSD 8(IIRC) on 400G disk, and got upgraded....)
Out of curiosity, what does "zpool status -D poolname" show? http://serverfault.com/questions/533877/how-large-is-my-zfs-dedupe-table-at-the-moment You might be burning a significant chunk of ARC metadata space to hold the dedupe table in memory. If I had to guess at this point, I would guess that there are a couple of corrupt records in the on-disk dedup table for one or more of your data sets. Since this is an old pool, there's been plenty of opportunities over the years for this to accumulate while zfs support matured. From your comment, it sounds like you have 1TB of data spread over a pool made of 2TB disks? Is that deduplicated? Is one of the drives a spare? (if so, can you take the spare offline and make a pool on it to facilitate a dump/restore?)
borg.lerctr.org /home/ler $ zpool status -D zroot pool: zroot state: ONLINE scan: scrub repaired 0 in 5h8m with 0 errors on Sat Aug 30 13:53:49 2014 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/71e84d97-043a-11e2-8cd6-003048f2299c ONLINE 0 0 0 gptid/3540b18c-0db2-11e2-8ef2-003048f2299c ONLINE 0 0 0 gptid/b0a645ed-0db9-11e2-8ef2-003048f2299c ONLINE 0 0 0 gptid/dc133fc1-0dc0-11e2-8ef2-003048f2299c ONLINE 0 0 0 gptid/21bef448-0dc8-11e2-8ef2-003048f2299c ONLINE 0 0 0 gptid/46033e1c-0432-11e2-8cd6-003048f2299c ONLINE 0 0 0 errors: No known data errors dedup: DDT entries 1415219, size 2676 on disk, 540 in core bucket allocated referenced ______ ______________________________ ______________________________ refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE ------ ------ ----- ----- ----- ------ ----- ----- ----- 1 1.26M 116G 66.3G 69.2G 1.26M 116G 66.3G 69.2G 2 86.1K 6.84G 6.04G 6.20G 174K 13.8G 12.1G 12.4G 4 3.15K 253M 243M 249M 14.6K 1.00G 985M 1019M 8 196 2.32M 823K 1.78M 1.96K 24.5M 8.42M 18.4M 16 109 1.23M 1024K 1.52M 2.50K 31.9M 26.2M 38.2M 32 33 330K 318K 498K 1.37K 18.1M 17.5M 24.9M 64 3 1.50K 1.50K 19.2K 249 124K 124K 1.55M 128 3 28K 28K 38.3K 470 4.96M 4.96M 6.40M 256 4 162K 162K 173K 1.33K 64.3M 64.3M 67.6M Total 1.35M 123G 72.6G 75.7G 1.45M 131G 79.5G 82.8G borg.lerctr.org /home/ler $ I'm actually using 500G on a 6 disk raid-Z1 pool of 2TB disks. That's the "logical Used" (approx). I went out and bought a 4TB NAS to use for this, and then for other things.
"dedup: DDT entries 1415219, size 2676 on disk, 540 in core" -> 728MB of ram to cache this. Fortunately that's a relatively small portion of your system ram. I would still be doing a zfs set dedup=off anywhere it's on. While it won't fix the on-disk dedup corruption, at least you won't risk adding to it any more. Your system is crashing trying to parse the on-disk dedup records. That's what the *_ddt_* functions are about in the traces. I can't imagine this ending well. Does a scrub find the problem? or does the scrub trigger the assertions?
scrub triggers the assertions..... currently rsyncing the volatile stuff to the new 4TB NAS mounted via NFS.... then will blow this system up and rebuild it withOUT dedupe....
Rebuilding ports -- no more assertions on a non-modified kernel. Thanks Peter Wemm for the input on dedup. I think we can close this one out.
Looks like an OLD dedup corruption/mis-feature -- rebuilding the entire system from scratch cleaned it up.