Bug 186112 - [zfs] [panic] ZFS Panic/Solaris Assert/zap.c:479
Summary: [zfs] [panic] ZFS Panic/Solaris Assert/zap.c:479
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-01-25 22:40 UTC by Larry Rosenman
Modified: 2014-08-31 20:49 UTC (History)
1 user (show)

See Also:


Attachments
Latest panic core.txt. I **DO** have the vmcore available (93.96 KB, text/plain)
2014-08-30 21:04 UTC, Larry Rosenman
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Larry Rosenman freebsd_committer freebsd_triage 2014-01-25 22:40:01 UTC
Crashes on below rev stays up on above rev

borg.lerctr.org dumped core - see /var/crash/vmcore.2

Sat Jan 25 15:25:30 CST 2014

FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #3 r261166: Sat Jan 25 14:55:24 CST 2014     root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER  amd64

panic: solaris assert: l->l_phys->l_hdr.lh_pad1 == 0 (0xffff000000000000 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c, line: 479

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #3 r261166: Sat Jan 25 14:55:24 CST 2014
    root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
info: [drm] Initialized drm 1.1.0 20060810
CPU: Intel(R) Xeon(R) CPU           E5410  @ 2.33GHz (2327.55-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x10676  Family=0x6  Model=0x17  Stepping=6
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 65641644032 (62600 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
random: <Software, Yarrow> initialized
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2000-0x201f mem 0xd9220000-0xd923ffff,0xd9200000-0xd921ffff irq 18 at device 0.0 on pci6
em0: Using an MSI interrupt
em0: Ethernet address: 00:30:48:f2:29:9c
em1: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2020-0x203f mem 0xd9260000-0xd927ffff,0xd9240000-0xd925ffff irq 19 at device 0.1 on pci6
em1: Using an MSI interrupt
em1: Ethernet address: 00:30:48:f2:29:9d
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> port 0x3000-0x307f mem 0xd8000000-0xd8ffffff,0xc0000000-0xc7ffffff,0xc8000000-0xc9ffffff irq 16 at device 0.0 on pci8
nvidia0: <GeForce GT 610> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
hdac0: <NVIDIA GF119 HDA Controller> mem 0xd9000000-0xd9003fff irq 17 at device 0.1 on pci8
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> irq 16 at device 0.0 on pci10
pci11: <PCI bus> on pcib11
pcm0: <Envy24HT audio (Generic)> port 0x4080-0x409f,0x4000-0x407f irq 16 at device 0.0 on pci11
pcm0: system configuration
  SubVendorID: 0x1412, SubDeviceID: 0x2403
  XIN2 Clock Source: 24.576MHz(96kHz*256)
  MPU-401 UART(s) #: not implemented
  ADC #: 1 and SPDIF receiver connected
  DAC #: 4
  Multi-track converter type: AC'97(SDATA_OUT:packed)
  S/PDIF(IN/OUT): 1/1 ID# 0x00
  GPIO(mask/dir/state): 0xff/0xff/0xff
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
usbus0 on uhci0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
usbus1 on uhci1
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
usbus2 on uhci2
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd9600400-0xd96007ff irq 17 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci12: <ACPI PCI bus> on pcib12
vgapci1: <VGA-compatible display> port 0x5000-0x50ff mem 0xd0000000-0xd7ffffff,0xd9300000-0xd930ffff irq 18 at device 1.0 on pci12
drmn1: <ATI ES1000 RN50> on vgapci1
info: [drm] RADEON_IS_PCI
info: [drm] initializing kernel modesetting (RV100 0x1002:0x515E 0x15D9:0x8080).
info: [drm] register mmio base: 0xD9300000
info: [drm] register mmio size: 65536
info: [drm] radeon_atrm_get_bios: ===> Try ATRM...
info: [drm] radeon_atrm_get_bios: pci_find_class() found: 0:8:0:0, vendor=10de, device=104a
info: [drm] radeon_atrm_get_bios: Get ACPI device handle
info: [drm] radeon_acpi_vfct_bios: ===> Try VFCT...
info: [drm] radeon_acpi_vfct_bios: Get "VFCT" ACPI table
info: [drm] radeon_acpi_vfct_bios: Failed to get "VFCT" table: AE_NOT_FOUND
info: [drm] igp_read_bios_from_vram: ===> Try IGP's VRAM...
info: [drm] igp_read_bios_from_vram: VRAM base address: 0xd0000000
info: [drm] igp_read_bios_from_vram: Map address: 0xfffff800d0000000 (262144 bytes)
info: [drm] igp_read_bios_from_vram: Incorrect BIOS signature: 0x0000
info: [drm] radeon_read_bios: ===> Try PCI Expansion ROM...
info: [drm] radeon_read_bios: Map address: 0xfffff800000c0000 (131072 bytes)
drmn1: info: VRAM: 128M 0x00000000D0000000 - 0x00000000D7FFFFFF (16M used)
drmn1: info: GTT: 512M 0x00000000B0000000 - 0x00000000CFFFFFFF
info: [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
info: [drm] Driver supports precise vblank timestamp query.
info: [drm] radeon: irq initialized.
info: [drm] Detected VRAM RAM=128M, BAR=128M
info: [drm] RAM width 64bits SDR
[TTM] Zone  kernel: Available graphics memory: 33006090 kiB
[TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[TTM] Initializing pool allocator
info: [drm] radeon: 16M of VRAM memory ready
info: [drm] radeon: 512M of GTT memory ready.
info: [drm] GART: num cpu pages 131072, num gpu pages 131072
info: [drm] PCI GART of 512M enabled (table at 0x0000000011200000).
drmn1: info: WB disabled
drmn1: info: fence driver on ring 0 use gpu addr 0x00000000b0000000 and cpu addr 0x0xfffff8001083c000
info: [drm] Loading R100 Microcode
info: [drm] radeon: ring at 0x00000000B0001000
info: [drm] ring test succeeded in 1 usecs
info: [drm] ib test succeeded in 0 usecs
info: [drm] radeon_device_init: Taking over the fictitious range 0xd0000000-0xd4000000
iicbus0: <Philips I2C bus> on iicbb0 addr 0xff
iic0: <I2C generic I/O> on iicbus0
iicbus1: <Philips I2C bus> on iicbb1 addr 0xff
iic1: <I2C generic I/O> on iicbus1
iicbus2: <Philips I2C bus> on iicbb2 addr 0xff
iic2: <I2C generic I/O> on iicbus2
iicbus3: <Philips I2C bus> on iicbb3 addr 0xff
iic3: <I2C generic I/O> on iicbus3
info: [drm] Radeon Display Connectors
info: [drm] Connector 0:
info: [drm]   VGA-1
info: [drm]   DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
info: [drm]   Encoders:
info: [drm]     CRT1: INTERNAL_DAC1
info: [drm] Connector 1:
info: [drm]   DVI-I-1
info: [drm]   HPD2
info: [drm]   DDC: 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c
info: [drm]   Encoders:
info: [drm]     CRT2: INTERNAL_DAC2
info: [drm]     DFP2: INTERNAL_DVO1
composite sync not supported
composite sync not supported
info: [drm] fb mappable at 0xD0040000
info: [drm] vram apper at 0xD0000000
info: [drm] size 2076672
info: [drm] fb depth is 8
info: [drm]    pitch is 1920
fbd1 on drmn1
vt_allocate: Replace existing VT driver.
info: [drm] Initialized radeon 2.29.0 20080528
vgapci1: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ahci0: <Intel ESB2 AHCI SATA controller> port 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f mem 0xd9600800-0xd9600bff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x1100-0x111f irq 19 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_button0: <Power Button> on acpi0
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ichwd0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x387 offMax=0x50f
hdacc0: <NVIDIA (0x001c) HDA CODEC> at cad 0 on hdac0
hdaa0: <NVIDIA (0x001c) Audio Function Group> at nid 1 on hdacc0
pcm1: <NVIDIA (0x001c) (HDMI/DP 8ch)> at nid 4 on hdaa0
pcm2: <NVIDIA (0x001c) (HDMI/DP 8ch)> at nid 5 on hdaa0
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen3.1: <Intel> at usbus3
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen2.1: <Intel> at usbus2
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen1.1: <Intel> at usbus1
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen0.1: <Intel> at usbus0
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ipmi0: IPMI device rev. 1, firmware rev. 1.64, version 2.0
ata0: DMA limited to UDMA33, controller found non-ATA66 cable
ipmi0: Number of channels 8
ipmi0: Attached watchdog
uhub2: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada0: Serial Number 5YDA1ZL4
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
ada1: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada1: Serial Number 5YD6FPLG
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada2: Serial Number 5YDA3PC5
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada3: Serial Number 5YD9Y0P4
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada4: Serial Number 5YDA1R5W
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: quirks=0x1<4K>
ada4: Previously was known as ad12
ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
ada5: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada5: Serial Number 5YD5RBS8
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5: quirks=0x1<4K>
ada5: Previously was known as ad14
cd0 at ata0 bus 0 scbus0 target 0 lun 0
cd0: <Memorex DVD+-RAM 510L v1 MWS7> Removable CD-ROM SCSI-0 device 
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
Netvsc initializing... done!
SMP: AP CPU #7 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
Root mount waiting for: usbus3
uhub0: 6 ports with 6 removable, self powered
Root mount waiting for: usbus3
Root mount waiting for: usbus3
ugen3.2: <Peppercon AG> at usbus3
ukbd0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
kbd2 at ukbd0
Trying to mount root from zfs:zroot/ROOT/default []...
ugen0.2: <American Power Conversion> at usbus0
ugen1.2: <Prolific Technology Inc.> at usbus1
<118>Setting hostuuid: 53d19f64-d663-a017-8922-0030488e9ff3.
<118>Setting hostid: 0xf53a926e.
<118>Entropy harvesting: interrupts ethernet point_to_point swi.
<118>Starting file system checks:
<118>Mounting local file systems:.
panic: solaris assert: l->l_phys->l_hdr.lh_pad1 == 0 (0xffff000000000000 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c, line: 479
cpuid = 7
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe100c55d350
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe100c55d400
vpanic() at vpanic+0x126/frame 0xfffffe100c55d440
panic() at panic+0x43/frame 0xfffffe100c55d4a0
assfail3() at assfail3+0x2f/frame 0xfffffe100c55d4c0
zap_get_leaf_byblk() at zap_get_leaf_byblk+0x2ea/frame 0xfffffe100c55d510
zap_deref_leaf() at zap_deref_leaf+0xd1/frame 0xfffffe100c55d560
fzap_length() at fzap_length+0x31/frame 0xfffffe100c55d5d0
zap_length_uint64() at zap_length_uint64+0xbf/frame 0xfffffe100c55d630
ddt_zap_lookup() at ddt_zap_lookup+0x3d/frame 0xfffffe100c55d790
ddt_lookup() at ddt_lookup+0x17e/frame 0xfffffe100c55d960
zio_ddt_write() at zio_ddt_write+0xf2/frame 0xfffffe100c55dad0
zio_execute() at zio_execute+0x1eb/frame 0xfffffe100c55db30
taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame 0xfffffe100c55db80
taskqueue_thread_loop() at taskqueue_thread_loop+0x9b/frame 0xfffffe100c55dbb0
fork_exit() at fork_exit+0x84/frame 0xfffffe100c55dbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe100c55dbf0
--- trap 0, rip = 0, rsp = 0xfffffe100c55dcb0, rbp = 0 ---
KDB: enter: panic
Uptime: 23s
Dumping 2330 out of 64465 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
Loaded symbols for /boot/kernel/if_lagg.ko.symbols
Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel/ichsmb.ko.symbols
Reading symbols from /boot/kernel/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel/smbus.ko.symbols
Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
Loaded symbols for /boot/kernel/ichwd.ko.symbols
Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
Loaded symbols for /boot/kernel/cpuctl.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel/cryptodev.ko.symbols
Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
Loaded symbols for /boot/kernel/dtraceall.ko.symbols
Reading symbols from /boot/kernel/profile.ko.symbols...done.
Loaded symbols for /boot/kernel/profile.ko.symbols
Reading symbols from /boot/kernel/cyclic.ko.symbols...done.
Loaded symbols for /boot/kernel/cyclic.ko.symbols
Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
Loaded symbols for /boot/kernel/dtrace.ko.symbols
Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
Reading symbols from /boot/kernel/systrace.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace.ko.symbols
Reading symbols from /boot/kernel/sdt.ko.symbols...done.
Loaded symbols for /boot/kernel/sdt.ko.symbols
Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
Loaded symbols for /boot/kernel/lockstat.ko.symbols
Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
Loaded symbols for /boot/kernel/fasttrap.ko.symbols
Reading symbols from /boot/kernel/fbt.ko.symbols...done.
Loaded symbols for /boot/kernel/fbt.ko.symbols
Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi.ko.symbols
Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkms.ko.symbols
Reading symbols from /boot/kernel/iicbb.ko.symbols...done.
Loaded symbols for /boot/kernel/iicbb.ko.symbols
Reading symbols from /boot/kernel/iicbus.ko.symbols...done.
Loaded symbols for /boot/kernel/iicbus.ko.symbols
Reading symbols from /boot/kernel/iic.ko.symbols...done.
Loaded symbols for /boot/kernel/iic.ko.symbols
Reading symbols from /boot/kernel/drm2.ko.symbols...done.
Loaded symbols for /boot/kernel/drm2.ko.symbols
Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols
Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
Loaded symbols for /boot/kernel/fdescfs.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
#0  doadump (textdump=1) at pcpu.h:219
219	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=1) at pcpu.h:219
#1  0xffffffff809b9c67 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:452
#2  0xffffffff809ba175 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff809ba1c3 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:688
#4  0xffffffff80327abf in assfail3 (a=<value optimized out>, 
    lv=<value optimized out>, op=<value optimized out>, 
    rv=<value optimized out>, f=<value optimized out>, 
    l=<value optimized out>)
    at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
#5  0xffffffff803bac1a in zap_get_leaf_byblk (zap=<value optimized out>, 
    blkid=4858, tx=0x0, lt=RW_READER, lp=0xfffffe100c55d5a0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:479
#6  0xffffffff803b84f1 in zap_deref_leaf (zap=0xfffff800220bf680, 
    h=1610088556361285632, tx=0x0, lt=RW_READER, lp=0xfffffe100c55d5a0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:585
#7  0xffffffff803b9481 in fzap_length (zn=0xfffff80028e95600, 
    integer_size=0xfffffe100c55d658, num_integers=0xfffffe100c55d650)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:897
#8  0xffffffff803bf5bf in zap_length_uint64 (os=<value optimized out>, 
    zapobj=<value optimized out>, key=<value optimized out>, 
    key_numints=<value optimized out>, integer_size=0xfffffe100c55d658, 
    num_integers=0xfffffe100c55d650)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:934
#9  0xffffffff80352e7d in ddt_zap_lookup (os=0xfffff80020793c00, object=155, 
    dde=0xfffff80028e96600)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/ddt_zap.c:64
#10 0xffffffff8035146e in ddt_lookup (ddt=0xfffffe000d515000, 
    bp=<value optimized out>, add=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/ddt.c:178
#11 0xffffffff803fb242 in zio_ddt_write (zio=0xfffff800288a0730)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2242
#12 0xffffffff803f893b in zio_execute (zio=0xfffff800288a0730)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1355
#13 0xffffffff80a01970 in taskqueue_run_locked (queue=0xfffff80020127000)
    at /usr/src/sys/kern/subr_taskqueue.c:355
#14 0xffffffff80a022cb in taskqueue_thread_loop (arg=<value optimized out>)
    at /usr/src/sys/kern/subr_taskqueue.c:576
#15 0xffffffff80989324 in fork_exit (
    callout=0xffffffff80a02230 <taskqueue_thread_loop>, 
    arg=0xfffff800200fd4e0, frame=0xfffffe100c55dc00)
    at /usr/src/sys/kern/kern_fork.c:977
#16 0xffffffff80d95b0e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:605
#17 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------
ps -axlww

UID PID PPID CPU PRI NI   VSZ RSS MWCHAN   STAT TT     TIME COMMAND
  0   0    0   0  -8  0     0   0 -        DLs   -  0:00.10 [kernel]
  0   1    0   0  24  0  9368   0 wait     DLs   -  0:00.00 [init]
  0   2    0   0 -16  0     0   0 ftcl     DL    -  0:00.00 [ftcleanup]
  0   3    0   0 -16  0     0   0 crypto_w DL    -  0:00.00 [crypto]
  0   4    0   0 -16  0     0   0 crypto_r DL    -  0:00.00 [crypto returns]
  0   5    0   0 -16  0     0   0 ccb_scan DL    -  0:00.00 [cam]
  0   6    0   0 -16  0     0   0 -        DL    -  0:00.00 [fdc0]
  0   7    0   0  -8  0     0   0 zio->io_ DL    -  0:00.00 [zfskern]
  0   8    0   0 -16  0     0   0 waiting_ DL    -  0:00.00 [sctp_iterator]
  0   9    0   0 -16  0     0   0 ipmireq  DL    -  0:00.00 [ipmi0: kcs]
  0  10    0   0 -16  0     0   0 audit_wo DL    -  0:00.00 [audit]
  0  11    0   0 155  0     0   0 -        RL    -  0:00.00 [idle]
  0  12    0   0 -84  0     0   0 -        WL    -  0:00.00 [intr]
  0  13    0   0  -8  0     0   0 -        DL    -  0:00.00 [geom]
  0  14    0   0 -16  0     0   0 -        DL    -  0:00.00 [rand_harvestq]
  0  15    0   0 -68  0     0   0 -        DL    -  0:00.00 [usb]
  0  16    0   0 -20  0     0   0 VBoxIS   DL    -  0:00.00 [TIMER]
  0  17    0   0 -16  0     0   0 psleep   DL    -  0:00.00 [pagedaemon]
  0  18    0   0 -16  0     0   0 psleep   DL    -  0:00.00 [vmdaemon]
  0  19    0   0 155  0     0   0 pgzero   DL    -  0:00.00 [pagezero]
  0  20    0   0 -16  0     0   0 psleep   DL    -  0:00.00 [bufdaemon]
  0  21    0   0  16  0     0   0 syncer   DL    -  0:00.00 [syncer]
  0  22    0   0 -16  0     0   0 vlruwt   DL    -  0:00.00 [vnlru]
  0  23    0   0 -16  0     0   0 sdflush  DL    -  0:00.00 [softdepflush]
  0  24    1   0  52  0 16928   0 wait     Ds+   -  0:00.01 [sh]
  0  72   24   0  52  0 16928   0 wait     D+    -  0:00.00 [sh]
  0  75   72   0  52  0 40000   0 tx->tx_s D+    -  0:00.00 [zfs]

------------------------------------------------------------------------
vmstat -s

   158237 cpu context switches
    22164 device interrupts
      800 software interrupts
     5019 traps
     5843 system calls
       23 kernel threads created
       37  fork() calls
       15 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
      136 vnode pager pageins
      911 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 pages reactivated
     1771 copy-on-write faults
       15 copy-on-write optimized faults
     2232 zero fill pages zeroed
        0 zero fill pages prezeroed
        0 intransit blocking page faults
     4649 total VM faults taken
      113 page faults requiring I/O
        0 pages affected by kernel thread creation
     1334 pages affected by  fork()
      525 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
    55169 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
      465 pages active
      860 pages inactive
        0 pages in VM cache
   190921 pages wired down
 15883166 pages free
     4096 bytes per page
     1886 total name lookups
          cache hits (80% pos + 4% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
         cdev     8     2K       -        8  256
     ppbusdev     2     1K       -        2  256
     filedesc    27    54K       -       76  2048
      kdtrace   323    76K       -      515  64,256
         kenv    86    11K       -      114  16,32,64,128
    proc-args     3     1K       -       31  32,64,128
        hhook     2     1K       -        2  256
      ithread   125    22K       -      125  32,128,256
       KTRACE   100    13K       -      100  128
      entropy  1026    65K       -     1026  32,64
     acpiintr     1     1K       -        1  64
       linker   403   101K       -      533  16,32,64,128,256,512,1024,2048,4096
        lockf     2     1K       -        2  64,128
   loginclass     2     1K       -        2  64
        cache     1     1K       -        1  32
       devbuf 17314 34391K       -    18687  16,32,64,128,256,512,1024,2048,4096
         temp    57    18K       -      538  16,32,64,128,256,512,1024,2048
       ip6ndp     3     1K       -        3  64
      ata_pci     1     1K       -        1  64
       acpica  1824   187K       -    63680  16,32,64,128,256,512,1024,2048
       kbdmux     7    18K       -        7  16,512,1024,2048
       module   521    66K       -      521  128
     mtx_pool     2    16K       -        2  
          LED     4     1K       -        4  16,128
      CAM XPT    58     4K       -      308  16,32,64,128,512,1024,2048
          osd     5     1K       -       11  16,32,64,128
     pmchooks     1     1K       -        1  128
      CAM DEV    15    30K       -       24  2048
     acpitask     1     8K       -        1  
         hdaa     4     5K       -        4  512,1024,2048
         pgrp     2     1K       -        2  128
      session     2     1K       -        2  128
         proc     2   256K       -        2  
      subproc   100   145K       -      149  512,4096
         cred    40     7K       -      240  64,256
       plimit     2     1K       -        2  256
      uidinfo     2    33K       -        2  128
         hdac     1     1K       -        1  512
        hdacc     1     1K       -        1  32
        vtbuf    24  5712K       -       24  4096
      acpisem    22     3K       -       22  128
           vt    11     3K       -       11  256
       sysctl     0     0K       -       43  16,32,64
    sysctloid  7607   378K       -     7779  16,32,64,128
    sysctltmp     0     0K       -        6  32,64
      tidhash     1   256K       -        1  
      callout     9  3208K       -        9  
         umtx   666    84K       -      666  128
     p1003.1b     1     1K       -        1  16
         SWAP    12 19681K       -       12  64
          bus  2257   362K       -     8440  16,32,64,128,256,512,1024
       bus-sc   157   310K       -     5479  16,32,64,128,256,512,1024,2048,4096
      devstat    32    65K       -       32  32,4096
 eventhandler   104     9K       -      104  64,128
       DEVFS3   230    58K       -      270  256
         kobj   349  1396K       -     1111  4096
       DEVFS1   202   101K       -      238  512
      Per-cpu     1     1K       -        1  32
        DEVFS    41     1K       -       42  16,128
         rman   360    41K       -      754  16,32,128
         sbuf     0     0K       -     2295  16,32,64,128,256
       sglist     3     1K       -        3  32
    taskqueue   115    17K       -      159  16,32,64,128,256
     terminal    11     3K       -       11  256
       Unitno    28     2K       -      120  32,64
         vmem     2   160K       -        3  
     ioctlops     0     0K       -      158  16,32,64
          iov     1     1K       -       27  64,256,512
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1  
          tty    14    14K       -       14  1024
        shmfd     1     8K       -        1  
          pcb    12  8341K       -       12  16,128,1024,2048
     vfscache     1 16384K       -        1  
     vfs_hash     1  8192K       -        1  
       vnodes     1     1K       -        1  256
        mount   194     7K       -      402  16,32,64,128,256
  vnodemarker     0     0K       -        8  512
          BPF     3     1K       -        3  128
        ifnet     4     7K       -        4  128,2048
       ifaddr    34    10K       -       34  32,64,256,512,2048,4096
        clone     8     1K       -        8  128
       arpcom     2     1K       -        2  16
      lltable     6     3K       -        6  512
     routetbl     5     2K       -        5  256,512
         igmp     3     1K       -        3  256
     sctp_vrf     1     1K       -        1  64
    hostcache     1    28K       -        1  
     syncache     1    64K       -        1  
          mld     3     1K       -        3  128
          rpc     2     1K       -        2  512
audit_evclass   187     6K       -      228  32
    ufs_quota     1  8192K       -        1  
    vm_pgdata     7  8193K       -        7  128
      UMAHash    19    10K       -       19  512
    pfs_nodes    73    19K       -       73  256
  pfs_vncache     2     1K       -        2  64
      scsi_cd     0     0K       -       11  16
         GEOM   404    67K       -     3278  16,32,64,128,256,512,1024,2048
      memdesc     1     4K       -        1  4096
       feeder    20     2K       -       24  32,128
      acpidev    36     3K       -       36  64
     atkbddev     2     1K       -        2  64
      CAM CCB     1     2K       -    22943  2048
        mixer     3    12K       -        3  4096
    raid_data     0     0K       -      480  32,128,256
     CAM path    22     1K       -      110  32
      solaris 326492 166906K       -   672607  16,32,64,128,256,512,1024,2048,4096
         UART     6     5K       -        6  16,1024
   kstat_data     6     1K       -        6  64
   CAM periph    16     4K       -       40  16,32,64,128,256
md_nvidia_data     0     0K       -       78  512
     pci_link    16     2K       -       16  32,64,128
  md_sii_data     0     0K       -       78  512
    acpi_perf     8     1K       -        8  64
    CAM queue    23     8K       -       57  16,32,512
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        1  4096
CAM dev queue     8     1K       -        8  64
      CAM SIM     8     2K       -        8  256
  ddb_capture     1    48K       -        1  
      io_apic     2     4K       -        2  2048
          MCA     8     1K       -        8  128
          msi     4     1K       -        4  128
     nexusdev     4     1K       -        4  16
       isadev     5     1K       -        5  128
          USB    46    59K       -       52  16,32,64,128,256,512,1024,2048,4096
       USBdev    35     4K       -       35  32,64,128,256
        linux    17     2K       -       17  64
       spicds     7     1K       -        7  128
     envy24ht    15   195K       -       15  64,2048
       cpuctl     1     1K       -        1  64
       crypto     1     1K       -        1  512
       cyclic    32     3K       -       32  16,64,128
          SDT    32     2K       -       32  16,64
          fbt 60661  7839K       -    60661  128
     iprtheap    17    54K       -       17  32,64,128,256
       nvidia   194   864K       -      195  16,32,64,128,256,512,1024,2048,4096
         ipmi     0     0K       -       11  128
     gem_name    59    10K       -       62  32,4096
   drm_global     2     1K       -        2  128,256
   drm_vblank     7     1K       -        7  16,64
      drm_dma     2     1K       -        2  32
    drm_sarea     1     1K       -        1  16
   drm_driver    32  2203K       -       36  16,32,64,128,256,512,1024,4096
drm_ctxbitmap     1     4K       -        1  4096
     drm_sman    13     2K       -       13  128
  drm_hashtab     1  4096K       -        1  
      drm_kms    90    18K       -      108  16,32,64,128,256,2048
       ttm_pd     6     9K       -        6  16,2048
     ttm_rman     2     1K       -        2  256
     ttm_zone     2     1K       -        2  64
  ttm_poolmgr     1     1K       -        1  512
  fdesc_mount     1     1K       -        1  16

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     267,       3,     267,   0,   0
UMA Zones:             1664,      0,     267,       1,     267,   0,   0
UMA Slabs:              112,      0,   12325,      30,   14833,   0,   0
UMA RCntSlabs:          120,      0,       0,       0,       0,   0,   0
UMA Hash:               256,      0,      58,      32,      77,   0,   0
4 Bucket:                32,      0,    1296,    1184,    9186,   0,   0
6 Bucket:                48,      0,     310,    1018,    1646,   0,   0
8 Bucket:                64,      0,     103,    1199,    2674,   0,   0
12 Bucket:               96,      0,      64,    1084,    1327,   0,   0
16 Bucket:              128,      0,     100,    1171,    1676,  17,   0
32 Bucket:              256,      0,     240,     630,    4097,  49,   0
64 Bucket:              512,      0,     238,     287,    1922,  49,   0
128 Bucket:            1024,      0,    1190,     102,    6750,   0,   0
vmem btag:               56,      0,   14856,     409,   14954, 215,   0
VM OBJECT:              256,      0,     583,     407,    1328,   0,   0
RADIX NODE:             144,      0,    8773,     272,   13246,   0,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,       8,     271,       8,   0,   0
MAP ENTRY:              128,      0,     103,    1044,    1717,   0,   0
VMSPACE:                448,      0,       4,     172,      54,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     405,       0,     405,   0,   0
16:                      16,      0,       6,     243,       6,   0,   0
16:                      16,      0,    2762,     724,    3015,   0,   0
16:                      16,      0,  154129,     749,  173890,   0,   0
16:                      16,      0,      54,     693,   28214,   0,   0
16:                      16,      0,      62,     436,      63,   0,   0
16:                      16,      0,     690,     804,    5386,   0,   0
16:                      16,      0,      30,     219,      30,   0,   0
16:                      16,      0,       1,     746,      18,   0,   0
32:                      32,      0,      30,     962,     116,   0,   0
32:                      32,      0,    3092,     876,    3286,   0,   0
32:                      32,      0,   74071,     577,  140388,   0,   0
32:                      32,      0,      66,     430,    4216,   0,   0
32:                      32,      0,      18,    1098,      98,   0,   0
32:                      32,      0,     833,     903,    2868,   0,   0
32:                      32,      0,      94,     526,     186,   0,   0
32:                      32,      0,       8,     612,     164,   0,   0
64:                      64,      0,       5,     181,       6,   0,   0
64:                      64,      0,     516,     786,     586,   0,   0
64:                      64,      0,   20714,    1172,   73124,   0,   0
64:                      64,      0,     767,     659,    1637,   0,   0
64:                      64,      0,      80,     106,      80,   0,   0
64:                      64,      0,    9171,     749,    9688,   0,   0
64:                      64,      0,     131,     303,     131,   0,   0
64:                      64,      0,    1038,     884,    1071,   0,   0
128:                    128,      0,      34,     865,      95,   0,   0
128:                    128,      0,    1899,     364,    1908,   0,   0
128:                    128,      0,  121894,   13359,  190707,   0,   0
128:                    128,      0,    1077,     318,   27008,   0,   0
128:                    128,      0,     112,     911,     192,   0,   0
128:                    128,      0,    1877,     882,    4086,   0,   0
128:                    128,      0,      29,     126,      30,   0,   0
128:                    128,      0,     524,     499,     526,   0,   0
256:                    256,      0,       2,      73,       2,   0,   0
256:                    256,      0,     301,     254,     444,   0,   0
256:                    256,      0,    7514,    4696,   54916,   0,   0
256:                    256,      0,      64,     131,     740,   0,   0
256:                    256,      0,      28,     287,     348,   0,   0
256:                    256,      0,     903,     447,    1957,   0,   0
256:                    256,      0,      31,      44,      31,   0,   0
256:                    256,      0,       4,     251,      18,   0,   0
512:                    512,      0,       2,      33,       9,   0,   0
512:                    512,      0,      88,     206,     244,   0,   0
512:                    512,      0,     347,     500,   78882,   0,   0
512:                    512,      0,      16,      47,      35,   0,   0
512:                    512,      0,       3,      32,       3,   0,   0
512:                    512,      0,     327,     198,    2807,   0,   0
512:                    512,      0,      47,      16,      47,   0,   0
512:                    512,      0,       1,      62,       2,   0,   0
1024:                  1024,      0,       0,      92,      99,   0,   0
1024:                  1024,      0,       9,      19,       9,   0,   0
1024:                  1024,      0,    8351,     125,   23312,   0,   0
1024:                  1024,      0,       5,      23,    2074,   0,   0
1024:                  1024,      0,      16,      12,      16,   0,   0
1024:                  1024,      0,     230,      82,    1273,   0,   0
1024:                  1024,      0,       2,      14,       2,   0,   0
1024:                  1024,      0,       0,       0,       0,   0,   0
2048:                  2048,      0,       2,       4,       9,   0,   0
2048:                  2048,      0,       6,       4,       6,   0,   0
2048:                  2048,      0,      11,      25,      39,   0,   0
2048:                  2048,      0,       8,      58,   22955,   0,   0
2048:                  2048,      0,      21,       9,      30,   0,   0
2048:                  2048,      0,      48,      38,     466,   0,   0
2048:                  2048,      0,       5,       1,       5,   0,   0
2048:                  2048,      0,       0,       0,       0,   0,   0
4096:                  4096,      0,       0,       0,       0,   0,   0
4096:                  4096,      0,       5,       0,       5,   0,   0
4096:                  4096,      0,     298,       2,     497,   0,   0
4096:                  4096,      0,      10,       0,      12,   0,   0
4096:                  4096,      0,      16,       0,      17,   0,   0
4096:                  4096,      0,      25,       0,     415,   0,   0
4096:                  4096,      0,      15,       0,      15,   0,   0
4096:                  4096,      0,     349,       9,    1111,   0,   0
uint64 pcpu:              8,      0,    1398,     138,    1398,   0,   0
SLEEPQUEUE:              88,      0,     334,     565,     334,   0,   0
Files:                   80,      0,       8,    1021,     395,   0,   0
rl_entry:                40,      0,       6,     588,       6,   0,   0
TURNSTILE:              136,      0,     334,     226,     334,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,      26,      46,      75,   0,   0
THREAD:                1168,      0,     295,      38,     438,   0,   0
cpuset:                  72,      0,     282,     488,     425,   0,   0
cyclic_id_cache:         64,      0,       0,       0,       0,   0,   0
audit_record:          1248,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 25720710,       0,      10,       0,   0,   0
mbuf:                   256, 25720710,       1,     134,       1,   0,   0
mbuf_cluster:          2048, 4018860,       0,       0,       0,   0,   0
mbuf_jumbo_page:       4096, 2009429,       0,       0,       0,   0,   0
mbuf_jumbo_9k:         9216, 1786158,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 1339616,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
sendfile_sync:          128,      0,       0,       0,       0,   0,   0
dtrace_state_cache:    4096,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       3,     669,   68266,   0,   0
DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0
ttyinq:                 160,      0,      15,      57,      15,   0,   0
ttyoutq:                256,      0,       8,      67,       8,   0,   0
ata_request:            336,      0,       0,      88,      42,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
nv_stack_t:           12288,      0,       2,       1,       6,   0,   0
FPU_save_area:          512,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,     664,      47,   0,   0
taskq_zone:              48,      0,       0,       0,       0,   0,   0
VNODE:                  472,      0,    4245,     139,    4246,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
BUF TRIE:               144,      0,       0,  105948,       0,   0,   0
NAMEI:                 1024,      0,       0,     104,     887,   0,   0
S VFS Cache:            108,      0,     247,     908,     285,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,       0,       0,       0,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
reference_cache:         40,      0,       0,       0,       0,   0,   0
reference_history_cache:      8,      0,       0,       0,       0,   0,   0
range_seg_cache:         64,      0,   15246,   81288,  176756,   0,   0
zio_cache:              920,      0,    2874,    3294,  146114,   0,   0
zio_link_cache:          48,      0,    2867,    4686,  145725,   0,   0
zio_buf_512:            512,      0,    4917,     221,    5752,   0,   0
zio_data_buf_512:       512,      0,     362,     247,     801,   0,   0
zio_buf_1024:          1024,      0,      95,     113,     269,   0,   0
zio_data_buf_1024:     1024,      0,     421,      99,     521,   0,   0
zio_buf_1536:          1536,      0,      35,     127,     335,   0,   0
zio_data_buf_1536:     1536,      0,     717,      85,     845,   0,   0
zio_buf_2048:          2048,      0,      45,      91,     262,   0,   0
zio_data_buf_2048:     2048,      0,     295,      51,     408,   0,   0
zio_buf_2560:          2560,      0,      11,      70,     131,   0,   0
zio_data_buf_2560:     2560,      0,     202,      32,     291,   0,   0
zio_buf_3072:          3072,      0,       4,      53,      79,   0,   0
zio_data_buf_3072:     3072,      0,     151,      29,     225,   0,   0
zio_buf_3584:          3584,      0,       5,      28,      49,   0,   0
zio_data_buf_3584:     3584,      0,     106,      11,     153,   0,   0
zio_buf_4096:          4096,      0,    3767,    1199,   45373,   0,   0
zio_data_buf_4096:     4096,      0,      97,     116,    2911,   0,   0
zio_buf_5120:          5120,      0,       2,      55,      89,   0,   0
zio_data_buf_5120:     5120,      0,     185,      21,     272,   0,   0
zio_buf_6144:          6144,      0,       5,      35,      62,   0,   0
zio_data_buf_6144:     6144,      0,     150,      13,     206,   0,   0
zio_buf_7168:          7168,      0,       2,      37,      51,   0,   0
zio_data_buf_7168:     7168,      0,     110,      10,     164,   0,   0
zio_buf_8192:          8192,      0,       2,     286,    1766,   0,   0
zio_data_buf_8192:     8192,      0,     102,       9,     192,   0,   0
zio_buf_10240:        10240,      0,       1,      49,      59,   0,   0
zio_data_buf_10240:   10240,      0,     145,      10,     215,   0,   0
zio_buf_12288:        12288,      0,       1,     176,    1011,   0,   0
zio_data_buf_12288:   12288,      0,     102,      10,     149,   0,   0
zio_buf_14336:        14336,      0,       0,      24,      27,   0,   0
zio_data_buf_14336:   14336,      0,      67,      12,     100,   0,   0
zio_buf_16384:        16384,      0,     718,      40,    2238,   0,   0
zio_data_buf_16384:   16384,      0,      57,       8,      86,   0,   0
zio_buf_20480:        20480,      0,       0,      57,     571,   0,   0
zio_data_buf_20480:   20480,      0,      59,       7,     102,   0,   0
zio_buf_24576:        24576,      0,       0,      40,     565,   0,   0
zio_data_buf_24576:   24576,      0,      44,       6,      67,   0,   0
zio_buf_28672:        28672,      0,       0,     131,    1560,   0,   0
zio_data_buf_28672:   28672,      0,      40,      13,      60,   0,   0
zio_buf_32768:        32768,      0,       0,      27,     290,   0,   0
zio_data_buf_32768:   32768,      0,      31,       5,      43,   0,   0
zio_buf_36864:        36864,      0,       1,      16,     244,   0,   0
zio_data_buf_36864:   36864,      0,      32,       3,      40,   0,   0
zio_buf_40960:        40960,      0,       0,      14,     139,   0,   0
zio_data_buf_40960:   40960,      0,      18,       1,      19,   0,   0
zio_buf_45056:        45056,      0,       0,      13,      99,   0,   0
zio_data_buf_45056:   45056,      0,      21,       3,      25,   0,   0
zio_buf_49152:        49152,      0,       0,      18,      89,   0,   0
zio_data_buf_49152:   49152,      0,      16,       3,      22,   0,   0
zio_buf_53248:        53248,      0,       1,      17,     148,   0,   0
zio_data_buf_53248:   53248,      0,      22,       3,      26,   0,   0
zio_buf_57344:        57344,      0,       0,      11,      73,   0,   0
zio_data_buf_57344:   57344,      0,      14,       1,      15,   0,   0
zio_buf_61440:        61440,      0,       0,      12,      56,   0,   0
zio_data_buf_61440:   61440,      0,      10,       3,      13,   0,   0
zio_buf_65536:        65536,      0,       0,      11,      46,   0,   0
zio_data_buf_65536:   65536,      0,       4,       1,       5,   0,   0
zio_buf_69632:        69632,      0,       0,       8,      27,   0,   0
zio_data_buf_69632:   69632,      0,       8,       1,       9,   0,   0
zio_buf_73728:        73728,      0,       0,       9,      27,   0,   0
zio_data_buf_73728:   73728,      0,       5,       0,       5,   0,   0
zio_buf_77824:        77824,      0,       0,       9,      41,   0,   0
zio_data_buf_77824:   77824,      0,       6,       1,       7,   0,   0
zio_buf_81920:        81920,      0,       0,      13,      84,   0,   0
zio_data_buf_81920:   81920,      0,       9,       1,      10,   0,   0
zio_buf_86016:        86016,      0,       0,       8,      25,   0,   0
zio_data_buf_86016:   86016,      0,       5,       3,       8,   0,   0
zio_buf_90112:        90112,      0,       0,      10,      33,   0,   0
zio_data_buf_90112:   90112,      0,       3,       1,       4,   0,   0
zio_buf_94208:        94208,      0,       0,       6,      15,   0,   0
zio_data_buf_94208:   94208,      0,       4,       1,       5,   0,   0
zio_buf_98304:        98304,      0,       1,       5,      20,   0,   0
zio_data_buf_98304:   98304,      0,       9,       0,       9,   0,   0
zio_buf_102400:      102400,      0,       0,       7,      13,   0,   0
zio_data_buf_102400: 102400,      0,       2,       1,       3,   0,   0
zio_buf_106496:      106496,      0,       0,      16,      99,   0,   0
zio_data_buf_106496: 106496,      0,      12,       2,      14,   0,   0
zio_buf_110592:      110592,      0,       0,      12,      72,   0,   0
zio_data_buf_110592: 110592,      0,       6,       1,       7,   0,   0
zio_buf_114688:      114688,      0,       0,      22,      79,   0,   0
zio_data_buf_114688: 114688,      0,       6,       0,       6,   0,   0
zio_buf_118784:      118784,      0,       0,       8,      22,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       9,      28,   0,   0
zio_data_buf_122880: 122880,      0,       6,       0,       6,   0,   0
zio_buf_126976:      126976,      0,       0,      10,      22,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,     160,      79,     862,   0,   0
zio_data_buf_131072: 131072,      0,     383,       8,     556,   0,   0
lz4_ctx:              16384,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,    4199,     848,    4199,   0,   0
dnode_t:               1096,      0,    4649,      31,    5833,   0,   0
dmu_buf_impl_t:         336,      0,    9678,      68,   14441,   0,   0
arc_buf_hdr_t:          328,      0,   10162,     134,   11502,   0,   0
arc_buf_t:               72,      0,    9396,     614,   12289,   0,   0
zil_lwb_cache:          192,      0,       0,       0,       0,   0,   0
zfs_znode_cache:        368,      0,    4199,     151,    4199,   0,   0
pipe:                   744,      0,       0,      55,      13,   0,   0
Mountpoints:            816,      0,      17,      23,      17,   0,   0
procdesc:               128,      0,       0,       0,       0,   0,   0
ksiginfo:               112,      0,      24,    1026,      24,   0,   0
itimer:                 352,      0,       0,       0,       0,   0,   0
KNOTE:                  128,      0,       0,       0,       0,   0,   0
socket:                 696, 2062880,       0,       0,       0,   0,   0
ipq:                     56, 125599,       0,       0,       0,   0,   0
udp_inpcb:              392, 2062880,       0,       0,       0,   0,   0
udpcb:                   16, 2062965,       0,       0,       0,   0,   0
tcp_inpcb:              392, 2062880,       0,       0,       0,   0,   0
tcpcb:                 1024, 2062880,       0,       0,       0,   0,   0
tcptw:                   88,  27810,       0,       0,       0,   0,   0
syncache:               160,  15360,       0,       0,       0,   0,   0
hostcache:              136,  15370,       0,       0,       0,   0,   0
tcpreass:                40, 251262,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
sctp_ep:               1408, 2062880,       0,       0,       0,   0,   0
sctp_asoc:             2352,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,       0,       0,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400026,       0,       0,       0,   0,   0
sctp_readq:             104, 400026,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400059,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
ripcb:                  392, 2062880,       0,       0,       0,   0,   0
unpcb:                  240, 2062880,       0,       0,       0,   0,   0
rtentry:                200,      0,       0,       0,       0,   0,   0
selfd:                   56,      0,       0,       0,       0,   0,   0
SWAPMETA:               288, 8037731,       0,       0,       0,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq6: fdc0                            22          0
irq14: ata0                           61          0
irq17: uhci0 ehci0                    54          0
irq18: uhci2+                        306          2
irq19: uhci1 ahci0+                21712        146
cpu0:timer                          3524         23
irq259: hdac0                          9          0
cpu7:timer                          2042         13
cpu1:timer                          1450          9
cpu3:timer                          3609         24
cpu2:timer                          1414          9
cpu6:timer                          1949         13
cpu4:timer                          1408          9
cpu5:timer                          1575         10
Total                              39135        264

------------------------------------------------------------------------
pstat -T

  8/2062880 files
0M/147455M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/#C:0xc0      50331392        0 50331392     0%
/dev/ad14p2       50331392        0 50331392     0%
/dev/usb/3.2.1    50331392        0 50331392     0%
/dev/#C:0xd0      50331392        0 50331392     0%
/dev/#C:0xd8      50331392        0 50331392     0%
/dev/#C:0xe0      50331392        0 50331392     0%
Total            301988352        0 301988352     0%

------------------------------------------------------------------------
iostat

       tty            ada0             ada1             ada2             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0     9 15.20 161  2.39  14.99 169  2.47  15.24 167  2.49   0  0  4  1 96

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          632	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        2         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         1         1         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         4
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
	0 packets sent
		0 data packets (0 bytes)
		0 data packets (0 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		0 ack-only packets (0 delayed)
		0 URG only packets
		0 window probe packets
		0 window update packets
		0 control packets
	0 packets received
		0 acks (for 0 bytes)
		0 duplicate acks
		0 acks for unsent data
		0 packets (0 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		0 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	0 connection requests
	0 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	0 ignored RSTs in the windows
	0 connections established (including accepts)
	0 connections closed (including 0 drops)
		0 connections updated cached RTT on close
		0 connections updated cached RTT variance on close
		0 connections updated cached ssthresh on close
	0 embryonic connections dropped
	0 segments updated rtt (of 0 attempts)
	0 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	0 correct ACK header predictions
	0 correct data packet header predictions
	0 syncache entries added
		0 retransmitted
		0 dupsyn
		0 dropped
		0 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	0 cookies sent
	0 cookies received
	0 hostcache entries added
		0 bucket overflow
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	0 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
udp:
	0 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	0 with no checksum
	0 dropped due to no socket
	0 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	0 delivered
	0 datagrams output
	0 times multicast source filter matched
ip:
	0 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	0 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	0 packets not forwardable
	0 packets received for
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2014-04-16 02:18:05 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-fs

Over to maintainer(s).
Comment 2 Larry Rosenman freebsd_committer freebsd_triage 2014-06-03 17:11:45 UTC
I've wound up commenting out a bunch of these ASSERTS, and haven't seen any negative consequences, HOWEVER, I'd like someone to let me know how I could look at the FS(s)/POOL and see if there is a real issue.
Comment 3 Larry Rosenman freebsd_committer freebsd_triage 2014-08-30 21:03:41 UTC
I'm STILL seeing these, and have commented out:
===================================================================
--- zap.c       (revision 270765)
+++ zap.c       (working copy)
@@ -476,7 +476,9 @@
         * chain.  There should be no chained leafs (as we have removed
         * support for them).
         */
+#if 0 /*LER: to see what else blows up */
        ASSERT0(l->l_phys->l_hdr.lh_pad1);
+#endif

        /*
         * There should be more hash entries than there can be
@@ -531,7 +533,9 @@
        ASSERT3U(l->l_blkid, ==, blkid);
        ASSERT3P(l->l_dbuf, ==, db);
        ASSERT3P(l->l_phys, ==, l->l_dbuf->db_data);
+#if 0 /* LER */
        ASSERT3U(l->l_phys->l_hdr.lh_block_type, ==, ZBT_LEAF);
+#endif
        ASSERT3U(l->l_phys->l_hdr.lh_magic, ==, ZAP_LEAF_MAGIC);

        *lp = l;
borg.lerctr.org /usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs $


Can I PLEASE get someone to look at these, and tell me what I need to do to fix the on-disk image?
Comment 4 Larry Rosenman freebsd_committer freebsd_triage 2014-08-30 21:04:45 UTC
Created attachment 146572 [details]
Latest panic core.txt.  I **DO** have the vmcore available
Comment 5 Peter Wemm freebsd_committer freebsd_triage 2014-08-30 22:14:25 UTC
If you look at the whole comment for one of the assertions you disabled:

        /*
         * lhr_pad was previously used for the next leaf in the leaf
         * chain.  There should be no chained leafs (as we have removed
         * support for them).
         */
        ASSERT0(l->l_phys->l_hdr.lh_pad1);

Or looking at the annotation:
168404        pjd       /*
168404        pjd        * lhr_pad was previously used for the next leaf in the leaf
168404        pjd        * chain.  There should be no chained leafs (as we have removed
168404        pjd        * support for them).
168404        pjd        */
240415         mm       ASSERT0(l->l_phys->l_hdr.lh_pad1);

One of those is the initial commit:
------------------------------------------------------------------------
r168404 | pjd | 2007-04-05 18:09:06 -0700 (Thu, 05 Apr 2007) | 11 lines

Please welcome ZFS - The last word in file systems.

ZFS file system was ported from OpenSolaris operating system. The code in under
CDDL license.
------------------------------------------------------------------------


The second is:
------------------------------------------------------------------------
r240415 | mm | 2012-09-12 11:05:43 -0700 (Wed, 12 Sep 2012) | 21 lines

Merge recent zfs vendor changes, sync code and adjust userland DEBUG.
------------------------------------------------------------------------

The change is just a syntax one:
-       ASSERT3U(l->l_phys->l_hdr.lh_pad1, ==, 0);
+       ASSERT0(l->l_phys->l_hdr.lh_pad1);

So.. ever since ZFS existed in FreeBSD, that field should be zero, on disk.  You are tripping those asserts because it isn't the case on your machine.

This suggests to me that you've either got a very old file system that pre-dates freebsd, or you've got some evil on-disk corruption.

I suspect the latter.  If this was my machine, I'd be contemplating a backup/restore.  I see from the disk sizes you may have a lot of data, but that's what I would be doing.

I also see the stack traces come from the zfs de-dupe code.  There are a great number of people who have a very dim view of that code, even the authors apologized for it.  If it were my machine, I would be turning off dedup immediately, if not sooner.
Comment 6 Larry Rosenman freebsd_committer freebsd_triage 2014-08-30 22:17:36 UTC
This pool has been around a LONG time -- 

I'm wondering where I can borrow a Terabyte of disk to backup what I care about and rebuild it one more time...

(this pool started on FreeBSD 8(IIRC) on 400G disk, and got upgraded....)
Comment 7 Peter Wemm freebsd_committer freebsd_triage 2014-08-30 22:51:16 UTC
Out of curiosity, what does "zpool status -D poolname" show?

http://serverfault.com/questions/533877/how-large-is-my-zfs-dedupe-table-at-the-moment

You might be burning a significant chunk of ARC metadata space to hold the dedupe table in memory.

If I had to guess at this point, I would guess that there are a couple of corrupt records in the on-disk dedup table for one or more of your data sets.  Since this is an old pool, there's been plenty of opportunities over the years for this to accumulate while zfs support matured.

From your comment, it sounds like you have 1TB of data spread over a pool made of 2TB disks?  Is that deduplicated?  Is one of the drives a spare? (if so, can you take the spare offline and make a pool on it to facilitate a dump/restore?)
Comment 8 Larry Rosenman freebsd_committer freebsd_triage 2014-08-30 23:49:58 UTC
borg.lerctr.org /home/ler $ zpool status -D zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 5h8m with 0 errors on Sat Aug 30 13:53:49 2014
config:

        NAME                                            STATE     READ WRITE CKSUM
        zroot                                           ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/71e84d97-043a-11e2-8cd6-003048f2299c  ONLINE       0     0     0
            gptid/3540b18c-0db2-11e2-8ef2-003048f2299c  ONLINE       0     0     0
            gptid/b0a645ed-0db9-11e2-8ef2-003048f2299c  ONLINE       0     0     0
            gptid/dc133fc1-0dc0-11e2-8ef2-003048f2299c  ONLINE       0     0     0
            gptid/21bef448-0dc8-11e2-8ef2-003048f2299c  ONLINE       0     0     0
            gptid/46033e1c-0432-11e2-8cd6-003048f2299c  ONLINE       0     0     0

errors: No known data errors

 dedup: DDT entries 1415219, size 2676 on disk, 540 in core

bucket              allocated                       referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    1.26M    116G   66.3G   69.2G    1.26M    116G   66.3G   69.2G
     2    86.1K   6.84G   6.04G   6.20G     174K   13.8G   12.1G   12.4G
     4    3.15K    253M    243M    249M    14.6K   1.00G    985M   1019M
     8      196   2.32M    823K   1.78M    1.96K   24.5M   8.42M   18.4M
    16      109   1.23M   1024K   1.52M    2.50K   31.9M   26.2M   38.2M
    32       33    330K    318K    498K    1.37K   18.1M   17.5M   24.9M
    64        3   1.50K   1.50K   19.2K      249    124K    124K   1.55M
   128        3     28K     28K   38.3K      470   4.96M   4.96M   6.40M
   256        4    162K    162K    173K    1.33K   64.3M   64.3M   67.6M
 Total    1.35M    123G   72.6G   75.7G    1.45M    131G   79.5G   82.8G

borg.lerctr.org /home/ler $


I'm actually using 500G on a 6 disk raid-Z1 pool of 2TB disks.  That's the "logical Used" (approx). 

I went out and bought a 4TB NAS to use for this, and then for other things.
Comment 9 Peter Wemm freebsd_committer freebsd_triage 2014-08-31 00:17:52 UTC
 "dedup: DDT entries 1415219, size 2676 on disk, 540 in core"

->  728MB of ram to cache this.  Fortunately that's a relatively small portion of your system ram.

I would still be doing a zfs set dedup=off anywhere it's on.  While it won't fix the on-disk dedup corruption, at least you won't risk adding to it any more.

Your system is crashing trying to parse the on-disk dedup records.  That's what the *_ddt_* functions are about in the traces.  I can't imagine this ending well.

Does a scrub find the problem? or does the scrub trigger the assertions?
Comment 10 Larry Rosenman freebsd_committer freebsd_triage 2014-08-31 00:25:13 UTC
scrub triggers the assertions.....

currently rsyncing the volatile stuff to the new 4TB NAS mounted via NFS....

then will blow this system up and rebuild it withOUT dedupe....
Comment 11 Larry Rosenman freebsd_committer freebsd_triage 2014-08-31 20:47:58 UTC
Rebuilding ports -- no more assertions on a non-modified kernel.

Thanks Peter Wemm for the input on dedup. 

I think we can close this one out.
Comment 12 Larry Rosenman freebsd_committer freebsd_triage 2014-08-31 20:49:34 UTC
Looks like an OLD dedup corruption/mis-feature -- rebuilding the entire system from scratch cleaned it up.