Since upgrading from 8.0 to 8-STABLE, we get a panic at least every week once. The affected machine is a backup storage server, so we were able to set up a debugging kernel on that. In my opinion, this panic is ZFS related, the machine is equipped with 2x6 drives raidz. I hope that the following information will be helpful to you. First, the trace after panic: db> trace Tracing pid 45 tid 100154 td 0xffffff000741f8c0 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b trap_fatal() at trap_fatal Xdblfault() at Xdblfault+0xad --- trap 0x17, rip = 0xffffffff8037a87e, rsp = 0xffffffff809b4b50, rbp = 0xffffff82476f6030 --- witness_checkorder() at witness_checkorder+0x1a _mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x80 ioapic_disable_source() at ioapic_disable_source+0x30 intr_event_handle() at intr_event_handle+0xc4 intr_execute_handlers() at intr_execute_handlers+0x4f lapic_handle_intr() at lapic_handle_intr+0x37 Xapic_isr1() at Xapic_isr1+0xa5 --- interrupt, rip = 0xffffffff80329447, rsp = 0xffffff82476f6200, rbp=0xffffff82476f6240 --- _mtx_lock_flags() at _mtx_lock_flags+0x79 uma_zalloc_arg() at uma_zalloc_arg+0x2a7 vm_map_entry_create() at vm_map_entry_create+0x20 _vm_map_clip_start() at _vm_map_clip_start+0x52 vm_map_delete() at vm_map_delete+0x3b6 vm_map_remove() at vm_map_remove+0x52 uma_large_free() at uma_large_free+0x55 free() at free+0x7e vdev_cache_evict() at vdev_cache_evict+0x39 vdev_cache_read() at vdev_cache_read+0x238 zio_vdev_io_start() at zio_vdev_io_start+x01a8 zio_execute() at zio_execute+0x70 vdev_raidz_io_start() at vdev_raidz_io_start+0x404 zio_execute() at zio_execute+0x70 vdev_mirror_io_start() at vdev_mirror_io_start+x015c zio_execute() at zio_execute+0x70 arc_read_nolock() at arc_read_nolock+0x5a0 arc_read() at arc_read+0x71 dbuf_prefetch() at dbuf_prefetch+0x135 dmu_zfetch_dofetch() at dmu_zfetch_dofetch+0xe3 dmu_zfetch() ad dmu_zfetch+0xb7a dbuf_read() at dbuf_read+0x5ce dnode_hold_impl() at dnode_hold_impl+0xdd dmu_buf_hold() at dmu_buf_hold+0x2b bplist_enqueue() at bplist_enqueue+0x6b dsl_dataset_block_kill() at dsl_dataset_block_kill+0x209 dbuf_write() at dbuf_write+0x1eb dbuf_sync_list() at dbuf_sync_list+0x1f2 dbuf_sync_list() at dbuf_sync_list+0x100 dbuf_sync_list() at dbuf_sync_list+0x100 dbuf_sync_list() at dbuf_sync_list+0x100 dbuf_sync_list() at dbuf_sync_list+0x100 dbuf_sync_list() at dbuf_sync_list+0x100 dbuf_sync_list() at dbuf_sync_list+0x100 dnode_sync() at dnode_sync+0x46f dmu_objset_sync() at dmu_objset_sync+0xb1 dsl_pool_sync() at dsl_pool_sync+0xb1 spa_sync() at spa_sync+0x3c8 txg_sync_thread() at txg_sync_thread+0x2c6 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82476f8cf0, rbp = 0 --- /etc/sysctl.conf: security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 kern.maxvnodes=200000 /boot/loader.conf geom_mirror_load="YES" vfs.zfs.arc_max="5G" kernel configuration: cpu HAMMER ident GENERIC makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options COMPAT_FREEBSD32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework options FLOWTABLE # per-cpu routing cache options INCLUDE_CONFIG_FILE # Include this file in kernel options KDB # Kernel debugger related code options KDB_TRACE # Print a stack trace for a panic # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # Debugging makeoptions DEBUG=-g options KDB options DDB options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC # CPU frequency control device cpufreq device coretemp device ahci device pf # Bus support. device acpi device pci # ATA and ATAPI devices device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering # SCSI peripherals device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) # RAID controllers interfaced to the SCSI subsystem device twa # 3ware 9000 series PATA/SATA RAID # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc device agp # support several AGP chipsets # Serial (COM) ports device uart # Generic UART driver # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer device plip # TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da # If you've got a "dumb" serial or parallel PCI card that is # supported by the puc(4) glue driver, uncomment the following # line to enable it (connects to sio, uart and/or ppc drivers): #device puc # PCI Ethernet NICs. device em # Intel PRO/1000 Gigabit Ethernet Family # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices device uhid # "Human Interface Devices" device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da dmesg after boot: Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-RC3 #3: Mon Jan 31 12:57:26 CET 2011 root@xxxxx:/usr/obj/usr/src/sys/BAK amd64 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2394.01-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8239063040 (7857 MB) ACPI APIC Table: <PTLTD APIC > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard ioapic2 <Version 2.0> irqs 48-71 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD XSDT> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1 pci2: <ACPI PCI bus> on pcib2 3ware device driver for 9000 series storage controllers, version: 3.80.06.003 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem 0xd2000000-0xd3ffffff,0xd0200000-0xd0200fff irq 24 at device 1.0 on pci2 twa0: [ITHREAD] twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue: twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue: twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SXU-8LP, 8 ports, Firmware FE9X 3.08.00.029, BIOS BE9X 3.10.00.003 pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1 pci3: <ACPI PCI bus> on pcib3 twa1: <3ware 9000 series Storage Controller> port 0x3000-0x303f mem 0xd4000000-0xd5ffffff,0xd0300000-0xd0300fff irq 52 at device 2.0 on pci3 twa1: [ITHREAD] twa1: INFO: (0x04: 0x0053): Battery capacity test is overdue: twa1: INFO: (0x04: 0x0053): Battery capacity test is overdue: twa1: INFO: (0x15: 0x1300): Controller details:: Model 9550SXU-8LP, 8 ports, Firmware FE9X 3.08.00.029, BIOS BE9X 3.10.00.003 uhci0: <Intel 82801I (ICH9) USB controller> port 0x1820-0x183f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] usbus0: <Intel 82801I (ICH9) USB controller> on uhci0 uhci1: <Intel 82801I (ICH9) USB controller> port 0x1840-0x185f irq 17 at device 26.1 on pci0 uhci1: [ITHREAD] usbus1: <Intel 82801I (ICH9) USB controller> on uhci1 uhci2: <Intel 82801I (ICH9) USB controller> port 0x1860-0x187f irq 18 at device 26.2 on pci0 uhci2: [ITHREAD] usbus2: <Intel 82801I (ICH9) USB controller> on uhci2 ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xd0000000-0xd00003ff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: <Intel 82801I (ICH9) USB 2.0 controller> on ehci0 pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pci5: <ACPI PCI bus> on pcib4 em0: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0x4000-0x401f mem 0xd0420000-0xd043ffff,0xd0400000-0xd041ffff irq 16 at device 0.0 on pci5 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1b:21:15:3a:7d pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 pci13: <ACPI PCI bus> on pcib5 em1: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0x5000-0x501f mem 0xd0500000-0xd051ffff irq 16 at device 0.0 on pci13 em1: Using an MSI interrupt em1: [FILTER] em1: Ethernet address: 00:30:48:b8:84:9a pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0 pci15: <ACPI PCI bus> on pcib6 em2: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0x6000-0x601f mem 0xd0600000-0xd061ffff irq 17 at device 0.0 on pci15 em2: Using an MSI interrupt em2: [FILTER] em2: Ethernet address: 00:30:48:b8:84:9b uhci3: <Intel 82801I (ICH9) USB controller> port 0x1880-0x189f irq 23 at device 29.0 on pci0 uhci3: [ITHREAD] usbus4: <Intel 82801I (ICH9) USB controller> on uhci3 uhci4: <Intel 82801I (ICH9) USB controller> port 0x18a0-0x18bf irq 22 at device 29.1 on pci0 uhci4: [ITHREAD] usbus5: <Intel 82801I (ICH9) USB controller> on uhci4 uhci5: <Intel 82801I (ICH9) USB controller> port 0x18c0-0x18df irq 18 at device 29.2 on pci0 uhci5: [ITHREAD] usbus6: <Intel 82801I (ICH9) USB controller> on uhci5 ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xd0000400-0xd00007ff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus7: EHCI version 1.0 usbus7: <Intel 82801I (ICH9) USB 2.0 controller> on ehci1 pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci17: <ACPI PCI bus> on pcib7 vgapci0: <VGA-compatible display> port 0x7000-0x70ff mem 0xd8000000-0xdfffffff,0xd0700000-0xd070ffff irq 22 at device 4.0 on pci17 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel ICH9 AHCI SATA controller> port 0x1c30-0x1c37,0x1c24-0x1c27,0x1c28-0x1c2f,0x1c20-0x1c23,0x18e0-0x18ff mem 0xd0000800-0xd0000fff irq 17 at device 31.2 on pci0 ahci0: [ITHREAD] ahci0: AHCI v1.20 with 4 3Gbps ports, Port Multiplier supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich1: [ITHREAD] ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich2: [ITHREAD] ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich3: [ITHREAD] pci0: <serial bus, SMBus> at device 31.3 (no driver attached) pci0: <dasp> at device 31.6 (no driver attached) acpi_button0: <Power Button> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcb7ff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range coretemp0: <CPU On-Die Thermal Sensors> on cpu0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 p4tcc0: <CPU Frequency Thermal Control> on cpu0 coretemp1: <CPU On-Die Thermal Sensors> on cpu1 est1: <Enhanced SpeedStep Frequency Control> on cpu1 p4tcc1: <CPU Frequency Thermal Control> on cpu1 coretemp2: <CPU On-Die Thermal Sensors> on cpu2 est2: <Enhanced SpeedStep Frequency Control> on cpu2 p4tcc2: <CPU Frequency Thermal Control> on cpu2 coretemp3: <CPU On-Die Thermal Sensors> on cpu3 est3: <Enhanced SpeedStep Frequency Control> on cpu3 p4tcc3: <CPU Frequency Thermal Control> on cpu3 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ugen0.1: <Intel> at usbus0 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen1.1: <Intel> at usbus1 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 ugen2.1: <Intel> at usbus2 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 ugen3.1: <Intel> at usbus3 uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 ugen4.1: <Intel> at usbus4 uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 ugen5.1: <Intel> at usbus5 uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 ugen6.1: <Intel> at usbus6 uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6 ugen7.1: <Intel> at usbus7 uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered uhub3: 6 ports with 6 removable, self powered uhub7: 6 ports with 6 removable, self powered ada0 at ahcich0 bus 0 scbus2 target 0 lun 0 ada0: <Hitachi HTE545025B9A300 PB2OC64G> ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich1 bus 0 scbus3 target 0 lun 0 ada1: <Hitachi HTE545025B9A300 PB2OC64G> ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C) da0 at twa0 bus 0 scbus0 target 0 lun 0 da0: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) da1 at twa0 bus 0 scbus0 target 1 lun 0 da1: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da1: 100.000MB/s transfers da1: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) da2 at twa0 bus 0 scbus0 target 2 lun 0 da2: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) da3 at twa0 bus 0 scbus0 target 3 lun 0 da3: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da3: 100.000MB/s transfers da3: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) da4 at twa0 bus 0 scbus0 target 4 lun 0 da4: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da4: 100.000MB/s transfers da4: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) da5 at twa0 bus 0 scbus0 target 5 lun 0 da5: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da5: 100.000MB/s transfers da5: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) da6 at twa1 bus 0 scbus1 target 0 lun 0 da6: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da6: 100.000MB/s transfers da6: 1907338MB (3906228224 512 byte sectors: 255H 63S/T 243151C) da7 at twa1 bus 0 scbus1 target 1 lun 0 da7: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da7: 100.000MB/s transfers da7: 1907338MB (3906228224 512 byte sectors: 255H 63S/T 243151C) da8 at twa1 bus 0 scbus1 target 2 lun 0 da8: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da8: 100.000MB/s transfers da8: 1907338MB (3906228224 512 byte sectors: 255H 63S/T 243151C) da9 at twa1 bus 0 scbus1 target 3 lun 0 da9: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da9: 100.000MB/s transfers da9: 1907338MB (3906228224 512 byte sectors: 255H 63S/T 243151C) da10 at twa1 bus 0 scbus1 target 4 lun 0 da10: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da10: 100.000MB/s transfers da10: 1907338MB (3906228224 512 byte sectors: 255H 63S/T 243151C) da11 at twa1 bus 0 scbus1 target 5 lun 0 da11: <AMCC 9550SXU-8L DISK 3.08> Fixed Direct Access SCSI-5 device da11: 100.000MB/s transfers da11: 1907338MB (3906228224 512 byte sectors: 255H 63S/T 243151C) SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. GEOM_MIRROR: Device mirror/gm0 launched (2/2). GEOM: mirror/gm0s1: geometry does not match label (16h,63s != 255h,63s). Trying to mount root from ufs:/dev/mirror/gm0s1a WARNING: / was not properly dismounted ZFS filesystem version 4 ZFS storage pool version 15 em0: link state changed to UP em2: link state changed to UP em1: link state changed to UP zpool: NAME STATE READ WRITE CKSUM backup1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0
Responsible Changed From-To: freebsd-bugs->freebsd-fs Submitter suspects ZFS in this case.
Hi, today once again a panic - the same situation. Today I viewed the whole output of "ps" and I can confirm that the panic is in the process "zfskern". with kind regards, Robert Schulze
Hi, bumping that up, the problem still exists with FreeBSD-8.2-STABLE csup'd and built yesterday. In the debugger, I did a "continue", maybe these information are valuable: --- Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x29 fault code = supervisort read data, page not present instruction pointer = 0x20:0xffffffff80173330 stack pointer = 0x28:0xffffff80000b6af0 frame pointer = 0x28:0xffffff80000b6b20 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def 32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi2: cambio) [thread pid 12 tid 100024 ] Stopped at camisr_runqueue+0x6e: cmpq %rax,0x28(%rdx%) --- I may provider further debugging information if I get a hint which information and how to do so. with kind regards, Robert Schulze
Author: avg Date: Fri Mar 11 19:27:31 2011 New Revision: 219526 URL: http://svn.freebsd.org/changeset/base/219526 Log: use even larger stack size for ZFS txg_sync_thread While the stack size was larger than the default stack size on i386, it was smaller than the default stack size on amd64 and apparently that wasn't enough. So, bump the size to 4 pages. Upcoming ZFSv28 code uses 8 pages for this stack size. This is a direct commit to stable/8. PR: kern/154681 Discussed with: pjd Modified: stable/8/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c Modified: stable/8/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c ============================================================================== --- stable/8/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c Fri Mar 11 19:21:42 2011 (r219525) +++ stable/8/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c Fri Mar 11 19:27:31 2011 (r219526) @@ -146,7 +146,7 @@ txg_sync_start(dsl_pool_t *dp) * 32-bit x86. This is due in part to nested pools and * scrub_visitbp() recursion. */ - tx->tx_sync_thread = thread_create(NULL, 12<<10, txg_sync_thread, + tx->tx_sync_thread = thread_create(NULL, 16<<10, txg_sync_thread, dp, 0, &p0, TS_RUN, minclsyspri); mutex_exit(&tx->tx_sync_lock); _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
State Changed From-To: open->closed A fix is committed.
Responsible Changed From-To: freebsd-fs->avg I am hanlding this issue.
This little fix indeed did it. Since we first tried it with 24<<10, I first had to pull stable to see whether the actual committed fix is working and wait some days without seeing the machine crashing. Thanks to avg@ for the help! Robert Schulze