Created attachment 144355 [details] Panic 1/24 I ran into a kernel panic with vdev_file_io_start(..) where it tried to dereference a NULL pointer. Screenshots attached.
Created attachment 144356 [details] Panic 2/24
Created attachment 144357 [details] Panic screenshots
Created attachment 144359 [details] Textdump
Over to maintainers.
Looks like this may be from an old version of current could you check with the latest source to see if it still exists?
Sure!
Still occurs with r268351 on i386 built/booted today.
Please note that this might be a malloc failure because in both cases I was running ZFS on VMs with less than <=4GB of RAM.
Its not recommended to run on i386 due to stack size, can you try with either am64 or increase your stack size with: options KSTACK_PAGES=4
Crashes on amd64 with 64GB of RAM, so the architecture and memory doesn't appear to matter. The tests are triggering a panic scenario with ZFS.
Slight correction. I meant 256GB of RAM. db> show msgbuf msgbufp = 0xfffff8403fffffb8 magic = 63062, size = 98232, r= 21200, w = 21678, ptr = 0xfffff8403ffe8000, cksum= 1683664 Fatal trap 12: page fault while in kernel mode cpuid = 6; apic id = 20 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80d1c5a6 stack pointer = 0x28:0xfffffe3fcf2090a0 frame pointer = 0x28:0xfffffe3fcf209110 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1894 (zpool) Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #0 a32b4d9(master)-dirty: Thu Jul 17 03:09:09 PDT 2014 root@mina3020.localdomain:/usr/obj/scratch/freebsd/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 WARNING: WITNESS option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.24-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant, performance statistics real memory = 274877906944 (262144 MB) avail memory = 267130585088 (254755 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <INTEL S2600GL> FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 cpu4 (AP): APIC ID: 8 cpu5 (AP): APIC ID: 10 cpu6 (AP): APIC ID: 32 cpu7 (AP): APIC ID: 34 cpu8 (AP): APIC ID: 36 cpu9 (AP): APIC ID: 38 cpu10 (AP): APIC ID: 40 cpu11 (AP): APIC ID: 42 ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20130823/tbfadt-682) ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard ioapic2 <Version 2.0> irqs 48-71 on motherboard kbd1 at kbdmux0 random: <Software, Yarrow> initialized acpi0: <INTEL S2600GL> on motherboard acpi0: Power Button (fixed) acpi0: reservation of 0, 9d000 (3) failed cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 cpu4: <ACPI CPU> on acpi0 cpu5: <ACPI CPU> on acpi0 cpu6: <ACPI CPU> on acpi0 cpu7: <ACPI CPU> on acpi0 cpu8: <ACPI CPU> on acpi0 cpu9: <ACPI CPU> on acpi0 cpu10: <ACPI CPU> on acpi0 cpu11: <ACPI CPU> on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> irq 47 at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> irq 47 at device 1.1 on pci0 pci2: <ACPI PCI bus> on pcib2 igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x3060-0x307f mem 0xd0b60000-0xd0b7ffff,0xd0bb0000-0xd0bb3fff irq 27 at device 0.0 on pci2 igb0: Using MSIX interrupts with 9 vectors igb0: Ethernet address: 00:1e:67:6d:7a:0c igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 igb0: Bound queue 4 to cpu 4 igb0: Bound queue 5 to cpu 5 igb0: Bound queue 6 to cpu 6 igb0: Bound queue 7 to cpu 7 igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x3040-0x305f mem 0xd0b40000-0xd0b5ffff,0xd0ba0000-0xd0ba3fff irq 30 at device 0.1 on pci2 igb1: Using MSIX interrupts with 9 vectors igb1: Ethernet address: 00:1e:67:6d:7a:0d igb1: Bound queue 0 to cpu 8 igb1: Bound queue 1 to cpu 9 igb1: Bound queue 2 to cpu 10 igb1: Bound queue 3 to cpu 11 igb1: Bound queue 4 to cpu 0 igb1: Bound queue 5 to cpu 1 igb1: Bound queue 6 to cpu 2 igb1: Bound queue 7 to cpu 3 igb2: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x3020-0x303f mem 0xd0b20000-0xd0b3ffff,0xd0b90000-0xd0b93fff irq 28 at device 0.2 on pci2 igb2: Using MSIX interrupts with 9 vectors igb2: Ethernet address: 00:1e:67:6d:7a:0e igb2: Bound queue 0 to cpu 4 igb2: Bound queue 1 to cpu 5 igb2: Bound queue 2 to cpu 6 igb2: Bound queue 3 to cpu 7 igb2: Bound queue 4 to cpu 8 igb2: Bound queue 5 to cpu 9 igb2: Bound queue 6 to cpu 10 igb2: Bound queue 7 to cpu 11 igb3: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x3000-0x301f mem 0xd0b00000-0xd0b1ffff,0xd0b80000-0xd0b83fff irq 29 at device 0.3 on pci2 igb3: Using MSIX interrupts with 9 vectors igb3: Ethernet address: 00:1e:67:6d:7a:0f igb3: Bound queue 0 to cpu 0 igb3: Bound queue 1 to cpu 1 igb3: Bound queue 2 to cpu 2 igb3: Bound queue 3 to cpu 3 igb3: Bound queue 4 to cpu 4 igb3: Bound queue 5 to cpu 5 igb3: Bound queue 6 to cpu 6 igb3: Bound queue 7 to cpu 7 pcib3: <ACPI PCI-PCI bridge> irq 47 at device 2.0 on pci0 pci4: <ACPI PCI bus> on pcib3 ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15> port 0x2020-0x203f mem 0xd0e20000-0xd0e3ffff,0xd0e50000-0xd0e53fff irq 32 at device 0.0 on pci4 ix0: Using MSIX interrupts with 9 vectors ix0: Bound queue 0 to cpu 0 ix0: Bound queue 1 to cpu 1 ix0: Bound queue 2 to cpu 2 ix0: Bound queue 3 to cpu 3 ix0: Bound queue 4 to cpu 4 ix0: Bound queue 5 to cpu 5 ix0: Bound queue 6 to cpu 6 ix0: Bound queue 7 to cpu 7 ix0: Ethernet address: 00:1e:67:5b:8b:f4 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15> port 0x2000-0x201f mem 0xd0e00000-0xd0e1ffff,0xd0e40000-0xd0e43fff irq 36 at device 0.1 on pci4 ix1: Using MSIX interrupts with 9 vectors ix1: Bound queue 0 to cpu 0 ix1: Bound queue 1 to cpu 1 ix1: Bound queue 2 to cpu 2 ix1: Bound queue 3 to cpu 3 ix1: Bound queue 4 to cpu 4 ix1: Bound queue 5 to cpu 5 ix1: Bound queue 6 to cpu 6 ix1: Bound queue 7 to cpu 7 ix1: Ethernet address: 00:1e:67:5b:8b:f5 ix1: PCI Express Bus: Speed 5.0GT/s Width x8 pcib4: <ACPI PCI-PCI bridge> irq 47 at device 2.2 on pci0 pci6: <ACPI PCI bus> on pcib4 mps0: <Intel(R) Integrated RAID Module RMS25JB080> port 0x1000-0x10ff mem 0xd0a40000-0xd0a4ffff,0xd0a00000-0xd0a3ffff irq 34 at device 0.0 on pci6 mps0: Firmware: 13.00.57.00, Driver: 16.00.00.00-fbsd mps0: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR> pcib5: <ACPI PCI-PCI bridge> irq 16 at device 3.0 on pci0 pci7: <ACPI PCI bus> on pcib5 pci7: <network> at device 0.0 (no driver attached) pcib6: <ACPI PCI-PCI bridge> irq 16 at device 17.0 on pci0 pci8: <ACPI PCI bus> on pcib6 pci0: <simple comms> at device 22.0 (no driver attached) pci0: <simple comms> at device 22.1 (no driver attached) ehci0: <EHCI (generic) USB 2.0 controller> mem 0xd0f20000-0xd0f203ff irq 22 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 pcib7: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pci9: <ACPI PCI bus> on pcib7 pcib8: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0 pci10: <ACPI PCI bus> on pcib8 vgapci0: <VGA-compatible display> mem 0xea000000-0xeaffffff,0xd0810000-0xd0813fff,0xd0000000-0xd07fffff irq 19 at device 0.0 on pci10 vgapci0: Boot video device ehci1: <EHCI (generic) USB 2.0 controller> mem 0xd0f10000-0xd0f103ff irq 20 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci1 pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci11: <ACPI PCI bus> on pcib9 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel Patsburg AHCI SATA controller> port 0x4070-0x4077,0x4060-0x4063,0x4050-0x4057,0x4040-0x4043,0x4020-0x403f mem 0xd0f00000-0xd0f007ff irq 21 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 ahciem0: <AHCI enclosure management bridge> on ahci0 pcib10: <ACPI Host-PCI bridge> on acpi0 pci128: <ACPI PCI bus> on pcib10 pcib11: <ACPI PCI-PCI bridge> irq 71 at device 1.0 on pci128 pci129: <ACPI PCI bus> on pcib11 pcib12: <ACPI PCI-PCI bridge> irq 71 at device 2.0 on pci128 pci130: <ACPI PCI bus> on pcib12 pcib13: <ACPI PCI-PCI bridge> irq 71 at device 3.0 on pci128 pci131: <ACPI PCI bus> on pcib13 pcib14: <ACPI PCI-PCI bridge> irq 71 at device 3.2 on pci128 pci132: <ACPI PCI bus> on pcib14 pcib15: <ACPI Host-PCI bridge> on acpi0 pci127: <ACPI PCI bus> on pcib15 pcib16: <ACPI Host-PCI bridge> on acpi0 pci255: <ACPI PCI bus> on pcib16 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range est0: <Enhanced SpeedStep Frequency Control> on cpu0 est1: <Enhanced SpeedStep Frequency Control> on cpu1 est2: <Enhanced SpeedStep Frequency Control> on cpu2 est3: <Enhanced SpeedStep Frequency Control> on cpu3 est4: <Enhanced SpeedStep Frequency Control> on cpu4 est5: <Enhanced SpeedStep Frequency Control> on cpu5 est6: <Enhanced SpeedStep Frequency Control> on cpu6 est7: <Enhanced SpeedStep Frequency Control> on cpu7 est8: <Enhanced SpeedStep Frequency Control> on cpu8 est9: <Enhanced SpeedStep Frequency Control> on cpu9 est10: <Enhanced SpeedStep Frequency Control> on cpu10 est11: <Enhanced SpeedStep Frequency Control> on cpu11 Timecounters tick every 1.000 msec random: unblocking device. usbus0: 480Mbps High Speed USB v2.0 usbus1: 480Mbps High Speed USB v2.0 ugen0.1: <Intel> at usbus0 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 ugen1.1: <Intel> at usbus1 uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen0.2: <vendor 0x8087> at usbus0 uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0 ugen1.2: <vendor 0x8087> at usbus1 uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1 uhub2: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered ugen1.3: <American Megatrends Inc.> at usbus1 ukbd0: <Keyboard Interface> on usbus1 kbd0 at ukbd0 da0 at mps0 bus 0 scbus0 target 3 lun 0 da0: <HITACHI HUC10603 CLAR300 C330> Fixed Direct Access SCSI-6 device da0: Serial Number PWHKHGPD da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C) da1 at mps0 bus 0 scbus0 target 5 lun 0 da1: <HITACHI HUC10603 CLAR300 C330> Fixed Direct Access SCSI-6 device da1: Serial Number PWHL87KF da1: 600.000MB/s transfers da1: Command Queueing enabled da1: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C) ses0 at ahciem0 bus 0 scbus7 target 0 lun 0 ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device ses0: SEMB SES Device SMP: AP CPU #9 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC" frequency 1995238860 Hz quality 1000 SMP: AP CPU #7 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC" frequency 1995238860 Hz quality 1000 WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0p2 [rw]... ... lock order reversal: 1st 0xfffffe3e57ffcea8 bufwait (bufwait) @ /scratch/freebsd/sys/kern/vfs_bio.c:3088 2nd 0xfffff8009a9fcc00 dirhash (dirhash) @ /scratch/freebsd/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe3fcf065510 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf0655c0 witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe3fcf065650 _sx_xlock() at _sx_xlock+0x75/frame 0xfffffe3fcf065690 ufsdirhash_remove() at ufsdirhash_remove+0x37/frame 0xfffffe3fcf0656c0 ufs_dirremove() at ufs_dirremove+0x11b/frame 0xfffffe3fcf065720 ufs_remove() at ufs_remove+0x75/frame 0xfffffe3fcf065780 VOP_REMOVE_APV() at VOP_REMOVE_APV+0xf7/frame 0xfffffe3fcf0657b0 kern_unlinkat() at kern_unlinkat+0x209/frame 0xfffffe3fcf0659a0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe3fcf065ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe3fcf065ab0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x8014dd8aa, rsp = 0x7fffffffce58, rbp = 0x7fffffffcf10 --- ZFS filesystem version: 5 ZFS storage pool version: features support (5000)
Thats actually a crash in UFS there not in ZFS. Can you try on a ZFS only install to see if it is strictly a UFS issue please?
The witness warning is from UFS (and it's one of the well known ones), but the panic was in the ZFS code.
The panic didn't occur on 9.3-RELEASE. Going to try an 10-STABLE/11-CURRENT snapshot..
Panic didn't occur with 10.0-STABLE @ r268571.
Confirmed that it occurs on 11-CURRENT @ r268111 when run against a ZFS root only install on an amd64 VM with 2GB of RAM, without forcing 4kB sectors on the virtual drives, and with 512MB of swap. Procedure (from scratch): 1. Install ZFS on root with http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-20140714-r268622-disc1.iso.xz . 2. Reboot the install. 3. Login as root. 4. Either fix bug 191574, or check out my code forked FreeBSD tree from GitHub: i. pkg install -y git ii. cd /usr/src iii. git clone http://github.com/yaneurabeya/freebsd . iv. git checkout isilon-atf 5. Run the `prove -rv tools/regression/zfs` as root.
A commit references this bug: Author: ngie Date: Sat Oct 25 06:10:02 UTC 2014 New revision: 273630 URL: https://svnweb.freebsd.org/changeset/base/273630 Log: Bail out of the script on FreeBSD due to deterministic panic issue PR: 191573 Sponsored by: EMC / Isilon Storage Division Changes: head/tools/regression/zfs/zpool/add/files.t
A commit references this bug: Author: smh Date: Mon Nov 17 11:32:12 UTC 2014 New revision: 274619 URL: https://svnweb.freebsd.org/changeset/base/274619 Log: Disable TRIM on file backed ZFS vdevs and fix TRIM on init After r265152 TRIM requests are ZIO_TYPE_FREE instead of ZIO_TYPE_IOCTL this meant file backed vdevs to attempted to process the ZIO as a write causing a panic. We now disable TRIM on file backed vdevs and ASSERT the ZIO types supported by each vdev type to ensure we explicity support the ZIO type being processed. Also ensure that TRIM on init is not procesed for devices which declare they didn't support TRIM via vdev_notrim. PR: 195061, 194976, 191573 Sponsored by: Multiplay Changes: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_disk.c head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_file.c head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c
A commit references this bug: Author: smh Date: Mon Nov 17 11:35:30 UTC 2014 New revision: 274620 URL: https://svnweb.freebsd.org/changeset/base/274620 Log: Revert r273630 as the panic was fixed by r274619 The panic was caused by TRIM requests run against file based vdevs as write requests. PR: 191573 Sponsored by: Multiplay Changes: head/tools/regression/zfs/zpool/add/files.t
*** This bug has been marked as a duplicate of bug 195061 ***