Steps to reproduce: dd if=/dev/zero of=/tmp/test.bin bs=1G count=1 zpool create test /tmp/test.bin *instant kernel panic* Some guy mentioned this on irc, so me and a few others tried this. Everyone reported an instant kernel panic. So multiple 10.1 installs on different hardware. 10.1-release panics (3 different hardware setups) 10.1-rc3 panics (virtualbox) 10.0/stable r268824 (July 2014) is unaffected (hardware) 10.0/stable r266585 (May 2014) is unaffected (hardware) freebsd 10.0 is unaffected (virtualbox) So somewhere between July 2014 and 10.1-rc3 a kernel panic has been introduced. At the following link you can download the content of /var/crash of the 10.1-rc3 virtual machine. http://media.indylix.nl/crash.tar.gz If you need any more information, ask away. Kind Regards, Robert Sevat
vdev_file_io_start() is broken with respect to TRIM operation.
Created attachment 149477 [details] Can you try this?
Possibly related bugs: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194976 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191573
Trying this with /tmp on zfs works, so given this and the trace I suspect your /tmp was a UFS formatted volume could you confirm?
Created attachment 149483 [details] Disable free on file backed vdevs
Created attachment 149484 [details] Disable free on file backed vdevs Looking at what vdev_file_io_start is doing (calling vn_rdwr) its not clear to me why there should be any difference between in behaviour ZFS backed and UFS backed as they would both end up doing a write, which is of course incorrect in both cases. The Disable free on file backed vdevs patch extends Xin's patch to ensure ZIO_TYPE_FREE can never get through to a file backed vdev even if we're called without checking vdev_notrim.
Created attachment 149486 [details] Disable free on file backed vdevs Reattach to obsolete old version
For reference this would be been caused by r265152 which change frees from being zio_type ZIO_TYPE_IOCTL to ZIO_TYPE_FREE.
The dump was indeed from a default ufs formatted install. But I've done it on multiple machines that have zfs as rootfs too: robert@summus:~ % zfs get all zroot/tmp NAME PROPERTY VALUE SOURCE zroot/tmp type filesystem - zroot/tmp creation Thu Nov 6 22:38 2014 - zroot/tmp used 128K - zroot/tmp available 225G - zroot/tmp referenced 128K - zroot/tmp compressratio 1.00x - zroot/tmp mounted yes - zroot/tmp quota none default zroot/tmp reservation none default zroot/tmp recordsize 128K default zroot/tmp mountpoint /tmp local zroot/tmp sharenfs off default zroot/tmp checksum on default zroot/tmp compression lz4 inherited from zroot zroot/tmp atime off inherited from zroot zroot/tmp devices on default zroot/tmp exec on local zroot/tmp setuid off local zroot/tmp readonly off default zroot/tmp jailed off default zroot/tmp snapdir hidden default zroot/tmp aclmode discard default zroot/tmp aclinherit restricted default zroot/tmp canmount on default zroot/tmp xattr off temporary zroot/tmp copies 1 default zroot/tmp version 5 - zroot/tmp utf8only off - zroot/tmp normalization none - zroot/tmp casesensitivity sensitive - zroot/tmp vscan off default zroot/tmp nbmand off default zroot/tmp sharesmb off default zroot/tmp refquota none default zroot/tmp refreservation none default zroot/tmp primarycache all default zroot/tmp secondarycache all default zroot/tmp usedbysnapshots 0 - zroot/tmp usedbydataset 128K - zroot/tmp usedbychildren 0 - zroot/tmp usedbyrefreservation 0 - zroot/tmp logbias latency default zroot/tmp dedup off default zroot/tmp mlslabel - zroot/tmp sync standard default zroot/tmp refcompressratio 1.00x - zroot/tmp written 128K - zroot/tmp logicalused 12K - zroot/tmp logicalreferenced 12K - zroot/tmp volmode default default zroot/tmp filesystem_limit none default zroot/tmp snapshot_limit none default zroot/tmp filesystem_count none default zroot/tmp snapshot_count none default zroot/tmp redundant_metadata all default
Created attachment 149495 [details] Disable free on file backed vdevs (Against releng/10.1) The original is against head this is against releng/10.1 so try this one
(In reply to Robert Sevat from comment #9) > The dump was indeed from a default ufs formatted install. But I've done it > on multiple machines that have zfs as rootfs too: Thanks Robert, think I must have got very lucky here not to trip it, or in this case unlucky as it would have been caught in testing otherwise :(
On the 10.1-rc3 virtual machine, I've just checked out: /base/releng/10.1 then applied: https://bugs.freebsd.org/bugzilla/attachment.cgi?id=149495&action=diff then build a new kernel and rebooted. I've then tried to reproduce it: dd if=/dev/zero of=/tmp/test.bin bs=1G count=1 zpool create test /tmp/test.bin works as expected. No kernel panic. root@:/tmp # zpool create test /tmp/test.bin root@:/tmp # zpool status pool: test state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 /tmp/test.bin ONLINE 0 0 0 errors: No known data errors Thanks for fixing it!
(In reply to Steven Hartland from comment #10) What I would love to see is an assertion that io_type has a value that vdev_file_io_start() can handle, so that a new zio type is not silently handled as one of the existing types.
This was fixed by https://svnweb.freebsd.org/base?view=revision&revision=274619 Commit hook didn't trigger to note this for some reason.
Backtrace from the tarball (to aid Google searches): (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219 #1 0xffffffff80926b32 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff80926ef4 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80d23ccf in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865 #4 0xffffffff80d23fe8 in trap_pfault (frame=0xfffffe01f2f8b110, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676 #5 0xffffffff80d2364a in trap (frame=0xfffffe01f2f8b110) at /usr/src/sys/amd64/amd64/trap.c:440 #6 0xffffffff80d09502 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #7 0xffffffff80d219a6 in bcopy () at /usr/src/sys/amd64/amd64/support.S:112 #8 0xffffffff80d24c19 in uiomove_fromphys (ma=<value optimized out>, offset=<value optimized out>, n=<value optimized out>, uio=<value optimized out>) at /usr/src/sys/amd64/amd64/uio_machdep.c:103 #9 0xffffffff80b7e37b in ffs_write (ap=<value optimized out>) at /usr/src/sys/ufs/ffs/ffs_vnops.c:764 #10 0xffffffff80e416a5 in VOP_WRITE_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:999 #11 0xffffffff809d6139 in vn_rdwr (rw=<value optimized out>, vp=0xfffff8002f4e11d8, base=<value optimized out>, len=<value optimized out>, offset=<value optimized out>, segflg=<value optimized out>, ioflg=512, active_cred=<value optimized out>, file_cred=<value optimized out>, aresid=<value optimized out>, td=<value optimized out>) at vnode_if.h:413 #12 0xffffffff81a8ce55 in vdev_file_io_start (zio=0xfffff8002f960ac8) at vnode.h:237 #13 0xffffffff81aaa676 in zio_vdev_io_start (zio=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2701 #14 0xffffffff81aa7382 in zio_execute (zio=0xfffff8002f960ac8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1414 #15 0xffffffff81aa67d3 in zio_wait (zio=0xfffff8002f960ac8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1438 #16 0xffffffff81a8d93d in vdev_label_init (vd=0xfffff8002f4ee000, crtxg=4, reason=VDEV_LABEL_CREATE) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c:718 #17 0xffffffff81a8d767 in vdev_label_init (vd=0xfffff800cf821000, crtxg=4, reason=VDEV_LABEL_CREATE) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c:642 #18 0xffffffff81a89394 in vdev_create (vd=0xfffff800cf821000, txg=4, isreplacing=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:1653 #19 0xffffffff81a74477 in spa_create (pool=<value optimized out>, nvroot=0xfffff80004ceaa20, props=0xfffff80004ceaa00, zplprops=0xfffff80004cea9e0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3584 #20 0xffffffff81abd3dd in zfs_ioc_pool_create (zc=0xfffffe0002cb4000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1543 #21 0xffffffff81abb8a0 in zfsdev_ioctl (dev=<value optimized out>, zcmd=<value optimized out>, arg=<value optimized out>, flag=<value optimized out>, td=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6157 #22 0xffffffff808167c4 in devfs_ioctl_f (fp=0xfffff80004e12e60, com=3222821376, data=0xfffff80004d6f1c0, cred=<value optimized out>, td=0xfffff80004e91000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #23 0xffffffff80979845 in kern_ioctl (td=0xfffff80004e91000, fd=<value optimized out>, com=0) at file.h:320 #24 0xffffffff8097959c in sys_ioctl (td=0xfffff80004e91000, uap=0xfffffe01f2f8bb80) at /usr/src/sys/kern/sys_generic.c:702 #25 0xffffffff80d24601 in amd64_syscall (td=0xfffff80004e91000, traced=0) at subr_syscall.c:134 #26 0xffffffff80d097eb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 #27 0x00000008019f9b9a in ?? () Previous frame inner to this frame (corrupt stack?)
*** Bug 196498 has been marked as a duplicate of this bug. ***
*** Bug 194976 has been marked as a duplicate of this bug. ***
*** Bug 191573 has been marked as a duplicate of this bug. ***
May I kindly suggest that the title be modified to mention file-backed pools to keep people, including myself from submitting duplicate reports?
changed "kernel panic on 10.1-Release when creating a file backed zpool".
Fix was included in FreeBSD 10.1-RELEASE-p11: https://www.freebsd.org/security/advisories/FreeBSD-EN-15:07.zfs.asc