Bug 195061

Summary: kernel panic on 10.1-Release when creating a file backed zpool
Product: Base System Reporter: Robert Sevat <robert>
Component: kernAssignee: Steven Hartland <smh>
Status: Closed FIXED    
Severity: Affects Many People CC: delphij, editor, eugene.kiev, girgen, ja, ngie, smh, will
Priority: --- Flags: bugmeister: mfc-stable10?
bugmeister: mfc-stable9?
bugmeister: mfc-stable8?
Version: 10.1-RC2   
Hardware: amd64   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194976
Bug Depends on:    
Bug Blocks: 197156    
Description Flags
Can you try this?
Disable free on file backed vdevs
Disable free on file backed vdevs
Disable free on file backed vdevs
Disable free on file backed vdevs (Against releng/10.1) none

Description Robert Sevat 2014-11-16 02:26:56 UTC
Steps to reproduce:

dd if=/dev/zero of=/tmp/test.bin bs=1G count=1

zpool create test /tmp/test.bin

*instant kernel panic*

Some guy mentioned this on irc, so me and a few others tried this. Everyone reported an instant kernel panic. So multiple 10.1 installs on different hardware.

10.1-release panics (3 different hardware setups)
10.1-rc3 panics (virtualbox)

10.0/stable r268824 (July 2014) is unaffected (hardware)
10.0/stable r266585 (May 2014) is unaffected (hardware)
freebsd 10.0 is unaffected (virtualbox)

So somewhere between July 2014 and 10.1-rc3 a kernel panic has been introduced.

At the following link you can download the content of /var/crash of the 10.1-rc3 virtual machine.


If you need any more information, ask away.

Kind Regards,
Robert Sevat
Comment 1 Andriy Gapon freebsd_committer 2014-11-16 08:40:27 UTC
vdev_file_io_start() is broken with respect to TRIM operation.
Comment 2 Xin LI freebsd_committer 2014-11-16 09:24:25 UTC
Created attachment 149477 [details]
Can you try this?
Comment 4 Steven Hartland freebsd_committer 2014-11-16 15:25:11 UTC
Trying this with /tmp on zfs works, so given this and the trace I suspect your /tmp was a UFS formatted volume could you confirm?
Comment 5 Steven Hartland freebsd_committer 2014-11-16 15:47:35 UTC
Created attachment 149483 [details]
Disable free on file backed vdevs
Comment 6 Steven Hartland freebsd_committer 2014-11-16 15:57:11 UTC
Created attachment 149484 [details]
Disable free on file backed vdevs

Looking at what vdev_file_io_start is doing (calling vn_rdwr) its not clear to me why there should be any difference between in behaviour ZFS backed and UFS backed as they would both end up doing a write, which is of course incorrect in both cases.

The Disable free on file backed vdevs patch extends Xin's patch to ensure ZIO_TYPE_FREE can never get through to a file backed vdev even if we're called without checking vdev_notrim.
Comment 7 Steven Hartland freebsd_committer 2014-11-16 15:58:04 UTC
Created attachment 149486 [details]
Disable free on file backed vdevs

Reattach to obsolete old version
Comment 8 Steven Hartland freebsd_committer 2014-11-16 16:03:51 UTC
For reference this would be been caused by r265152 which change frees from being zio_type ZIO_TYPE_IOCTL to ZIO_TYPE_FREE.
Comment 9 Robert Sevat 2014-11-16 18:55:22 UTC
The dump was indeed from a default ufs formatted install. But I've done it on multiple machines that have zfs as rootfs too:

robert@summus:~ % zfs get all zroot/tmp
NAME       PROPERTY              VALUE                  SOURCE
zroot/tmp  type                  filesystem             -
zroot/tmp  creation              Thu Nov  6 22:38 2014  -
zroot/tmp  used                  128K                   -
zroot/tmp  available             225G                   -
zroot/tmp  referenced            128K                   -
zroot/tmp  compressratio         1.00x                  -
zroot/tmp  mounted               yes                    -
zroot/tmp  quota                 none                   default
zroot/tmp  reservation           none                   default
zroot/tmp  recordsize            128K                   default
zroot/tmp  mountpoint            /tmp                   local
zroot/tmp  sharenfs              off                    default
zroot/tmp  checksum              on                     default
zroot/tmp  compression           lz4                    inherited from zroot
zroot/tmp  atime                 off                    inherited from zroot
zroot/tmp  devices               on                     default
zroot/tmp  exec                  on                     local
zroot/tmp  setuid                off                    local
zroot/tmp  readonly              off                    default
zroot/tmp  jailed                off                    default
zroot/tmp  snapdir               hidden                 default
zroot/tmp  aclmode               discard                default
zroot/tmp  aclinherit            restricted             default
zroot/tmp  canmount              on                     default
zroot/tmp  xattr                 off                    temporary
zroot/tmp  copies                1                      default
zroot/tmp  version               5                      -
zroot/tmp  utf8only              off                    -
zroot/tmp  normalization         none                   -
zroot/tmp  casesensitivity       sensitive              -
zroot/tmp  vscan                 off                    default
zroot/tmp  nbmand                off                    default
zroot/tmp  sharesmb              off                    default
zroot/tmp  refquota              none                   default
zroot/tmp  refreservation        none                   default
zroot/tmp  primarycache          all                    default
zroot/tmp  secondarycache        all                    default
zroot/tmp  usedbysnapshots       0                      -
zroot/tmp  usedbydataset         128K                   -
zroot/tmp  usedbychildren        0                      -
zroot/tmp  usedbyrefreservation  0                      -
zroot/tmp  logbias               latency                default
zroot/tmp  dedup                 off                    default
zroot/tmp  mlslabel                                     -
zroot/tmp  sync                  standard               default
zroot/tmp  refcompressratio      1.00x                  -
zroot/tmp  written               128K                   -
zroot/tmp  logicalused           12K                    -
zroot/tmp  logicalreferenced     12K                    -
zroot/tmp  volmode               default                default
zroot/tmp  filesystem_limit      none                   default
zroot/tmp  snapshot_limit        none                   default
zroot/tmp  filesystem_count      none                   default
zroot/tmp  snapshot_count        none                   default
zroot/tmp  redundant_metadata    all                    default
Comment 10 Steven Hartland freebsd_committer 2014-11-16 21:18:56 UTC
Created attachment 149495 [details]
Disable free on file backed vdevs (Against releng/10.1)

The original is against head this is against releng/10.1 so try this one
Comment 11 Steven Hartland freebsd_committer 2014-11-16 21:20:51 UTC
(In reply to Robert Sevat from comment #9)
> The dump was indeed from a default ufs formatted install. But I've done it
> on multiple machines that have zfs as rootfs too:

Thanks Robert, think I must have got very lucky here not to trip it, or in this case unlucky as it would have been caught in testing otherwise :(
Comment 12 Robert Sevat 2014-11-16 21:57:18 UTC
On the 10.1-rc3 virtual machine, I've just checked out: /base/releng/10.1

then applied: https://bugs.freebsd.org/bugzilla/attachment.cgi?id=149495&action=diff

then build a new kernel and rebooted. 

I've then tried to reproduce it:

dd if=/dev/zero of=/tmp/test.bin bs=1G count=1

zpool create test /tmp/test.bin

works as expected. No kernel panic.

root@:/tmp # zpool create test /tmp/test.bin 
root@:/tmp # zpool status
  pool: test
 state: ONLINE
  scan: none requested

	test             ONLINE       0     0     0
	  /tmp/test.bin  ONLINE       0     0     0

errors: No known data errors

Thanks for fixing it!
Comment 13 Andriy Gapon freebsd_committer 2014-11-16 22:11:04 UTC
(In reply to Steven Hartland from comment #10)
What I would love to see is an assertion that io_type has a value that vdev_file_io_start() can handle, so that a new zio type is not silently handled as one of the existing types.
Comment 14 Steven Hartland freebsd_committer 2014-11-17 11:37:33 UTC
This was fixed by https://svnweb.freebsd.org/base?view=revision&revision=274619

Commit hook didn't trigger to note this for some reason.
Comment 15 Gavin Atkinson freebsd_committer freebsd_triage 2014-11-21 13:51:00 UTC
Backtrace from the tarball (to aid Google searches):

(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff80926b32 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:452
#2  0xffffffff80926ef4 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80d23ccf in trap_fatal (frame=<value optimized out>, 
    eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865
#4  0xffffffff80d23fe8 in trap_pfault (frame=0xfffffe01f2f8b110, 
    usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676
#5  0xffffffff80d2364a in trap (frame=0xfffffe01f2f8b110)
    at /usr/src/sys/amd64/amd64/trap.c:440
#6  0xffffffff80d09502 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff80d219a6 in bcopy () at /usr/src/sys/amd64/amd64/support.S:112
#8  0xffffffff80d24c19 in uiomove_fromphys (ma=<value optimized out>, 
    offset=<value optimized out>, n=<value optimized out>, 
    uio=<value optimized out>) at /usr/src/sys/amd64/amd64/uio_machdep.c:103
#9  0xffffffff80b7e37b in ffs_write (ap=<value optimized out>)
    at /usr/src/sys/ufs/ffs/ffs_vnops.c:764
#10 0xffffffff80e416a5 in VOP_WRITE_APV (vop=<value optimized out>, 
    a=<value optimized out>) at vnode_if.c:999
#11 0xffffffff809d6139 in vn_rdwr (rw=<value optimized out>, 
    vp=0xfffff8002f4e11d8, base=<value optimized out>, 
    len=<value optimized out>, offset=<value optimized out>, 
    segflg=<value optimized out>, ioflg=512, 
    active_cred=<value optimized out>, file_cred=<value optimized out>, 
    aresid=<value optimized out>, td=<value optimized out>) at vnode_if.h:413
#12 0xffffffff81a8ce55 in vdev_file_io_start (zio=0xfffff8002f960ac8)
    at vnode.h:237
#13 0xffffffff81aaa676 in zio_vdev_io_start (zio=<value optimized out>)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2701
#14 0xffffffff81aa7382 in zio_execute (zio=0xfffff8002f960ac8)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1414
#15 0xffffffff81aa67d3 in zio_wait (zio=0xfffff8002f960ac8)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1438
#16 0xffffffff81a8d93d in vdev_label_init (vd=0xfffff8002f4ee000, crtxg=4, 
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c:718
#17 0xffffffff81a8d767 in vdev_label_init (vd=0xfffff800cf821000, crtxg=4, 
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c:642
#18 0xffffffff81a89394 in vdev_create (vd=0xfffff800cf821000, txg=4, 
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:1653
#19 0xffffffff81a74477 in spa_create (pool=<value optimized out>, 
    nvroot=0xfffff80004ceaa20, props=0xfffff80004ceaa00, 
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3584
#20 0xffffffff81abd3dd in zfs_ioc_pool_create (zc=0xfffffe0002cb4000)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1543
#21 0xffffffff81abb8a0 in zfsdev_ioctl (dev=<value optimized out>, 
    zcmd=<value optimized out>, arg=<value optimized out>, 
    flag=<value optimized out>, td=<value optimized out>)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6157
#22 0xffffffff808167c4 in devfs_ioctl_f (fp=0xfffff80004e12e60, 
    com=3222821376, data=0xfffff80004d6f1c0, cred=<value optimized out>, 
    td=0xfffff80004e91000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
#23 0xffffffff80979845 in kern_ioctl (td=0xfffff80004e91000, 
    fd=<value optimized out>, com=0) at file.h:320
#24 0xffffffff8097959c in sys_ioctl (td=0xfffff80004e91000, 
    uap=0xfffffe01f2f8bb80) at /usr/src/sys/kern/sys_generic.c:702
#25 0xffffffff80d24601 in amd64_syscall (td=0xfffff80004e91000, traced=0)
    at subr_syscall.c:134
#26 0xffffffff80d097eb in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:391
#27 0x00000008019f9b9a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Comment 16 Xin LI freebsd_committer 2015-01-28 21:27:34 UTC
*** Bug 196498 has been marked as a duplicate of this bug. ***
Comment 17 Xin LI freebsd_committer 2015-01-28 21:29:19 UTC
*** Bug 194976 has been marked as a duplicate of this bug. ***
Comment 18 Xin LI freebsd_committer 2015-01-28 21:29:43 UTC
*** Bug 191573 has been marked as a duplicate of this bug. ***
Comment 19 Michael Dexter 2015-01-28 21:38:09 UTC
May I kindly suggest that the title be modified to mention file-backed pools to keep people, including myself from submitting duplicate reports?
Comment 20 Robert Sevat 2015-01-28 21:42:36 UTC
changed "kernel panic on 10.1-Release when creating a file backed zpool".
Comment 21 Will Green 2015-06-10 19:47:26 UTC
Fix was included in FreeBSD 10.1-RELEASE-p11: https://www.freebsd.org/security/advisories/FreeBSD-EN-15:07.zfs.asc