Bug 195061 - kernel panic on 10.1-Release when creating a file backed zpool
Summary: kernel panic on 10.1-Release when creating a file backed zpool
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.1-RC2
Hardware: amd64 Any
: --- Affects Many People
Assignee: Steven Hartland
URL:
Keywords:
: 191573 194976 196498 (view as bug list)
Depends on:
Blocks: 197156
  Show dependency treegraph
 
Reported: 2014-11-16 02:26 UTC by Robert Sevat
Modified: 2015-07-20 14:17 UTC (History)
8 users (show)

See Also:
bugmeister: mfc-stable10?
bugmeister: mfc-stable9?
bugmeister: mfc-stable8?


Attachments
Can you try this? (510 bytes, patch)
2014-11-16 09:24 UTC, Xin LI
no flags Details | Diff
Disable free on file backed vdevs (992 bytes, patch)
2014-11-16 15:47 UTC, Steven Hartland
no flags Details | Diff
Disable free on file backed vdevs (1.17 KB, patch)
2014-11-16 15:57 UTC, Steven Hartland
no flags Details | Diff
Disable free on file backed vdevs (1.17 KB, patch)
2014-11-16 15:58 UTC, Steven Hartland
no flags Details | Diff
Disable free on file backed vdevs (Against releng/10.1) (1.20 KB, patch)
2014-11-16 21:18 UTC, Steven Hartland
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Sevat 2014-11-16 02:26:56 UTC
Steps to reproduce:

dd if=/dev/zero of=/tmp/test.bin bs=1G count=1

zpool create test /tmp/test.bin

*instant kernel panic*

Some guy mentioned this on irc, so me and a few others tried this. Everyone reported an instant kernel panic. So multiple 10.1 installs on different hardware.


10.1-release panics (3 different hardware setups)
10.1-rc3 panics (virtualbox)

10.0/stable r268824 (July 2014) is unaffected (hardware)
10.0/stable r266585 (May 2014) is unaffected (hardware)
freebsd 10.0 is unaffected (virtualbox)

So somewhere between July 2014 and 10.1-rc3 a kernel panic has been introduced.

At the following link you can download the content of /var/crash of the 10.1-rc3 virtual machine.

http://media.indylix.nl/crash.tar.gz

If you need any more information, ask away.

Kind Regards,
Robert Sevat
Comment 1 Andriy Gapon freebsd_committer 2014-11-16 08:40:27 UTC
vdev_file_io_start() is broken with respect to TRIM operation.
Comment 2 Xin LI freebsd_committer 2014-11-16 09:24:25 UTC
Created attachment 149477 [details]
Can you try this?
Comment 4 Steven Hartland freebsd_committer 2014-11-16 15:25:11 UTC
Trying this with /tmp on zfs works, so given this and the trace I suspect your /tmp was a UFS formatted volume could you confirm?
Comment 5 Steven Hartland freebsd_committer 2014-11-16 15:47:35 UTC
Created attachment 149483 [details]
Disable free on file backed vdevs
Comment 6 Steven Hartland freebsd_committer 2014-11-16 15:57:11 UTC
Created attachment 149484 [details]
Disable free on file backed vdevs

Looking at what vdev_file_io_start is doing (calling vn_rdwr) its not clear to me why there should be any difference between in behaviour ZFS backed and UFS backed as they would both end up doing a write, which is of course incorrect in both cases.

The Disable free on file backed vdevs patch extends Xin's patch to ensure ZIO_TYPE_FREE can never get through to a file backed vdev even if we're called without checking vdev_notrim.
Comment 7 Steven Hartland freebsd_committer 2014-11-16 15:58:04 UTC
Created attachment 149486 [details]
Disable free on file backed vdevs

Reattach to obsolete old version
Comment 8 Steven Hartland freebsd_committer 2014-11-16 16:03:51 UTC
For reference this would be been caused by r265152 which change frees from being zio_type ZIO_TYPE_IOCTL to ZIO_TYPE_FREE.
Comment 9 Robert Sevat 2014-11-16 18:55:22 UTC
The dump was indeed from a default ufs formatted install. But I've done it on multiple machines that have zfs as rootfs too:

robert@summus:~ % zfs get all zroot/tmp
NAME       PROPERTY              VALUE                  SOURCE
zroot/tmp  type                  filesystem             -
zroot/tmp  creation              Thu Nov  6 22:38 2014  -
zroot/tmp  used                  128K                   -
zroot/tmp  available             225G                   -
zroot/tmp  referenced            128K                   -
zroot/tmp  compressratio         1.00x                  -
zroot/tmp  mounted               yes                    -
zroot/tmp  quota                 none                   default
zroot/tmp  reservation           none                   default
zroot/tmp  recordsize            128K                   default
zroot/tmp  mountpoint            /tmp                   local
zroot/tmp  sharenfs              off                    default
zroot/tmp  checksum              on                     default
zroot/tmp  compression           lz4                    inherited from zroot
zroot/tmp  atime                 off                    inherited from zroot
zroot/tmp  devices               on                     default
zroot/tmp  exec                  on                     local
zroot/tmp  setuid                off                    local
zroot/tmp  readonly              off                    default
zroot/tmp  jailed                off                    default
zroot/tmp  snapdir               hidden                 default
zroot/tmp  aclmode               discard                default
zroot/tmp  aclinherit            restricted             default
zroot/tmp  canmount              on                     default
zroot/tmp  xattr                 off                    temporary
zroot/tmp  copies                1                      default
zroot/tmp  version               5                      -
zroot/tmp  utf8only              off                    -
zroot/tmp  normalization         none                   -
zroot/tmp  casesensitivity       sensitive              -
zroot/tmp  vscan                 off                    default
zroot/tmp  nbmand                off                    default
zroot/tmp  sharesmb              off                    default
zroot/tmp  refquota              none                   default
zroot/tmp  refreservation        none                   default
zroot/tmp  primarycache          all                    default
zroot/tmp  secondarycache        all                    default
zroot/tmp  usedbysnapshots       0                      -
zroot/tmp  usedbydataset         128K                   -
zroot/tmp  usedbychildren        0                      -
zroot/tmp  usedbyrefreservation  0                      -
zroot/tmp  logbias               latency                default
zroot/tmp  dedup                 off                    default
zroot/tmp  mlslabel                                     -
zroot/tmp  sync                  standard               default
zroot/tmp  refcompressratio      1.00x                  -
zroot/tmp  written               128K                   -
zroot/tmp  logicalused           12K                    -
zroot/tmp  logicalreferenced     12K                    -
zroot/tmp  volmode               default                default
zroot/tmp  filesystem_limit      none                   default
zroot/tmp  snapshot_limit        none                   default
zroot/tmp  filesystem_count      none                   default
zroot/tmp  snapshot_count        none                   default
zroot/tmp  redundant_metadata    all                    default
Comment 10 Steven Hartland freebsd_committer 2014-11-16 21:18:56 UTC
Created attachment 149495 [details]
Disable free on file backed vdevs (Against releng/10.1)

The original is against head this is against releng/10.1 so try this one
Comment 11 Steven Hartland freebsd_committer 2014-11-16 21:20:51 UTC
(In reply to Robert Sevat from comment #9)
> The dump was indeed from a default ufs formatted install. But I've done it
> on multiple machines that have zfs as rootfs too:

Thanks Robert, think I must have got very lucky here not to trip it, or in this case unlucky as it would have been caught in testing otherwise :(
Comment 12 Robert Sevat 2014-11-16 21:57:18 UTC
On the 10.1-rc3 virtual machine, I've just checked out: /base/releng/10.1

then applied: https://bugs.freebsd.org/bugzilla/attachment.cgi?id=149495&action=diff

then build a new kernel and rebooted. 

I've then tried to reproduce it:


dd if=/dev/zero of=/tmp/test.bin bs=1G count=1

zpool create test /tmp/test.bin

works as expected. No kernel panic.

root@:/tmp # zpool create test /tmp/test.bin 
root@:/tmp # zpool status
  pool: test
 state: ONLINE
  scan: none requested
config:

	NAME             STATE     READ WRITE CKSUM
	test             ONLINE       0     0     0
	  /tmp/test.bin  ONLINE       0     0     0

errors: No known data errors


Thanks for fixing it!
Comment 13 Andriy Gapon freebsd_committer 2014-11-16 22:11:04 UTC
(In reply to Steven Hartland from comment #10)
What I would love to see is an assertion that io_type has a value that vdev_file_io_start() can handle, so that a new zio type is not silently handled as one of the existing types.
Comment 14 Steven Hartland freebsd_committer 2014-11-17 11:37:33 UTC
This was fixed by https://svnweb.freebsd.org/base?view=revision&revision=274619

Commit hook didn't trigger to note this for some reason.
Comment 15 Gavin Atkinson freebsd_committer freebsd_triage 2014-11-21 13:51:00 UTC
Backtrace from the tarball (to aid Google searches):

(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff80926b32 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:452
#2  0xffffffff80926ef4 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80d23ccf in trap_fatal (frame=<value optimized out>, 
    eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865
#4  0xffffffff80d23fe8 in trap_pfault (frame=0xfffffe01f2f8b110, 
    usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676
#5  0xffffffff80d2364a in trap (frame=0xfffffe01f2f8b110)
    at /usr/src/sys/amd64/amd64/trap.c:440
#6  0xffffffff80d09502 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff80d219a6 in bcopy () at /usr/src/sys/amd64/amd64/support.S:112
#8  0xffffffff80d24c19 in uiomove_fromphys (ma=<value optimized out>, 
    offset=<value optimized out>, n=<value optimized out>, 
    uio=<value optimized out>) at /usr/src/sys/amd64/amd64/uio_machdep.c:103
#9  0xffffffff80b7e37b in ffs_write (ap=<value optimized out>)
    at /usr/src/sys/ufs/ffs/ffs_vnops.c:764
#10 0xffffffff80e416a5 in VOP_WRITE_APV (vop=<value optimized out>, 
    a=<value optimized out>) at vnode_if.c:999
#11 0xffffffff809d6139 in vn_rdwr (rw=<value optimized out>, 
    vp=0xfffff8002f4e11d8, base=<value optimized out>, 
    len=<value optimized out>, offset=<value optimized out>, 
    segflg=<value optimized out>, ioflg=512, 
    active_cred=<value optimized out>, file_cred=<value optimized out>, 
    aresid=<value optimized out>, td=<value optimized out>) at vnode_if.h:413
#12 0xffffffff81a8ce55 in vdev_file_io_start (zio=0xfffff8002f960ac8)
    at vnode.h:237
#13 0xffffffff81aaa676 in zio_vdev_io_start (zio=<value optimized out>)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2701
#14 0xffffffff81aa7382 in zio_execute (zio=0xfffff8002f960ac8)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1414
#15 0xffffffff81aa67d3 in zio_wait (zio=0xfffff8002f960ac8)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1438
#16 0xffffffff81a8d93d in vdev_label_init (vd=0xfffff8002f4ee000, crtxg=4, 
    reason=VDEV_LABEL_CREATE)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c:718
#17 0xffffffff81a8d767 in vdev_label_init (vd=0xfffff800cf821000, crtxg=4, 
    reason=VDEV_LABEL_CREATE)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c:642
#18 0xffffffff81a89394 in vdev_create (vd=0xfffff800cf821000, txg=4, 
    isreplacing=0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:1653
#19 0xffffffff81a74477 in spa_create (pool=<value optimized out>, 
    nvroot=0xfffff80004ceaa20, props=0xfffff80004ceaa00, 
    zplprops=0xfffff80004cea9e0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3584
#20 0xffffffff81abd3dd in zfs_ioc_pool_create (zc=0xfffffe0002cb4000)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1543
#21 0xffffffff81abb8a0 in zfsdev_ioctl (dev=<value optimized out>, 
    zcmd=<value optimized out>, arg=<value optimized out>, 
    flag=<value optimized out>, td=<value optimized out>)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6157
#22 0xffffffff808167c4 in devfs_ioctl_f (fp=0xfffff80004e12e60, 
    com=3222821376, data=0xfffff80004d6f1c0, cred=<value optimized out>, 
    td=0xfffff80004e91000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
#23 0xffffffff80979845 in kern_ioctl (td=0xfffff80004e91000, 
    fd=<value optimized out>, com=0) at file.h:320
#24 0xffffffff8097959c in sys_ioctl (td=0xfffff80004e91000, 
    uap=0xfffffe01f2f8bb80) at /usr/src/sys/kern/sys_generic.c:702
#25 0xffffffff80d24601 in amd64_syscall (td=0xfffff80004e91000, traced=0)
    at subr_syscall.c:134
#26 0xffffffff80d097eb in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:391
#27 0x00000008019f9b9a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Comment 16 Xin LI freebsd_committer 2015-01-28 21:27:34 UTC
*** Bug 196498 has been marked as a duplicate of this bug. ***
Comment 17 Xin LI freebsd_committer 2015-01-28 21:29:19 UTC
*** Bug 194976 has been marked as a duplicate of this bug. ***
Comment 18 Xin LI freebsd_committer 2015-01-28 21:29:43 UTC
*** Bug 191573 has been marked as a duplicate of this bug. ***
Comment 19 Michael Dexter 2015-01-28 21:38:09 UTC
May I kindly suggest that the title be modified to mention file-backed pools to keep people, including myself from submitting duplicate reports?
Comment 20 Robert Sevat 2015-01-28 21:42:36 UTC
changed "kernel panic on 10.1-Release when creating a file backed zpool".
Comment 21 Will Green 2015-06-10 19:47:26 UTC
Fix was included in FreeBSD 10.1-RELEASE-p11: https://www.freebsd.org/security/advisories/FreeBSD-EN-15:07.zfs.asc