Bug 201859 - Kernel panic after every reboot (ZFS)
Summary: Kernel panic after every reboot (ZFS)
Status: Closed Not A Bug
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 10.1-STABLE
Hardware: i386 Any
: --- Affects Only Me
Assignee: freebsd-fs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-07-24 19:29 UTC by Licho
Modified: 2015-07-29 08:19 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Licho 2015-07-24 19:29:58 UTC
After every reboot of my EeePC i see kernel panic when FreeBSD tries to mount root filesystem (ZFS). Only booting from live cd and typing "zpool import -fR /mnt root && zpool export root" fixes this issue, but it's repeating on every reboot and many (not all) shutdowns.

Shutdown procedure didn't show any errors on console, so I'm thinking that disk could be not fully synced before powering off.

I have ZFS on whole diks, configuration below.

--------------------------------------------------------------------------------

└──> gpart show ada0
=>      34  31522109  ada0  GPT  (15G)
        34      1024     1  freebsd-boot  (512K)
      1058  31456256     2  freebsd-zfs  (15G)
  31457314     64829        - free -  (32M)

--------------------------------------------------------------------------------

└──> zpool status root
  pool: root
 state: ONLINE
  scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        root           ONLINE       0     0     0
          gpt/freebsd  ONLINE       0     0     0

errors: No known data errors

--------------------------------------------------------------------------------

└──> zpool get all root
NAME  PROPERTY                       VALUE                          SOURCE
root  size                           14.9G                          -
root  capacity                       39%                            -
root  altroot                        -                              default
root  health                         ONLINE                         -
root  guid                           16556967936092608241           default
root  version                        -                              default
root  bootfs                         root                           local
root  delegation                     on                             default
root  autoreplace                    off                            default
root  cachefile                      -                              default
root  failmode                       wait                           default
root  listsnapshots                  off                            default
root  autoexpand                     on                             local
root  dedupditto                     0                              default
root  dedupratio                     1.00x                          -
root  free                           9.05G                          -
root  allocated                      5.82G                          -
root  readonly                       off                            -
root  comment                        -                              default
root  expandsize                     0                              -
root  freeing                        0                              default
root  fragmentation                  21%                            -
root  leaked                         0                              default
root  feature@async_destroy          enabled                        local
root  feature@empty_bpobj            active                         local
root  feature@lz4_compress           active                         local
root  feature@multi_vdev_crash_dump  enabled                        local
root  feature@spacemap_histogram     active                         local
root  feature@enabled_txg            active                         local
root  feature@hole_birth             active                         local
root  feature@extensible_dataset     enabled                        local
root  feature@embedded_data          active                         local
root  feature@bookmarks              enabled                        local
root  feature@filesystem_limits      enabled                        local

--------------------------------------------------------------------------------

└──> zfs get all root
NAME  PROPERTY              VALUE                  SOURCE
root  type                  filesystem             -
root  creation              Thu Jul 23  1:27 2015  -
root  used                  7.88G                  -
root  available             6.53G                  -
root  referenced            179M                   -
root  compressratio         1.94x                  -
root  mounted               yes                    -
root  quota                 none                   default
root  reservation           none                   default
root  recordsize            128K                   default
root  mountpoint            /                      local
root  sharenfs              off                    default
root  checksum              sha256                 local
root  compression           lz4                    local
root  atime                 off                    local
root  devices               on                     default
root  exec                  on                     default
root  setuid                on                     default
root  readonly              off                    default
root  jailed                off                    default
root  snapdir               visible                local
root  aclmode               discard                default
root  aclinherit            restricted             default
root  canmount              on                     default
root  xattr                 off                    temporary
root  copies                1                      default
root  version               5                      -
root  utf8only              on                     -
root  normalization         none                   -
root  casesensitivity       sensitive              -
root  vscan                 off                    default
root  nbmand                off                    default
root  sharesmb              off                    default
root  refquota              none                   default
root  refreservation        none                   default
root  primarycache          all                    default
root  secondarycache        all                    default
root  usedbysnapshots       0                      -
root  usedbydataset         179M                   -
root  usedbychildren        7.71G                  -
root  usedbyrefreservation  0                      -
root  logbias               latency                default
root  dedup                 off                    default
root  mlslabel                                     -
root  sync                  standard               default
root  refcompressratio      1.71x                  -
root  written               179M                   -
root  logicalused           11.2G                  -
root  logicalreferenced     307M                   -
root  volmode               default                default
root  filesystem_limit      none                   default
root  snapshot_limit        none                   default
root  filesystem_count      none                   default
root  snapshot_count        none                   default
root  redundant_metadata    all                    default

--------------------------------------------------------------------------------

└──> cat /boot/loader.conf 
#Bootloader
beastie_disable="YES"
autoboot_delay=-1


#Modules
zfs_load="YES"
aio_load="YES"
i915kms_load="YES"


#Kernel memory
vm.kmem_size=512M
vm.kmem_size_min=512M
vm.kmem_size_max=512M


#ZFS
vfs.zfs.recover=1
vfs.zfs.check_hostid=0

vfs.zfs.zio.use_uma=1
vfs.zfs.prefetch_disable=1

vfs.zfs.arc_min=256M
vfs.zfs.arc_max=256M
vfs.zfs.vdev.cache.size=5M


#Display
kern.vty=vt


#HW tunables
hw.pci.do_power_nodriver=1
hw.psm.synaptics_support=1
kern.timecounter.hardware=TSC
drm.i915.enable_rc6=7


#Limits
kern.maxfiles=65536
kern.ipc.maxsockets=65536


#Shared memory
kern.ipc.shmseg=4096
kern.ipc.shmmni=4096

--------------------------------------------------------------------------------

And obviously zfs_enable="YES" in /etc/rc.conf.
Comment 1 Licho 2015-07-24 19:34:43 UTC
PC is EeePC 900 with Pentium M 900MHz, 2GB of RAM and 16GB SSD running on GENERIC kernel.
Comment 2 Steven Hartland freebsd_committer 2015-07-28 08:39:09 UTC
You cant run ZFS with GENERIC on i386 as you need the custom option:
options KSTACK_PAGES=4

Of your kernel config, see the UPDATING entry 20121223.
Comment 3 Licho 2015-07-28 09:50:04 UTC
(In reply to Steven Hartland from comment #2)

>> FreeBSD/i386 10.1-RELEASE configured with a multi-disk ZFS dataset (mirror, raidz1, raidz2, raidz3) may crash during boot when the ZFS pool mount is attempted while booting an unmodified GENERIC kernel. (https://www.freebsd.org/releases/10.1R/errata.html)

It's single drive pool, also panics are only after machine reboot after I set kern.shutdown.poweroff_delay=60000 - on shutdown and cold boot now it works fine every time.
Comment 4 Fabian Keil 2015-07-28 09:55:56 UTC
Given how many users run into this, I think the zfs module should emit a warning when KSTACK_PAGES is too low, just like it already does if there's less RAM than recommended in general or for prefetching.
Comment 5 Steven Hartland freebsd_committer 2015-07-28 09:58:39 UTC
It could still be stack space, impossible to tell without the panic info; so try with the correct kernel options and see if that fixes the issue.

If it still fails please attach the stack trace from the panic.

You may need to set the following in /etc/rc.conf to get a crash dump for the full panic information:
dumpdev="AUTO"
Comment 6 Steven Hartland freebsd_committer 2015-07-28 11:23:02 UTC
(In reply to fk from comment #4)
Agreed, committed as https://svnweb.freebsd.org/changeset/base/285947
Comment 7 Licho 2015-07-28 17:45:17 UTC
Seems like KSTACK_PAGES works for me. Is it possible that this will be default behavior on generic kernels in next release (10.2 or 11.0)?
Comment 8 Glen Barber freebsd_committer 2015-07-28 17:55:36 UTC
(In reply to Licho from comment #7)
> Seems like KSTACK_PAGES works for me. Is it possible that this will be
> default behavior on generic kernels in next release (10.2 or 11.0)?

(After referring to email archives from the 10.1-RELEASE cycle...)

This cannot be made the default for i386, because increasing the stack pages significantly limits the number of userland threads, which will eventually lead to KVA exhaustion.
Comment 9 Steven Hartland freebsd_committer 2015-07-29 08:19:41 UTC
Closing at is was a KSTACK_PAGES issue, which is documented.