Bug 249055

Summary: bectl and libbe tests on openzfs panics i386 kernel
Product: Base System Reporter: Li-Wen Hsu <lwhsu>
Component: testsAssignee: freebsd-testing (Nobody) <testing>
Status: Open ---    
Severity: Affects Only Me CC: freqlabs, kevans, mmacy
Priority: --- Keywords: crash
Version: CURRENT   
Hardware: i386   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249229

Description Li-Wen Hsu freebsd_committer freebsd_triage 2020-09-01 19:20:10 UTC
https://ci.freebsd.org/job/FreeBSD-head-i386-test/10388/console

sbin/bectl/bectl_test:bectl_create  ->  can't handle raw ops yet!!!
panic: deadlres_td_on_lock: possible deadlock detected for 0xb8883a0 (swi4: clock (0)), blocked for 90005 ticks

cpuid = 1
time = 1598333085
KDB: stack backtrace:
db_trace_self_wrapper(69742035,a736b63,d566300,d4c046,1c8b8a00,...) at db_trace_self_wrapper+0x28/frame 0xd566368
kdb_backtrace(1d09fae0,1,b8883a0,15f95,b5dba84,...) at kdb_backtrace+0x2b/frame 0xd5663c0
vpanic(132102e,d5663fc,d5663fc,d566414,ea0d79,...) at vpanic+0x130/frame 0xd5663dc
panic(132102e,13176ea,b8883a0,b88856c,15f95,...) at panic+0x14/frame 0xd5663f0
deadlres_td_on_lock(15f90) at deadlres_td_on_lock+0xa9/frame 0xd566414
deadlkres(0,d566468) at deadlkres+0x10c/frame 0xd566438
fork_exit(ea0b00,0,d566468,0,0,...) at fork_exit+0x68/frame 0xd566454
fork_trampoline() at 0xffc0340e/frame 0xd566454
--- kthread start
KDB: enter: panic
[ thread pid 0 tid 100097 ]
Stopped at      breakpoint+0x4: popl    %ebp
db:0:kdb.enter.panic> show pcpu
cpuid        = 1
dynamic pcpu = 0x9e8f9c0
curthread    = 0x1d09fae0: pid 0 tid 100097 critnest 1 "deadlkres"
curpcb       = 0xd5664c0
fpcurthread  = none
idlethread   = 0xb5ea000: tid 100004 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
trampstk     = 0xffc0aff0
kesp0        = 0xd5664b0
common_tssp  = 0xffc01198
tlb gen      = 111093
curvnet      = 0
spin locks held:
db:0:kdb.enter.panic> alltrace
Comment 1 Li-Wen Hsu freebsd_committer freebsd_triage 2020-09-02 18:52:45 UTC
https://ci.freebsd.org/job/FreeBSD-head-i386-test/10389/console

lib/libbe/be_create:libbe_create  ->  can't handle raw ops yet!!!
Comment 2 commit-hook freebsd_committer freebsd_triage 2020-09-02 19:18:53 UTC
A commit references this bug:

Author: lwhsu
Date: Wed Sep  2 19:18:27 UTC 2020
New revision: 365271
URL: https://svnweb.freebsd.org/changeset/base/365271

Log:
  Temporarily skip tests panic i386 kernel in CI

  lib.libbe.be_create.libbe_create
  sbin.bectl.bectl_test.bectl_create

  PR:		249055
  Sponsored by:	The FreeBSD Foundation

Changes:
  head/lib/libbe/tests/be_create.sh
  head/sbin/bectl/tests/bectl_test.sh
Comment 3 Matt Macy freebsd_committer freebsd_triage 2020-09-02 22:38:03 UTC
Before I can proceed:

- How much memory do the CI VMs have?

- What is ARC max configured to?

- How many cores?


I tried installing the latest i386 snapshot in a VM and got checksum errors so I can't reproduce on i386 right now. I might, however, be able to configure an x86_64 VM to be crippled in a manner equivalent to i386. I have to admit that this is incredibly low priority for me. Using ZFS on i386 was fiddly at the best of times.
Comment 4 Li-Wen Hsu freebsd_committer freebsd_triage 2020-09-03 04:13:28 UTC
(In reply to Matt Macy from comment #3)
> - How much memory do the CI VMs have?

4G for i386

> - What is ARC max configured to?

Oh this might be the problem, this is the global setting and obviously not suitable for i386:

vfs.zfs.arc_max=4294967296

I'll adjust this and try again.

> - How many cores?

2

> I tried installing the latest i386 snapshot in a VM and got checksum errors so I can't reproduce on i386 right now. I might, however, be able to configure an x86_64 VM to be crippled in a manner equivalent to i386. I have to admit that this is incredibly low priority for me. Using ZFS on i386 was fiddly at the best of times.

No problem, I'm just tracking the failures and let the experts to check if they need more examination.

BTW, if you want to check the VM image used in CI, please use:
https://artifact.ci.freebsd.org/snapshot/head/latest/i386/i386/disk-test.img.zst

("latest" should be "latest_testvm" but it was gone, I'm checking what's wrong here)
Comment 5 Li-Wen Hsu freebsd_committer freebsd_triage 2020-09-03 08:09:54 UTC
It turns out that i386 hangs while loading zfs module:

root@:~ # uname -a
FreeBSD  13.0-CURRENT FreeBSD 13.0-CURRENT #0 r365284: Thu Sep  3 00:34:08 UTC 2020     root@FreeBSD-head-i386-build.jail.ci.FreeBSD.org:/usr/obj/usr/src/i386.i386/sys/GENERIC  i386

root@:~ # kldload zfs
can't handle raw ops yet!!!

amd64 has similar messages:

root@:~ # uname -a
FreeBSD  13.0-CURRENT FreeBSD 13.0-CURRENT #0 r365287: Thu Sep  3 06:17:36 UTC 2020     root@FreeBSD-head-amd64-build.jail.ci.FreeBSD.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64                                                                                                       

root@:~ # kldload zfs
can't handle raw ops yet!!!
can't handle raw ops yet!!!
can't handle raw ops yet!!!
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
can't handle raw ops yet!!!
root@:~ #
Comment 6 Matt Macy freebsd_committer freebsd_triage 2020-09-03 18:26:04 UTC
(In reply to Li-Wen Hsu from comment #4)
I need to be able to instrument the kernel to debug issues. The last snapshot does not install in vbox. I'll keep trying once a week until a snapshot comes along with an installer functional enough to install in vbox.
Comment 7 Matt Macy freebsd_committer freebsd_triage 2020-09-03 18:27:29 UTC
(In reply to Li-Wen Hsu from comment #5)
Those messages are no more than an expected reminder of missing functionality. RAW_KSTAT_TYPE support has been merged upstream and will be MFVed as soon as the corresponding changes are merged from master to 2.0.
Comment 8 commit-hook freebsd_committer freebsd_triage 2020-09-07 18:16:05 UTC
A commit references this bug:

Author: lwhsu
Date: Mon Sep  7 18:15:59 UTC 2020
New revision: 365424
URL: https://svnweb.freebsd.org/changeset/base/365424

Log:
  Call atf_skip function in cleanup procedure as it also triggers zfs.ko loading

  PR:		249055
  Sponsored by:	The FreeBSD Foundation

Changes:
  head/sbin/bectl/tests/bectl_test.sh