https://ci.freebsd.org/job/FreeBSD-head-i386-test/10388/console sbin/bectl/bectl_test:bectl_create -> can't handle raw ops yet!!! panic: deadlres_td_on_lock: possible deadlock detected for 0xb8883a0 (swi4: clock (0)), blocked for 90005 ticks cpuid = 1 time = 1598333085 KDB: stack backtrace: db_trace_self_wrapper(69742035,a736b63,d566300,d4c046,1c8b8a00,...) at db_trace_self_wrapper+0x28/frame 0xd566368 kdb_backtrace(1d09fae0,1,b8883a0,15f95,b5dba84,...) at kdb_backtrace+0x2b/frame 0xd5663c0 vpanic(132102e,d5663fc,d5663fc,d566414,ea0d79,...) at vpanic+0x130/frame 0xd5663dc panic(132102e,13176ea,b8883a0,b88856c,15f95,...) at panic+0x14/frame 0xd5663f0 deadlres_td_on_lock(15f90) at deadlres_td_on_lock+0xa9/frame 0xd566414 deadlkres(0,d566468) at deadlkres+0x10c/frame 0xd566438 fork_exit(ea0b00,0,d566468,0,0,...) at fork_exit+0x68/frame 0xd566454 fork_trampoline() at 0xffc0340e/frame 0xd566454 --- kthread start KDB: enter: panic [ thread pid 0 tid 100097 ] Stopped at breakpoint+0x4: popl %ebp db:0:kdb.enter.panic> show pcpu cpuid = 1 dynamic pcpu = 0x9e8f9c0 curthread = 0x1d09fae0: pid 0 tid 100097 critnest 1 "deadlkres" curpcb = 0xd5664c0 fpcurthread = none idlethread = 0xb5ea000: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0xd5664b0 common_tssp = 0xffc01198 tlb gen = 111093 curvnet = 0 spin locks held: db:0:kdb.enter.panic> alltrace
https://ci.freebsd.org/job/FreeBSD-head-i386-test/10389/console lib/libbe/be_create:libbe_create -> can't handle raw ops yet!!!
A commit references this bug: Author: lwhsu Date: Wed Sep 2 19:18:27 UTC 2020 New revision: 365271 URL: https://svnweb.freebsd.org/changeset/base/365271 Log: Temporarily skip tests panic i386 kernel in CI lib.libbe.be_create.libbe_create sbin.bectl.bectl_test.bectl_create PR: 249055 Sponsored by: The FreeBSD Foundation Changes: head/lib/libbe/tests/be_create.sh head/sbin/bectl/tests/bectl_test.sh
Before I can proceed: - How much memory do the CI VMs have? - What is ARC max configured to? - How many cores? I tried installing the latest i386 snapshot in a VM and got checksum errors so I can't reproduce on i386 right now. I might, however, be able to configure an x86_64 VM to be crippled in a manner equivalent to i386. I have to admit that this is incredibly low priority for me. Using ZFS on i386 was fiddly at the best of times.
(In reply to Matt Macy from comment #3) > - How much memory do the CI VMs have? 4G for i386 > - What is ARC max configured to? Oh this might be the problem, this is the global setting and obviously not suitable for i386: vfs.zfs.arc_max=4294967296 I'll adjust this and try again. > - How many cores? 2 > I tried installing the latest i386 snapshot in a VM and got checksum errors so I can't reproduce on i386 right now. I might, however, be able to configure an x86_64 VM to be crippled in a manner equivalent to i386. I have to admit that this is incredibly low priority for me. Using ZFS on i386 was fiddly at the best of times. No problem, I'm just tracking the failures and let the experts to check if they need more examination. BTW, if you want to check the VM image used in CI, please use: https://artifact.ci.freebsd.org/snapshot/head/latest/i386/i386/disk-test.img.zst ("latest" should be "latest_testvm" but it was gone, I'm checking what's wrong here)
It turns out that i386 hangs while loading zfs module: root@:~ # uname -a FreeBSD 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r365284: Thu Sep 3 00:34:08 UTC 2020 root@FreeBSD-head-i386-build.jail.ci.FreeBSD.org:/usr/obj/usr/src/i386.i386/sys/GENERIC i386 root@:~ # kldload zfs can't handle raw ops yet!!! amd64 has similar messages: root@:~ # uname -a FreeBSD 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r365287: Thu Sep 3 06:17:36 UTC 2020 root@FreeBSD-head-amd64-build.jail.ci.FreeBSD.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 root@:~ # kldload zfs can't handle raw ops yet!!! can't handle raw ops yet!!! can't handle raw ops yet!!! ZFS filesystem version: 5 ZFS storage pool version: features support (5000) can't handle raw ops yet!!! root@:~ #
(In reply to Li-Wen Hsu from comment #4) I need to be able to instrument the kernel to debug issues. The last snapshot does not install in vbox. I'll keep trying once a week until a snapshot comes along with an installer functional enough to install in vbox.
(In reply to Li-Wen Hsu from comment #5) Those messages are no more than an expected reminder of missing functionality. RAW_KSTAT_TYPE support has been merged upstream and will be MFVed as soon as the corresponding changes are merged from master to 2.0.
A commit references this bug: Author: lwhsu Date: Mon Sep 7 18:15:59 UTC 2020 New revision: 365424 URL: https://svnweb.freebsd.org/changeset/base/365424 Log: Call atf_skip function in cleanup procedure as it also triggers zfs.ko loading PR: 249055 Sponsored by: The FreeBSD Foundation Changes: head/sbin/bectl/tests/bectl_test.sh