Bug 247153

Summary: emulators/virtualbox-ose-kmod System crash on VM resume after system upgrade
Product: Base System Reporter: rkoberman
Component: kernAssignee: Mark Johnston <markj>
Status: Closed FIXED    
Severity: Affects Only Me CC: markj
Priority: ---    
Version: 12.1-STABLE   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
proposed patch none

Description rkoberman 2020-06-10 21:59:27 UTC
After I updated my 12-STABLE system yesterday I can no longer resume a paused VM (Windows7). I get a system crash. I rolled back to my previous version, r361980 but the VM shows "Powered off" and the system still crashes at some point in the VM boot. Full dumps available.

Sue to all of the drm related stuff in hte backtrace, I am going to try backingupto a slightly earlier kernel and see if I can get the VM to boot.

Both drm-fbsd12.0-kmod-4.16.g20200221  and virtualbox-ose-kmod-5.2.34 were rebuilt after the kernel build using PORTS_MODULE.

Here is the backtrace from the first crash.
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x105887000
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff82a93ce0
stack pointer           = 0x28:0xfffffe004e780390
frame pointer           = 0x28:0xfffffe004e7803f0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 2161 (VirtualBox)
trap number             = 12
WARNING !drm_modeset_is_locked(&crtc->mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:577
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff828686bd at drm_atomic_helper_check_modeset+0xcd
#2 0xffffffff827b135f at intel_atomic_check+0x1f
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
#11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
#15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
WARNING !drm_modeset_is_locked(&crtc->mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:577
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff828686bd at drm_atomic_helper_check_modeset+0xcd
#2 0xffffffff827b135f at intel_atomic_check+0x1f
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
#11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
#15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
WARNING !drm_modeset_is_locked(&dev->mode_config.connection_mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:622
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff82868840 at drm_atomic_helper_check_modeset+0x250
#2 0xffffffff827b135f at intel_atomic_check+0x1f
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
#11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:821
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff82869917 at drm_atomic_helper_check_planes+0xa7
#2 0xffffffff827b207c at intel_atomic_check+0xd3c
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
#11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
#15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:821
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff82869917 at drm_atomic_helper_check_planes+0xa7
#2 0xffffffff827b207c at intel_atomic_check+0xd3c
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
#11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
#15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:821
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff82869917 at drm_atomic_helper_check_planes+0xa7
#2 0xffffffff827b207c at intel_atomic_check+0xd3c
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
#11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
#15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:821
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff82869917 at drm_atomic_helper_check_planes+0xa7
#2 0xffffffff827b207c at intel_atomic_check+0xd3c
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
#11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
#15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:821
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff82869917 at drm_atomic_helper_check_planes+0xa7
#2 0xffffffff827b207c at intel_atomic_check+0xd3c
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
#11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
#15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/obj/usr/src/amd64.amd64/sys/GENERIC.4BSD/usr/ports/graphics/drm-fbsd12.0-kmod/work/kms-drm-99da0ba/drivers/gpu/drm/drm_atomic_helper.c:821
#0 0xffffffff828df413 at linux_dump_stack+0x23
#1 0xffffffff82869917 at drm_atomic_helper_check_planes+0xa7
#2 0xffffffff827b207c at intel_atomic_check+0xd3c
#3 0xffffffff828666bb at drm_atomic_check_only+0x2ab
#4 0xffffffff82866a63 at drm_atomic_commit+0x13
#5 0xffffffff82892a88 at restore_fbdev_mode_atomic+0x1c8
#6 0xffffffff8288edaa at drm_fb_helper_restore_fbdev_mode_unlocked+0x7a
#7 0xffffffff828b4767 at vt_kms_postswitch+0x127
#8 0xffffffff80a21a7a at vt_window_switch+0x13a
#9 0xffffffff80a1eaef at vtterm_cngrab+0x1f
#10 0xffffffff80b54fc6 at cngrab+0x16
11 0xffffffff80bb52d5 at vpanic+0xe5
#12 0xffffffff80bb51e3 at panic+0x43
#13 0xffffffff8107f331 at trap_fatal+0x391
#14 0xffffffff8107f38f at trap_pfault+0x4f
#15 0xffffffff8107e9d6 at trap+0x286
#16 0xffffffff81058d48 at calltrap+0x8
#17 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
<4>WARN_ON(!mutex_is_locked(&dev->struct_mutex))

<4>WARN_ON(!mutex_is_locked(&fbc->lock))WARN_ON(!mutex_is_locked(&fbc->lock))
panic: page fault
cpuid = 1
time = 1591763184
KDB: stack backtrace:
#0 0xffffffff80bfe565 at kdb_backtrace+0x65
#1 0xffffffff80bb536b at vpanic+0x17b
#2 0xffffffff80bb51e3 at panic+0x43
#3 0xffffffff8107f331 at trap_fatal+0x391
#4 0xffffffff8107f38f at trap_pfault+0x4f
#5 0xffffffff8107e9d6 at trap+0x286
#6 0xffffffff81058d48 at calltrap+0x8
#7 0xffffffff82a81867 at logo_update.ypos+0xcfe1b
#8 0xffffffff829d5ba0 at logo_update.ypos+0x24154
#9 0xffffffff829ce173 at logo_update.ypos+0x1c727
#10 0xffffffff829c4831 at logo_update.ypos+0x12de5
#11 0xffffffff82a9ab46 at logo_update.ypos+0xe90fa
#12 0xffffffff829d7b24 at logo_update.ypos+0x260d8
#13 0xffffffff824f0f77 at supdrvIOCtlFast+0xb7
#14 0xffffffff8250029b at VBoxDrvFreeBSDIOCtl+0x5b
#15 0xffffffff80a6fd30 at devfs_ioctl+0xb0
#16 0xffffffff811ff36b at VOP_IOCTL_APV+0x7b
#17 0xffffffff80c9058a at vn_ioctl+0x16a
Uptime: 6m35s
Comment 1 rkoberman 2020-06-12 06:45:18 UTC
First, I did get the working revision VERY wrong. The good version was 361412. The failing one was 3861980. Sorry!

After further testing, there is no problem with r361412. I had made an error in my rollback and, after correcting it, the VM runs normally.

I then updated again today and confirmed the failure. Tomorrow I will start bisecting kernels and try to track down the exact revision that triggers the crash.
Comment 2 rkoberman 2020-06-14 17:03:14 UTC
I have now tracked the crash to r361350, "Provide separate accounting for user-wired pages." Revert this commit and everything works. With this commit, the system crashes when I start my Windows7 VM. All backtraces are essentially identical to the one in the initial report.

A bit more detail on the trigger. If the VM is in a "Saved" state, the crash happens immediately after the image is loaded and started. If the VM is "Powered off", the system boots up until it does its second window resize. The initial window is the console window at 640x480. As soon as the system boot starts, it resizes to 800x600 for the splash screen and the animated Wind7 logo. After a few seconds, it expands to the size I have set for the running system, about 1480x816. At that time, the system panics.

Any ideas?Much as I hate to admit it, I do need my Windows system regularly.
Comment 3 Mark Johnston freebsd_committer freebsd_triage 2020-06-14 17:20:00 UTC
(In reply to rkoberman from comment #2)
I take it you have not rebuilt the virtualbox kernel modules?  That is, you are using whatever modules that were installed by pkg?
Comment 4 Mark Johnston freebsd_committer freebsd_triage 2020-06-14 17:22:28 UTC
(In reply to Mark Johnston from comment #3)
And a follow-up question: how much RAM does the system have, and how much do you give to the VM?
Comment 5 Mark Johnston freebsd_committer freebsd_triage 2020-06-14 17:33:39 UTC
(In reply to Mark Johnston from comment #4)
Never mind, I managed to reproduce it on head.  It looks like my change exposed an error-handling bug in the virtualbox kernel code.

As a workaround, please try increasing vm.max_wired to the value hw.physmem / 4096 before starting the VM.
Comment 6 rkoberman 2020-06-14 18:39:30 UTC
(In reply to Mark Johnston from comment #4)
The system has 8G and the VM is allowed 4G.
Comment 7 rkoberman 2020-06-14 19:07:20 UTC
Wow, that's a lot of wired memory, but that did the trick! VM now runs fine.

Let me know if I can test any patch.

Since this bumps FreeBSD_version, this commit requires a full rebuild of userland at 3.5+ hours on my 9 year old system; 2.5 just to build the LLVM stuff. At least I'll only need one more full rebuild to get up to date. I may start letting the system build use the full LLVM from ports to make this sort of thing less painful!

Thanks so much for the VERY quick response! Awesome!
Comment 8 Mark Johnston freebsd_committer freebsd_triage 2020-06-14 19:26:17 UTC
Created attachment 215559 [details]
proposed patch

Thanks.  Could you try the attached patch?  It is only necessary to recompile the kernel, no need to rebuild world or any ports.  This is true for the original commit too, __FreeBSD_version bump notwithstanding.
Comment 9 rkoberman 2020-06-15 15:36:00 UTC
Patch applied & kernel rebuilt. No problems at all. My VMs now are starting properly.

Thanks so much for such a quick response and fix. Once committed, feel free to close the ticket.
Comment 10 Mark Johnston freebsd_committer freebsd_triage 2020-06-15 15:59:03 UTC
(In reply to rkoberman from comment #9)
Thanks.  To be clear, did you test with the vm.max_wired reverted to the default value?
Comment 11 rkoberman 2020-06-15 16:08:13 UTC
(In reply to rkoberman from comment #9)
Yes. Saw no point in testing with the huge max_wired.
Comment 12 Mark Johnston freebsd_committer freebsd_triage 2020-06-15 16:13:23 UTC
(In reply to rkoberman from comment #11)
Great, thanks again.
Comment 13 Mark Johnston freebsd_committer freebsd_triage 2020-07-11 18:39:35 UTC
Sorry for the delay.  I haven't forgotten about this, I will try to get a fix committed soon.
Comment 14 rkoberman 2020-09-14 00:04:47 UTC
Mark, Has this been resolved? If not, any hope?
Comment 15 Mark Johnston freebsd_committer freebsd_triage 2020-09-14 21:02:25 UTC
(In reply to rkoberman from comment #14)
Not yet, sorry.  I posted a patch to simply increase the default vm_max_user_wired value, which I aim to get into 12.2.  I'll also try to fix virtualbox to avoid panicking the kernel if it hits the limit/
Comment 16 Mark Johnston freebsd_committer freebsd_triage 2020-09-23 13:03:55 UTC
I ended up committing a patch to increase the default user-wired memory limit, which effectively works around the problem for "reasonable" VM configurations.  It was merged to 12.2.  I also have a patch to virtualbox to avoid crashing when the limit is hit which I'll submit this week.
Comment 17 Mark Johnston freebsd_committer freebsd_triage 2020-09-24 15:06:20 UTC
The virtualbox patch was committed to ports as r549922.