Bug 220923 - [boot] [panic] 11.1-RC3 Kernel panic during boot
Summary: [boot] [panic] 11.1-RC3 Kernel panic during boot
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Josh Paetzel
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2017-07-22 11:10 UTC by Zhenlei Huang
Modified: 2019-03-21 15:12 UTC (History)
10 users (show)

See Also:


Attachments
Full crash log (9.13 KB, text/plain)
2017-07-22 11:10 UTC, Zhenlei Huang
no flags Details
DDB prompt (314.40 KB, image/png)
2017-08-28 08:31 UTC, Zhenlei Huang
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Zhenlei Huang 2017-07-22 11:10:01 UTC
Created attachment 184595 [details]
Full crash log

I upgraded my test vm from 11.1-RC2 to 11.1-RC3, and the system crashs during system boot.

Environment:
rMBP, OS X 10.9.5, VMware Fusion, 2 process core, 512MB memeory, 2G virtual hard disk for system install and two virtual hard disks for data, bridged VMXNET3 nic

System is installed with ZFS root. The /boot/loader.conf is as following,

kern.peom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
vfs.zfs.min_auto_ashift=12
zfs_load="YES"

hw.pci.honor_msi_blacklist="0"
hw.vmx.txnqueue="4"
hw.vmx.rxnqueue="4"

Crash log:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address   = 0x18
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff808eac3a
stack pointer           = 0x28:0xfffffe001d3f2000
frame pointer           = 0x28:0xfffffe001d3f2050
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 4 (doneq0)
trap number             = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec3641 at calltrap+0x8
#7 0xffffffff808f26d8 at vt_flush+0x398
#8 0xffffffff80ac09cf at termcn_cnputc+0xff
#9 0xffffffff80a100ed at cnputc+0x7d
#10 0xffffffff80a103f8 at cnputs+0xb8
#11 0xffffffff80ab36ad at putchar+0x14d
#12 0xffffffff80ab323d at kvprintf+0x103d
#13 0xffffffff80ab3ea7 at vprintf+0x87
#14 0xffffffff80ab3e13 at printf+0x43
#15 0xffffffff80307c52 at xpt_announce_periph+0x62
#16 0xffffffff8032e089 at cddone+0x249
#17 0xffffffff8030dbe7 at xpt_done_process+0x677

Full crash log is attached.

Anyone has a clue about this panic?
Comment 1 Zhenlei Huang 2017-07-26 13:41:06 UTC
Same bugs for 11.1-RELEASE, need set affects version to 11.0-RELEASE ?
Comment 2 ferdinand.goldmann 2017-07-28 09:38:08 UTC
This affects me as well. I just tried upgrading a FreeBSD system from 11.0 to 11.1. The system would panic after rebooting. Strange thing though: After the 2nd panic, the kernel actually booted.

Environment is VMware ESXi 6.0. Kernel backtrace:

kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address   = 0x3c
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff808eac3a
stack pointer           = 0x28:0xfffffe03e6d14f70
frame pointer           = 0x28:0xfffffe03e6d14fc0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 4 (doneq0)
trap number             = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec3641 at calltrap+0x8
#7 0xffffffff808f26d8 at vt_flush+0x398
#8 0xffffffff80ac09cf at termcn_cnputc+0xff
#9 0xffffffff80a100ed at cnputc+0x7d
#10 0xffffffff80a103f8 at cnputs+0xb8
#11 0xffffffff80ab36ad at putchar+0x14d
#12 0xffffffff80ab323d at kvprintf+0x103d
#13 0xffffffff80ab3ea7 at vprintf+0x87
#14 0xffffffff80ab3e13 at printf+0x43
#15 0xffffffff80322e24 at scsi_print_inquiry+0x144
#16 0xffffffff80307c9e at xpt_announce_periph+0xae
#17 0xffffffff8033bb48 at dadone+0x16f8
Uptime: 3s
Comment 3 Cata Lin 2017-08-10 13:32:37 UTC
I have same issues on ESXi 6.5 , on VM reboot some times randomly crashes one time / several times with Fatal trap 12: page fault while in kernel mode then boots normally .
Comment 4 Cata Lin 2017-08-10 13:34:43 UTC
os version : 

FreeBSD hyperyon 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

log 

cd0: <NECVMWar VMware SATA CD00 1.00> Removable CD-ROM SCSI device
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x11c
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff808eac3a
stack pointer           = 0x28:0xfffffe011af73f80
frame pointer           = 0x28:0xfffffe011af73fd0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 4 (doneq1)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec3641 at calltrap+0x8
#7 0xffffffff808f26d8 at vt_flush+0x398
#8 0xffffffff80ac09cf at termcn_cnputc+0xff
#9 0xffffffff80a100ed at cnputc+0x7d
#10 0xffffffff80a103f8 at cnputs+0xb8
#11 0xffffffff80ab36ad at putchar+0x14d
#12 0xffffffff80ab323d at kvprintf+0x103d
#13 0xffffffff80ab3ea7 at vprintf+0x87
#14 0xffffffff80ab3e13 at printf+0x43
#15 0xffffffff80322e24 at scsi_print_inquiry+0x144
#16 0xffffffff80307c9e at xpt_announce_periph+0xae
#17 0xffffffff8032e089 at cddone+0x249
Uptime: 3s
Comment 5 Zhenlei Huang 2017-08-27 04:05:51 UTC
Umm... I set up a new vm, repeat rebooting it with 11.1-RC2 installing ISO, and it also crashes randomly.
Comment 6 Josh Paetzel freebsd_committer 2017-08-27 04:33:12 UTC
Can we get a crashdump or text dump for one of these panics please?
Comment 7 Zhenlei Huang 2017-08-27 07:25:55 UTC
(In reply to Josh Paetzel from comment #6)
The system reboot instantly when crash, not leaving any vmcores in /var/crash on reboot. I've set dumpdev=AUTO in rc.conf, and swapinfo shows the dump device is /dev/da0p2. @Josh Paetzel, how can I get a crashdump then?
Comment 8 Josh Paetzel freebsd_committer 2017-08-27 15:34:18 UTC
So what's happening is you are getting the kernel panic early before any userland config for dumping is happening.

You'll need to install a kernel on this machine with "options DDB" in the kernel config file.

That way when it panics you will get dropped to a ddb prompt and can dump manually.
Comment 9 Zhenlei Huang 2017-08-28 08:31:08 UTC
Created attachment 185839 [details]
DDB prompt

I've got DDB prompt now, but it seems not doing a dump. Am I wrong? @Josh Paetzel
Comment 10 Josh Paetzel freebsd_committer 2017-08-28 15:08:18 UTC
Try typing "dump" without the quotes. I'm traveling today and will look at this tonight.
Comment 11 Zhenlei Huang 2017-08-29 12:46:17 UTC
(In reply to Josh Paetzel from comment #10)
Thanks for that. Hmm, I need config dump device first...

````
Stopped at      vga_bitblt_text+0x14a:  movl    (%rax,%rcx,4),%r15d
db> bt
Tracing pid 4 tid 100030 td 0xfffff800032bc560
vga_bitblt_text() at vga_bitblt_text+0x14a/frame 0xfffffe001d3dafd0
vt_flush() at vt_flush+0x398/frame 0xfffffe001d3db020
termcn_cnputc() at termcn_cnputc+0xff/frame 0xfffffe001d3db060
cnputc() at cnputc+0x7d/frame 0xfffffe001d3db090
cnputs() at cnputs+0xb8/frame 0xfffffe001d3db0b0
putchar() at putchar+0x14d/frame 0xfffffe001d3db130
kvprintf() at kvprintf+0x103d/frame 0xfffffe001d3db220
vprintf() at vprintf+0x87/frame 0xfffffe001d3db2f0
printf() at printf+0x43/frame 0xfffffe001d3db350
scsi_print_inquiry() at scsi_print_inquiry+0x144/frame 0xfffffe001d3db3f0
xpt_announce_periph() at xpt_announce_periph+0xae/frame 0xfffffe001d3db420
cddone() at cddone+0x249/frame 0xfffffe001d3db9e0
xpt_done_process() at xpt_done_process+0x677/frame 0xfffffe001d3dba20
xpt_done_td() at xpt_done_td+0x196/frame 0xfffffe001d3dba70
fork_exit() at fork_exit+0x85/frame 0xfffffe001d3dbab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001d3dbab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db> dump
Cannot dump: no dump device specified.
db> 
````
Comment 12 Zhenlei Huang 2017-08-29 14:12:31 UTC
I've set `dumpdev="/dev/da0p2"` in /boot/loader.conf as per loader(8), but it seems not take effect when I do dumping in DDB prompt. It's so weird...
Comment 13 ferdinand.goldmann 2017-08-31 11:47:09 UTC
FWIW, I tried this with VMware ESXi, 6.5.0, 5310538 and 11.1-RELEASE-p1 today and was not able to reproduce the problem.

Were there any significant kernel changes between 11.1-RELEASE and 11.1-RELEASE-p1 that might have resolved the problem already? I tried booting the VM dozens of times and was not able to get it to panic. :-/
Comment 14 Zhenlei Huang 2017-08-31 12:05:42 UTC
(In reply to ferdinand.goldmann from comment #13)
You need try another dozens of times, :)

Changes between 11.1-RELEASE and 11.1-RELEASE-p111.1-RELEASE-p1 seems not related to this bug.

https://github.com/freebsd/freebsd/compare/release/11.1.0...releng/11.1
Comment 15 Zhenlei Huang 2017-09-08 10:34:52 UTC
I'm getting this with DDB, hope it helps. :) @Josh Paetzel

db> bt
Tracing pid 4 tid 100030 td 0xfffff800032bc560
vga_bitblt_text() at vga_bitblt_text+0x14a/frame 0xfffffe001d3db050
vt_flush() at vt_flush+0x398/frame 0xfffffe001d3db0a0
termcn_cnputc() at termcn_cnputc+0xff/frame 0xfffffe001d3db0e0
cnputc() at cnputc+0x7d/frame 0xfffffe001d3db110
cnputs() at cnputs+0xb8/frame 0xfffffe001d3db130
putchar() at putchar+0x14d/frame 0xfffffe001d3db1b0
kvprintf() at kvprintf+0x103d/frame 0xfffffe001d3db2a0
vprintf() at vprintf+0x87/frame 0xfffffe001d3db370
printf() at printf+0x43/frame 0xfffffe001d3db3d0
xpt_announce_periph() at xpt_announce_periph+0x62/frame 0xfffffe001d3db420
cddone() at cddone+0x249/frame 0xfffffe001d3db9e0
xpt_done_process() at xpt_done_process+0x677/frame 0xfffffe001d3dba20
xpt_done_td() at xpt_done_td+0x196/frame 0xfffffe001d3dba70
fork_exit() at fork_exit+0x85/frame 0xfffffe001d3dbab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001d3dbab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

db> show register
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x28
ss                        0x28
rax                          0
rcx                       0x20
rdx                       0xdd
rbx         0xfffffe001d3db070
rsp         0xfffffe001d3db000
rbp         0xfffffe001d3db050
rsi                       0x15
rdi                          0
r8                        0x63
r9          0xfffffe001d3db070
r10                      0x12c
r11                          0
r12         0xffffffff81b48fc8  vga_conssoftc
r13                       0x20
r14         0xffffffff81970cc0  vt_conswindow
r15                 0x1c200043
rip         0xffffffff808f57ba  vga_bitblt_text+0x14a
rflags                 0x10046
vga_bitblt_text+0x14a:  movl    (%rax,%rcx,4),%r15d

db> x/ia vga_bitblt_text,0x14a
vga_bitblt_text:        pushq   %rbp
vga_bitblt_text+0x1:    movq    %rsp,%rbp
vga_bitblt_text+0x4:    pushq   %r15
vga_bitblt_text+0x6:    pushq   %r14
vga_bitblt_text+0x8:    pushq   %r13
vga_bitblt_text+0xa:    pushq   %r12
vga_bitblt_text+0xc:    pushq   %rbx
vga_bitblt_text+0xd:    subq    $0x28,%rsp
vga_bitblt_text+0x11:   movq    %rdx,%rbx
vga_bitblt_text+0x14:   movq    %rsi,%r14
vga_bitblt_text+0x17:   movq    %rdi,%r12
vga_bitblt_text+0x1a:   testb   $0x1,0x14c(%r12)
vga_bitblt_text+0x23:   jnz     vga_bitblt_text+0xe3
vga_bitblt_text+0x29:   movq    0x70(%r14),%rcx
vga_bitblt_text+0x2d:   movzwl  (%rbx),%r15d
vga_bitblt_text+0x31:   movl    0x28(%rcx),%eax
vga_bitblt_text+0x34:   movq    %rcx,0xffffffffffffffc8(%rbp)
vga_bitblt_text+0x38:   movl    0x2c(%rcx),%r8d
vga_bitblt_text+0x3c:   movzwl  0x7a(%r14),%edi
vga_bitblt_text+0x41:   imull   %eax,%r15d
vga_bitblt_text+0x45:   movzwl  0x78(%r14),%edx
vga_bitblt_text+0x4a:   addl    %edx,%r15d
vga_bitblt_text+0x4d:   movzwl  0x6(%rbx),%esi
vga_bitblt_text+0x51:   movzwl  0x4(%rbx),%ecx
vga_bitblt_text+0x55:   imull   %r8d,%esi
vga_bitblt_text+0x59:   leal    0x7(%rdi,%rsi,1),%r13d
vga_bitblt_text+0x5e:   andl    $-0x8,%r13d
vga_bitblt_text+0x62:   imull   %eax,%ecx
vga_bitblt_text+0x65:   addl    %edx,%ecx
vga_bitblt_text+0x67:   movzwl  0x7e(%r14),%edx
vga_bitblt_text+0x6c:   cmpl    %edx,%r13d
vga_bitblt_text+0x6f:   cmovnb  %edx,%r13d
vga_bitblt_text+0x73:   movzwl  0x7c(%r14),%edx
vga_bitblt_text+0x78:   cmpl    %edx,%ecx
vga_bitblt_text+0x7a:   cmovnb  %edx,%ecx
vga_bitblt_text+0x7d:   cmpl    %ecx,%r15d
vga_bitblt_text+0x80:   jnb     vga_bitblt_text+0x25c
vga_bitblt_text+0x86:   movzwl  0x2(%rbx),%edx
vga_bitblt_text+0x8a:   imull   %edx,%r8d
vga_bitblt_text+0x8e:   addl    %r8d,%edi
vga_bitblt_text+0x91:   andl    $-0x8,%edi
vga_bitblt_text+0x94:   movq    %rdi,0xffffffffffffffb8(%rbp)
vga_bitblt_text+0x98:   movl    %ecx,0xffffffffffffffc0(%rbp)
vga_bitblt_text+0x9b:   nopl
vga_bitblt_text+0xa0:   cmpl    %r13d,%edi
vga_bitblt_text+0xa3:   jnb     vga_bitblt_text+0xd6
vga_bitblt_text+0xa5:   movl    %edi,%ebx
vga_bitblt_text+0xa7:   nopw
vga_bitblt_text+0xb0:   movq    %r12,%rdi
vga_bitblt_text+0xb3:   movq    %r14,%rsi
vga_bitblt_text+0xb6:   movl    %ebx,%edx
vga_bitblt_text+0xb8:   movl    %r15d,%ecx
vga_bitblt_text+0xbb:   call    vga_bitblt_one_text_pixels_block
vga_bitblt_text+0xc0:   addl    $0x8,%ebx
vga_bitblt_text+0xc3:   cmpl    %r13d,%ebx
vga_bitblt_text+0xc6:   jb      vga_bitblt_text+0xb0
vga_bitblt_text+0xc8:   movq    0xffffffffffffffc8(%rbp),%rax
vga_bitblt_text+0xcc:   movl    0x28(%rax),%eax
vga_bitblt_text+0xcf:   movq    0xffffffffffffffb8(%rbp),%rdi
vga_bitblt_text+0xd3:   movl    0xffffffffffffffc0(%rbp),%ecx
vga_bitblt_text+0xd6:   addl    %eax,%r15d
vga_bitblt_text+0xd9:   cmpl    %ecx,%r15d
vga_bitblt_text+0xdc:   jb      vga_bitblt_text+0xa0
vga_bitblt_text+0xde:   jmp     vga_bitblt_text+0x25c
vga_bitblt_text+0xe3:   movzwl  (%rbx),%eax
vga_bitblt_text+0xe6:   movzwl  0x4(%rbx),%ecx
vga_bitblt_text+0xea:   movl    %eax,0xffffffffffffffc8(%rbp)
vga_bitblt_text+0xed:   cmpw    %cx,%ax
vga_bitblt_text+0xf0:   jnb     vga_bitblt_text+0x25c
vga_bitblt_text+0xf6:   movq    0x88(%r12),%r12
vga_bitblt_text+0xfe:   leaq    0x10(%r14),%rax
vga_bitblt_text+0x102:  movq    %rax,0xffffffffffffffc0(%rbp)
vga_bitblt_text+0x106:  movw    0x6(%rbx),%ax
vga_bitblt_text+0x10a:  nopw
vga_bitblt_text+0x110:  movzwl  0x2(%rbx),%r13d
vga_bitblt_text+0x115:  cmpw    %ax,%r13w
vga_bitblt_text+0x119:  jnb     vga_bitblt_text+0x249
vga_bitblt_text+0x11f:  movl    0xffffffffffffffc8(%rbp),%eax
vga_bitblt_text+0x122:  shll    $0x4,%eax
vga_bitblt_text+0x125:  leal    (%rax,%rax,4),%eax
vga_bitblt_text+0x128:  movq    %rax,0xffffffffffffffb8(%rbp)
vga_bitblt_text+0x12c:  nopl
vga_bitblt_text+0x130:  movq    0x68(%r14),%rcx
vga_bitblt_text+0x134:  movl    0x3c(%r14),%eax
vga_bitblt_text+0x138:  movl    0xffffffffffffffc8(%rbp),%esi
vga_bitblt_text+0x13b:  addl    %esi,%eax
vga_bitblt_text+0x13d:  xorl    %edx,%edx
vga_bitblt_text+0x13f:  divl    0x38(%r14),%eax
vga_bitblt_text+0x143:  movq    (%rcx,%rdx,8),%rax
vga_bitblt_text+0x147:  movl    %r13d,%ecx
vga_bitblt_text+0x14a:  movl    (%rax,%rcx,4),%r15d
vga_bitblt_text+0x14e:  movq    0xffffffffffffffc0(%rbp),%rdi
vga_bitblt_text+0x152:  movl    %r13d,%edx
vga_bitblt_text+0x155:  call    vtbuf_iscursor
vga_bitblt_text+0x15a:  movl    %r15d,%edi
vga_bitblt_text+0x15d:  movl    %eax,%esi
vga_bitblt_text+0x15f:  leaq    0xffffffffffffffd6(%rbp),%rdx
vga_bitblt_text+0x163:  leaq    0xffffffffffffffd7(%rbp),%rcx
vga_bitblt_text+0x167:  call    vt_determine_colors
vga_bitblt_text+0x16c:  movl    %r15d,%r8d
vga_bitblt_text+0x16f:  andl    $0x1fffff,%r8d
vga_bitblt_text+0x176:  leal    0xffffffffffffffe0(%r8),%ecx
vga_bitblt_text+0x17a:  movb    $0x3f,%rax
vga_bitblt_text+0x17c:  cmpl    $0x264c,%ecx
vga_bitblt_text+0x182:  jnbe    vga_bitblt_text+0x1e3
vga_bitblt_text+0x184:  movq    %rbx,%r9
vga_bitblt_text+0x187:  xorl    %esi,%esi
vga_bitblt_text+0x189:  movl    $0xa8,%rdx
vga_bitblt_text+0x18e:  nop
vga_bitblt_text+0x190:  leal    (%rdx,%rsi,1),%ecx
vga_bitblt_text+0x193:  movl    %ecx,%edi
vga_bitblt_text+0x195:  shrl    $0x1f,%edi
vga_bitblt_text+0x198:  addl    %ecx,%edi
vga_bitblt_text+0x19a:  sarl    $1,%edi
vga_bitblt_text+0x19c:  movslq  %edi,%rbx
vga_bitblt_text+0x19f:  movzwl  cp437table(,%rbx,4),%ecx
vga_bitblt_text+0x1a7:  cmpl    %ecx,%r8d
vga_bitblt_text+0x1aa:  jnb     vga_bitblt_text+0x1b5
vga_bitblt_text+0x1ac:  leal    0xffffffffffffffff(%rdi),%edx
vga_bitblt_text+0x1af:  cmpl    %esi,%edi
vga_bitblt_text+0x1b1:  jnle    vga_bitblt_text+0x190
vga_bitblt_text+0x1b3:  jmp     vga_bitblt_text+0x1e0
vga_bitblt_text+0x1b5:  movzbl  cp437table+0x3(,%rbx,4),%esi
vga_bitblt_text+0x1bd:  addl    %ecx,%esi
vga_bitblt_text+0x1bf:  cmpl    %r8d,%esi
vga_bitblt_text+0x1c2:  jnb     vga_bitblt_text+0x1cd
vga_bitblt_text+0x1c4:  leal    0x1(%rdi),%esi
vga_bitblt_text+0x1c7:  cmpl    %edi,%edx
vga_bitblt_text+0x1c9:  jnle    vga_bitblt_text+0x190
vga_bitblt_text+0x1cb:  jmp     vga_bitblt_text+0x1e0
vga_bitblt_text+0x1cd:  subl    %ecx,%r15d
vga_bitblt_text+0x1d0:  movzbl  cp437table+0x2(,%rbx,4),%eax
vga_bitblt_text+0x1d8:  addl    %r15d,%eax
vga_bitblt_text+0x1db:  nopl
vga_bitblt_text+0x1e0:  movq    %r9,%rbx
vga_bitblt_text+0x1e3:  movb    0xffffffffffffffd7(%rbp),%cl
vga_bitblt_text+0x1e6:  shlb    $0x4,%cl
vga_bitblt_text+0x1e9:  orb     0xffffffffffffffd6(%rbp),%cl
vga_bitblt_text+0x1ec:  movq    0xffffffffffffffb8(%rbp),%rdx
vga_bitblt_text+0x1f0:  leal    (%r13,%rdx,1),%esi
vga_bitblt_text+0x1f5:  addl    %esi,%esi
vga_bitblt_text+0x1f7:  movq    0x8(%r12),%rdx
vga_bitblt_text+0x1fc:  addq    %rsi,%rdx
vga_bitblt_text+0x1ff:  cmpq    $0,(%r12)
vga_bitblt_text+0x204:  jz      vga_bitblt_text+0x210
vga_bitblt_text+0x206:  movb    %al,(%rdx)
vga_bitblt_text+0x208:  jmp     vga_bitblt_text+0x211
vga_bitblt_text+0x20a:  nopw
vga_bitblt_text+0x210:  outb    %al,%dx
vga_bitblt_text+0x211:  orl     $0x1,%esi
vga_bitblt_text+0x214:  addq    0x8(%r12),%rsi
vga_bitblt_text+0x219:  cmpq    $0,(%r12)
vga_bitblt_text+0x21e:  jz      vga_bitblt_text+0x230
vga_bitblt_text+0x220:  movb    %cl,(%rsi)
vga_bitblt_text+0x222:  jmp     vga_bitblt_text+0x235
vga_bitblt_text+0x224:  nopw
vga_bitblt_text+0x230:  movl    %ecx,%eax
vga_bitblt_text+0x232:  movl    %esi,%edx
vga_bitblt_text+0x234:  outb    %al,%dx
vga_bitblt_text+0x235:  incl    %r13d
vga_bitblt_text+0x238:  movzwl  0x6(%rbx),%eax
vga_bitblt_text+0x23c:  cmpl    %eax,%r13d
vga_bitblt_text+0x23f:  jb      vga_bitblt_text+0x130
vga_bitblt_text+0x245:  movzwl  0x4(%rbx),%ecx
vga_bitblt_text+0x249:  movl    0xffffffffffffffc8(%rbp),%esi
vga_bitblt_text+0x24c:  incl    %esi
vga_bitblt_text+0x24e:  movzwl  %cx,%edx
vga_bitblt_text+0x251:  movl    %esi,0xffffffffffffffc8(%rbp)
vga_bitblt_text+0x254:  cmpl    %edx,%esi
vga_bitblt_text+0x256:  jb      vga_bitblt_text+0x110
vga_bitblt_text+0x25c:  addq    $0x28,%rsp
vga_bitblt_text+0x260:  popq    %rbx
vga_bitblt_text+0x261:  popq    %r12
vga_bitblt_text+0x263:  popq    %r13
vga_bitblt_text+0x265:  popq    %r14
vga_bitblt_text+0x267:  popq    %r15
vga_bitblt_text+0x269:  popq    %rbp
vga_bitblt_text+0x26a:  ret
vga_bitblt_text+0x26b:  nopl
Comment 16 Mahmoud Al-Qudsi 2017-10-20 02:32:20 UTC
This wasn't resolved in -p1; I just experienced it myself on an ESXi 6.5.0U1 host. It doesn't happen in any deterministic fashion, so far as I can see:

kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x114
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff808eac3a
stack pointer           = 0x28:0xfffffe022f438000
frame pointer           = 0x28:0xfffffe022f438050
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 4 (doneq0)
trap number             = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec3641 at calltrap+0x8
#7 0xffffffff808f26d8 at vt_flush+0x398
#8 0xffffffff80ac09cf at termcn_cnputc+0xff
#9 0xffffffff80a100ed at cnputc+0x7d
#10 0xffffffff80a103f8 at cnputs+0xb8
#11 0xffffffff80ab36ad at putchar+0x14d
#12 0xffffffff80ab323d at kvprintf+0x103d
#13 0xffffffff80ab3ea7 at vprintf+0x87
#14 0xffffffff80ab3e13 at printf+0x43
#15 0xffffffff80307c52 at xpt_announce_periph+0x62
#16 0xffffffff8032e089 at cddone+0x249
#17 0xffffffff8030dbe7 at xpt_done_process+0x677
Uptime: 2s
Comment 17 Toshimichi Masubuchi 2017-11-30 04:53:08 UTC
In my environment I can avoid it by setting the "kern.cam.scsi_delay" option.

/boot/loader.conf:
kern.cam.scsi_delay="5000"

OS Version:
# uname -v
FreeBSD 11.1-RELEASE-p4 #0: Tue Nov 14 06:12:40 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
# freebsd-version
11.1-RELEASE-p5

Environment:
VMware ESXi 6.0U3 on HP DL380 G7
Comment 18 Toshimichi Masubuchi 2017-12-01 16:12:13 UTC
(In reply to Toshimichi Masubuchi from comment #17)

If there is no effect with the kern.cam.scsi_delay option, try the kern.cam.boot_delay option.

/boot/loader.conf:
kern.cam.boot_delay="5000"
or
kern.cam.boot_delay="10000"
Comment 19 Gert Doering 2018-01-24 20:16:37 UTC
I'm observing this as well with 11.1-RELEASE-p6 on ESXi 6.5.0U1.

Will try the loader.conf settings next...
Comment 20 Josh Paetzel freebsd_committer 2018-06-06 18:46:58 UTC
is the mitigation working for anyone else?
Comment 21 Scott Aitken 2019-03-09 04:03:19 UTC
I can confirm that this is still happening on ESXi 6.7.0 #1 SMP Release build-11675023 with FreeBSD 11.2-RELEASE.

Did anyone confirm if those loader.conf tuneables helped?
Comment 22 Josh Paetzel freebsd_committer 2019-03-21 15:07:50 UTC
I never heard back on whether the loader tunables helped or not.  cddone() is in the callstack of every crash dump I saw.  You can mitigate that by removing the CD hardware from the VM.

(The virtual CD is the only device that causes that code path to be run)