Bug 220923

Summary: [boot] [panic] 11.1-RC3 Kernel panic during boot
Product: Base System Reporter: Zhenlei Huang <zlei>
Component: kernAssignee: Josh Paetzel <jpaetzel>
Status: Closed DUPLICATE    
Severity: Affects Some People CC: ferdinand.goldmann, franco, freebsd-bugzilla, gert, jcasale, jpaetzel, loos, mqudsi, t.masub, zlei
Priority: --- Keywords: regression
Version: 11.0-STABLE   
Hardware: amd64   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217282
Attachments:
Description Flags
Full crash log
none
DDB prompt none

Description Zhenlei Huang freebsd_committer freebsd_triage 2017-07-22 11:10:01 UTC
Created attachment 184595 [details]
Full crash log

I upgraded my test vm from 11.1-RC2 to 11.1-RC3, and the system crashs during system boot.

Environment:
rMBP, OS X 10.9.5, VMware Fusion, 2 process core, 512MB memeory, 2G virtual hard disk for system install and two virtual hard disks for data, bridged VMXNET3 nic

System is installed with ZFS root. The /boot/loader.conf is as following,

kern.peom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
vfs.zfs.min_auto_ashift=12
zfs_load="YES"

hw.pci.honor_msi_blacklist="0"
hw.vmx.txnqueue="4"
hw.vmx.rxnqueue="4"

Crash log:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address   = 0x18
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff808eac3a
stack pointer           = 0x28:0xfffffe001d3f2000
frame pointer           = 0x28:0xfffffe001d3f2050
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 4 (doneq0)
trap number             = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec3641 at calltrap+0x8
#7 0xffffffff808f26d8 at vt_flush+0x398
#8 0xffffffff80ac09cf at termcn_cnputc+0xff
#9 0xffffffff80a100ed at cnputc+0x7d
#10 0xffffffff80a103f8 at cnputs+0xb8
#11 0xffffffff80ab36ad at putchar+0x14d
#12 0xffffffff80ab323d at kvprintf+0x103d
#13 0xffffffff80ab3ea7 at vprintf+0x87
#14 0xffffffff80ab3e13 at printf+0x43
#15 0xffffffff80307c52 at xpt_announce_periph+0x62
#16 0xffffffff8032e089 at cddone+0x249
#17 0xffffffff8030dbe7 at xpt_done_process+0x677

Full crash log is attached.

Anyone has a clue about this panic?
Comment 1 Zhenlei Huang freebsd_committer freebsd_triage 2017-07-26 13:41:06 UTC
Same bugs for 11.1-RELEASE, need set affects version to 11.0-RELEASE ?
Comment 2 ferdinand.goldmann 2017-07-28 09:38:08 UTC
This affects me as well. I just tried upgrading a FreeBSD system from 11.0 to 11.1. The system would panic after rebooting. Strange thing though: After the 2nd panic, the kernel actually booted.

Environment is VMware ESXi 6.0. Kernel backtrace:

kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address   = 0x3c
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff808eac3a
stack pointer           = 0x28:0xfffffe03e6d14f70
frame pointer           = 0x28:0xfffffe03e6d14fc0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 4 (doneq0)
trap number             = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec3641 at calltrap+0x8
#7 0xffffffff808f26d8 at vt_flush+0x398
#8 0xffffffff80ac09cf at termcn_cnputc+0xff
#9 0xffffffff80a100ed at cnputc+0x7d
#10 0xffffffff80a103f8 at cnputs+0xb8
#11 0xffffffff80ab36ad at putchar+0x14d
#12 0xffffffff80ab323d at kvprintf+0x103d
#13 0xffffffff80ab3ea7 at vprintf+0x87
#14 0xffffffff80ab3e13 at printf+0x43
#15 0xffffffff80322e24 at scsi_print_inquiry+0x144
#16 0xffffffff80307c9e at xpt_announce_periph+0xae
#17 0xffffffff8033bb48 at dadone+0x16f8
Uptime: 3s
Comment 3 Cata Lin 2017-08-10 13:32:37 UTC
I have same issues on ESXi 6.5 , on VM reboot some times randomly crashes one time / several times with Fatal trap 12: page fault while in kernel mode then boots normally .
Comment 4 Cata Lin 2017-08-10 13:34:43 UTC
os version : 

FreeBSD hyperyon 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

log 

cd0: <NECVMWar VMware SATA CD00 1.00> Removable CD-ROM SCSI device
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x11c
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff808eac3a
stack pointer           = 0x28:0xfffffe011af73f80
frame pointer           = 0x28:0xfffffe011af73fd0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 4 (doneq1)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec3641 at calltrap+0x8
#7 0xffffffff808f26d8 at vt_flush+0x398
#8 0xffffffff80ac09cf at termcn_cnputc+0xff
#9 0xffffffff80a100ed at cnputc+0x7d
#10 0xffffffff80a103f8 at cnputs+0xb8
#11 0xffffffff80ab36ad at putchar+0x14d
#12 0xffffffff80ab323d at kvprintf+0x103d
#13 0xffffffff80ab3ea7 at vprintf+0x87
#14 0xffffffff80ab3e13 at printf+0x43
#15 0xffffffff80322e24 at scsi_print_inquiry+0x144
#16 0xffffffff80307c9e at xpt_announce_periph+0xae
#17 0xffffffff8032e089 at cddone+0x249
Uptime: 3s
Comment 5 Zhenlei Huang freebsd_committer freebsd_triage 2017-08-27 04:05:51 UTC
Umm... I set up a new vm, repeat rebooting it with 11.1-RC2 installing ISO, and it also crashes randomly.
Comment 6 Josh Paetzel freebsd_committer freebsd_triage 2017-08-27 04:33:12 UTC
Can we get a crashdump or text dump for one of these panics please?
Comment 7 Zhenlei Huang freebsd_committer freebsd_triage 2017-08-27 07:25:55 UTC
(In reply to Josh Paetzel from comment #6)
The system reboot instantly when crash, not leaving any vmcores in /var/crash on reboot. I've set dumpdev=AUTO in rc.conf, and swapinfo shows the dump device is /dev/da0p2. @Josh Paetzel, how can I get a crashdump then?
Comment 8 Josh Paetzel freebsd_committer freebsd_triage 2017-08-27 15:34:18 UTC
So what's happening is you are getting the kernel panic early before any userland config for dumping is happening.

You'll need to install a kernel on this machine with "options DDB" in the kernel config file.

That way when it panics you will get dropped to a ddb prompt and can dump manually.
Comment 9 Zhenlei Huang freebsd_committer freebsd_triage 2017-08-28 08:31:08 UTC
Created attachment 185839 [details]
DDB prompt

I've got DDB prompt now, but it seems not doing a dump. Am I wrong? @Josh Paetzel
Comment 10 Josh Paetzel freebsd_committer freebsd_triage 2017-08-28 15:08:18 UTC
Try typing "dump" without the quotes. I'm traveling today and will look at this tonight.
Comment 11 Zhenlei Huang freebsd_committer freebsd_triage 2017-08-29 12:46:17 UTC
(In reply to Josh Paetzel from comment #10)
Thanks for that. Hmm, I need config dump device first...

````
Stopped at      vga_bitblt_text+0x14a:  movl    (%rax,%rcx,4),%r15d
db> bt
Tracing pid 4 tid 100030 td 0xfffff800032bc560
vga_bitblt_text() at vga_bitblt_text+0x14a/frame 0xfffffe001d3dafd0
vt_flush() at vt_flush+0x398/frame 0xfffffe001d3db020
termcn_cnputc() at termcn_cnputc+0xff/frame 0xfffffe001d3db060
cnputc() at cnputc+0x7d/frame 0xfffffe001d3db090
cnputs() at cnputs+0xb8/frame 0xfffffe001d3db0b0
putchar() at putchar+0x14d/frame 0xfffffe001d3db130
kvprintf() at kvprintf+0x103d/frame 0xfffffe001d3db220
vprintf() at vprintf+0x87/frame 0xfffffe001d3db2f0
printf() at printf+0x43/frame 0xfffffe001d3db350
scsi_print_inquiry() at scsi_print_inquiry+0x144/frame 0xfffffe001d3db3f0
xpt_announce_periph() at xpt_announce_periph+0xae/frame 0xfffffe001d3db420
cddone() at cddone+0x249/frame 0xfffffe001d3db9e0
xpt_done_process() at xpt_done_process+0x677/frame 0xfffffe001d3dba20
xpt_done_td() at xpt_done_td+0x196/frame 0xfffffe001d3dba70
fork_exit() at fork_exit+0x85/frame 0xfffffe001d3dbab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001d3dbab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db> dump
Cannot dump: no dump device specified.
db> 
````
Comment 12 Zhenlei Huang freebsd_committer freebsd_triage 2017-08-29 14:12:31 UTC
I've set `dumpdev="/dev/da0p2"` in /boot/loader.conf as per loader(8), but it seems not take effect when I do dumping in DDB prompt. It's so weird...
Comment 13 ferdinand.goldmann 2017-08-31 11:47:09 UTC
FWIW, I tried this with VMware ESXi, 6.5.0, 5310538 and 11.1-RELEASE-p1 today and was not able to reproduce the problem.

Were there any significant kernel changes between 11.1-RELEASE and 11.1-RELEASE-p1 that might have resolved the problem already? I tried booting the VM dozens of times and was not able to get it to panic. :-/
Comment 14 Zhenlei Huang freebsd_committer freebsd_triage 2017-08-31 12:05:42 UTC
(In reply to ferdinand.goldmann from comment #13)
You need try another dozens of times, :)

Changes between 11.1-RELEASE and 11.1-RELEASE-p111.1-RELEASE-p1 seems not related to this bug.

https://github.com/freebsd/freebsd/compare/release/11.1.0...releng/11.1
Comment 15 Zhenlei Huang freebsd_committer freebsd_triage 2017-09-08 10:34:52 UTC
I'm getting this with DDB, hope it helps. :) @Josh Paetzel

db> bt
Tracing pid 4 tid 100030 td 0xfffff800032bc560
vga_bitblt_text() at vga_bitblt_text+0x14a/frame 0xfffffe001d3db050
vt_flush() at vt_flush+0x398/frame 0xfffffe001d3db0a0
termcn_cnputc() at termcn_cnputc+0xff/frame 0xfffffe001d3db0e0
cnputc() at cnputc+0x7d/frame 0xfffffe001d3db110
cnputs() at cnputs+0xb8/frame 0xfffffe001d3db130
putchar() at putchar+0x14d/frame 0xfffffe001d3db1b0
kvprintf() at kvprintf+0x103d/frame 0xfffffe001d3db2a0
vprintf() at vprintf+0x87/frame 0xfffffe001d3db370
printf() at printf+0x43/frame 0xfffffe001d3db3d0
xpt_announce_periph() at xpt_announce_periph+0x62/frame 0xfffffe001d3db420
cddone() at cddone+0x249/frame 0xfffffe001d3db9e0
xpt_done_process() at xpt_done_process+0x677/frame 0xfffffe001d3dba20
xpt_done_td() at xpt_done_td+0x196/frame 0xfffffe001d3dba70
fork_exit() at fork_exit+0x85/frame 0xfffffe001d3dbab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001d3dbab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

db> show register
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x28
ss                        0x28
rax                          0
rcx                       0x20
rdx                       0xdd
rbx         0xfffffe001d3db070
rsp         0xfffffe001d3db000
rbp         0xfffffe001d3db050
rsi                       0x15
rdi                          0
r8                        0x63
r9          0xfffffe001d3db070
r10                      0x12c
r11                          0
r12         0xffffffff81b48fc8  vga_conssoftc
r13                       0x20
r14         0xffffffff81970cc0  vt_conswindow
r15                 0x1c200043
rip         0xffffffff808f57ba  vga_bitblt_text+0x14a
rflags                 0x10046
vga_bitblt_text+0x14a:  movl    (%rax,%rcx,4),%r15d

db> x/ia vga_bitblt_text,0x14a
vga_bitblt_text:        pushq   %rbp
vga_bitblt_text+0x1:    movq    %rsp,%rbp
vga_bitblt_text+0x4:    pushq   %r15
vga_bitblt_text+0x6:    pushq   %r14
vga_bitblt_text+0x8:    pushq   %r13
vga_bitblt_text+0xa:    pushq   %r12
vga_bitblt_text+0xc:    pushq   %rbx
vga_bitblt_text+0xd:    subq    $0x28,%rsp
vga_bitblt_text+0x11:   movq    %rdx,%rbx
vga_bitblt_text+0x14:   movq    %rsi,%r14
vga_bitblt_text+0x17:   movq    %rdi,%r12
vga_bitblt_text+0x1a:   testb   $0x1,0x14c(%r12)
vga_bitblt_text+0x23:   jnz     vga_bitblt_text+0xe3
vga_bitblt_text+0x29:   movq    0x70(%r14),%rcx
vga_bitblt_text+0x2d:   movzwl  (%rbx),%r15d
vga_bitblt_text+0x31:   movl    0x28(%rcx),%eax
vga_bitblt_text+0x34:   movq    %rcx,0xffffffffffffffc8(%rbp)
vga_bitblt_text+0x38:   movl    0x2c(%rcx),%r8d
vga_bitblt_text+0x3c:   movzwl  0x7a(%r14),%edi
vga_bitblt_text+0x41:   imull   %eax,%r15d
vga_bitblt_text+0x45:   movzwl  0x78(%r14),%edx
vga_bitblt_text+0x4a:   addl    %edx,%r15d
vga_bitblt_text+0x4d:   movzwl  0x6(%rbx),%esi
vga_bitblt_text+0x51:   movzwl  0x4(%rbx),%ecx
vga_bitblt_text+0x55:   imull   %r8d,%esi
vga_bitblt_text+0x59:   leal    0x7(%rdi,%rsi,1),%r13d
vga_bitblt_text+0x5e:   andl    $-0x8,%r13d
vga_bitblt_text+0x62:   imull   %eax,%ecx
vga_bitblt_text+0x65:   addl    %edx,%ecx
vga_bitblt_text+0x67:   movzwl  0x7e(%r14),%edx
vga_bitblt_text+0x6c:   cmpl    %edx,%r13d
vga_bitblt_text+0x6f:   cmovnb  %edx,%r13d
vga_bitblt_text+0x73:   movzwl  0x7c(%r14),%edx
vga_bitblt_text+0x78:   cmpl    %edx,%ecx
vga_bitblt_text+0x7a:   cmovnb  %edx,%ecx
vga_bitblt_text+0x7d:   cmpl    %ecx,%r15d
vga_bitblt_text+0x80:   jnb     vga_bitblt_text+0x25c
vga_bitblt_text+0x86:   movzwl  0x2(%rbx),%edx
vga_bitblt_text+0x8a:   imull   %edx,%r8d
vga_bitblt_text+0x8e:   addl    %r8d,%edi
vga_bitblt_text+0x91:   andl    $-0x8,%edi
vga_bitblt_text+0x94:   movq    %rdi,0xffffffffffffffb8(%rbp)
vga_bitblt_text+0x98:   movl    %ecx,0xffffffffffffffc0(%rbp)
vga_bitblt_text+0x9b:   nopl
vga_bitblt_text+0xa0:   cmpl    %r13d,%edi
vga_bitblt_text+0xa3:   jnb     vga_bitblt_text+0xd6
vga_bitblt_text+0xa5:   movl    %edi,%ebx
vga_bitblt_text+0xa7:   nopw
vga_bitblt_text+0xb0:   movq    %r12,%rdi
vga_bitblt_text+0xb3:   movq    %r14,%rsi
vga_bitblt_text+0xb6:   movl    %ebx,%edx
vga_bitblt_text+0xb8:   movl    %r15d,%ecx
vga_bitblt_text+0xbb:   call    vga_bitblt_one_text_pixels_block
vga_bitblt_text+0xc0:   addl    $0x8,%ebx
vga_bitblt_text+0xc3:   cmpl    %r13d,%ebx
vga_bitblt_text+0xc6:   jb      vga_bitblt_text+0xb0
vga_bitblt_text+0xc8:   movq    0xffffffffffffffc8(%rbp),%rax
vga_bitblt_text+0xcc:   movl    0x28(%rax),%eax
vga_bitblt_text+0xcf:   movq    0xffffffffffffffb8(%rbp),%rdi
vga_bitblt_text+0xd3:   movl    0xffffffffffffffc0(%rbp),%ecx
vga_bitblt_text+0xd6:   addl    %eax,%r15d
vga_bitblt_text+0xd9:   cmpl    %ecx,%r15d
vga_bitblt_text+0xdc:   jb      vga_bitblt_text+0xa0
vga_bitblt_text+0xde:   jmp     vga_bitblt_text+0x25c
vga_bitblt_text+0xe3:   movzwl  (%rbx),%eax
vga_bitblt_text+0xe6:   movzwl  0x4(%rbx),%ecx
vga_bitblt_text+0xea:   movl    %eax,0xffffffffffffffc8(%rbp)
vga_bitblt_text+0xed:   cmpw    %cx,%ax
vga_bitblt_text+0xf0:   jnb     vga_bitblt_text+0x25c
vga_bitblt_text+0xf6:   movq    0x88(%r12),%r12
vga_bitblt_text+0xfe:   leaq    0x10(%r14),%rax
vga_bitblt_text+0x102:  movq    %rax,0xffffffffffffffc0(%rbp)
vga_bitblt_text+0x106:  movw    0x6(%rbx),%ax
vga_bitblt_text+0x10a:  nopw
vga_bitblt_text+0x110:  movzwl  0x2(%rbx),%r13d
vga_bitblt_text+0x115:  cmpw    %ax,%r13w
vga_bitblt_text+0x119:  jnb     vga_bitblt_text+0x249
vga_bitblt_text+0x11f:  movl    0xffffffffffffffc8(%rbp),%eax
vga_bitblt_text+0x122:  shll    $0x4,%eax
vga_bitblt_text+0x125:  leal    (%rax,%rax,4),%eax
vga_bitblt_text+0x128:  movq    %rax,0xffffffffffffffb8(%rbp)
vga_bitblt_text+0x12c:  nopl
vga_bitblt_text+0x130:  movq    0x68(%r14),%rcx
vga_bitblt_text+0x134:  movl    0x3c(%r14),%eax
vga_bitblt_text+0x138:  movl    0xffffffffffffffc8(%rbp),%esi
vga_bitblt_text+0x13b:  addl    %esi,%eax
vga_bitblt_text+0x13d:  xorl    %edx,%edx
vga_bitblt_text+0x13f:  divl    0x38(%r14),%eax
vga_bitblt_text+0x143:  movq    (%rcx,%rdx,8),%rax
vga_bitblt_text+0x147:  movl    %r13d,%ecx
vga_bitblt_text+0x14a:  movl    (%rax,%rcx,4),%r15d
vga_bitblt_text+0x14e:  movq    0xffffffffffffffc0(%rbp),%rdi
vga_bitblt_text+0x152:  movl    %r13d,%edx
vga_bitblt_text+0x155:  call    vtbuf_iscursor
vga_bitblt_text+0x15a:  movl    %r15d,%edi
vga_bitblt_text+0x15d:  movl    %eax,%esi
vga_bitblt_text+0x15f:  leaq    0xffffffffffffffd6(%rbp),%rdx
vga_bitblt_text+0x163:  leaq    0xffffffffffffffd7(%rbp),%rcx
vga_bitblt_text+0x167:  call    vt_determine_colors
vga_bitblt_text+0x16c:  movl    %r15d,%r8d
vga_bitblt_text+0x16f:  andl    $0x1fffff,%r8d
vga_bitblt_text+0x176:  leal    0xffffffffffffffe0(%r8),%ecx
vga_bitblt_text+0x17a:  movb    $0x3f,%rax
vga_bitblt_text+0x17c:  cmpl    $0x264c,%ecx
vga_bitblt_text+0x182:  jnbe    vga_bitblt_text+0x1e3
vga_bitblt_text+0x184:  movq    %rbx,%r9
vga_bitblt_text+0x187:  xorl    %esi,%esi
vga_bitblt_text+0x189:  movl    $0xa8,%rdx
vga_bitblt_text+0x18e:  nop
vga_bitblt_text+0x190:  leal    (%rdx,%rsi,1),%ecx
vga_bitblt_text+0x193:  movl    %ecx,%edi
vga_bitblt_text+0x195:  shrl    $0x1f,%edi
vga_bitblt_text+0x198:  addl    %ecx,%edi
vga_bitblt_text+0x19a:  sarl    $1,%edi
vga_bitblt_text+0x19c:  movslq  %edi,%rbx
vga_bitblt_text+0x19f:  movzwl  cp437table(,%rbx,4),%ecx
vga_bitblt_text+0x1a7:  cmpl    %ecx,%r8d
vga_bitblt_text+0x1aa:  jnb     vga_bitblt_text+0x1b5
vga_bitblt_text+0x1ac:  leal    0xffffffffffffffff(%rdi),%edx
vga_bitblt_text+0x1af:  cmpl    %esi,%edi
vga_bitblt_text+0x1b1:  jnle    vga_bitblt_text+0x190
vga_bitblt_text+0x1b3:  jmp     vga_bitblt_text+0x1e0
vga_bitblt_text+0x1b5:  movzbl  cp437table+0x3(,%rbx,4),%esi
vga_bitblt_text+0x1bd:  addl    %ecx,%esi
vga_bitblt_text+0x1bf:  cmpl    %r8d,%esi
vga_bitblt_text+0x1c2:  jnb     vga_bitblt_text+0x1cd
vga_bitblt_text+0x1c4:  leal    0x1(%rdi),%esi
vga_bitblt_text+0x1c7:  cmpl    %edi,%edx
vga_bitblt_text+0x1c9:  jnle    vga_bitblt_text+0x190
vga_bitblt_text+0x1cb:  jmp     vga_bitblt_text+0x1e0
vga_bitblt_text+0x1cd:  subl    %ecx,%r15d
vga_bitblt_text+0x1d0:  movzbl  cp437table+0x2(,%rbx,4),%eax
vga_bitblt_text+0x1d8:  addl    %r15d,%eax
vga_bitblt_text+0x1db:  nopl
vga_bitblt_text+0x1e0:  movq    %r9,%rbx
vga_bitblt_text+0x1e3:  movb    0xffffffffffffffd7(%rbp),%cl
vga_bitblt_text+0x1e6:  shlb    $0x4,%cl
vga_bitblt_text+0x1e9:  orb     0xffffffffffffffd6(%rbp),%cl
vga_bitblt_text+0x1ec:  movq    0xffffffffffffffb8(%rbp),%rdx
vga_bitblt_text+0x1f0:  leal    (%r13,%rdx,1),%esi
vga_bitblt_text+0x1f5:  addl    %esi,%esi
vga_bitblt_text+0x1f7:  movq    0x8(%r12),%rdx
vga_bitblt_text+0x1fc:  addq    %rsi,%rdx
vga_bitblt_text+0x1ff:  cmpq    $0,(%r12)
vga_bitblt_text+0x204:  jz      vga_bitblt_text+0x210
vga_bitblt_text+0x206:  movb    %al,(%rdx)
vga_bitblt_text+0x208:  jmp     vga_bitblt_text+0x211
vga_bitblt_text+0x20a:  nopw
vga_bitblt_text+0x210:  outb    %al,%dx
vga_bitblt_text+0x211:  orl     $0x1,%esi
vga_bitblt_text+0x214:  addq    0x8(%r12),%rsi
vga_bitblt_text+0x219:  cmpq    $0,(%r12)
vga_bitblt_text+0x21e:  jz      vga_bitblt_text+0x230
vga_bitblt_text+0x220:  movb    %cl,(%rsi)
vga_bitblt_text+0x222:  jmp     vga_bitblt_text+0x235
vga_bitblt_text+0x224:  nopw
vga_bitblt_text+0x230:  movl    %ecx,%eax
vga_bitblt_text+0x232:  movl    %esi,%edx
vga_bitblt_text+0x234:  outb    %al,%dx
vga_bitblt_text+0x235:  incl    %r13d
vga_bitblt_text+0x238:  movzwl  0x6(%rbx),%eax
vga_bitblt_text+0x23c:  cmpl    %eax,%r13d
vga_bitblt_text+0x23f:  jb      vga_bitblt_text+0x130
vga_bitblt_text+0x245:  movzwl  0x4(%rbx),%ecx
vga_bitblt_text+0x249:  movl    0xffffffffffffffc8(%rbp),%esi
vga_bitblt_text+0x24c:  incl    %esi
vga_bitblt_text+0x24e:  movzwl  %cx,%edx
vga_bitblt_text+0x251:  movl    %esi,0xffffffffffffffc8(%rbp)
vga_bitblt_text+0x254:  cmpl    %edx,%esi
vga_bitblt_text+0x256:  jb      vga_bitblt_text+0x110
vga_bitblt_text+0x25c:  addq    $0x28,%rsp
vga_bitblt_text+0x260:  popq    %rbx
vga_bitblt_text+0x261:  popq    %r12
vga_bitblt_text+0x263:  popq    %r13
vga_bitblt_text+0x265:  popq    %r14
vga_bitblt_text+0x267:  popq    %r15
vga_bitblt_text+0x269:  popq    %rbp
vga_bitblt_text+0x26a:  ret
vga_bitblt_text+0x26b:  nopl
Comment 16 Mahmoud Al-Qudsi 2017-10-20 02:32:20 UTC
This wasn't resolved in -p1; I just experienced it myself on an ESXi 6.5.0U1 host. It doesn't happen in any deterministic fashion, so far as I can see:

kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x114
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff808eac3a
stack pointer           = 0x28:0xfffffe022f438000
frame pointer           = 0x28:0xfffffe022f438050
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 4 (doneq0)
trap number             = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec3641 at calltrap+0x8
#7 0xffffffff808f26d8 at vt_flush+0x398
#8 0xffffffff80ac09cf at termcn_cnputc+0xff
#9 0xffffffff80a100ed at cnputc+0x7d
#10 0xffffffff80a103f8 at cnputs+0xb8
#11 0xffffffff80ab36ad at putchar+0x14d
#12 0xffffffff80ab323d at kvprintf+0x103d
#13 0xffffffff80ab3ea7 at vprintf+0x87
#14 0xffffffff80ab3e13 at printf+0x43
#15 0xffffffff80307c52 at xpt_announce_periph+0x62
#16 0xffffffff8032e089 at cddone+0x249
#17 0xffffffff8030dbe7 at xpt_done_process+0x677
Uptime: 2s
Comment 17 Toshimichi Masubuchi 2017-11-30 04:53:08 UTC
In my environment I can avoid it by setting the "kern.cam.scsi_delay" option.

/boot/loader.conf:
kern.cam.scsi_delay="5000"

OS Version:
# uname -v
FreeBSD 11.1-RELEASE-p4 #0: Tue Nov 14 06:12:40 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
# freebsd-version
11.1-RELEASE-p5

Environment:
VMware ESXi 6.0U3 on HP DL380 G7
Comment 18 Toshimichi Masubuchi 2017-12-01 16:12:13 UTC
(In reply to Toshimichi Masubuchi from comment #17)

If there is no effect with the kern.cam.scsi_delay option, try the kern.cam.boot_delay option.

/boot/loader.conf:
kern.cam.boot_delay="5000"
or
kern.cam.boot_delay="10000"
Comment 19 Gert Doering 2018-01-24 20:16:37 UTC
I'm observing this as well with 11.1-RELEASE-p6 on ESXi 6.5.0U1.

Will try the loader.conf settings next...
Comment 20 Josh Paetzel freebsd_committer freebsd_triage 2018-06-06 18:46:58 UTC
is the mitigation working for anyone else?
Comment 21 Scott Aitken 2019-03-09 04:03:19 UTC
I can confirm that this is still happening on ESXi 6.7.0 #1 SMP Release build-11675023 with FreeBSD 11.2-RELEASE.

Did anyone confirm if those loader.conf tuneables helped?
Comment 22 Josh Paetzel freebsd_committer freebsd_triage 2019-03-21 15:07:50 UTC
I never heard back on whether the loader tunables helped or not.  cddone() is in the callstack of every crash dump I saw.  You can mitigate that by removing the CD hardware from the VM.

(The virtual CD is the only device that causes that code path to be run)
Comment 23 Zhenlei Huang freebsd_committer freebsd_triage 2020-10-29 06:57:58 UTC
This duplicate with https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217282, fixed in stable/11 by MFC r314116.
Comment 24 Zhenlei Huang freebsd_committer freebsd_triage 2020-10-29 07:04:38 UTC

*** This bug has been marked as a duplicate of bug 217282 ***