Comment 1 Conrad Meyer 2018-06-22 01:14:34 UTC

> 8  0xffffffff80df76ee in aesni_encrypt_cbc (rounds=10, key_schedule=0xfffff8001446cc00, len=2,

If this is accurate (may not be), this is a bogus invocation of AES-CBC (and probably should be validated higher in OCF).  AES-CBC is a block cipher; it operates in AES block unit sizes (multiples of 16 bytes).

Comment 2 Konstantin Belousov 2018-06-22 11:08:37 UTC

I agree that the length is weird.

Disassemble the faulting instruction, and show the registers content.
From kgdb, perhaps you need one installed from ports,
(kgdb) disassemble 0xffffffff80df76ee
(kgdb) p/x *(struct frame *)0xfffffe0688f57d30

(In reply to Konstantin Belousov from comment #2)
Thank-you for looking into this.  I should clarify an earlier observation.  It appears that the ssh connection was initiated from within an i386 jail that resides on the amd64 base system.  i386 jails run openssl and not libressl.

To you request Konstantin
(kgdb) disassemble 0xffffffff80df76ee
Dump of assembler code for function aesni_encrypt_cbc:
0xffffffff80df76b0 <aesni_encrypt_cbc+0>:       push   %rbp
0xffffffff80df76b1 <aesni_encrypt_cbc+1>:       mov    %rsp,%rbp
0xffffffff80df76b4 <aesni_encrypt_cbc+4>:       sub    $0x90,%rsp
0xffffffff80df76bb <aesni_encrypt_cbc+11>:      mov    %edi,-0x2c(%rbp)
0xffffffff80df76be <aesni_encrypt_cbc+14>:      mov    %rsi,-0x38(%rbp)
0xffffffff80df76c2 <aesni_encrypt_cbc+18>:      mov    %rdx,-0x40(%rbp)
0xffffffff80df76c6 <aesni_encrypt_cbc+22>:      mov    %rcx,-0x48(%rbp)
0xffffffff80df76ca <aesni_encrypt_cbc+26>:      mov    %r8,-0x50(%rbp)
0xffffffff80df76ce <aesni_encrypt_cbc+30>:      mov    %r9,-0x58(%rbp)
0xffffffff80df76d2 <aesni_encrypt_cbc+34>:      mov    -0x40(%rbp),%rcx
0xffffffff80df76d6 <aesni_encrypt_cbc+38>:      shr    $0x4,%rcx
0xffffffff80df76da <aesni_encrypt_cbc+42>:      mov    %rcx,-0x40(%rbp)
0xffffffff80df76de <aesni_encrypt_cbc+46>:      mov    -0x58(%rbp),%rcx
0xffffffff80df76e2 <aesni_encrypt_cbc+50>:      mov    %rcx,-0x28(%rbp)
0xffffffff80df76e6 <aesni_encrypt_cbc+54>:      mov    -0x28(%rbp),%rcx
0xffffffff80df76ea <aesni_encrypt_cbc+58>:      movdqu (%rcx),%xmm0
0xffffffff80df76ee <aesni_encrypt_cbc+62>:      movdqa %xmm0,-0x80(%rbp)
0xffffffff80df76f3 <aesni_encrypt_cbc+67>:      movq   $0x0,-0x88(%rbp)
0xffffffff80df76fe <aesni_encrypt_cbc+78>:      mov    -0x88(%rbp),%rax
0xffffffff80df7705 <aesni_encrypt_cbc+85>:      cmp    -0x40(%rbp),%rax
0xffffffff80df7709 <aesni_encrypt_cbc+89>:      jae    0xffffffff80df7795 <aesni_encrypt_cbc+229>
0xffffffff80df770f <aesni_encrypt_cbc+95>:      mov    -0x2c(%rbp),%eax
0xffffffff80df7712 <aesni_encrypt_cbc+98>:      sub    $0x1,%eax
0xffffffff80df7715 <aesni_encrypt_cbc+101>:     mov    -0x38(%rbp),%rcx
0xffffffff80df7719 <aesni_encrypt_cbc+105>:     mov    -0x48(%rbp),%rdx
0xffffffff80df771d <aesni_encrypt_cbc+109>:     mov    %rdx,-0x8(%rbp)
0xffffffff80df7721 <aesni_encrypt_cbc+113>:     mov    -0x8(%rbp),%rdx
0xffffffff80df7725 <aesni_encrypt_cbc+117>:     movdqu (%rdx),%xmm0
0xffffffff80df7729 <aesni_encrypt_cbc+121>:     pxor   -0x80(%rbp),%xmm0
0xffffffff80df772e <aesni_encrypt_cbc+126>:     mov    %eax,%edi
0xffffffff80df7730 <aesni_encrypt_cbc+128>:     mov    %rcx,%rsi
0xffffffff80df7733 <aesni_encrypt_cbc+131>:     callq  0xffffffff80df77a0 <aesni_enc>
0xffffffff80df7738 <aesni_encrypt_cbc+136>:     movdqa %xmm0,-0x70(%rbp)
0xffffffff80df773d <aesni_encrypt_cbc+141>:     movdqa -0x70(%rbp),%xmm0
0xffffffff80df7742 <aesni_encrypt_cbc+146>:     movdqa %xmm0,-0x80(%rbp)
0xffffffff80df7747 <aesni_encrypt_cbc+151>:     mov    -0x50(%rbp),%rcx
---Type <return> to continue, or q <return> to quit---
0xffffffff80df774b <aesni_encrypt_cbc+155>:     movdqa -0x70(%rbp),%xmm0
0xffffffff80df7750 <aesni_encrypt_cbc+160>:     mov    %rcx,-0x10(%rbp)
0xffffffff80df7754 <aesni_encrypt_cbc+164>:     movdqa %xmm0,-0x20(%rbp)
0xffffffff80df7759 <aesni_encrypt_cbc+169>:     movdqa -0x20(%rbp),%xmm0
0xffffffff80df775e <aesni_encrypt_cbc+174>:     mov    -0x10(%rbp),%rcx
0xffffffff80df7762 <aesni_encrypt_cbc+178>:     movdqu %xmm0,(%rcx)
0xffffffff80df7766 <aesni_encrypt_cbc+182>:     mov    -0x48(%rbp),%rcx
0xffffffff80df776a <aesni_encrypt_cbc+186>:     add    $0x10,%rcx
0xffffffff80df776e <aesni_encrypt_cbc+190>:     mov    %rcx,-0x48(%rbp)
0xffffffff80df7772 <aesni_encrypt_cbc+194>:     mov    -0x50(%rbp),%rcx
0xffffffff80df7776 <aesni_encrypt_cbc+198>:     add    $0x10,%rcx
0xffffffff80df777a <aesni_encrypt_cbc+202>:     mov    %rcx,-0x50(%rbp)
0xffffffff80df777e <aesni_encrypt_cbc+206>:     mov    -0x88(%rbp),%rax
0xffffffff80df7785 <aesni_encrypt_cbc+213>:     add    $0x1,%rax
0xffffffff80df7789 <aesni_encrypt_cbc+217>:     mov    %rax,-0x88(%rbp)
0xffffffff80df7790 <aesni_encrypt_cbc+224>:     jmpq   0xffffffff80df76fe <aesni_encrypt_cbc+78>
0xffffffff80df7795 <aesni_encrypt_cbc+229>:     add    $0x90,%rsp
0xffffffff80df779c <aesni_encrypt_cbc+236>:     pop    %rbp
0xffffffff80df779d <aesni_encrypt_cbc+237>:     retq
End of assembler dump.
Current language:  auto; currently minimal
(kgdb) p/x *(struct frame *)0xfffffe0688f57d30
No struct type named frame.

Happy to provide any information that will help. I'm GMT+10, 23:18 local, so may take awhile.

Comment 4 Konstantin Belousov 2018-06-22 13:40:20 UTC

(In reply to dewayne from comment #3)
The structure is called trapframe.

(In reply to Konstantin Belousov from comment #4)
(kgdb) p/x *(struct trapframe *)0xfffffe0688f57d30
$1 = {tf_rdi = 0xa, tf_rsi = 0xfffff8001446cc00, tf_rdx = 0x20, tf_rcx = 0xfffffe0688f57fa8,
  tf_r8 = 0xfffff8001408bb80, tf_r9 = 0xfffffe0688f57fa8, tf_rax = 0xfffff803905493f0,
  tf_rbx = 0xfffffe0688f58698, tf_rbp = 0xfffffe0688f57e88, tf_r10 = 0x28, tf_r11 = 0x0,
  tf_r12 = 0x0, tf_r13 = 0x0, tf_r14 = 0x400000000000080, tf_r15 = 0x0, tf_trapno = 0x9,
  tf_fs = 0x13, tf_gs = 0x1b, tf_addr = 0x0, tf_flags = 0x1, tf_es = 0x3b, tf_ds = 0x3b,
  tf_err = 0x0, tf_rip = 0xffffffff80df76ee, tf_cs = 0x20, tf_rflags = 0x10202,
  tf_rsp = 0xfffffe0688f57df8, tf_ss = 0x0}

Thanks Kib, I appreciate your patience, as I'm really just your fingers, its been 28 years since I had to use a C debugger.

Comment 6 Konstantin Belousov 2018-06-22 14:50:28 UTC

Created attachment 194505 [details]
MFC r333059

(In reply to Konstantin Belousov from comment #6)
Unfortunately not a solution - performed svn update, then successfully applied the patch attachment; rebuilt,installed and running the non-debugging kernel which has crashed.  Need to fsck and build fresh debugging kernel.

Comment 8 Konstantin Belousov 2018-06-22 23:40:34 UTC

(In reply to dewayne from comment #7)
Crashed at the same place, I expect ?  I.e. same instruction in the disassembly ?

The issue is that the stack is not aligned to 16b.  The patch I attached should fixed some case where the stack could be unaligned (but usually not) on the first C function entry from the exception asm handler.  Since it did not helped, my highest suspect is the compiler.

(In reply to Konstantin Belousov from comment #8)
After performing buildworld, buildkernel & installkernel - I'm trying to induce a crash which may take a few hours.  Yes, I thought that the compiler had a part to play but I did a partial revert (mentioned in initial PR); which didn't prevent crashing.

(In reply to Konstantin Belousov from comment #8)
Just crashed, results being:
kgdb /pd2/tmp/destQ/usr/lib/debug/boot/kernel/kernel.debug /var/crash/vmcore.10
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
[9024]
[9024]
[9024] Fatal trap 9: general protection fault while in kernel mode
[9024] cpuid = 2; apic id = 02
[9024] instruction pointer      = 0x20:0xffffffff80df7abe
[9024] stack pointer            = 0x0:0xfffffe0688b39df8
[9024] frame pointer            = 0x0:0xfffffe0688b39e88
[9024] code segment             = base rx0, limit 0xfffff, type 0x1b
[9024]                  = DPL 0, pres 1, long 1, def32 0, gran 1
[9024] processor eflags = interrupt enabled, resume, IOPL = 0
[9024] current process          = 53068 (ssh)
[9024] trap number              = 9
[9024] panic: general protection fault
[9024] cpuid = 2
[9024] Uptime: 2h30m24s
[9024] Dumping 1861 out of 24501 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /pd2/tmp/destQ/usr/lib/debug/boot/kernel/mac_ifoff.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/mac_ifoff.ko.debug
Reading symbols from /pd2/tmp/destQ/usr/lib/debug/boot/kernel/cpufreq.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/cpufreq.ko.debug
Reading symbols from /pd2/tmp/destQ/usr/lib/debug/boot/kernel/coretemp.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/coretemp.ko.debug
Reading symbols from /pd2/tmp/destQ/usr/lib/debug/boot/kernel/uplcom.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/uplcom.ko.debug
Reading symbols from /pd2/tmp/destQ/usr/lib/debug/boot/kernel/ucom.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/ucom.ko.debug
#0  doadump (textdump=1) at /smallblocks/src/sys/kern/kern_shutdown.c:315
315             dumptid = curthread->td_tid;
(kgdb) list *0xffffffff80df7abe
0xffffffff80df7abe is in aesni_encrypt_cbc (/smallblocks/src/sys/crypto/aesni/aesni_wrap.c:64).
59      {
60              __m128i tot, ivreg;
61              size_t i;
62
63              len /= AES_BLOCK_LEN;
64              ivreg = _mm_loadu_si128((const __m128i *)iv);
65              for (i = 0; i < len; i++) {
66                      tot = aesni_enc(rounds - 1, key_schedule,
67                          _mm_loadu_si128((const __m128i *)from) ^ ivreg);
68                      ivreg = tot;
Current language:  auto; currently minimal
(kgdb) disassemble 0xffffffff80df7abe
Dump of assembler code for function aesni_encrypt_cbc:
0xffffffff80df7a80 <aesni_encrypt_cbc+0>:       push   %rbp
0xffffffff80df7a81 <aesni_encrypt_cbc+1>:       mov    %rsp,%rbp
0xffffffff80df7a84 <aesni_encrypt_cbc+4>:       sub    $0x90,%rsp
0xffffffff80df7a8b <aesni_encrypt_cbc+11>:      mov    %edi,-0x2c(%rbp)
0xffffffff80df7a8e <aesni_encrypt_cbc+14>:      mov    %rsi,-0x38(%rbp)
0xffffffff80df7a92 <aesni_encrypt_cbc+18>:      mov    %rdx,-0x40(%rbp)
0xffffffff80df7a96 <aesni_encrypt_cbc+22>:      mov    %rcx,-0x48(%rbp)
0xffffffff80df7a9a <aesni_encrypt_cbc+26>:      mov    %r8,-0x50(%rbp)
0xffffffff80df7a9e <aesni_encrypt_cbc+30>:      mov    %r9,-0x58(%rbp)
0xffffffff80df7aa2 <aesni_encrypt_cbc+34>:      mov    -0x40(%rbp),%rcx
0xffffffff80df7aa6 <aesni_encrypt_cbc+38>:      shr    $0x4,%rcx
0xffffffff80df7aaa <aesni_encrypt_cbc+42>:      mov    %rcx,-0x40(%rbp)
0xffffffff80df7aae <aesni_encrypt_cbc+46>:      mov    -0x58(%rbp),%rcx
0xffffffff80df7ab2 <aesni_encrypt_cbc+50>:      mov    %rcx,-0x28(%rbp)
0xffffffff80df7ab6 <aesni_encrypt_cbc+54>:      mov    -0x28(%rbp),%rcx
0xffffffff80df7aba <aesni_encrypt_cbc+58>:      movdqu (%rcx),%xmm0
0xffffffff80df7abe <aesni_encrypt_cbc+62>:      movdqa %xmm0,-0x80(%rbp)
0xffffffff80df7ac3 <aesni_encrypt_cbc+67>:      movq   $0x0,-0x88(%rbp)
0xffffffff80df7ace <aesni_encrypt_cbc+78>:      mov    -0x88(%rbp),%rax
0xffffffff80df7ad5 <aesni_encrypt_cbc+85>:      cmp    -0x40(%rbp),%rax
0xffffffff80df7ad9 <aesni_encrypt_cbc+89>:      jae    0xffffffff80df7b65 <aesni_encrypt_cbc+229>
0xffffffff80df7adf <aesni_encrypt_cbc+95>:      mov    -0x2c(%rbp),%eax
0xffffffff80df7ae2 <aesni_encrypt_cbc+98>:      sub    $0x1,%eax
0xffffffff80df7ae5 <aesni_encrypt_cbc+101>:     mov    -0x38(%rbp),%rcx
0xffffffff80df7ae9 <aesni_encrypt_cbc+105>:     mov    -0x48(%rbp),%rdx
0xffffffff80df7aed <aesni_encrypt_cbc+109>:     mov    %rdx,-0x8(%rbp)
0xffffffff80df7af1 <aesni_encrypt_cbc+113>:     mov    -0x8(%rbp),%rdx
0xffffffff80df7af5 <aesni_encrypt_cbc+117>:     movdqu (%rdx),%xmm0
0xffffffff80df7af9 <aesni_encrypt_cbc+121>:     pxor   -0x80(%rbp),%xmm0
0xffffffff80df7afe <aesni_encrypt_cbc+126>:     mov    %eax,%edi
0xffffffff80df7b00 <aesni_encrypt_cbc+128>:     mov    %rcx,%rsi
0xffffffff80df7b03 <aesni_encrypt_cbc+131>:     callq  0xffffffff80df7b70 <aesni_enc>
0xffffffff80df7b08 <aesni_encrypt_cbc+136>:     movdqa %xmm0,-0x70(%rbp)
0xffffffff80df7b0d <aesni_encrypt_cbc+141>:     movdqa -0x70(%rbp),%xmm0
0xffffffff80df7b12 <aesni_encrypt_cbc+146>:     movdqa %xmm0,-0x80(%rbp)
0xffffffff80df7b17 <aesni_encrypt_cbc+151>:     mov    -0x50(%rbp),%rcx
0xffffffff80df7b1b <aesni_encrypt_cbc+155>:     movdqa -0x70(%rbp),%xmm0
0xffffffff80df7b20 <aesni_encrypt_cbc+160>:     mov    %rcx,-0x10(%rbp)
0xffffffff80df7b24 <aesni_encrypt_cbc+164>:     movdqa %xmm0,-0x20(%rbp)
0xffffffff80df7b29 <aesni_encrypt_cbc+169>:     movdqa -0x20(%rbp),%xmm0
0xffffffff80df7b2e <aesni_encrypt_cbc+174>:     mov    -0x10(%rbp),%rcx
0xffffffff80df7b32 <aesni_encrypt_cbc+178>:     movdqu %xmm0,(%rcx)
0xffffffff80df7b36 <aesni_encrypt_cbc+182>:     mov    -0x48(%rbp),%rcx
0xffffffff80df7b3a <aesni_encrypt_cbc+186>:     add    $0x10,%rcx
---Type <return> to continue, or q <return> to quit---
0xffffffff80df7b3e <aesni_encrypt_cbc+190>:     mov    %rcx,-0x48(%rbp)
0xffffffff80df7b42 <aesni_encrypt_cbc+194>:     mov    -0x50(%rbp),%rcx
0xffffffff80df7b46 <aesni_encrypt_cbc+198>:     add    $0x10,%rcx
0xffffffff80df7b4a <aesni_encrypt_cbc+202>:     mov    %rcx,-0x50(%rbp)
0xffffffff80df7b4e <aesni_encrypt_cbc+206>:     mov    -0x88(%rbp),%rax
0xffffffff80df7b55 <aesni_encrypt_cbc+213>:     add    $0x1,%rax
0xffffffff80df7b59 <aesni_encrypt_cbc+217>:     mov    %rax,-0x88(%rbp)
0xffffffff80df7b60 <aesni_encrypt_cbc+224>:     jmpq   0xffffffff80df7ace <aesni_encrypt_cbc+78>
0xffffffff80df7b65 <aesni_encrypt_cbc+229>:     add    $0x90,%rsp
0xffffffff80df7b6c <aesni_encrypt_cbc+236>:     pop    %rbp
0xffffffff80df7b6d <aesni_encrypt_cbc+237>:     retq
End of assembler dump.
(kgdb) backtrace
#0  doadump (textdump=1) at /smallblocks/src/sys/kern/kern_shutdown.c:315
#1  0xffffffff8080984b in kern_reboot (howto=260)
    at /smallblocks/src/sys/kern/kern_shutdown.c:383
#2  0xffffffff8080a08c in vpanic (fmt=0xffffffff80e89d2a "%s", ap=0xfffffe0688b398e0)
    at /smallblocks/src/sys/kern/kern_shutdown.c:776
#3  0xffffffff80809e70 in panic (fmt=0xffffffff80e89d2a "%s")
    at /smallblocks/src/sys/kern/kern_shutdown.c:707
#4  0xffffffff80de9967 in trap_fatal (frame=0xfffffe0688b39d30, eva=0)
    at /smallblocks/src/sys/amd64/amd64/trap.c:877
#5  0xffffffff80de8ff2 in trap (frame=0xfffffe0688b39d30)
    at /smallblocks/src/sys/amd64/amd64/trap.c:610
#6  0xffffffff80de9db5 in trap_check (frame=0xfffffe0688b39d30)
    at /smallblocks/src/sys/amd64/amd64/trap.c:659
#7  0xffffffff80dbe8be in calltrap () at /smallblocks/src/sys/amd64/amd64/exception.S:231
#8  0xffffffff80df7abe in aesni_encrypt_cbc (rounds=10, key_schedule=0xfffff8001420a800, len=2,
    from=0xfffff8001203ede0 "\016\005", to=0xfffff8001203ede0 "\016\005",
    iv=0xfffffe0688b39fa8 "▒:▒▒\214:d▒w}4o▒\232\235\214")
    at /smallblocks/src/sys/crypto/aesni/aesni_wrap.c:64
#9  0xffffffff80defedb in aesni_cipher_process (ses=0xfffff8001420a800,
    enccrd=0xfffff802de6533f0, authcrd=0x0, crp=0xfffff801c28d8bb0)
    at /smallblocks/src/sys/crypto/aesni/aesni.c:606
#10 0xffffffff80def420 in aesni_process (dev=0xfffff8000567de00, crp=0xfffff801c28d8bb0, hint=0)
    at /smallblocks/src/sys/crypto/aesni/aesni.c:413
#11 0xffffffff80c7a085 in CRYPTODEV_PROCESS (dev=0xfffff8000567de00, op=0xfffff801c28d8bb0,
    flags=0) at cryptodev_if.h:53
#12 0xffffffff80c78a64 in crypto_invoke (cap=0xfffff80005023078, crp=0xfffff801c28d8bb0, hint=0)
    at /smallblocks/src/sys/opencrypto/crypto.c:1083
#13 0xffffffff80c787ea in crypto_dispatch (crp=0xfffff801c28d8bb0)
    at /smallblocks/src/sys/opencrypto/crypto.c:844
#14 0xffffffff80c7cbca in cryptodev_op (cse=0xfffff8028fb70e00, cop=0xfffffe0688b3a3f0,
    active_cred=0xfffff8028fb71000, td=0xfffff80146055620)
    at /smallblocks/src/sys/opencrypto/cryptodev.c:849
#15 0xffffffff80c7bc6f in cryptof_ioctl (fp=0xfffff802de7e2690, cmd=3223085927,
    data=0xfffffe0688b3a720, active_cred=0xfffff8028fb71000, td=0xfffff80146055620)
    at /smallblocks/src/sys/opencrypto/cryptodev.c:633
#16 0xffffffff80898dfc in fo_ioctl (fp=0xfffff802de7e2690, com=3223085927,
    data=0xfffffe0688b3a720, active_cred=0xfffff8028fb71000, td=0xfffff80146055620)
    at file.h:323
#17 0xffffffff80898bf4 in kern_ioctl (td=0xfffff80146055620, fd=5, com=3223085927,
    data=0xfffffe0688b3a720 "") at /smallblocks/src/sys/kern/sys_generic.c:836
#18 0xffffffff808987ba in sys_ioctl (td=0xfffff80146055620, uap=0xfffffe0688b3a878)
    at /smallblocks/src/sys/kern/sys_generic.c:745
#19 0xffffffff8030e351 in freebsd32_ioctl (td=0xfffff80146055620, uap=0xfffff80146055b58)
    at /smallblocks/src/sys/compat/freebsd32/freebsd32_ioctl.c:470
#20 0xffffffff80e3198e in syscallenter (td=0xfffff80146055620) at subr_syscall.c:132
#21 0xffffffff80e3130f in ia32_syscall (frame=0xfffffe0688b3aab8)
---Type <return> to continue, or q <return> to quit---
    at /smallblocks/src/sys/amd64/ia32/ia32_syscall.c:218
#22 0xffffffff80dbf906 in int0x80_syscall_common () at ia32_exception.S:76
#23 0x0000000000000000 in ?? ()
(kgdb) p/x *(struct trapframe *)0xfffffe0688b39d30
$1 = {tf_rdi = 0xa, tf_rsi = 0xfffff8001420a800, tf_rdx = 0x20, tf_rcx = 0xfffffe0688b39fa8,
  tf_r8 = 0xfffff8001203ede0, tf_r9 = 0xfffffe0688b39fa8, tf_rax = 0xfffff802de6533f0,
  tf_rbx = 0xfffffe0688b3a698, tf_rbp = 0xfffffe0688b39e88, tf_r10 = 0x28, tf_r11 = 0x0,
  tf_r12 = 0x0, tf_r13 = 0x0, tf_r14 = 0x400000000000080, tf_r15 = 0x0, tf_trapno = 0x9,
  tf_fs = 0x13, tf_gs = 0x1b, tf_addr = 0x0, tf_flags = 0x1, tf_es = 0x3b, tf_ds = 0x3b,
  tf_err = 0x0, tf_rip = 0xffffffff80df7abe, tf_cs = 0x20, tf_rflags = 0x10202,
  tf_rsp = 0xfffffe0688b39df8, tf_ss = 0x0}

CFLAGS, COPTFLAGS, LDFLAGS are as follows:
# make -VCFLAGS -Vsp -VCOPTFLAGS -Vsp -VLDFLAGS
-pipe -g0 -ggdb0 -DSTRIP_FBSDID -UDEBUGGING -UDEBUG -DUSB_HAVE_DISABLE_ENUM -O3 -fno-math-errno -fomit-frame-pointer -Wno-error=unused-command-line-argument  -Wl,--hash-style=sysv -march=haswell

-g0 -ggdb0 -DSTRIP_FBSDID -UDEBUGGING -UDEBUG -DUSB_HAVE_DISABLE_ENUM -fno-math-errno -fomit-frame-pointer -Wno-error=unused-command-line-argument -Wl,--hash-style=sysv -O2


# ie no LDFLAGS.  

Differences in make.conf files between the working kernel/world builds (prior to May) and now is that we:
- removed -O3 from COPTFLAGS, we explicitly state -O2
- Added TARGET=amd64

Comment 12 Konstantin Belousov 2018-06-23 09:19:01 UTC

(In reply to dewayne from comment #11)
Build with *all default* settings and reproduce the issue first.

(In reply to Konstantin Belousov from comment #12)
Using "make __MAKE_CONF=/dev/null" so no influence of make.conf settings.  The uptime is less than 10 minutes.

kgdb /pd2/tmp/destR/usr/lib/debug/boot/kernel/kernel.debug /var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
[490]
[490]
[490] Fatal trap 9: general protection fault while in kernel mode
[490] cpuid = 4; apic id = 04
[490] instruction pointer       = 0x20:0xffffffff808e7d89
[490] stack pointer             = 0x0:0xfffffe06892d9068
[490] frame pointer             = 0x0:0xfffffe06892d9138
[490] code segment              = base rx0, limit 0xfffff, type 0x1b
[490]                   = DPL 0, pres 1, long 1, def32 0, gran 1
[490] processor eflags  = interrupt enabled, resume, IOPL = 0
[490] current process           = 22887 (ssh)
[490] trap number               = 9
[490] panic: general protection fault
[490] cpuid = 4
[490] Uptime: 8m10s
[490] Dumping 1125 out of 24507 MB:..2%..12%..22%..32%..42%..52%..62%..72%..82%..92%

Reading symbols from /pd2/tmp/destR/usr/lib/debug/boot/kernel/mac_ifoff.ko.debug...done.
Loaded symbols for /pd2/tmp/destR/usr/lib/debug/boot/kernel/mac_ifoff.ko.debug
Reading symbols from /pd2/tmp/destR/usr/lib/debug/boot/kernel/cpufreq.ko.debug...done.
Loaded symbols for /pd2/tmp/destR/usr/lib/debug/boot/kernel/cpufreq.ko.debug
Reading symbols from /pd2/tmp/destR/usr/lib/debug/boot/kernel/coretemp.ko.debug...done.
Loaded symbols for /pd2/tmp/destR/usr/lib/debug/boot/kernel/coretemp.ko.debug
Reading symbols from /pd2/tmp/destR/usr/lib/debug/boot/kernel/uplcom.ko.debug...done.
Loaded symbols for /pd2/tmp/destR/usr/lib/debug/boot/kernel/uplcom.ko.debug
Reading symbols from /pd2/tmp/destR/usr/lib/debug/boot/kernel/ucom.ko.debug...done.
Loaded symbols for /pd2/tmp/destR/usr/lib/debug/boot/kernel/ucom.ko.debug
#0  doadump (textdump=<value optimized out>) at pcpu.h:230
230     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) list *0xffffffff808e7d89
0xffffffff808e7d89 is in aesni_decrypt_cbc (aesencdec.h:102).
97                      f = _mm_aesdec_si128(f, keysched[i + 1]);
98                      g = _mm_aesdec_si128(g, keysched[i + 1]);
99                      h = _mm_aesdec_si128(h, keysched[i + 1]);
100             }
101
102             out[0] = _mm_aesdeclast_si128(a, keysched[i + 1]);
103             out[1] = _mm_aesdeclast_si128(b, keysched[i + 1]);
104             out[2] = _mm_aesdeclast_si128(c, keysched[i + 1]);
105             out[3] = _mm_aesdeclast_si128(d, keysched[i + 1]);
106             out[4] = _mm_aesdeclast_si128(e, keysched[i + 1]);
Current language:  auto; currently minimal
(kgdb)
107             out[5] = _mm_aesdeclast_si128(f, keysched[i + 1]);
108             out[6] = _mm_aesdeclast_si128(g, keysched[i + 1]);
109             out[7] = _mm_aesdeclast_si128(h, keysched[i + 1]);
110     }
111
112     /* rounds is passed in as rounds - 1 */
113     static inline __m128i
114     aesni_enc(int rounds, const __m128i *keysched, const __m128i from)
115     {
116             __m128i tmp;
(kgdb) disassemble 0xffffffff808e7d89
Dump of assembler code for function aesni_decrypt_cbc:
0xffffffff808e7ba0 <aesni_decrypt_cbc+0>:       push   %rbp
0xffffffff808e7ba1 <aesni_decrypt_cbc+1>:       mov    %rsp,%rbp
0xffffffff808e7ba4 <aesni_decrypt_cbc+4>:       push   %r15
0xffffffff808e7ba6 <aesni_decrypt_cbc+6>:       push   %r14
0xffffffff808e7ba8 <aesni_decrypt_cbc+8>:       push   %r13
0xffffffff808e7baa <aesni_decrypt_cbc+10>:      push   %r12
0xffffffff808e7bac <aesni_decrypt_cbc+12>:      push   %rbx
0xffffffff808e7bad <aesni_decrypt_cbc+13>:      sub    $0xa8,%rsp
0xffffffff808e7bb4 <aesni_decrypt_cbc+20>:      mov    %rcx,%r9
0xffffffff808e7bb7 <aesni_decrypt_cbc+23>:      movdqu (%r8),%xmm8
0xffffffff808e7bbc <aesni_decrypt_cbc+28>:      mov    %rdx,%r8
0xffffffff808e7bbf <aesni_decrypt_cbc+31>:      shr    $0x4,%r8
0xffffffff808e7bc3 <aesni_decrypt_cbc+35>:      mov    %rdx,%r11
0xffffffff808e7bc6 <aesni_decrypt_cbc+38>:      shr    $0x7,%r11
0xffffffff808e7bca <aesni_decrypt_cbc+42>:      je     0xffffffff808e7e96 <aesni_decrypt_cbc+758>
0xffffffff808e7bd0 <aesni_decrypt_cbc+48>:      lea    -0x1(%rdi),%eax
0xffffffff808e7bd3 <aesni_decrypt_cbc+51>:      mov    %rdx,%rbx
0xffffffff808e7bd6 <aesni_decrypt_cbc+54>:      and    $0xffffffffffffff80,%rbx
0xffffffff808e7bda <aesni_decrypt_cbc+58>:      test   %eax,%eax
0xffffffff808e7bdc <aesni_decrypt_cbc+60>:      jle    0xffffffff808e7eab <aesni_decrypt_cbc+779>
0xffffffff808e7be2 <aesni_decrypt_cbc+66>:      mov    %rbx,-0x30(%rbp)
0xffffffff808e7be6 <aesni_decrypt_cbc+70>:      mov    %rdx,-0x38(%rbp)
0xffffffff808e7bea <aesni_decrypt_cbc+74>:      movslq %edi,%rax
0xffffffff808e7bed <aesni_decrypt_cbc+77>:      shl    $0x4,%rax
0xffffffff808e7bf1 <aesni_decrypt_cbc+81>:      add    %rsi,%rax
0xffffffff808e7bf4 <aesni_decrypt_cbc+84>:      mov    %rax,-0x40(%rbp)
0xffffffff808e7bf8 <aesni_decrypt_cbc+88>:      mov    $0x200000000,%rax
0xffffffff808e7c02 <aesni_decrypt_cbc+98>:      mov    %edi,%ebx
0xffffffff808e7c04 <aesni_decrypt_cbc+100>:     mov    %edi,%r14d
0xffffffff808e7c07 <aesni_decrypt_cbc+103>:     not    %r14d
0xffffffff808e7c0a <aesni_decrypt_cbc+106>:     and    $0x1,%r14d
0xffffffff808e7c0e <aesni_decrypt_cbc+110>:     sub    %r14,%rbx
0xffffffff808e7c11 <aesni_decrypt_cbc+113>:     xor    %r15d,%r15d
0xffffffff808e7c14 <aesni_decrypt_cbc+116>:     mov    %r9,%rcx
0xffffffff808e7c17 <aesni_decrypt_cbc+119>:     mov    %rdi,-0x48(%rbp)
0xffffffff808e7c1b <aesni_decrypt_cbc+123>:     nopl   0x0(%rax,%rax,1)
0xffffffff808e7c20 <aesni_decrypt_cbc+128>:     movdqu (%rcx),%xmm9
0xffffffff808e7c25 <aesni_decrypt_cbc+133>:     movdqu 0x10(%rcx),%xmm0
0xffffffff808e7c2a <aesni_decrypt_cbc+138>:     movdqu 0x20(%rcx),%xmm7
0xffffffff808e7c2f <aesni_decrypt_cbc+143>:     movdqu 0x30(%rcx),%xmm6
0xffffffff808e7c34 <aesni_decrypt_cbc+148>:     movdqu 0x40(%rcx),%xmm5
0xffffffff808e7c39 <aesni_decrypt_cbc+153>:     movdqu 0x50(%rcx),%xmm4
0xffffffff808e7c3e <aesni_decrypt_cbc+158>:     movdqu 0x60(%rcx),%xmm3
---Type <return> to continue, or q <return> to quit---
0xffffffff808e7c43 <aesni_decrypt_cbc+163>:     movdqu 0x70(%rcx),%xmm2
0xffffffff808e7c48 <aesni_decrypt_cbc+168>:     movdqa (%rsi),%xmm10
0xffffffff808e7c4d <aesni_decrypt_cbc+173>:     movdqa 0x10(%rsi),%xmm1
0xffffffff808e7c52 <aesni_decrypt_cbc+178>:     pxor   %xmm10,%xmm9
0xffffffff808e7c57 <aesni_decrypt_cbc+183>:     pxor   %xmm10,%xmm0
0xffffffff808e7c5c <aesni_decrypt_cbc+188>:     pxor   %xmm10,%xmm7
0xffffffff808e7c61 <aesni_decrypt_cbc+193>:     pxor   %xmm10,%xmm6
0xffffffff808e7c66 <aesni_decrypt_cbc+198>:     pxor   %xmm10,%xmm5
0xffffffff808e7c6b <aesni_decrypt_cbc+203>:     pxor   %xmm10,%xmm4
0xffffffff808e7c70 <aesni_decrypt_cbc+208>:     pxor   %xmm10,%xmm3
0xffffffff808e7c75 <aesni_decrypt_cbc+213>:     pxor   %xmm10,%xmm2
0xffffffff808e7c7a <aesni_decrypt_cbc+218>:     cmp    $0x2,%edi
0xffffffff808e7c7d <aesni_decrypt_cbc+221>:     jne    0xffffffff808e7ca0 <aesni_decrypt_cbc+256>
0xffffffff808e7c7f <aesni_decrypt_cbc+223>:     mov    $0x1,%r13d
0xffffffff808e7c85 <aesni_decrypt_cbc+229>:     test   %r14d,%r14d
0xffffffff808e7c88 <aesni_decrypt_cbc+232>:     jne    0xffffffff808e7d3f <aesni_decrypt_cbc+415>
0xffffffff808e7c8e <aesni_decrypt_cbc+238>:     jmpq   0xffffffff808e7d83 <aesni_decrypt_cbc+483>
0xffffffff808e7c93 <aesni_decrypt_cbc+243>:     nopw   %cs:0x0(%rax,%rax,1)
0xffffffff808e7ca0 <aesni_decrypt_cbc+256>:     mov    $0x300000000,%rdi
0xffffffff808e7caa <aesni_decrypt_cbc+266>:     mov    %rax,%r10
0xffffffff808e7cad <aesni_decrypt_cbc+269>:     mov    $0x1,%r13d
0xffffffff808e7cb3 <aesni_decrypt_cbc+275>:     nopw   %cs:0x0(%rax,%rax,1)
0xffffffff808e7cc0 <aesni_decrypt_cbc+288>:     aesdec %xmm1,%xmm9
0xffffffff808e7cc6 <aesni_decrypt_cbc+294>:     aesdec %xmm1,%xmm0
0xffffffff808e7ccb <aesni_decrypt_cbc+299>:     aesdec %xmm1,%xmm7
0xffffffff808e7cd0 <aesni_decrypt_cbc+304>:     aesdec %xmm1,%xmm6
0xffffffff808e7cd5 <aesni_decrypt_cbc+309>:     aesdec %xmm1,%xmm5
0xffffffff808e7cda <aesni_decrypt_cbc+314>:     aesdec %xmm1,%xmm4
0xffffffff808e7cdf <aesni_decrypt_cbc+319>:     aesdec %xmm1,%xmm3
0xffffffff808e7ce4 <aesni_decrypt_cbc+324>:     aesdec %xmm1,%xmm2
0xffffffff808e7ce9 <aesni_decrypt_cbc+329>:     mov    %r10,%r12
0xffffffff808e7cec <aesni_decrypt_cbc+332>:     sar    $0x1c,%r12
0xffffffff808e7cf0 <aesni_decrypt_cbc+336>:     movdqa (%rsi,%r12,1),%xmm1
0xffffffff808e7cf6 <aesni_decrypt_cbc+342>:     aesdec %xmm1,%xmm9
0xffffffff808e7cfc <aesni_decrypt_cbc+348>:     aesdec %xmm1,%xmm0
0xffffffff808e7d01 <aesni_decrypt_cbc+353>:     aesdec %xmm1,%xmm7
0xffffffff808e7d06 <aesni_decrypt_cbc+358>:     aesdec %xmm1,%xmm6
0xffffffff808e7d0b <aesni_decrypt_cbc+363>:     aesdec %xmm1,%xmm5
0xffffffff808e7d10 <aesni_decrypt_cbc+368>:     aesdec %xmm1,%xmm4
0xffffffff808e7d15 <aesni_decrypt_cbc+373>:     aesdec %xmm1,%xmm3
0xffffffff808e7d1a <aesni_decrypt_cbc+378>:     aesdec %xmm1,%xmm2
0xffffffff808e7d1f <aesni_decrypt_cbc+383>:     add    $0x2,%r13
0xffffffff808e7d23 <aesni_decrypt_cbc+387>:     mov    %rdi,%rdx
---Type <return> to continue, or q <return> to quit---
0xffffffff808e7d26 <aesni_decrypt_cbc+390>:     sar    $0x1c,%rdx
0xffffffff808e7d2a <aesni_decrypt_cbc+394>:     movdqa (%rsi,%rdx,1),%xmm1
0xffffffff808e7d2f <aesni_decrypt_cbc+399>:     add    %rax,%r10
0xffffffff808e7d32 <aesni_decrypt_cbc+402>:     add    %rax,%rdi
0xffffffff808e7d35 <aesni_decrypt_cbc+405>:     cmp    %r13,%rbx
0xffffffff808e7d38 <aesni_decrypt_cbc+408>:     jne    0xffffffff808e7cc0 <aesni_decrypt_cbc+288>
0xffffffff808e7d3a <aesni_decrypt_cbc+410>:     test   %r14d,%r14d
0xffffffff808e7d3d <aesni_decrypt_cbc+413>:     je     0xffffffff808e7d83 <aesni_decrypt_cbc+483>
0xffffffff808e7d3f <aesni_decrypt_cbc+415>:     shl    $0x20,%r13
0xffffffff808e7d43 <aesni_decrypt_cbc+419>:     mov    $0x100000000,%rdx
0xffffffff808e7d4d <aesni_decrypt_cbc+429>:     add    %rdx,%r13
0xffffffff808e7d50 <aesni_decrypt_cbc+432>:     aesdec %xmm1,%xmm2
0xffffffff808e7d55 <aesni_decrypt_cbc+437>:     aesdec %xmm1,%xmm3
0xffffffff808e7d5a <aesni_decrypt_cbc+442>:     aesdec %xmm1,%xmm4
0xffffffff808e7d5f <aesni_decrypt_cbc+447>:     aesdec %xmm1,%xmm5
0xffffffff808e7d64 <aesni_decrypt_cbc+452>:     aesdec %xmm1,%xmm6
0xffffffff808e7d69 <aesni_decrypt_cbc+457>:     aesdec %xmm1,%xmm7
0xffffffff808e7d6e <aesni_decrypt_cbc+462>:     aesdec %xmm1,%xmm0
0xffffffff808e7d73 <aesni_decrypt_cbc+467>:     sar    $0x1c,%r13
0xffffffff808e7d77 <aesni_decrypt_cbc+471>:     aesdec %xmm1,%xmm9
0xffffffff808e7d7d <aesni_decrypt_cbc+477>:     movdqa (%rsi,%r13,1),%xmm1
0xffffffff808e7d83 <aesni_decrypt_cbc+483>:     aesdeclast %xmm1,%xmm9
0xffffffff808e7d89 <aesni_decrypt_cbc+489>:     movdqa %xmm9,-0xd0(%rbp)
0xffffffff808e7d92 <aesni_decrypt_cbc+498>:     mov    -0x40(%rbp),%rdx
0xffffffff808e7d96 <aesni_decrypt_cbc+502>:     movdqa (%rdx),%xmm1
0xffffffff808e7d9a <aesni_decrypt_cbc+506>:     aesdeclast %xmm1,%xmm0
0xffffffff808e7d9f <aesni_decrypt_cbc+511>:     movdqa %xmm0,-0xc0(%rbp)
0xffffffff808e7da7 <aesni_decrypt_cbc+519>:     aesdeclast %xmm1,%xmm7
0xffffffff808e7dac <aesni_decrypt_cbc+524>:     movdqa %xmm7,-0xb0(%rbp)
0xffffffff808e7db4 <aesni_decrypt_cbc+532>:     aesdeclast %xmm1,%xmm6
0xffffffff808e7db9 <aesni_decrypt_cbc+537>:     movdqa %xmm6,-0xa0(%rbp)
0xffffffff808e7dc1 <aesni_decrypt_cbc+545>:     aesdeclast %xmm1,%xmm5
0xffffffff808e7dc6 <aesni_decrypt_cbc+550>:     movdqa %xmm5,-0x90(%rbp)
0xffffffff808e7dce <aesni_decrypt_cbc+558>:     aesdeclast %xmm1,%xmm4
0xffffffff808e7dd3 <aesni_decrypt_cbc+563>:     movdqa %xmm4,-0x80(%rbp)
0xffffffff808e7dd8 <aesni_decrypt_cbc+568>:     aesdeclast %xmm1,%xmm3
0xffffffff808e7ddd <aesni_decrypt_cbc+573>:     movdqa %xmm3,-0x70(%rbp)
0xffffffff808e7de2 <aesni_decrypt_cbc+578>:     aesdeclast %xmm1,%xmm2
0xffffffff808e7de7 <aesni_decrypt_cbc+583>:     movdqa %xmm2,-0x60(%rbp)
0xffffffff808e7dec <aesni_decrypt_cbc+588>:     pxor   %xmm8,%xmm9
0xffffffff808e7df1 <aesni_decrypt_cbc+593>:     movups (%rcx),%xmm0
0xffffffff808e7df4 <aesni_decrypt_cbc+596>:     movups 0x10(%rcx),%xmm1
0xffffffff808e7df8 <aesni_decrypt_cbc+600>:     movdqu 0x20(%rcx),%xmm2
0xffffffff808e7dfd <aesni_decrypt_cbc+605>:     movdqu 0x30(%rcx),%xmm3
---Type <return> to continue, or q <return> to quit---
0xffffffff808e7e02 <aesni_decrypt_cbc+610>:     movdqu %xmm9,(%rcx)
0xffffffff808e7e07 <aesni_decrypt_cbc+615>:     xorps  -0xc0(%rbp),%xmm0
0xffffffff808e7e0e <aesni_decrypt_cbc+622>:     movups %xmm0,0x10(%rcx)
0xffffffff808e7e12 <aesni_decrypt_cbc+626>:     xorps  -0xb0(%rbp),%xmm1
0xffffffff808e7e19 <aesni_decrypt_cbc+633>:     movups %xmm1,0x20(%rcx)
0xffffffff808e7e1d <aesni_decrypt_cbc+637>:     pxor   -0xa0(%rbp),%xmm2
0xffffffff808e7e25 <aesni_decrypt_cbc+645>:     movdqu %xmm2,0x30(%rcx)
0xffffffff808e7e2a <aesni_decrypt_cbc+650>:     movups 0x40(%rcx),%xmm0
0xffffffff808e7e2e <aesni_decrypt_cbc+654>:     pxor   -0x90(%rbp),%xmm3
0xffffffff808e7e36 <aesni_decrypt_cbc+662>:     movdqu %xmm3,0x40(%rcx)
0xffffffff808e7e3b <aesni_decrypt_cbc+667>:     movdqu 0x50(%rcx),%xmm1
0xffffffff808e7e40 <aesni_decrypt_cbc+672>:     xorps  -0x80(%rbp),%xmm0
0xffffffff808e7e44 <aesni_decrypt_cbc+676>:     movups %xmm0,0x50(%rcx)
0xffffffff808e7e48 <aesni_decrypt_cbc+680>:     pxor   -0x70(%rbp),%xmm1
0xffffffff808e7e4d <aesni_decrypt_cbc+685>:     movdqu 0x60(%rcx),%xmm0
0xffffffff808e7e52 <aesni_decrypt_cbc+690>:     movdqu %xmm1,0x60(%rcx)
0xffffffff808e7e57 <aesni_decrypt_cbc+695>:     movdqu 0x70(%rcx),%xmm8
0xffffffff808e7e5d <aesni_decrypt_cbc+701>:     pxor   -0x60(%rbp),%xmm0
0xffffffff808e7e62 <aesni_decrypt_cbc+706>:     movdqu %xmm0,0x70(%rcx)
0xffffffff808e7e67 <aesni_decrypt_cbc+711>:     sub    $0xffffffffffffff80,%rcx
0xffffffff808e7e6b <aesni_decrypt_cbc+715>:     add    $0x1,%r15
0xffffffff808e7e6f <aesni_decrypt_cbc+719>:     cmp    %r11,%r15
0xffffffff808e7e72 <aesni_decrypt_cbc+722>:     mov    -0x48(%rbp),%rdi
0xffffffff808e7e76 <aesni_decrypt_cbc+726>:     jne    0xffffffff808e7c20 <aesni_decrypt_cbc+128>
0xffffffff808e7e7c <aesni_decrypt_cbc+732>:     add    -0x30(%rbp),%r9
0xffffffff808e7e80 <aesni_decrypt_cbc+736>:     mov    -0x38(%rbp),%rdx
0xffffffff808e7e84 <aesni_decrypt_cbc+740>:     shl    $0x3,%r11
0xffffffff808e7e88 <aesni_decrypt_cbc+744>:     cmp    %r11,%r8
0xffffffff808e7e8b <aesni_decrypt_cbc+747>:     ja     0xffffffff808e8012 <aesni_decrypt_cbc+1138>
0xffffffff808e7e91 <aesni_decrypt_cbc+753>:     jmpq   0xffffffff808e8160 <aesni_decrypt_cbc+1472>
0xffffffff808e7e96 <aesni_decrypt_cbc+758>:     xor    %r11d,%r11d
0xffffffff808e7e99 <aesni_decrypt_cbc+761>:     shl    $0x3,%r11
0xffffffff808e7e9d <aesni_decrypt_cbc+765>:     cmp    %r11,%r8
0xffffffff808e7ea0 <aesni_decrypt_cbc+768>:     ja     0xffffffff808e8012 <aesni_decrypt_cbc+1138>
0xffffffff808e7ea6 <aesni_decrypt_cbc+774>:     jmpq   0xffffffff808e8160 <aesni_decrypt_cbc+1472>
0xffffffff808e7eab <aesni_decrypt_cbc+779>:     mov    %r11,%rax
0xffffffff808e7eae <aesni_decrypt_cbc+782>:     mov    %r9,%rcx
0xffffffff808e7eb1 <aesni_decrypt_cbc+785>:     nopw   %cs:0x0(%rax,%rax,1)
0xffffffff808e7ec0 <aesni_decrypt_cbc+800>:     movdqu (%rcx),%xmm0
0xffffffff808e7ec4 <aesni_decrypt_cbc+804>:     movdqu 0x10(%rcx),%xmm7
0xffffffff808e7ec9 <aesni_decrypt_cbc+809>:     movdqu 0x20(%rcx),%xmm1
---Type <return> to continue, or q <return> to quit---
0xffffffff808e7ece <aesni_decrypt_cbc+814>:     movdqu 0x30(%rcx),%xmm6
0xffffffff808e7ed3 <aesni_decrypt_cbc+819>:     movdqu 0x40(%rcx),%xmm5
0xffffffff808e7ed8 <aesni_decrypt_cbc+824>:     movdqu 0x50(%rcx),%xmm4
0xffffffff808e7edd <aesni_decrypt_cbc+829>:     movdqu 0x60(%rcx),%xmm10
0xffffffff808e7ee3 <aesni_decrypt_cbc+835>:     movdqu 0x70(%rcx),%xmm9
0xffffffff808e7ee9 <aesni_decrypt_cbc+841>:     movdqa (%rsi),%xmm2
0xffffffff808e7eed <aesni_decrypt_cbc+845>:     movdqa 0x10(%rsi),%xmm3
0xffffffff808e7ef2 <aesni_decrypt_cbc+850>:     pxor   %xmm2,%xmm0
0xffffffff808e7ef6 <aesni_decrypt_cbc+854>:     pxor   %xmm2,%xmm7
0xffffffff808e7efa <aesni_decrypt_cbc+858>:     pxor   %xmm2,%xmm1
0xffffffff808e7efe <aesni_decrypt_cbc+862>:     pxor   %xmm2,%xmm6
0xffffffff808e7f02 <aesni_decrypt_cbc+866>:     pxor   %xmm2,%xmm5
0xffffffff808e7f06 <aesni_decrypt_cbc+870>:     pxor   %xmm2,%xmm4
0xffffffff808e7f0a <aesni_decrypt_cbc+874>:     pxor   %xmm2,%xmm10
0xffffffff808e7f0f <aesni_decrypt_cbc+879>:     pxor   %xmm2,%xmm9
0xffffffff808e7f14 <aesni_decrypt_cbc+884>:     aesdeclast %xmm3,%xmm0
0xffffffff808e7f19 <aesni_decrypt_cbc+889>:     movdqa %xmm0,-0xd0(%rbp)
0xffffffff808e7f21 <aesni_decrypt_cbc+897>:     aesdeclast %xmm3,%xmm7
0xffffffff808e7f26 <aesni_decrypt_cbc+902>:     movdqa %xmm7,-0xc0(%rbp)
0xffffffff808e7f2e <aesni_decrypt_cbc+910>:     aesdeclast %xmm3,%xmm1
0xffffffff808e7f33 <aesni_decrypt_cbc+915>:     movdqa %xmm1,-0xb0(%rbp)
0xffffffff808e7f3b <aesni_decrypt_cbc+923>:     aesdeclast %xmm3,%xmm6
0xffffffff808e7f40 <aesni_decrypt_cbc+928>:     movdqa %xmm6,-0xa0(%rbp)
0xffffffff808e7f48 <aesni_decrypt_cbc+936>:     aesdeclast %xmm3,%xmm5
0xffffffff808e7f4d <aesni_decrypt_cbc+941>:     movdqa %xmm5,-0x90(%rbp)
0xffffffff808e7f55 <aesni_decrypt_cbc+949>:     aesdeclast %xmm3,%xmm4
0xffffffff808e7f5a <aesni_decrypt_cbc+954>:     movdqa %xmm4,-0x80(%rbp)
0xffffffff808e7f5f <aesni_decrypt_cbc+959>:     aesdeclast %xmm3,%xmm10
0xffffffff808e7f65 <aesni_decrypt_cbc+965>:     movdqa %xmm10,-0x70(%rbp)
0xffffffff808e7f6b <aesni_decrypt_cbc+971>:     aesdeclast %xmm3,%xmm9
0xffffffff808e7f71 <aesni_decrypt_cbc+977>:     movdqa %xmm9,-0x60(%rbp)
0xffffffff808e7f77 <aesni_decrypt_cbc+983>:     pxor   %xmm8,%xmm0
0xffffffff808e7f7c <aesni_decrypt_cbc+988>:     movups (%rcx),%xmm1
0xffffffff808e7f7f <aesni_decrypt_cbc+991>:     movdqu 0x10(%rcx),%xmm2
0xffffffff808e7f84 <aesni_decrypt_cbc+996>:     movdqu 0x20(%rcx),%xmm3
0xffffffff808e7f89 <aesni_decrypt_cbc+1001>:    movdqu 0x30(%rcx),%xmm4
0xffffffff808e7f8e <aesni_decrypt_cbc+1006>:    movdqu %xmm0,(%rcx)
0xffffffff808e7f92 <aesni_decrypt_cbc+1010>:    xorps  -0xc0(%rbp),%xmm1
0xffffffff808e7f99 <aesni_decrypt_cbc+1017>:    movups %xmm1,0x10(%rcx)
0xffffffff808e7f9d <aesni_decrypt_cbc+1021>:    pxor   -0xb0(%rbp),%xmm2
0xffffffff808e7fa5 <aesni_decrypt_cbc+1029>:    movdqu %xmm2,0x20(%rcx)
0xffffffff808e7faa <aesni_decrypt_cbc+1034>:    pxor   -0xa0(%rbp),%xmm3
0xffffffff808e7fb2 <aesni_decrypt_cbc+1042>:    movdqu %xmm3,0x30(%rcx)
0xffffffff808e7fb7 <aesni_decrypt_cbc+1047>:    movups 0x40(%rcx),%xmm0
0xffffffff808e7fbb <aesni_decrypt_cbc+1051>:    pxor   -0x90(%rbp),%xmm4
0xffffffff808e7fc3 <aesni_decrypt_cbc+1059>:    movdqu %xmm4,0x40(%rcx)
---Type <return> to continue, or q <return> to quit---
0xffffffff808e7fc8 <aesni_decrypt_cbc+1064>:    xorps  -0x80(%rbp),%xmm0
0xffffffff808e7fcc <aesni_decrypt_cbc+1068>:    movdqu 0x50(%rcx),%xmm1
0xffffffff808e7fd1 <aesni_decrypt_cbc+1073>:    movups %xmm0,0x50(%rcx)
0xffffffff808e7fd5 <aesni_decrypt_cbc+1077>:    movdqu 0x60(%rcx),%xmm0
0xffffffff808e7fda <aesni_decrypt_cbc+1082>:    pxor   -0x70(%rbp),%xmm1
0xffffffff808e7fdf <aesni_decrypt_cbc+1087>:    movdqu %xmm1,0x60(%rcx)
0xffffffff808e7fe4 <aesni_decrypt_cbc+1092>:    movdqu 0x70(%rcx),%xmm8
0xffffffff808e7fea <aesni_decrypt_cbc+1098>:    pxor   -0x60(%rbp),%xmm0
0xffffffff808e7fef <aesni_decrypt_cbc+1103>:    movdqu %xmm0,0x70(%rcx)
0xffffffff808e7ff4 <aesni_decrypt_cbc+1108>:    sub    $0xffffffffffffff80,%rcx
0xffffffff808e7ff8 <aesni_decrypt_cbc+1112>:    add    $0xffffffffffffffff,%rax
0xffffffff808e7ffc <aesni_decrypt_cbc+1116>:    jne    0xffffffff808e7ec0 <aesni_decrypt_cbc+800>
0xffffffff808e8002 <aesni_decrypt_cbc+1122>:    add    %rbx,%r9
0xffffffff808e8005 <aesni_decrypt_cbc+1125>:    shl    $0x3,%r11
0xffffffff808e8009 <aesni_decrypt_cbc+1129>:    cmp    %r11,%r8
0xffffffff808e800c <aesni_decrypt_cbc+1132>:    jbe    0xffffffff808e8160 <aesni_decrypt_cbc+1472>
0xffffffff808e8012 <aesni_decrypt_cbc+1138>:    lea    -0x1(%rdi),%eax
0xffffffff808e8015 <aesni_decrypt_cbc+1141>:    movslq %eax,%rbx
0xffffffff808e8018 <aesni_decrypt_cbc+1144>:    mov    %rbx,%r10
0xffffffff808e801b <aesni_decrypt_cbc+1147>:    shl    $0x4,%r10
0xffffffff808e801f <aesni_decrypt_cbc+1151>:    add    %rsi,%r10
0xffffffff808e8022 <aesni_decrypt_cbc+1154>:    movslq %edi,%rcx
0xffffffff808e8025 <aesni_decrypt_cbc+1157>:    shl    $0x4,%rcx
0xffffffff808e8029 <aesni_decrypt_cbc+1161>:    add    %rsi,%rcx
0xffffffff808e802c <aesni_decrypt_cbc+1164>:    cmp    $0x1,%ebx
0xffffffff808e802f <aesni_decrypt_cbc+1167>:    jle    0xffffffff808e80b3 <aesni_decrypt_cbc+1299>
0xffffffff808e8035 <aesni_decrypt_cbc+1173>:    nopw   %cs:0x0(%rax,%rax,1)
0xffffffff808e8040 <aesni_decrypt_cbc+1184>:    movdqu (%r9),%xmm0
0xffffffff808e8045 <aesni_decrypt_cbc+1189>:    movdqa (%rsi),%xmm1
0xffffffff808e8049 <aesni_decrypt_cbc+1193>:    pxor   %xmm0,%xmm1
0xffffffff808e804d <aesni_decrypt_cbc+1197>:    mov    $0x1,%ebx
0xffffffff808e8052 <aesni_decrypt_cbc+1202>:    nopw   %cs:0x0(%rax,%rax,1)
0xffffffff808e8060 <aesni_decrypt_cbc+1216>:    movslq %ebx,%rbx
0xffffffff808e8063 <aesni_decrypt_cbc+1219>:    mov    %rbx,%rdi
0xffffffff808e8066 <aesni_decrypt_cbc+1222>:    shl    $0x4,%rdi
0xffffffff808e806a <aesni_decrypt_cbc+1226>:    aesdec (%rsi,%rdi,1),%xmm1
0xffffffff808e8070 <aesni_decrypt_cbc+1232>:    lea    0x1(%rbx),%edi
0xffffffff808e8073 <aesni_decrypt_cbc+1235>:    movslq %edi,%rdi
0xffffffff808e8076 <aesni_decrypt_cbc+1238>:    shl    $0x4,%rdi
0xffffffff808e807a <aesni_decrypt_cbc+1242>:    aesdec (%rsi,%rdi,1),%xmm1
0xffffffff808e8080 <aesni_decrypt_cbc+1248>:    add    $0x2,%ebx
0xffffffff808e8083 <aesni_decrypt_cbc+1251>:    cmp    %eax,%ebx
0xffffffff808e8085 <aesni_decrypt_cbc+1253>:    jl     0xffffffff808e8060 <aesni_decrypt_cbc+1216---Type <return> to continue, or q <return> to quit---
>
0xffffffff808e8087 <aesni_decrypt_cbc+1255>:    aesdec (%r10),%xmm1
0xffffffff808e808d <aesni_decrypt_cbc+1261>:    aesdeclast (%rcx),%xmm1
0xffffffff808e8092 <aesni_decrypt_cbc+1266>:    pxor   %xmm8,%xmm1
0xffffffff808e8097 <aesni_decrypt_cbc+1271>:    movdqu %xmm1,(%r9)
0xffffffff808e809c <aesni_decrypt_cbc+1276>:    add    $0x10,%r9
0xffffffff808e80a0 <aesni_decrypt_cbc+1280>:    add    $0x1,%r11
0xffffffff808e80a4 <aesni_decrypt_cbc+1284>:    movdqa %xmm0,%xmm8
0xffffffff808e80a9 <aesni_decrypt_cbc+1289>:    cmp    %r8,%r11
0xffffffff808e80ac <aesni_decrypt_cbc+1292>:    jne    0xffffffff808e8040 <aesni_decrypt_cbc+1184>
0xffffffff808e80ae <aesni_decrypt_cbc+1294>:    jmpq   0xffffffff808e8160 <aesni_decrypt_cbc+1472>
0xffffffff808e80b3 <aesni_decrypt_cbc+1299>:    lea    -0x1(%r8),%rax
0xffffffff808e80b7 <aesni_decrypt_cbc+1303>:    test   $0x10,%dl
0xffffffff808e80ba <aesni_decrypt_cbc+1306>:    jne    0xffffffff808e80c9 <aesni_decrypt_cbc+1321>
0xffffffff808e80bc <aesni_decrypt_cbc+1308>:    mov    %r11,%rdi
0xffffffff808e80bf <aesni_decrypt_cbc+1311>:    cmp    %r11,%rax
0xffffffff808e80c2 <aesni_decrypt_cbc+1314>:    jne    0xffffffff808e8100 <aesni_decrypt_cbc+1376>
0xffffffff808e80c4 <aesni_decrypt_cbc+1316>:    jmpq   0xffffffff808e8160 <aesni_decrypt_cbc+1472>
0xffffffff808e80c9 <aesni_decrypt_cbc+1321>:    movdqu (%r9),%xmm0
0xffffffff808e80ce <aesni_decrypt_cbc+1326>:    movdqa (%rsi),%xmm1
0xffffffff808e80d2 <aesni_decrypt_cbc+1330>:    pxor   %xmm0,%xmm1
0xffffffff808e80d6 <aesni_decrypt_cbc+1334>:    aesdec (%r10),%xmm1
0xffffffff808e80dc <aesni_decrypt_cbc+1340>:    aesdeclast (%rcx),%xmm1
0xffffffff808e80e1 <aesni_decrypt_cbc+1345>:    pxor   %xmm8,%xmm1
0xffffffff808e80e6 <aesni_decrypt_cbc+1350>:    movdqu %xmm1,(%r9)
0xffffffff808e80eb <aesni_decrypt_cbc+1355>:    add    $0x10,%r9
0xffffffff808e80ef <aesni_decrypt_cbc+1359>:    mov    %r11,%rdi
0xffffffff808e80f2 <aesni_decrypt_cbc+1362>:    or     $0x1,%rdi
0xffffffff808e80f6 <aesni_decrypt_cbc+1366>:    movdqa %xmm0,%xmm8
0xffffffff808e80fb <aesni_decrypt_cbc+1371>:    cmp    %r11,%rax
0xffffffff808e80fe <aesni_decrypt_cbc+1374>:    je     0xffffffff808e8160 <aesni_decrypt_cbc+1472>
0xffffffff808e8100 <aesni_decrypt_cbc+1376>:    sub    %rdi,%r8
0xffffffff808e8103 <aesni_decrypt_cbc+1379>:    nopw   %cs:0x0(%rax,%rax,1)
0xffffffff808e8110 <aesni_decrypt_cbc+1392>:    movdqu (%r9),%xmm0
0xffffffff808e8115 <aesni_decrypt_cbc+1397>:    movdqa (%rsi),%xmm1
0xffffffff808e8119 <aesni_decrypt_cbc+1401>:    pxor   %xmm0,%xmm1
0xffffffff808e811d <aesni_decrypt_cbc+1405>:    aesdec (%r10),%xmm1
0xffffffff808e8123 <aesni_decrypt_cbc+1411>:    aesdeclast (%rcx),%xmm1
0xffffffff808e8128 <aesni_decrypt_cbc+1416>:    pxor   %xmm8,%xmm1
0xffffffff808e812d <aesni_decrypt_cbc+1421>:    movdqu 0x10(%r9),%xmm8
---Type <return> to continue, or q <return> to quit---
0xffffffff808e8133 <aesni_decrypt_cbc+1427>:    movdqu %xmm1,(%r9)
0xffffffff808e8138 <aesni_decrypt_cbc+1432>:    movdqa (%rsi),%xmm1
0xffffffff808e813c <aesni_decrypt_cbc+1436>:    pxor   %xmm8,%xmm1
0xffffffff808e8141 <aesni_decrypt_cbc+1441>:    aesdec (%r10),%xmm1
0xffffffff808e8147 <aesni_decrypt_cbc+1447>:    aesdeclast (%rcx),%xmm1
0xffffffff808e814c <aesni_decrypt_cbc+1452>:    pxor   %xmm0,%xmm1
0xffffffff808e8150 <aesni_decrypt_cbc+1456>:    movdqu %xmm1,0x10(%r9)
0xffffffff808e8156 <aesni_decrypt_cbc+1462>:    add    $0x20,%r9
0xffffffff808e815a <aesni_decrypt_cbc+1466>:    add    $0xfffffffffffffffe,%r8
0xffffffff808e815e <aesni_decrypt_cbc+1470>:    jne    0xffffffff808e8110 <aesni_decrypt_cbc+1392>
0xffffffff808e8160 <aesni_decrypt_cbc+1472>:    add    $0xa8,%rsp
0xffffffff808e8167 <aesni_decrypt_cbc+1479>:    pop    %rbx
0xffffffff808e8168 <aesni_decrypt_cbc+1480>:    pop    %r12
0xffffffff808e816a <aesni_decrypt_cbc+1482>:    pop    %r13
0xffffffff808e816c <aesni_decrypt_cbc+1484>:    pop    %r14
0xffffffff808e816e <aesni_decrypt_cbc+1486>:    pop    %r15
0xffffffff808e8170 <aesni_decrypt_cbc+1488>:    pop    %rbp
0xffffffff808e8171 <aesni_decrypt_cbc+1489>:    retq
End of assembler dump.
(kgdb) backtrace
#0  doadump (textdump=<value optimized out>) at pcpu.h:230
#1  0xffffffff805aef86 in kern_reboot (howto=260)
    at /smallblocks/src/sys/kern/kern_shutdown.c:383
#2  0xffffffff805af2fd in vpanic (fmt=<value optimized out>, ap=<value optimized out>)
    at /smallblocks/src/sys/kern/kern_shutdown.c:776
#3  0xffffffff805af193 in panic (fmt=<value optimized out>)
    at /smallblocks/src/sys/kern/kern_shutdown.c:707
#4  0xffffffff808e097f in trap_fatal (frame=0xfffffe06892d8fa0, eva=0)
    at /smallblocks/src/sys/amd64/amd64/trap.c:877
#5  0xffffffff808dffa5 in trap (frame=0xfffffe06892d8fa0) at pcpu.h:230
#6  0xffffffff808c2bae in calltrap () at /smallblocks/src/sys/amd64/amd64/exception.S:231
#7  0xffffffff808e7d89 in aesni_decrypt_cbc (rounds=<value optimized out>,
    key_schedule=0xfffff800122508f0, len=160,
    buf=0xfffff80050ad6200 "?\025\rz▒▒&HF▒_▒>ov\bV\2068\005k\r{d▒C▒\220c▒j▒Ը▒▒~▒▒6▒▒▒\227K\217@▒▒▒ai▒▒ll\v\216#f▒▒\f▒)▒\211|▒▒▒9ep▒\031▒)V▒▒▒momo/\036,▒m▒\003\177-{▒.'\005RJnK▒▒r\"\226s▒\231▒\203▒ v\020P▒\202\r\207\037L▒3▒\004{\002g_o▒%\201ͨ▒\235e▒\235\206?▒@▒▒$D\"Y**\f\t\214eject=5.*", iv=<value optimized out>) at aesencdec.h:102
#8  0xffffffff808e41dc in aesni_process (dev=<value optimized out>, crp=0xfffff8015ea94bb0,
    hint=<value optimized out>) at /smallblocks/src/sys/crypto/aesni/aesni.c:609
#9  0xffffffff8080cab3 in crypto_dispatch (crp=0xfffff8015ea94bb0)
    at /smallblocks/src/sys/opencrypto/crypto.c:844
#10 0xffffffff8080f6a7 in cryptof_ioctl (fp=<value optimized out>, cmd=3223085927,
    data=0xfffffe06892d9410, active_cred=<value optimized out>, td=<value optimized out>)
    at /smallblocks/src/sys/opencrypto/cryptodev.c:849
#11 0xffffffff806046d2 in kern_ioctl (td=<value optimized out>, fd=6, com=3223085927,
    data=<value optimized out>) at file.h:323
#12 0xffffffff8060445d in sys_ioctl (td=0xfffff8015eb18000, uap=0xfffffe06892d94e8)
    at /smallblocks/src/sys/kern/sys_generic.c:745
#13 0xffffffff802dbe1c in freebsd32_ioctl (td=0xfffff8015eb18000, uap=0xfffff8015eb18538)
    at /smallblocks/src/sys/compat/freebsd32/freebsd32_ioctl.c:470
#14 0xffffffff809092fc in ia32_syscall (frame=0xfffffe06892d9ab8) at subr_syscall.c:132
#15 0xffffffff808c3bf6 in int0x80_syscall_common () at ia32_exception.S:76
#16 0x0000000000000000 in ?? ()
(kgdb) p/x *(struct trapframe *)0xfffffe06892d8fa0
$1 = {tf_rdi = 0xb00000000, tf_rsi = 0xfffff800122508f0, tf_rdx = 0x100000000,
  tf_rcx = 0xfffff80050ad6200, tf_r8 = 0xa, tf_r9 = 0xfffff80050ad6200, tf_rax = 0x200000000,
  tf_rbx = 0x9, tf_rbp = 0xfffffe06892d9138, tf_r10 = 0xa00000000, tf_r11 = 0x1, tf_r12 = 0x80,
  tf_r13 = 0xa0, tf_r14 = 0x1, tf_r15 = 0x0, tf_trapno = 0x9, tf_fs = 0x13, tf_gs = 0x1b,
  tf_addr = 0x0, tf_flags = 0x1, tf_es = 0x3b, tf_ds = 0x3b, tf_err = 0x0,
  tf_rip = 0xffffffff808e7d89, tf_cs = 0x20, tf_rflags = 0x10206, tf_rsp = 0xfffffe06892d9068,
  tf_ss = 0x0}
(kgdb) quit

Comment 14 Konstantin Belousov 2018-06-23 16:27:19 UTC

(In reply to dewayne from comment #13)
Ok.

With this kernel and core, please do the manual unwind:
starting from the tf_rbp = 0xfffffe06892d9138 value,
dereference *tf_rbp, i.e. the long value at the address contained
in the rbp register.  It points to the next frame address, and tf_rbp+8
is the address of the caller, it should match the gdb backtrace.
Do that with the next frame as well, up to the end of the trace.
The end result should be the frame address and source line number for
each frame in the backtrace.

What I am looking for is the oldest frame where the frame is only 8-bytes
aligned.

Comment 15 Conrad Meyer 2018-06-23 18:13:56 UTC

(In reply to Konstantin Belousov from comment #8)
_mm_loadu_si128 should not require aligned stack, no?  (The "u" stands for "unaligned".)

Comment 16 Conrad Meyer 2018-06-23 18:16:34 UTC

(In reply to Conrad Meyer from comment #15)
Ignore me, I see the several movdqa instructions in the disassembly.

(In reply to Konstantin Belousov from comment #14)
Ok.
Instruction pointer 0xffffffff808e7d89 which points to frame 7, and from bt's frame pointer we have 0xfffffe06892d9138

(kgdb) x *tf_rbp
No symbol "tf_rbp" in current context.
(kgdb) x 0xfffffe06892d9138		# frame pointer
0xfffffe06892d9138:     0x892d91d8	# Next frame reference
(kgdb) x 0xfffffe06892d9138+8		# Address of caller
0xfffffe06892d9140:     0x808e41dc	# Instruction pointer, ie previous caller (frame 8)

I'm guessing the next steps
(kgdb) x 0xfffff892d91d8+8
0xfffff892d91e0:        Cannot access memory at address 0xfffff892d91e0
(kgdb) x 0xfffff892d91d8
0xfffff892d91d8:        Cannot access memory at address 0xfffff892d91d8

Continuing to guess 
(kgdb) x 0xfffff808e41dc
0xfffff808e41dc:        Cannot access memory at address 0xfffff808e41dc
(kgdb) x 0xfffff808e41dc+8
0xfffff808e41e4:        Cannot access memory at address 0xfffff808e41e4

Sorry I need help.  (I work 70% in PROLOG, 10% in Erlang).

I've made the dumps available at http://www.heuristicsystems.com.au/dumps/ 
The files kernel.debug.xz-R vmcore.11.xz-R relate to the kernel created via make __MAKE_CONF=/dev/null, which is what we're currently examining.

Comment 18 Conrad Meyer 2018-06-24 03:12:57 UTC

(In reply to dewayne from comment #17)
The URL gives a 404?

(In reply to Conrad Meyer from comment #18)
http://www.heuristicsystems.com.au/dump not dumps.  I'd originally placed one dump there and... oops.

Comment 20 Konstantin Belousov 2018-06-24 09:48:14 UTC

(In reply to dewayne from comment #19)
Now the URL gives 'unauthorized'.

tf_rbp referred to the value from the struct trapframe printout.
You should do
(kgdb) p/x *(long *)0xfffffe06892d9138
<value> <- this is the next rbp to deref
(kgdb) p/x *(long *)(0xfffffe06892d9138 + 8)
<value> <- this is return pc
(kgdb) list *<value from the previous line>

now repeat with the 'next rbp' and so on until finished the unwind.
Sanity check is that the lines printed by the 'list' command should
match the lines from the 'bt' output.

(In reply to Konstantin Belousov from comment #20)
Kib, Apologies an unusual level of activity from multiple locations (& countries) occurred within a 90 second interval.  Please email me and I'll send the access detail.  (I supposed I shouldn't kernel dump on our main dev server)...


Response to Kib's instructions
(kgdb) p/x *(long *)0xfffffe06892d9138
$1 = 0xfffffe06892d91d8
Current language:  auto; currently minimal
(kgdb) p/x *(long *)(0xfffffe06892d9138 + 8)
$2 = 0xffffffff808e41dc
(kgdb) list *0xffffffff808e41dc
0xffffffff808e41dc is in aesni_process (/smallblocks/src/sys/crypto/aesni/aesni.c:609).
604             case CRYPTO_AES_CBC:
605                     if (encflag)
606                             aesni_encrypt_cbc(ses->rounds, ses->enc_schedule,
607                                 enccrd->crd_len, buf, buf, iv);
608                     else
609                             aesni_decrypt_cbc(ses->rounds, ses->dec_schedule,
610                                 enccrd->crd_len, buf, iv);
611                     break;
612             case CRYPTO_AES_ICM:
613                     /* encryption & decryption are the same */
(kgdb) p/x *(long *)0xfffffe06892d91d8
$3 = 0xfffffe06892d9208
(kgdb) p/x *(long *)(0xfffffe06892d91d8 + 8)
$4 = 0xffffffff8080cab3
(kgdb) list *0xffffffff8080cab3
0xffffffff8080cab3 is in crypto_dispatch (/smallblocks/src/sys/opencrypto/crypto.c:844).
839                      */
840                     cap = crypto_checkdriver(hid);
841                     /* Driver cannot disappeared when there is an active session. */
842                     KASSERT(cap != NULL, ("%s: Driver disappeared.", __func__));
843                     if (!cap->cc_qblocked) {
844                             result = crypto_invoke(cap, crp, 0);
845                             if (result != ERESTART)
846                                     return (result);
847                             /*
848                              * The driver ran out of resources, put the request on
(kgdb) p/x *(long *)0xfffffe06892d9208
$5 = 0xfffffe06892d9388
(kgdb) p/x *(long *)(0xfffffe06892d9208 + 8)
$6 = 0xffffffff8080f6a7
(kgdb) list *0xffffffff8080f6a7
0xffffffff8080f6a7 is in cryptof_ioctl (/smallblocks/src/sys/opencrypto/cryptodev.c:849).
844              * callback before checking if the operation completed and going
845              * to sleep.  This insures drivers don't inherit our lock which
846              * results in a lock order reversal between crypto_dispatch forced
847              * entry and the crypto_done callback into us.
848              */
849             error = crypto_dispatch(crp);
850             mtx_lock(&cse->lock);
851             if (error == 0 && (crp->crp_flags & CRYPTO_F_DONE) == 0)
852                     error = msleep(crp, &cse->lock, PWAIT, "crydev", 0);
853             mtx_unlock(&cse->lock);
(kgdb) p/x *(long *)0xfffffe06892d9388
$7 = 0xfffffe06892d93f8
(kgdb) p/x *(long *)(0xfffffe06892d9388 + 8)
$8 = 0xffffffff806046d2
(kgdb) list *0xffffffff806046d2
0xffffffff806046d2 is in kern_ioctl (file.h:323).
318     static __inline int
319     fo_ioctl(struct file *fp, u_long com, void *data, struct ucred *active_cred,
320         struct thread *td)
321     {
322
323             return ((*fp->f_ops->fo_ioctl)(fp, com, data, active_cred, td));
324     }
325
326     static __inline int
327     fo_poll(struct file *fp, int events, struct ucred *active_cred,
(kgdb) p/x *(long *)0xfffffe06892d93f8
$9 = 0xfffffe06892d94c8
(kgdb) p/x *(long *)(0xfffffe06892d93f8 + 8)
$10 = 0xffffffff8060445d
(kgdb) list *0xffffffff8060445d
0xffffffff8060445d is in sys_ioctl (/smallblocks/src/sys/kern/sys_generic.c:745).
740                      * gets back something deterministic.
741                      */
742                     bzero(data, size);
743             }
744
745             error = kern_ioctl(td, uap->fd, com, data);
746
747             if (error == 0 && (com & IOC_OUT))
748                     error = copyout(data, uap->data, (u_int)size);
749
(kgdb) p/x *(long *)0xfffffe06892d94c8
$11 = 0xfffffe06892d9968
(kgdb) p/x *(long *)(0xfffffe06892d94c8 + 8)
$12 = 0xffffffff802dbe1c
(kgdb) list *0xffffffff802dbe1c
0xffffffff802dbe1c is in freebsd32_ioctl (/smallblocks/src/sys/compat/freebsd32/freebsd32_ioctl.c:470).
465             default:
466                     fdrop(fp, td);
467                     ap.fd = uap->fd;
468                     ap.com = uap->com;
469                     PTRIN_CP(*uap, ap, data);
470                     return sys_ioctl(td, &ap);
471             }
472
473             fdrop(fp, td);
474             return error;
(kgdb) p/x *(long *)0xfffffe06892d9968
$13 = 0xfffffe06892d9aa8
(kgdb) p/x *(long *)(0xfffffe06892d9968 + 8)
$14 = 0xffffffff809092fc
(kgdb) list *0xffffffff809092fc
0xffffffff809092fc is in ia32_syscall (subr_syscall.c:132).
127                     if (systrace_probe_func != NULL && sa->callp->sy_entry != 0)
128                             (*systrace_probe_func)(sa, SYSTRACE_ENTRY, 0);
129     #endif
130
131                     AUDIT_SYSCALL_ENTER(sa->code, td);
132                     error = (sa->callp->sy_call)(td, sa->args);
133                     AUDIT_SYSCALL_EXIT(error, td);
134
135                     /* Save the latest error return value. */
136                     if ((td->td_pflags & TDP_NERRNO) == 0)
(kgdb) p/x *(long *)0xfffffe06892d9aa8
$15 = 0xffff9d58
(kgdb) p/x *(long *)(0xfffffe06892d9aa8 + 8)
$16 = 0xffffffff808c3bf6
(kgdb) list *0xffffffff808c3bf6
0xffffffff808c3bf6 is at ia32_exception.S:78.
73              cld
74              FAKE_MCOUNT(TF_RIP(%rsp))
75              movq    %rsp, %rdi
76              call    ia32_syscall
77              MEXITCOUNT
78              jmp     doreti
(kgdb) p/x *(long *)0xffff9d58
Cannot access memory at address 0xffff9d58

Comment 22 Konstantin Belousov 2018-06-24 12:38:19 UTC

Created attachment 194558 [details]
Do not leave stray qword on top of stack for interrupts and exceptions without error code. It mis-aligns the stack.

(In reply to Konstantin Belousov from comment #22)
Quick update - the system remained up overnight, and has been under normal load for the past two hours.
11:12AM  up 11:44, 6 users, load averages: 9.15, 11.08, 9.87
I'll report back in 24hours.
Thank-you for your help & patience.

Comment 24 Konstantin Belousov 2018-06-25 11:28:39 UTC

(In reply to dewayne from comment #23)
It is not a race, the situation is deterministic.  So I committed the fix.

Comment 25 commit-hook 2018-06-25 11:30:01 UTC

A commit references this bug:

Author: kib
Date: Mon Jun 25 11:29:04 UTC 2018
New revision: 335635
URL: https://svnweb.freebsd.org/changeset/base/335635

Log:
  Do not leave stray qword on top of stack for interrupts and exceptions
  without error code.  Doing so it mis-aligned the stack.

  Since the only consumer of the SSE instructions with the alignment
  requirements is AES-NI module, and since the FPU context cannot be
  accessed in interrupts, the only situation where the alignment matter
  are the compat32 syscalls, as reported in the PR.

  PR:	229222
  Reported and tested by:	 dewayne@heuristicsystems.com.au
  Sponsored by:	The FreeBSD Foundation
  MFC after:	1 week

Changes:
  head/sys/amd64/include/asmacros.h

Comment 26 Kubilay Kocak 2018-06-25 11:38:16 UTC

Comment on attachment 194558 [details]
Do not leave stray qword on top of stack for interrupts and exceptions without error code. It mis-aligns the stack.

@kib Assuming we want this in 11.2-RELEASE?

(In reply to Kubilay Kocak from comment #26)
I'm unsure of the convention as I usually close my (ports related) PR's when a fix has been provided.  As you, Koobs, have changed the status, I'll not change the status here.

The applied fix means that our amd64 platform can continue building for our legacy i386 systems and perform some rudimentary testing - in this case a sendmail log triggers an ssh connection to update a firewall table.  To avert a known condition where an event triggers a spontaineous crash should be something that, I hope, would be included in any upcoming release.  :)  

Now to catch up with the backlog (uptime & temp):
 2:32PM  up 1 day, 21 mins, 5 users, load averages: 20.47, 22.31, 22.31 46.0C
Yay!

Comment 28 commit-hook 2018-07-02 07:59:20 UTC

A commit references this bug:

Author: kib
Date: Mon Jul  2 07:58:58 UTC 2018
New revision: 335857
URL: https://svnweb.freebsd.org/changeset/base/335857

Log:
  MFC r335635:
  Do not leave stray qword on top of stack for interrupts and exceptions
  without error code.  Doing so it mis-aligned the stack.

  PR:	229222

Changes:
_U  stable/11/
  stable/11/sys/amd64/include/asmacros.h

I've closed this PR as Konstantin has provided a fix for the crashes, which has been incorporated into Stable.  Though I should note that I haven't yet pinned down why an i386 jail on an amd64 platform seems to hang when run at a nice priority. (no other clues so far) This is on FreeBSD 11.2-STABLE  r336359M  i386 1102501 1102501