FreeBSD panics almost immidiately after starting ctld. FreeBSD holds 24 disks for VM resources, which are organized on zfs pools; zvols are created which are distributed via iSCSI. FreeBSD was up for several days, and then (all of a sudden, after some good production use) this panic appeared. Panic screen is attached. Panic is 100% reproducible after rebooting. I can also provide the direct server access if needed. Disks: ===Cut=== [root@san1:~]# camcontrol devlist <LSI SAS3x40 0601> at scbus1 target 4 lun 0 (pass0,ses0) <ATA INTEL SSDSC2BW24 RG21> at scbus1 target 5 lun 0 (pass1,da0) <ATA ST2000NX0253 SN02> at scbus1 target 6 lun 0 (pass2,da1) <ATA ST2000NX0253 SN02> at scbus1 target 7 lun 0 (pass3,da2) <ATA ST2000NX0253 SN02> at scbus1 target 8 lun 0 (pass4,da3) <ATA ST2000NX0253 SN02> at scbus1 target 9 lun 0 (pass5,da4) <ATA ST2000NX0253 SN02> at scbus1 target 10 lun 0 (pass6,da5) <ATA ST2000NX0253 SN02> at scbus1 target 11 lun 0 (pass7,da6) <ATA ST2000NX0253 SN02> at scbus1 target 12 lun 0 (pass8,da7) <ATA ST2000NX0253 SN02> at scbus1 target 13 lun 0 (pass9,da8) <ATA ST2000NX0253 SN02> at scbus1 target 14 lun 0 (pass10,da9) <ATA ST2000NX0253 SN02> at scbus1 target 15 lun 0 (pass11,da10) <ATA ST2000NX0253 SN02> at scbus1 target 16 lun 0 (pass12,da11) <ATA ST2000NX0253 SN02> at scbus1 target 17 lun 0 (pass13,da12) <ATA ST2000NX0253 SN02> at scbus1 target 18 lun 0 (pass14,da13) <ATA ST2000NX0253 SN02> at scbus1 target 19 lun 0 (pass15,da14) <ATA ST2000NX0253 SN02> at scbus1 target 20 lun 0 (pass16,da15) <ATA ST2000NX0253 SN02> at scbus1 target 21 lun 0 (pass17,da16) <ATA INTEL SSDSC2BW24 RG21> at scbus1 target 22 lun 0 (pass18,da17) <ATA INTEL SSDSC2BW24 RG21> at scbus1 target 23 lun 0 (pass19,da18) <ATA INTEL SSDSC2BW24 RG21> at scbus1 target 24 lun 0 (pass20,da19) <ATA INTEL SSDSC2BW24 RG21> at scbus1 target 25 lun 0 (pass21,da20) <ATA INTEL SSDSC2BW24 RG21> at scbus1 target 26 lun 0 (pass22,da21) <ATA INTEL SSDSC2BW24 RG21> at scbus1 target 27 lun 0 (pass23,da22) <ATA INTEL SSDSC2BW24 RG21> at scbus1 target 28 lun 0 (pass24,da23) <AHCI SGPIO Enclosure 1.00 0001> at scbus6 target 0 lun 0 (pass25,ses1) <AHCI SGPIO Enclosure 1.00 0001> at scbus13 target 0 lun 0 (pass26,ses2) ===Cut=== Controller: ===Cut=== # MegaCli -CfgDsply -aAll ============================================================================== Adapter: 0 Product Name: LSI MegaRAID SAS 9341-4i Memory: 0MB BBU: Absent Serial No: SV62122829 ============================================================================== Number of DISK GROUPS: 0 Exit Code: 0x00 ===Cut=== disks are used in JBOD/Passthrough mode. I have a dumpdev configured, but savecore sayd there's no core. Either my swap device is smaller, or there's just no core.
Created attachment 171937 [details] trap screen from ipkvm. Added a panic screen.
There's more probably important information: this panic screen is cycling on the console, so I just was lucky enough to get one when it suddenly freezed.
Can't do much without a crashdump. Can you obtain one?
Yeah, sure, I'm trying at my best, but still no backtrace. I built a kernel with options KDB options KDB_TRACE but that didn't help me to get a backtrace, panic was still cycling on the screen. I've added options INVARIANT_SUPPORT options INVARIANTS options WITNESS options WITNESS_KDB to the kernel - and that kernel isn't able to boot past mountroot stage. Mountroot reports (even with boot -v) that it's mounting root from zfs, and then nothing happens (I waited for 7 minutes). I don't know if it's me who's impatient or it just hung. Do you have a decent way to capture a backtrace ?
Can you try to build just with DDB and KDB enabled? This should give you "db>" prompt on panic, type "where".
Finally I managed to get the backtrace via SOL. Here it is, along with the end of normal console output: ===Cut=== WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): waiting for CTL to terminate 1 tasks WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): waiting for CTL to terminate 1 tasks Fatal trap 12: page fault while in kernel mode Fatal trap 12: page fault while in kernel mode cpuid = 18; WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): tasks terminated apic id = 16 cpuid = 11; fault virtual address = 0x1e8 apic id = 0b fault code = supervisor write data, page not present fault virtual address = 0x1e8 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff80943603 instruction pointer = 0x20:0xffffffff80943603 stack pointer = 0x28:0xfffffe08560549d0 stack pointer = 0x28:0xfffffe08511b7680 frame pointer = 0x28:0xfffffe08560549f0 frame pointer = 0x28:0xfffffe08511b76a0 code segment = base rx0, limit 0xfffff, type 0x1b code segment = base rx0, limit 0xfffff, type 0x1b processor eflags = processor eflags = interrupt enabled, WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 resume, interrupt enabled, IOPL = 0 resume, current process = 0 (cfiscsitx) I[ thread pid 0 tid 102045 ] Stopped at 0xffffffff80943603 = __mtx_lock_flags+0x23: lock cmpxchgq %rbx,(%rsi) db> bt Tracing pid 0 tid 102045 td 0xfffff80108f51960 __mtx_lock_flags() at 0xffffffff80943603 = __mtx_lock_flags+0x23/frame 0xfffffe08560549f0 icl_send_thread() at 0xffffffff81e5df84 = icl_send_thread+0xe4/frame 0xfffffe0856054a70 fork_exit() at 0xffffffff8092627a = fork_exit+0x9a/frame 0xfffffe0856054ab0 fork_trampoline() at 0xffffffff80d578ae = fork_trampoline+0xe/frame 0xfffffe0856054ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> bt full Symbol not found KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at 18446744071565588123 = db_trace_self_wrapper+43/frame 0xfffffe0856054260 kdb_backtrace() at 18446744071572141785 = kdb_backtrace+57/frame 0xfffffe0856054310 kdb_reenter() at 18446744071572142467 = kdb_reenter+51/frame 0xfffffe0856054320 db_term() at 18446744071565583551 = db_term+143/frame 0xfffffe0856054340 db_mult_expr() at 18446744071565583035 = db_mult_expr+27/frame 0xfffffe0856054380 db_add_expr() at 18446744071565582875 = db_add_expr+27/frame 0xfffffe08560543c0 db_expression() at 18446744071565582669 = db_expression+29/frame 0xfffffe0856054410 db_stack_trace() at 18446744071565579904 = db_stack_trace+48/frame 0xfffffe0856054440 db_command() at 18446744071565578461 = db_command+621/frame 0xfffffe0856054510 db_command_loop() at 18446744071565577812 = db_command_loop+100/frame 0xfffffe0856054520 db_trap() at 18446744071565588448 = db_trap+224/frame 0xfffffe08560545b0 kdb_trap() at 18446744071572143417 = kdb_trap+297/frame 0xfffffe0856054600 trap_fatal() at 18446744071576166344 = trap_fatal+808/frame 0xfffffe0856054660 trap_pfault() at 18446744071576167181 = trap_pfault+749/frame 0xfffffe0856054700 trap() at 18446744071576164746 = trap+1146/frame 0xfffffe0856054910 calltrap() at 18446744071576056690 = calltrap+8/frame 0xfffffe0856054910 --- trap 12, rip = 18446744071571781123, rsp = 18446741910489221600, rbp = 18446741910489221616 --- __mtx_lock_flags() at 18446744071571781123 = __mtx_lock_flags+35/frame 0xfffffe08560549f0 icl_send_thread() at 18446744071593910148 = icl_send_thread+228/frame 0xfffffe0856054a70 fork_exit() at 18446744071571661434 = fork_exit+154/frame 0xfffffe0856054ab0 fork_trampoline() at 18446744071576058030 = fork_trampoline+14/frame 0xfffffe0856054ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> where Tracing pid 0 tid 102045 td 0xfffff80108f51960 __mtx_lock_flags() at 18446744071571781123 = __mtx_lock_flags+35/frame 0xfffffe08560549f0 icl_send_thread() at 18446744071593910148 = icl_send_thread+228/frame 0xfffffe0856054a70 fork_exit() at 18446744071571661434 = fork_exit+154/frame 0xfffffe0856054ab0 fork_trampoline() at 18446744071576058030 = fork_trampoline+14/frame 0xfffffe0856054ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> ===Cut===
Thanks! Now, can you try to do "call doadump" from ddb prompt? That should produce a crashdump.
Yup, sure. ===Cut=== db> call doadump Dumping 1602 out of 32498 MB:KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at 18446744071565588123 = db_trace_self_wrapper+43/frame 0xfffffe0856053ae0 kdb_backtrace() at 18446744071572141785 = kdb_backtrace+57/frame 0xfffffe0856053b90 kdb_reenter() at 18446744071572142467 = kdb_reenter+51/frame 0xfffffe0856053ba0 trap() at 18446744071576163684 = trap+84/frame 0xfffffe0856053db0 calltrap() at 18446744071576056690 = calltrap+8/frame 0xfffffe0856053db0 --- trap 12, rip = 18446744071568745081, rsp = 18446741910489218688, rbp = 18446741910489218704 --- mrsas_clear_intr() at 18446744071568745081 = mrsas_clear_intr+25/frame 0xfffffe0856053e90 mrsas_isr() at 18446744071568744983 = mrsas_isr+39/frame 0xfffffe0856053eb0 xpt_polled_action() at 18446744071565128195 = xpt_polled_action+579/frame 0xfffffe0856053f00 dadump() at 18446744071565287101 = dadump+365/frame 0xfffffe08560541b0 minidumpsys() at 18446744071576088795 = minidumpsys+1851/frame 0xfffffe0856054280 dumpsys() at 18446744071577338543 = dumpsys+63/frame 0xfffffe0856054380 doadump() at 18446744071571890095 = doadump+127/frame 0xfffffe08560543a0 db_fncall() at 18446744071565579253 = db_fncall+277/frame 0xfffffe0856054440 db_command() at 18446744071565578461 = db_command+621/frame 0xfffffe0856054510 db_command_loop() at 18446744071565577812 = db_command_loop+100/frame 0xfffffe0856054520 db_trap() at 18446744071565588448 = db_trap+224/frame 0xfffffe08560545b0 kdb_trap() at 18446744071572143417 = kdb_trap+297/frame 0xfffffe0856054600 trap_fatal() at 18446744071576166344 = trap_fatal+808/frame 0xfffffe0856054660 trap_pfault() at 18446744071576167181 = trap_pfault+749/frame 0xfffffe0856054700 trap() at 18446744071576164746 = trap+1146/frame 0xfffffe0856054910 calltrap() at 18446744071576056690 = calltrap+8/frame 0xfffffe0856054910 --- trap 12, rip = 18446744071571781123, rsp = 18446741910489221600, rbp = 18446741910489221616 --- __mtx_lock_flags() at 18446744071571781123 = __mtx_lock_flags+35/frame 0xfffffe08560549f0 icl_send_thread() at 18446744071593910148 = icl_send_thread+228/frame 0xfffffe0856054a70 fork_exit() at 18446744071571661434 = fork_exit+154/frame 0xfffffe0856054ab0 fork_trampoline() at 18446744071576058030 = fork_trampoline+14/frame 0xfffffe0856054ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- ===Cut===
Unfortunately, savecore sees nothing: [root@san1:~]# savecore -C No dump exists [root@san1:~]# gmirror status Name Status Components mirror/swap COMPLETE da1p2 (ACTIVE) da2p2 (ACTIVE) [root@san1:~]# savecore -C /dev/da1p2 No dump exists [root@san1:~]# savecore -C /dev/da2p2 No dump exists geom_mirror mode is prefer, preferences are set. Did I do something worng ? I guess you need that core to play with, right ?
I can provide an access to the live ddb, if that will do. Or I can try to find a way plug a disk without mrsas, directly accessible as dumpdev, if that's preferable.
I'd really need the core. Can you try to dump to a flash drive hooked up via USB? IIRC this should work. Also - if possible, add this: #pragma clang optimize off at the top of sys/dev/iscsi/icl_soft.c and sys/cam/ctl/ctl_frontend_iscsi.c and rebuild/reinstall the kernel, before testing. This makes debug information much more useful.
Will do asap, only that it will take some time from engineers at the remote DC to help me. Plus, I don't have icl_soft.c in stable/10, do I have to install HEAD, or may I put this line in icl.c ?
Hm, if it's 10, then it might already be fixed in HEAD. Could you try to upgrade to 11.0-CURRENT? At least the kernel and ctld.
Sure thing.
Reproduces on 11.0. Backtraces without sourec modifications (will do with sources too, I was just hoping it wont't crash on 11.x): WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): waiting for CTL to terminate 1 tasks WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): waiting for CTL to terminate 1 tasks WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): tasks terminated WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): tasks terminated Fatal trap 12: page fault while in kernel mode cpuid = 13; WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 apic id = 11 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 fault virtual address = 0x28 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): waiting for CTL to terminate 1 tasks WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 fault code = supervisor read data, page not present WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 instruction pointer = 0x20:0xffffffff8264232c WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 stack pointer = 0x28:0xfffffe0850b853c0 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 frame pointer = 0x28:0xfffffe0850b85480 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 code segment = base rx0, limit 0xfffff, type 0x1b WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 = DPL 0, pres 1, long 1, def32 0, gran 1 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 processor eflags = WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 interrupt enabled, WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 resume, WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 IOPL = 0 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 current process = 843 (work5) W[ thread pid 843 tid 102522 ] Stopped at 0xffffffff8264232c = cfiscsi_datamove+0x15c: movq (%rdx,%rax,8),%rcx db> where Tracing pid 843 tid 102522 td 0xfffff8002eaac500 cfiscsi_datamove() at 0xffffffff8264232c = cfiscsi_datamove+0x15c/frame 0xfffffe0850b85480 ctl_datamove() at 0xffffffff82623ce8 = ctl_datamove+0x188/frame 0xfffffe0850b85630 ctl_report_luns() at 0xffffffff8262b1d7 = ctl_report_luns+0x3e7/frame 0xfffffe0850b856c0 ctl_work_thread() at 0xffffffff8262e1cf = ctl_work_thread+0xb4f/frame 0xfffffe0850b85a70 fork_exit() at 0xffffffff80a9d365 = fork_exit+0x85/frame 0xfffffe0850b85ab0 fork_trampoline() at 0xffffffff80f9f4de = fork_trampoline+0xe/frame 0xfffffe0850b85ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> bt Tracing pid 843 tid 102522 td 0xfffff8002eaac500 cfiscsi_datamove() at 0xffffffff8264232c = cfiscsi_datamove+0x15c/frame 0xfffffe0850b85480 ctl_datamove() at 0xffffffff82623ce8 = ctl_datamove+0x188/frame 0xfffffe0850b85630 ctl_report_luns() at 0xffffffff8262b1d7 = ctl_report_luns+0x3e7/frame 0xfffffe0850b856c0 ctl_work_thread() at 0xffffffff8262e1cf = ctl_work_thread+0xb4f/frame 0xfffffe0850b85a70 fork_exit() at 0xffffffff80a9d365 = fork_exit+0x85/frame 0xfffffe0850b85ab0 fork_trampoline() at 0xffffffff80f9f4de = fork_trampoline+0xe/frame 0xfffffe0850b85ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> bt full Symbol not found KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at 18446744071565858987 = db_trace_self_wrapper+43/frame 0xfffffe0850b84b80 kdb_reenter() at 18446744071573801358 = kdb_reenter+142/frame 0xfffffe0850b84c30 db_term() at 18446744071565853847 = db_term+151/frame 0xfffffe0850b84c50 db_unary() at 18446744071565853585 = db_unary+161/frame 0xfffffe0850b84c70 db_mult_expr() at 18446744071565853159 = db_mult_expr+23/frame 0xfffffe0850b84ca0 db_add_expr() at 18446744071565852953 = db_add_expr+25/frame 0xfffffe0850b84ce0 db_shift_expr() at 18446744071565852765 = db_shift_expr+29/frame 0xfffffe0850b84d30 db_logical_relation_expr() at 18446744071565852461 = db_logical_relation_expr+29/frame 0xfffffe0850b84d80 db_logical_and_expr() at 18446744071565852297 = db_logical_and_expr+25/frame 0xfffffe0850b84dc0 db_expression() at 18446744071565852137 = db_expression+25/frame 0xfffffe0850b84e00 db_stack_trace() at 18446744071565848368 = db_stack_trace+48/frame 0xfffffe0850b84e30 db_command() at 18446744071565847721 = db_command+665/frame 0xfffffe0850b84f00 db_command_loop() at 18446744071565847044 = db_command_loop+100/frame 0xfffffe0850b84f10 db_trap() at 18446744071565859307 = db_trap+219/frame 0xfffffe0850b84fa0 kdb_trap() at 18446744071573802435 = kdb_trap+403/frame 0xfffffe0850b85030 trap_fatal() at 18446744071578567377 = trap_fatal+785/frame 0xfffffe0850b85090 trap_pfault() at 18446744071578567939 = trap_pfault+483/frame 0xfffffe0850b850f0 trap() at 18446744071578565292 = trap+620/frame 0xfffffe0850b85300 calltrap() at 18446744071578447777 = calltrap+8/frame 0xfffffe0850b85300 --- trap 12, rip = 18446744071602185004, rsp = 18446741910400291792, rbp = 18446741910400291968 --- cfiscsi_datamove() at 18446744071602185004 = cfiscsi_datamove+348/frame 0xfffffe0850b85480 ctl_datamove() at 18446744071602060520 = ctl_datamove+392/frame 0xfffffe0850b85630 ctl_report_luns() at 18446744071602090455 = ctl_report_luns+999/frame 0xfffffe0850b856c0 ctl_work_thread() at 18446744071602102735 = ctl_work_thread+2895/frame 0xfffffe0850b85a70 fork_exit() at 18446744071573197669 = fork_exit+133/frame 0xfffffe0850b85ab0 fork_trampoline() at 18446744071578449118 = fork_trampoline+14/frame 0xfffffe0850b85ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db>
vmcore on the way (7 gigs), compressing at the time, will put on the web asap. Then I will redo it with modified sources.
vmcore with stock sources: http://tank0.music.enaza.ru/vmcore.0.tar.gz (1.4 Gb)
backtrace with modified sources: ===Cut=== WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.130 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): waiting for CTL to terminate 1 tasks WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): waiting for CTL to terminate 1 tasks WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): tasks terminated Fatal trap 12: page fault while in kernel mode WARNING: 82.194.234.131 (iqn.1991-05.com.microsoft:worker02): tasks terminated cpuid = 17; apic id = 15 fault virtual address = 0x28 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff826435ca stack pointer = 0x28:0xfffffe0850b85380 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 frame pointer = 0x28:0xfffffe0850b853c0 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 code segment = base rx0, limit 0xfffff, type 0x1b WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 = DPL 0, pres 1, long 1, def32 0, gran 1 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 processor eflags = WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 interrupt enabled, WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 resume, WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 IOPL = 0 WARNING: 82.194.234.130 (iqn.1991-05.com.microsoft:worker02): session reinstatement from different address 82.194.234.131 current process = 810 (work5) W[ thread pid 810 tid 102522 ] Stopped at 0xffffffff826435ca = ICL_CONN_NEW_PDU+0x3a: movq (%rax),%rax db> where Tracing pid 810 tid 102522 td 0xfffff8014e717a00 ICL_CONN_NEW_PDU() at 0xffffffff826435ca = ICL_CONN_NEW_PDU+0x3a/frame 0xfffffe0850b853c0 icl_pdu_new() at 0xffffffff8264357b = icl_pdu_new+0x1b/frame 0xfffffe0850b853e0 cfiscsi_pdu_new_response() at 0xffffffff8264345f = cfiscsi_pdu_new_response+0x1f/frame 0xfffffe0850b85400 cfiscsi_datamove_in() at 0xffffffff82642a13 = cfiscsi_datamove_in+0x143/frame 0xfffffe0850b854a0 cfiscsi_datamove() at 0xffffffff826426b4 = cfiscsi_datamove+0x24/frame 0xfffffe0850b854c0 ctl_datamove() at 0xffffffff82623ce8 = ctl_datamove+0x188/frame 0xfffffe0850b85670 ctl_inquiry() at 0xffffffff8262c651 = ctl_inquiry+0x1071/frame 0xfffffe0850b856c0 ctl_work_thread() at 0xffffffff8262e1cf = ctl_work_thread+0xb4f/frame 0xfffffe0850b85a70 fork_exit() at 0xffffffff80a9d365 = fork_exit+0x85/frame 0xfffffe0850b85ab0 fork_trampoline() at 0xffffffff80f9f4de = fork_trampoline+0xe/frame 0xfffffe0850b85ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> bt Tracing pid 810 tid 102522 td 0xfffff8014e717a00 ICL_CONN_NEW_PDU() at 0xffffffff826435ca = ICL_CONN_NEW_PDU+0x3a/frame 0xfffffe0850b853c0 icl_pdu_new() at 0xffffffff8264357b = icl_pdu_new+0x1b/frame 0xfffffe0850b853e0 cfiscsi_pdu_new_response() at 0xffffffff8264345f = cfiscsi_pdu_new_response+0x1f/frame 0xfffffe0850b85400 cfiscsi_datamove_in() at 0xffffffff82642a13 = cfiscsi_datamove_in+0x143/frame 0xfffffe0850b854a0 cfiscsi_datamove() at 0xffffffff826426b4 = cfiscsi_datamove+0x24/frame 0xfffffe0850b854c0 ctl_datamove() at 0xffffffff82623ce8 = ctl_datamove+0x188/frame 0xfffffe0850b85670 ctl_inquiry() at 0xffffffff8262c651 = ctl_inquiry+0x1071/frame 0xfffffe0850b856c0 ctl_work_thread() at 0xffffffff8262e1cf = ctl_work_thread+0xb4f/frame 0xfffffe0850b85a70 fork_exit() at 0xffffffff80a9d365 = fork_exit+0x85/frame 0xfffffe0850b85ab0 fork_trampoline() at 0xffffffff80f9f4de = fork_trampoline+0xe/frame 0xfffffe0850b85ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> bt full Symbol not found KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at 18446744071565858987 = db_trace_self_wrapper+43/frame 0xfffffe0850b84b40 kdb_reenter() at 18446744071573801358 = kdb_reenter+142/frame 0xfffffe0850b84bf0 db_term() at 18446744071565853847 = db_term+151/frame 0xfffffe0850b84c10 db_unary() at 18446744071565853585 = db_unary+161/frame 0xfffffe0850b84c30 db_mult_expr() at 18446744071565853159 = db_mult_expr+23/frame 0xfffffe0850b84c60 db_add_expr() at 18446744071565852953 = db_add_expr+25/frame 0xfffffe0850b84ca0 db_shift_expr() at 18446744071565852765 = db_shift_expr+29/frame 0xfffffe0850b84cf0 db_logical_relation_expr() at 18446744071565852461 = db_logical_relation_expr+29/frame 0xfffffe0850b84d40 db_logical_and_expr() at 18446744071565852297 = db_logical_and_expr+25/frame 0xfffffe0850b84d80 db_expression() at 18446744071565852137 = db_expression+25/frame 0xfffffe0850b84dc0 db_stack_trace() at 18446744071565848368 = db_stack_trace+48/frame 0xfffffe0850b84df0 db_command() at 18446744071565847721 = db_command+665/frame 0xfffffe0850b84ec0 db_command_loop() at 18446744071565847044 = db_command_loop+100/frame 0xfffffe0850b84ed0 db_trap() at 18446744071565859307 = db_trap+219/frame 0xfffffe0850b84f60 kdb_trap() at 18446744071573802435 = kdb_trap+403/frame 0xfffffe0850b84ff0 trap_fatal() at 18446744071578567377 = trap_fatal+785/frame 0xfffffe0850b85050 trap_pfault() at 18446744071578567939 = trap_pfault+483/frame 0xfffffe0850b850b0 trap() at 18446744071578565292 = trap+620/frame 0xfffffe0850b852c0 calltrap() at 18446744071578447777 = calltrap+8/frame 0xfffffe0850b852c0 --- trap 12, rip = 18446744071602189770, rsp = 18446741910400291728, rbp = 18446741910400291776 --- ICL_CONN_NEW_PDU() at 18446744071602189770 = ICL_CONN_NEW_PDU+58/frame 0xfffffe0850b853c0 icl_pdu_new() at 18446744071602189691 = icl_pdu_new+27/frame 0xfffffe0850b853e0 cfiscsi_pdu_new_response() at 18446744071602189407 = cfiscsi_pdu_new_response+31/frame 0xfffffe0850b85400 cfiscsi_datamove_in() at 18446744071602186771 = cfiscsi_datamove_in+323/frame 0xfffffe0850b854a0 cfiscsi_datamove() at 18446744071602185908 = cfiscsi_datamove+36/frame 0xfffffe0850b854c0 ctl_datamove() at 18446744071602060520 = ctl_datamove+392/frame 0xfffffe0850b85670 ctl_inquiry() at 18446744071602095697 = ctl_inquiry+4209/frame 0xfffffe0850b856c0 ctl_work_thread() at 18446744071602102735 = ctl_work_thread+2895/frame 0xfffffe0850b85a70 fork_exit() at 18446744071573197669 = fork_exit+133/frame 0xfffffe0850b85ab0 fork_trampoline() at 18446744071578449118 = fork_trampoline+14/frame 0xfffffe0850b85ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> ===Cut===
Do you have a /var/crash/core.txt.0? If so, please upload it. (And for the meantime 'chmod 0' to the core file (the huge one), it can contain private data.)
Second core, from modified source: http://tank0.music.enaza.ru/vmcore.1.tar.gz (1.1 Gb)
Created attachment 171976 [details] core.txt.0 core.txt.0, stock sources
Created attachment 171977 [details] core.txt.1 core.txt.1, modified sources.
*** This bug has been marked as a duplicate of bug 222898 ***
MARKED AS SPAM