panic while bulk building in poudriere Backtrace --------- Tracing pid 0 tid 100265 td 0xffff000312c70640 db_trace_self() at db_trace_self db_stack_trace() at db_stack_trace+0xf8 db_command() at db_command+0x390 db_command_loop() at db_command_loop+0x58 db_trap() at db_trap+0x10c kdb_trap() at kdb_trap+0x364 handle_el1h_sync() at handle_el1h_sync+0x18 --- exception, esr 0xf2000000 kdb_enter() at kdb_enter+0x48 vpanic() at vpanic+0x208 panic() at panic+0x48 rack_proc_sack_blk() at rack_proc_sack_blk+0x1450 handle_el1h_sync() at handle_el1h_sync+0x18 --- exception, esr 0xf2000000 kdb_enter() at kdb_enter+0x48 vpanic() at vpanic+0x208 panic() at panic+0x48 rack_proc_sack_blk() at rack_proc_sack_blk+0x1450 rack_log_ack() at rack_log_ack+0x624 rack_process_ack() at rack_process_ack+0x30c rack_do_established() at rack_do_established+0xaac rack_do_segment_nounlock() at rack_do_segment_nounlock+0x1cdc rack_do_segment() at rack_do_segment+0x114 tcp_input_with_port() at tcp_input_with_port+0x1020 tcp_input() at tcp_input+0x10 ip_input() at ip_input+0x370 netisr_dispatch_src() at netisr_dispatch_src+0xec ether_demux() at ether_demux+0x180 ether_nh_input() at ether_nh_input+0x394 netisr_dispatch_src() at netisr_dispatch_src+0xec ether_input() at ether_input+0xe8 iflib_rxeof() at iflib_rxeof+0x800 _task_fn_rx() at _task_fn_rx+0xb4 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x170 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc0 fork_exit() at fork_exit+0x80 fork_trampoline() at fork_trampoline+0x18 Registers at kdb_enter ---------------------- spsr 0x60400009 x0 0x12 x1 0xffff00031332c078 x2 0xffff000000b3d23e x3 0x248 x4 0xf x5 0x10 x6 0xffff007fffe2a000 x7 0x254 x8 0x915746f4baa89085 x9 0x915746f4baa89085 x10 0x12800 x11 0xf x12 0 x13 0x4 x14 0x200 x15 0x1 x16 0x80 x17 0x220 x18 0xffff000047f60000 x19 0xffff00000113b000 epoch_array+0x1f00 x20 0 x21 0xffff000312c70640 x22 0x1 x23 0 x24 0x721c9922 x25 0x1a8cf0b2 x26 0xffffa800e7321a80 x27 0x1a8cf2be x28 0xffffa800e7321aa8 x29 0xffff00031332c200 lr 0xffff0000005588f0 kdb_enter+0x44 elr 0xffff0000005588f4 kdb_enter+0x48 sp 0xffff00031332c200 Function: rack_check_reorder_ack() [inlined into rack_proc_sack_blk()] Line: 9203-9204 KASSERT((rack->r_ctl.recovery_rxt_cnt >= (the_end - rsm->r_start)), ("rsm:%p rack:%p recovery_rxt_cnt would go negative " "recovery_rxt_cnt:%u sub:%u", rsm, rack, rack->r_ctl.recovery_rxt_cnt, (the_end - rsm->r_start))); Values at time of panic: rack->r_ctl.recovery_rxt_cnt = 0 the_end - rsm->r_start = 524
Hi Dave, is the panic reproducible? Could you run the system with BBLogging enabled as described in https://wiki.freebsd.org/MichaelTuexen? When the panic then happens most likely I can build a reproducer from the BBLog information. Best regards Michael
Created attachment 272905 [details] tcplog bb debug Sorry it took far too long to get a panic off the box, netdump was the eventual solution.
Created attachment 272906 [details] this time include how we produced BB log from output lets try to process this, I have an amd64 15.1-RELEASE box: dch@wintermute /t/wat> kgdb /usr/obj/usr/src/arm64.aarch64/sys/GENERIC/kernel.debug vmcore.172.16.1.12.1 GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd15.1". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/obj/usr/src/arm64.aarch64/sys/GENERIC/kernel.debug... warning: File "/tmp/wat/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". To enable execution of this file add add-auto-load-safe-path /tmp/wat/.gdbinit line to your configuration file "/home/dch/.config/gdb/gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/home/dch/.config/gdb/gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Error while mapping shared library sections: `/boot/kernel/tcp_rack.ko': Shared library architecture i386:x86-64 is not compatible with target architecture aarch64. … Reading symbols from /usr/obj/usr/src/arm64.aarch64/kernelstage/kernel/boot/kernel/nullfs.ko... (No debugging symbols found in /usr/obj/usr/src/arm64.aarch64/kernelstage/kernel/boot/kernel/nullfs.ko) (kgdb) (kgdb) source /usr/src/tools/tools/kgdb/tcplog.py (kgdb) tcplog_dump (struct tcpcb *)0xffffa80d22259540 processing struct tcpcb * 0xffffa80d22259540 _t_logstate: 1 _t_logpoint: 0 '\000' t_lognum: 1727 t_logsn: 1727 log written to 0xffffa80d22259540_tcp_log.bin (kgdb)
Thanks for the BBLog file. I was able to convert it to a .pcapng file and can read that. I will attend the IETF meeting next week, so it might take some time to have a reproducer.
Created attachment 272951 [details] BBLog information in .pcapng
Created attachment 272998 [details] packetdrill reproducer