Created attachment 192755 [details] KCONF and nanobsd config I'm running a FreeBSD 11.1-RELEASE after build using nanobsd.sh. Hardware is amd64 running in a virtual machine (vmware). Attached is kernel config and nanobsd config. The system is rebooting every 30 minutes to 8 hours. Sample dmesg output is: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff805bbe57 stack pointer = 0x28:0xfffffe0000242360 frame pointer = 0x28:0xfffffe00002424b0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 37669 (ppp) trap number = 12 panic: page fault cpuid = 0 Uptime: 8h6m0s Any additional information needed please let me know.
Do you have a stack trace?
(In reply to Conrad Meyer from comment #1) No I don't, what would be the best way to do that?
(In reply to Matt Allanson from comment #0) Can you attach ppp configs? Does it run as server or client of forms p2p tunnel? If it's a server, how many clients does it have at max? For crashdump you will need another (virtual) disk or add big enough additional partition to your nanobsd build equal to RAM size to be sure. Add it to rc.conf: dumpdev="/dev/ada1" # or /dev/ada0s1b See https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html#kerneldebug-obtain for details.
Created attachment 192809 [details] Client and server config Waiting for the next crash to get the stack trace, attached are client and server ppp configs. It is really simple; just pushing over tcp.
(In reply to Eugene Grosbein from comment #3) The machine that's failing is the server. We have 18 tunnels at max, currently.
(In reply to Matt Allanson from comment #4) Kernel crashdump will be much more useful if kernel config file has: options KDB options KDB_TRACE options KDB_UNATTENDED options INVARIANTS options INVARIANT_SUPPORT options WITHNESS options WITNESS_SKIPSPIN
#1 0xffffffff804c06b5 in mi_switch () #2 0xffffffff804ff2da in sleepq_wait () #3 0xffffffff804c0231 in _sleep () #4 0xffffffff80504211 in taskqueue_thread_loop () #5 0xffffffff804844e5 in fork_exit () #6 <signal handler called> Here is the stack trace from the most recent reboot. I have additional information (from crashinfo) if it'd be helpful. Unfortunately, this kernel doesn't have debuginfo built in, but I can rebuild if absolutely necessary. Note that since this is a production server, putting a debug kernel on it would require change control and Wednesday (5/2) would be the earliest I could do it.
(In reply to Matt Allanson from comment #7) This is not very useful. You should have kernel.debug in the kernel build directory, do you? It is used to obtain kgdb backtrace.
(In reply to Eugene Grosbein from comment #8) Sorry for the late response, kernel.debug does not exist on any of our machines.
GENERIC kernel has following line: makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols Add it to your kernel configuration file. It makes kernel build process to produce two kernel binaries instead of just one: ordinary kernel that gets installed by installkernel and additional kernel.debug that is not installed but is kept in the kernel build directory and used for kgdb later.
(In reply to Eugene Grosbein from comment #10) Hoping this is more what you were looking for... (kgdb) #0 __curthread () at ./machine/pcpu.h:222 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:298 #2 0xffffffff804bf360 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366 #3 0xffffffff804bf920 in vpanic (fmt=<optimized out>, ap=0xfffffe00968ee120) at /usr/src/sys/kern/kern_shutdown.c:759 #4 0xffffffff804bf963 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:690 #5 0xffffffff806e56a2 in trap_fatal (frame=0xfffffe00968ee2b0, eva=0) at /usr/src/sys/amd64/amd64/trap.c:801 #6 0xffffffff806e4d15 in trap (frame=0xfffffe00968ee2b0) at /usr/src/sys/amd64/amd64/trap.c:197 #7 <signal handler called> #8 0xffffffff805d4297 in sysctl_dumpentry (rn=0xfffff80012b12c30, vw=0xfffffe00968ee648) at /usr/src/sys/net/rtsock.c:1543 #9 0xffffffff805ce5e0 in rn_walktree (h=<optimized out>, f=0xffffffff805d41a0 <sysctl_dumpentry>, w=0xfffffe00968ee648) at /usr/src/sys/net/radix.c:1094 #10 0xffffffff805d393b in sysctl_rtsock (oidp=<optimized out>, arg1=<optimized out>, arg2=<optimized out>, req=<optimized out>) at /usr/src/sys/net/rtsock.c:1900 #11 0xffffffff804cbe00 in sysctl_root_handler_locked ( oid=0xffffffff80a5e818 <sysctl___net_routetable>, arg1=0xfffffe00968ee8b8, arg2=4, req=0xfffffe00968ee7f0, tracker=0xfffffe00968ee770) at /usr/src/sys/kern/kern_sysctl.c:165 #12 0xffffffff804cb622 in sysctl_root (oidp=<optimized out>, arg1=0x80, arg2=4, req=<optimized out>) at /usr/src/sys/kern/kern_sysctl.c:1877 #13 0xffffffff804cbb8d in userland_sysctl (td=<optimized out>, name=0xfffffe00968ee8b0, namelen=6, old=<optimized out>, oldlenp=<optimized out>, inkernel=<optimized out>, new=0x0, newlen=<optimized out>, retval=0xfffff80001e6b800, flags=0) at /usr/src/sys/kern/kern_sysctl.c:1980 #14 0xffffffff804cba2f in sys___sysctl (td=0xfffff800524e9000, uap=0xfffffe00968eea30) at /usr/src/sys/kern/kern_sysctl.c:1907 #15 0xffffffff806e5c60 in syscallenter (td=0xfffff800524e9000, sa=<optimized out>) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #16 amd64_syscall (td=0xfffff800524e9000, traced=0) at /usr/src/sys/amd64/amd64/trap.c:902 #17 <signal handler called> #18 0x0000000801dd22aa in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffdf08 (kgdb) Please let me know if you need anything else.
(In reply to Matt Allanson from comment #11) Yes, that's it. Can you please make files kernel.debug and vmcore.0 available for download? Preferably compressed.
(In reply to Eugene Grosbein from comment #12) Sorry for the late reply was ill last week, here is the info you requested. http://rdmpackage.trimedx.com/crashinfo.tar.bz2 Please let me know when you have retrieved it.
I've downloaded it, thanks. (kgdb) p *((struct rtentry *)rn)->rt_ifp $7 = {if_link = {tqe_next = 0xdeadc0dedeadc0de, tqe_prev = 0xdeadc0dedeadc0de}, if_clones = { le_next = 0xdeadc0dedeadc0de, le_prev = 0xdeadc0dedeadc0de}, if_groups = { tqh_first = 0xdeadc0dedeadc0de, tqh_last = 0xdeadc0dedeadc0de}, if_alloctype = 222 'ч', if_softc = 0xdeadc0dedeadc0de, if_llsoftc = 0xdeadc0dedeadc0de, if_l2com = 0xdeadc0dedeadc0de, if_dname = 0xdeadc0dedeadc0de <Address 0xdeadc0dedeadc0de out of bounds>, etc. This means race condition in the kernel between interface removal procedure when some tunnel is being disconnected and sysctl handler for "net.routetable" that ppp calls, or some subroutine this handler uses. Perhaps, this is guilt of sysctl_rtsock() function that uses RIB_RLOCK() before calling rnh->rnh_walktree(&rnh->head, sysctl_dumpentry, &w) but that does not protect from interface destruction: https://svnweb.freebsd.org/base/release/11.1.0/sys/net/rtsock.c?annotate=321354#l1898 We need some more eyes of networking people here.
(In reply to Eugene Grosbein from comment #14) Is there something that I need to do in the meantime?
(In reply to Matt Allanson from comment #15) You could try using net/mpd5 package instead of built-in ppp(8) utility for same task. It may occur more stable for your environment.
(In reply to Eugene Grosbein from comment #16) Unfortunately it would be quite a significant change to our production environment and we aren't able to do that "in the meantime" type fix.
(In reply to Matt Allanson from comment #17) It would be nice if you write in details how one can reproduce the problem.
(In reply to Eugene Grosbein from comment #18) 1. Install FreeBSD 11.1 (nanobsd built with provided KCONF) 2. Install stunnel 3. Configure ppp + stunnel to act as server on one machine and client on another 4. Connect client(s) to server 5. Communicate via network as normal 6. Wait for system to crash We are not doing anything special in any way re: networking -- we're just pushing network comms over PPP over TLS. This solution has been 100% solid in FreeBSD 10.3.
(In reply to Eugene Grosbein from comment #18) Apologies that is last stable in 10.2-RELEASE not 10.3-RELEASE
(In reply to Matt Allanson from comment #20) We went from 10.2-RELEASE to 11.1-RELEASE. Just wanting to clarify
(In reply to Eugene Grosbein from comment #18) Matt is a co-worker and he is on leave right now, I was wondering if you could give me an update on this BUG 227720? Thank you, Brian Murrey
(In reply to Brian Murrey from comment #22) I cannot. Hopefully some other networking people will take a look at this racy panic.
You can try to apply this patch https://people.freebsd.org/~ae/netgc.diff It does deferred free for deleted ifnet structure. So, when interface destroyed and some code does access to the ifnet stucture in the same time, it can be done without page fault.
> You can try to apply this patch https://people.freebsd.org/~ae/netgc.diff > > It does deferred free for deleted ifnet structure. So, when interface > destroyed and some code does access to the ifnet stucture in the same time, > it can be done without page fault. Note, you need to set sysctl net.gc.enable=1 to take an effect.
(In reply to Andrey V. Elsukov from comment #25) Just a heads up, replied in email forgot to comment it here. Sorry about that... We applied the patch to enable garbage collection. Rebuilt the image and installed it, then set net.gc.enabled=1 as well as in /etc/sysctl.conf and it continued to fail. Here is the most recent backtrace: (kgdb) bt #0 0xffffffff804e7866 in sched_switch () #1 0xffffffff804c8d38 in mi_switch () #2 0xffffffff8050acfe in sleepq_switch () #3 0xffffffff8050abb3 in sleepq_wait () #4 0xffffffff804c8684 in _sleep () #5 0xffffffff80510401 in taskqueue_thread_loop () #6 0xffffffff804878d4 in fork_exit () #7 <signal handler called> (kgdb) Let me know any addition information you need.
(In reply to Matt Allanson from comment #26) Just wanted to check in on this and see if there is any movement or anything you guys need from me.
(In reply to Matt Allanson from comment #27) Just checking in again, this is currently affecting production... We would really like to continue to use FreeBSD.
(In reply to Matt Allanson from comment #27) If you still use 11.1-RELEASE, please upgrade to 11.2-RELEASE first and re-try.
(In reply to Matt Allanson from comment #28) If 11.2-RELEASE still panices for you, I need kernel.debug and crashdump got same way as you provided in the comment 11.
(In reply to Matt Allanson from comment #19) I'm going trying to reproduce the panic. Can yo chare your configs for system interfaces, stunnel (server and client) and ppp (server and client)? That would speedup things greatly.
Created attachment 197207 [details] stunnel client and server config
(In reply to Eugene Grosbein from comment #31) The client and server config for ppp is already uploaded and has not changed. Just uploaded the stunnel configs. Getting ready to build 11.2 to test. Thank you!
(In reply to Matt Allanson from comment #32) I also need to know version of stunnel you use and how do you run stunnel and ppp - just configure them with /etc/rc.conf? If so, I need corresponding lines from /etc/rc.conf. If you use some custom startup scripts, I need to know which arguments they pass to stunnel and ppp starting commands.
Hi everyone, same kind of crash here on a 11.2-RELEASE-p4, it also crashes on 11.2-RELEASE, but it seems to be a long standing bug, we had similar issues on previous versions. I only have the crash logs for this latest release. # uname -a FreeBSD testpc 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 # dmidecode -s system-product-name OptiPlex 7010 # sysctl hw.model hw.model: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz The crash occurs when we play with forwarding, a PPP server and ARP proxy, but unfortunately we do not have a very precise procedure to reproduce it, also it crashes very consistently. We use FreeBSD for educational purposes, doing practical work in computer networking. This is a simple setup where we use 3 computers, configure PC1 as a PPP client, that connects to PC2 that is a PPP server and ARP proxy for PC1, and we finally check the IP connectivity from PC3 that is on the same Ethernet LAN as PC2 and observe ARP activity. If we activate forwarding, start the PPP server, connect the client, add an ARP published entry, it works ok as long as there is no mistake done (although I cannot guarantee that it would run for hours). But when students make several wrong tries, stop ppp, play with arp, restart ppp, at some point it crashes (we had dozens of crashes). It is rather easy to provoke this crash as I did to get the crash log, although there are various symptoms: ppp server not accepting connections anymore unless we reboot, arp failing with socket memory error, etc. until it crashes at some point. Quite hard to track this one down... This bug seems to be somewhat similar to bug #230498 with the same location and cause. The ppp client and server configs are trivial, over a serial line, plus proxy arp (arp -s 192.168.0.1 <eth@> pub) default: set log Phase Chat LCP IPCP CCP tun command set device /dev/cuau0 set speed 9600 set accmap 000a0000 set ctsrts off set cd off set timeout 0 server: set ifaddr 192.168.0.2 192.168.0.1 255.255.255.255 Here is the backtrace, I can provide the crash image and kernel.debug if needed. (kgdb) bt #0 doadump (textdump=<value optimized out>) at pcpu.h:229 #1 0xffffffff80af673b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:383 #2 0xffffffff80af6b61 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:776 #3 0xffffffff80af69a3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:707 #4 0xffffffff80f77fdf in trap_fatal (frame=0xfffffe04688ae320, eva=0) at /usr/src/sys/amd64/amd64/trap.c:875 #5 0xffffffff80f78039 in trap_pfault (frame=0xfffffe04688ae320, usermode=0) at pcpu.h:229 #6 0xffffffff80f77807 in trap (frame=0xfffffe04688ae320) at /usr/src/sys/amd64/amd64/trap.c:415 #7 0xffffffff80f57fbc in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231 #8 0xffffffff80c0ce96 in sysctl_dumpentry (rn=0xfffff8000bc48410, vw=0xfffffe04688ae690) at /usr/src/sys/net/rtsock.c:1559 #9 0xffffffff80c07aa0 in rn_walktree (h=<value optimized out>, f=<value optimized out>, w=<value optimized out>) at /usr/src/sys/net/radix.c:1094 #10 0xffffffff80c0c7ff in sysctl_rtsock (oidp=<value optimized out>, arg1=<value optimized out>, arg2=<value optimized out>, req=<value optimized out>) at /usr/src/sys/net/rtsock.c:1916 #11 0xffffffff80b03ccb in sysctl_root_handler_locked (oid=0xffffffff81a33f38, arg1=0xfffffe04688ae908, arg2=4, req=0xfffffe04688ae840, tracker=0xfffffe04688ae7b8) at /usr/src/sys/kern/kern_sysctl.c:165 #12 0xffffffff80b03521 in sysctl_root (arg1=0xfffffe04688ae908, arg2=4) at /usr/src/sys/kern/kern_sysctl.c:1915 #13 0xffffffff80b03a46 in userland_sysctl (td=<value optimized out>, name=0xfffffe04688ae900, namelen=6, old=0x0, oldlenp=<value optimized out>, inkernel=<value optimized out>, new=0x0, newlen=0, retval=0xfffffe04688ae968, flags=0) at /usr/src/sys/kern/kern_sysctl.c:2011 #14 0xffffffff80b038cf in sys___sysctl (td=0xfffff80098410000, uap=0xfffff80098410538) at /usr/src/sys/kern/kern_sysctl.c:1945 #15 0xffffffff80f79068 in amd64_syscall (td=0xfffff80098410000, traced=0) at subr_syscall.c:132 #16 0xffffffff80f5880d in fast_syscall_common () at /usr/src/sys/amd64/amd64/exception.S:479 #17 0x0000000801de047a in ?? () (kgdb) f 8 #8 0xffffffff80c0ce96 in sysctl_dumpentry (rn=0xfffff8000bc48410, vw=0xfffffe04688ae690) at /usr/src/sys/net/rtsock.c:1559 1559 info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; (kgdb) p rt->rt_ifp->if_addr $1 = (struct ifaddr *) 0x0
Please try the patch from https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230498
Thanks for the fast reply! Not sure if I continue here or in bug #230498 but since this is still related to PPP, I put it here. I only had 15 min to test, but it crashed right away on the first try. Here is the procedure: - setup PC3: configure address on Ethernet interface; - setup PC2: configure address on Ethernet interface, add ARP pub entry, activate forwarding, start ppp server and wait for connection; - setup PC3: start pinging PC3, obviously it fails, start ppp client and open connection, add default route, everything works correctly. Leave everything running as it is, then quit ppp on both sides, restart the server waiting for the connection, connect from client -> crash on PC2. Here is the trace, it crashes one call further line rtsock.c:1559 after the patch info.rti_info[RTAX_GENMASK] = 0; if (rt->rt_ifp) { - info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; + IF_ADDR_RLOCK(rt->rt_ifp); + if (rt->rt_ifp->if_addr != NULL) + info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; info.rti_info[RTAX_IFA] = rt->rt_ifa->ifa_addr; I also add a somewhat tidied up version of the (struct ifnet *) (kgdb) bt #0 doadump (textdump=<value optimized out>) at pcpu.h:229 #1 0xffffffff80af673b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:383 #2 0xffffffff80af6b61 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:776 #3 0xffffffff80af69a3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:707 #4 0xffffffff80f77fdf in trap_fatal (frame=0xfffffe0468486290, eva=1208) at /usr/src/sys/amd64/amd64/trap.c:875 #5 0xffffffff80f78039 in trap_pfault (frame=0xfffffe0468486290, usermode=0) at pcpu.h:229 #6 0xffffffff80f77807 in trap (frame=0xfffffe0468486290) at /usr/src/sys/amd64/amd64/trap.c:415 #7 0xffffffff80f57fdc in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231 #8 0xffffffff80af2893 in __rw_rlock_hard (rw=0xfffff800be4bc990, td=0xfffff80105056620, v=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:493 #9 0xffffffff80c0ce9b in sysctl_dumpentry (rn=0xfffff80008e74270, vw=0xfffffe0468486690) at /usr/src/sys/net/rtsock.c:1559 #10 0xffffffff80c07aa0 in rn_walktree (h=<value optimized out>, f=<value optimized out>, w=<value optimized out>) at /usr/src/sys/net/radix.c:1094 #11 0xffffffff80c0c7ff in sysctl_rtsock (oidp=<value optimized out>, arg1=<value optimized out>, arg2=<value optimized out>, req=<value optimized out>) at /usr/src/sys/net/rtsock.c:1919 #12 0xffffffff80b03ccb in sysctl_root_handler_locked (oid=0xffffffff81a33f38, arg1=0xfffffe0468486908, arg2=4, req=0xfffffe0468486840, tracker=0xfffffe04684867b8) at /usr/src/sys/kern/kern_sysctl.c:165 #13 0xffffffff80b03521 in sysctl_root (arg1=0xfffffe0468486908, arg2=4) at /usr/src/sys/kern/kern_sysctl.c:1915 #14 0xffffffff80b03a46 in userland_sysctl (td=<value optimized out>, name=0xfffffe0468486900, namelen=6, old=0x0, oldlenp=<value optimized out>, inkernel=<value optimized out>, new=0x0, newlen=0, retval=0xfffffe0468486968, flags=0) at /usr/src/sys/kern/kern_sysctl.c:2011 #15 0xffffffff80b038cf in sys___sysctl (td=0xfffff80105056620, uap=0xfffff80105056b58) at /usr/src/sys/kern/kern_sysctl.c:1945 #16 0xffffffff80f79068 in amd64_syscall (td=0xfffff80105056620, traced=0) at subr_syscall.c:132 #17 0xffffffff80f5882d in fast_syscall_common () at /usr/src/sys/amd64/amd64/exception.S:479 #18 0x0000000801de047a in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) f 8 #8 0xffffffff80af2893 in __rw_rlock_hard (rw=0xfffff800be4bc990, td=0xfffff80105056620, v=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:493 493 owner = (struct thread *)RW_OWNER(v); Current language: auto; currently minimal (kgdb) f 9 #9 0xffffffff80c0ce9b in sysctl_dumpentry (rn=0xfffff80008e74270, vw=0xfffffe0468486690) at /usr/src/sys/net/rtsock.c:1559 1559 IF_ADDR_RLOCK(rt->rt_ifp); (kgdb) p rt->rt_ifp->if_addr_lock $1 = {lock_object = {lo_name = 0xfffff800be4bc9f0 "P?K?", lo_flags = 3192637744, lo_data = 4294965248, lo_witness = 0xfffff80007085848}, rw_lock = 256} (kgdb) p rt->rt_ifp->if_addr->ifa_addr Cannot access memory at address 0x3700000018 (kgdb) p *rt->rt_ifp $2 = { if_link = { tqe_next = 0xfffff800be9c9210, tqe_prev = 0xfffff800be9c9000 }, if_clones = { le_next = 0xfffff800be4bc870, le_prev = 0xfffff800be4bcb70 }, if_groups = { tqh_first = 0xfffff800be9c9048, tqh_last = 0x100 }, if_alloctype = 0 '\0', if_softc = 0xfffff800be9c9000, if_llsoftc = 0x3e50000, if_l2com = 0x400000004, if_dname = 0x0, if_dunit = 51, if_index = 36, if_index_reserved = 0, if_xname = 0xfffff800be4bc860 "\020>y\b", if_description = 0xfffff800be4bc8d0 "0?K?", if_flags = -1102329840, if_drv_flags = -2048, if_capabilities = 142163016, if_capenable = -2048, if_linkmib = 0x100, if_linkmiblen = 0, if_refcount = 142162944, if_type = 0 '\0', if_addrlen = 248 '?', if_hdrlen = 255 '?', if_link_state = 255 '?', if_mtu = 1078468608, if_metric = 0, if_baudrate = 2, if_hwassist = 0, if_epoch = 90194313239, if_lastchange = { tv_sec = -8796001543664, tv_usec = -8796001544192 }, if_snd = { ifq_head = 0xfffff800be4bc930, ifq_tail = 0xfffff800be4bc870, ifq_len = 91478088, ifq_maxlen = -2048, ifq_mtx = { lock_object = { lo_name = 0x100 <Address 0x100 out of bounds>, lo_flags = 0, lo_data = 0, lo_witness = 0xfffff8000573d800}, mtx_lock = 1079562240 }, ifq_drv_head = 0x2, ifq_drv_tail = 0x0, ifq_drv_len = 149, ifq_drv_maxlen = 21, altq_type = 141323792, altq_flags = -2048, altq_disc = 0xfffff800086c6c00, altq_ifp = 0xfffff800be4bc990, altq_enqueue = 0xfffff800be4bc8d0, altq_dequeue = 0xfffff800086c6c48, altq_request = 0x100, altq_clfier = 0x0, altq_classify = 0xfffff800086c6c00, altq_tbr = 0x84a000, altq_cdnr = 0x4 }, if_linktask = { ta_link = { stqe_next = 0x0}, ta_pending = 6, ta_priority = 0, ta_func = 0xfffff80007085a10, ta_context = 0xfffff80007085800 }, if_addr_lock = { lock_object = { lo_name = 0xfffff800be4bc9f0 "P?K?", lo_flags = 3192637744, lo_data = 4294965248, lo_witness = 0xfffff80007085848 }, rw_lock = 256 }, if_addrhead = { tqh_first = 0x0, tqh_last = 0xfffff80007085800 }, if_multiaddrs = { tqh_first = 0xf7d000, tqh_last = 0x4 }, if_amcount = 0, if_addr = 0x3700000018, if_broadcastaddr = 0xfffff80007090a10 "\001", if_afdata_lock = { lock_object = { lo_name = 0xfffff80007090800 "", lo_flags = 3192638032, lo_data = 4294965248, lo_witness = 0xfffff800be4bc990 }, rw_lock = 18446735277734561864 }, if_afdata = 0xfffff800be4bca08, if_afdata_initialized = 63, if_fib = 55, if_vnet = 0xfffff800be3dd610, if_home_vnet = 0xfffff800be3dd400, if_vlantrunk = 0xfffff800be4bc810, if_bpf = 0xfffff800be4bccf0, if_pcount = -1103244216, if_bridge = 0x100, if_lagg = 0x0, if_pf_kif = 0xfffff800be3dd400, if_carp = 0x220a000, if_label = 0x400000004, if_netmap = 0x0, if_output = 0x2400000039, if_input = 0xfffff80007075a10, if_start = 0xfffff80007075800, if_ioctl = 0xfffff800be4bcc30, if_init = 0xfffff800be4bcb10, if_resolvemulti = 0xfffff80007075848, if_qflush = 0x100, if_transmit = 0, if_reassign = 0xfffff80007075800, if_get_counter = 0x40460000, if_requestencap = 0x2, if_counters = 0xfffff800be4bcc10, if_hw_tsomax = 0, if_hw_tsomaxsegcount = 0, if_hw_tsomaxsegsize = 17, if_pspare = 0xfffff800be4bcc80, if_hw_addr = 0xfffff800be4bcc30, if_pcp = 72 'H', if_bspare = 0xfffff800be4bcca1 "?\b\a", if_ispare = 0xfffff800be4bcca4 }
Hi all, some additional information on this crash. The procedure that I describe in the previous post crashes consistently, which is a good point to start debugging. I suspect the crash to come from internal structures that are left corrupted at some point, after which there are several symptoms, like cannot intuit interface, no memory allocated, and ultimately a kernel crash. I have compiled a new kernel with DDB support enabled, hopping to be able to inspect memory at runtime, but the address of the (struct rtentry *) is different each time. Does anyone has an idea on how can I get the address that I need to watch to track who is modifying the routing table?
A commit references this bug: Author: ae Date: Tue Nov 27 09:04:06 UTC 2018 New revision: 341008 URL: https://svnweb.freebsd.org/changeset/base/341008 Log: Fix possible panic during ifnet detach in rtsock. The panic can happen, when some application does dump of routing table using sysctl interface. To prevent this, set IFF_DYING flag in if_detach_internal() function, when ifnet under lock is removed from the chain. In sysctl_rtsock() take IFNET_RLOCK_NOSLEEP() to prevent ifnet detach during routes enumeration. In case, if some interface was detached in the time before we take the lock, add the check, that ifnet is not DYING. This prevents access to memory that could be freed after ifnet is unlinked. PR: 227720, 230498, 233306 Reviewed by: bz, eugen MFC after: 1 week Sponsored by: Yandex LLC Differential Revision: https://reviews.freebsd.org/D18338 Changes: head/sys/net/if.c head/sys/net/rtsock.c
(In reply to Franck Rousseau from comment #37) > Thanks for the fast reply! Not sure if I continue here or in bug #230498 but > since this is still related to PPP, I put it here. > > I only had 15 min to test, but it crashed right away on the first try. Here > is the procedure: > - setup PC3: configure address on Ethernet interface; > - setup PC2: configure address on Ethernet interface, add ARP pub entry, > activate forwarding, start ppp server and wait for connection; > - setup PC3: start pinging PC3, obviously it fails, start ppp client and > open connection, add default route, everything works correctly. > Leave everything running as it is, then quit ppp on both sides, restart the > server waiting for the connection, connect from client -> crash on PC2. > > Here is the trace, it crashes one call further line rtsock.c:1559 after the > patch > > info.rti_info[RTAX_GENMASK] = 0; > if (rt->rt_ifp) { > - info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; > + IF_ADDR_RLOCK(rt->rt_ifp); > + if (rt->rt_ifp->if_addr != NULL) > + info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; > info.rti_info[RTAX_IFA] = rt->rt_ifa->ifa_addr; If this patch is full version that you used, you missed IF_ADDR_RUNLOCK() here, and this is why it panics. > #8 0xffffffff80af2893 in __rw_rlock_hard (rw=0xfffff800be4bc990, > td=0xfffff80105056620, v=<value optimized out>) at > /usr/src/sys/kern/kern_rwlock.c:493 > #9 0xffffffff80c0ce9b in sysctl_dumpentry (rn=0xfffff80008e74270, > vw=0xfffffe0468486690) at /usr/src/sys/net/rtsock.c:1559
(In reply to Andrey V. Elsukov from comment #40) The patch used was attachment #199064 [details] I have tried all proposed patches on 11.2 and 12, and the latest on the svn devel, none works.
(In reply to Franck Rousseau from comment #41) > (In reply to Andrey V. Elsukov from comment #40) > > The patch used was attachment #199064 [details] > > I have tried all proposed patches on 11.2 and 12, and the latest on the svn > devel, none works. Ok, can you do the following and then report back? Assume, you use 11.2. 1. cd /usr/src # (or path where are your source code is) 2. svnlite revert -R . 3. svnlite up 4. fetch -o rtsock.diff "https://bz-attachments.freebsd.org/attachment.cgi?id=199450&action=diff&format=raw&headers=1" 5. svnlite patch rtsock.diff 6. svnlite diff # (to be sure that all looks good) 7. make buildkernel installkernel 8. shutdown -r now 9. Try you test and if it will panic, post the panic message and backtrace from kgdb.
(In reply to Andrey V. Elsukov from comment #42) This is what I report in bug #230498 at comment #20 and at comment #37 in this thread. I did it again from a clean SVN repo as you asked to be sure of the conclusion. How to crash : - boot with the new kernel - ifconfig bge0 192.168.0.2 - ppp server then term, wait for ppp open from client, with local server address set to the same 192.168.0.2 - connection ok, it pings, then quit - restart ppp server then term, wait for ppp open from client, after getting PPp at the prompt, IP config is starting I guess, I get the crash, trying to access a NULL pointer In the dump: (kgdb) bt #0 doadump (textdump=1) at pcpu.h:229 #1 0xffffffff80b072a0 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:383 #2 0xffffffff80b076e1 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:776 #3 0xffffffff80b07523 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:707 #4 0xffffffff803aefc7 in db_panic (addr=<value optimized out>, have_addr=<value optimized out>, count=<value optimized out>, modif=<value optimized out>) at /usr/src/sys/ddb/db_command.c:499 #5 0xffffffff803ae539 in db_command (cmd_table=<value optimized out>) at /usr/src/sys/ddb/db_command.c:466 #6 0xffffffff803ae2b4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:519 #7 0xffffffff803b14ff in db_trap (type=<value optimized out>, code=<value optimized out>) at /usr/src/sys/ddb/db_main.c:248 #8 0xffffffff80b4ed63 in kdb_trap (type=12, code=0, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:689 #9 0xffffffff80f99501 in trap_fatal (frame=0xfffffe0467edd320, eva=0) at /usr/src/sys/amd64/amd64/trap.c:867 #10 0xffffffff80f99609 in trap_pfault (frame=0xfffffe0467edd320, usermode=0) at pcpu.h:229 #11 0xffffffff80f98dd7 in trap (frame=0xfffffe0467edd320) at /usr/src/sys/amd64/amd64/trap.c:415 #12 0xffffffff80f78e6c in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231 #13 0xffffffff80c24da4 in sysctl_dumpentry (rn=0xfffff80008954410, vw=0xfffffe0467edd690) at /usr/src/sys/net/rtsock.c:1559 #14 0xffffffff80c1f990 in rn_walktree (h=<value optimized out>, f=<value optimized out>, w=<value optimized out>) at /usr/src/sys/net/radix.c:1094 #15 0xffffffff80c246fb in sysctl_rtsock (oidp=<value optimized out>, arg1=<value optimized out>, arg2=<value optimized out>, req=<value optimized out>) at /usr/src/sys/net/rtsock.c:1917 #16 0xffffffff80b14a6b in sysctl_root_handler_locked (oid=0xffffffff81a690d8, arg1=0xfffffe0467edd908, arg2=4, req=0xfffffe0467edd840, tracker=0xfffffe0467edd7b8) at /usr/src/sys/kern/kern_sysctl.c:165 #17 0xffffffff80b142c1 in sysctl_root (arg1=0xfffffe0467edd908, arg2=4) at /usr/src/sys/kern/kern_sysctl.c:1915 #18 0xffffffff80b147e6 in userland_sysctl (td=<value optimized out>, name=0xfffffe0467edd900, namelen=6, old=0x0, oldlenp=<value optimized out>, inkernel=<value optimized out>, new=0x0, newlen=0, retval=0xfffffe0467edd968, flags=0) at /usr/src/sys/kern/kern_sysctl.c:2011 #19 0xffffffff80b1466f in sys___sysctl (td=0xfffff80008837000, uap=0xfffff80008837538) at /usr/src/sys/kern/kern_sysctl.c:1945 #20 0xffffffff80f9a638 in amd64_syscall (td=0xfffff80008837000, traced=0) at subr_syscall.c:132 #21 0xffffffff80f796bd in fast_syscall_common () at /usr/src/sys/amd64/amd64/exception.S:479 #22 0x0000000801de047a in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) f 13 #13 0xffffffff80c24da4 in sysctl_dumpentry (rn=0xfffff80008954410, vw=0xfffffe0467edd690) at /usr/src/sys/net/rtsock.c:1559 1559 info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; (kgdb) print rt->rt_ifp->if_addr $1 = (struct ifaddr *) 0x0 (kgdb) print rt->rt_ifp->if_flags $2 = 0 (kgdb) print rt->rt_ifp->if_index $3 = 0 (kgdb) print rt->rt_ifp $4 = (struct ifnet *) 0xfffff8002be6c800 (kgdb) print *rt->rt_ifp $5 = {if_link = {tqe_next = 0xfffff800b0cfe050, tqe_prev = 0xfffff800b0cfe0a0}, if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {tqh_first = 0x0, tqh_last = 0x0}, if_alloctype = 0 '\0', if_softc = 0x0, if_llsoftc = 0x0, if_l2com = 0x0, if_dname = 0x0, if_dunit = 0, if_index = 0, if_index_reserved = 0, if_xname = 0xfffff8002be6c860 "", if_description = 0x0, if_flags = 0, if_drv_flags = 0, if_capabilities = -1325336224, if_capenable = -2048, if_linkmib = 0xfffff800b100f9b0, if_linkmiblen = 18446735280583750992, if_refcount = 2967221664, if_type = 0 '\0', if_addrlen = 248 '�', if_hdrlen = 255 '�', if_link_state = 255 '�', if_mtu = 2967221744, if_metric = 4294965248, if_baudrate = 18446735280583751232, if_hwassist = 18446735280582943280, if_epoch = -8793126608256, if_lastchange = { tv_sec = -8793126608176, tv_usec = 0}, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 0, ifq_mtx = {lock_object = {lo_name = 0x0, lo_flags = 503152064, lo_data = 4294965252, lo_witness = 0xfffff800053ee3c0}, mtx_lock = 18446735277704537104}, ifq_drv_head = 0xfffff800053ee460, ifq_drv_tail = 0x0, ifq_drv_len = -1326900496, ifq_drv_maxlen = -2048, altq_type = -1326900416, altq_flags = -2048, altq_disc = 0xfffff800b0cfe320, altq_ifp = 0xfffff800b0cfe370, altq_enqueue = 0xfffff800b0cfe3c0, altq_dequeue = 0xfffff800b0cfe410, altq_request = 0xfffff800b0dc3870, altq_clfier = 0xfffff800b100f8c0, altq_classify = 0xfffff800b100f910, altq_tbr = 0x0, altq_cdnr = 0x0}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xfffff800b100fa00, ta_context = 0x0}, if_addr_lock = {lock_object = {lo_name = 0xfffff800b0b8a1e0 "\200}�\035\004���\220���", lo_flags = 2964890160, lo_data = 4294965248, lo_witness = 0xfffff800b0b8a280}, rw_lock = 18446735280581419728}, if_addrhead = {tqh_first = 0x0, tqh_last = 0xfffff800b1044960}, if_multiaddrs = {tqh_first = 0x0, tqh_last = 0x0}, if_amcount = 0, if_addr = 0x0, if_broadcastaddr = 0xfffff800b0e91d70 "\200}�\035\004����\033��", if_afdata_lock = { lock_object = {lo_name = 0xfffff800b0e91dc0 "\200}�\035\004���p\035��", lo_flags = 2967222464, lo_data = 4294965248, lo_witness = 0xfffff800b0dc3910}, rw_lock = 18446735280583752032}, if_afdata = 0xfffff8002be6ca08, if_afdata_initialized = -1330076256, if_fib = 4294965248, if_vnet = 0xfffff800b0b8a5f0, if_home_vnet = 0xfffff800b0b8a640, if_vlantrunk = 0xfffff800b100fe60, if_bpf = 0xfffff800b100feb0, if_pcount = -1325334784, if_bridge = 0xfffff800b100ff50, if_lagg = 0x0, if_pf_kif = 0xfffff800b1072000, if_carp = 0xfffff800b1072050, if_label = 0xfffff800b10720a0, if_netmap = 0xfffff800b0b8a690, if_output = 0xfffff800b0b8a6e0, if_input = 0xfffff800b0b8a730, if_start = 0xfffff800b0f5c280, if_ioctl = 0xfffff800b0f5c2d0, if_init = 0, if_resolvemulti = 0, if_qflush = 0xfffff800b0cfea00, if_transmit = 0xfffff800b0cfea50, if_reassign = 0xfffff800b0cfeaa0, if_get_counter = 0xfffff800b0dc3f50, if_requestencap = 0xfffff800b1072320, if_counters = 0xfffff8002be6cc10, if_hw_tsomax = 2968896528, if_hw_tsomaxsegcount = 4294965248, if_hw_tsomaxsegsize = 2970036096, if_pspare = 0xfffff8002be6cc80, if_hw_addr = 0xfffff800b0cfebe0, if_pcp = 160 '�', if_bspare = 0xfffff8002be6cca1 "\020��", if_ispare = 0xfffff8002be6cca4}
(In reply to Franck Rousseau from comment #43) > (In reply to Andrey V. Elsukov from comment #42) > > This is what I report in bug #230498 at comment #20 and at comment #37 in > this thread. I did it again from a clean SVN repo as you asked to be sure of > the conclusion. > > How to crash : > - boot with the new kernel > - ifconfig bge0 192.168.0.2 > - ppp server then term, wait for ppp open from client, with local server > address set to the same 192.168.0.2 > - connection ok, it pings, then quit > - restart ppp server then term, wait for ppp open from client, after getting > PPp at the prompt, IP config is starting I guess, I get the crash, trying to > access a NULL pointer Can you show the output of these commands: # cd /usr/src # svnlite info # svnlite diff
(In reply to Andrey V. Elsukov from comment #44) [/usr/src]# svnlite info Path: . Working Copy Root Path: /usr/src URL: https://svn.freebsd.org/base/releng/11.2 Relative URL: ^/releng/11.2 Repository Root: https://svn.freebsd.org/base Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 341162 Node Kind: directory Schedule: normal Last Changed Author: gordon Last Changed Rev: 341093 Last Changed Date: 2018-11-27 20:45:25 +0100 (Tue, 27 Nov 2018) [/usr/src]# svnlite diff Index: sys/amd64/conf/GENERIC =================================================================== --- sys/amd64/conf/GENERIC (revision 341162) +++ sys/amd64/conf/GENERIC (working copy) @@ -82,6 +82,8 @@ # Debugging support. Always need this: options KDB # Enable kernel debugger support. options KDB_TRACE # Print a stack trace for a panic. +options DDB # Support DDB. +options GDB # Support remote GDB. # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel Index: sys/net/if.c =================================================================== --- sys/net/if.c (revision 341162) +++ sys/net/if.c (working copy) @@ -1032,6 +1032,8 @@ if (iter == ifp) { TAILQ_REMOVE(&V_ifnet, ifp, if_link); found = 1; + if (!vmove) + ifp->if_flags |= IFF_DYING; break; } IFNET_WUNLOCK(); Index: sys/net/rtsock.c =================================================================== --- sys/net/rtsock.c (revision 341162) +++ sys/net/rtsock.c (working copy) @@ -1555,7 +1555,7 @@ info.rti_info[RTAX_NETMASK] = rtsock_fix_netmask(rt_key(rt), rt_mask(rt), &ss); info.rti_info[RTAX_GENMASK] = 0; - if (rt->rt_ifp) { + if (rt->rt_ifp && !(rt->rt_ifp->if_flags & IFF_DYING)) { info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; info.rti_info[RTAX_IFA] = rt->rt_ifa->ifa_addr; if (rt->rt_ifp->if_flags & IFF_POINTOPOINT) @@ -1913,8 +1913,10 @@ rnh = rt_tables_get_rnh(fib, i); if (rnh != NULL) { RIB_RLOCK(rnh); + IFNET_RLOCK_NOSLEEP(); error = rnh->rnh_walktree(&rnh->head, sysctl_dumpentry, &w); + IFNET_RUNLOCK_NOSLEEP(); RIB_RUNLOCK(rnh); } else if (af != 0) error = EAFNOSUPPORT;
(In reply to Franck Rousseau from comment #43) > #13 0xffffffff80c24da4 in sysctl_dumpentry (rn=0xfffff80008954410, > vw=0xfffffe0467edd690) > at /usr/src/sys/net/rtsock.c:1559 > 1559 info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; > (kgdb) print rt->rt_ifp->if_addr > $1 = (struct ifaddr *) 0x0 > (kgdb) print rt->rt_ifp->if_flags > $2 = 0 > (kgdb) print rt->rt_ifp->if_index > $3 = 0 > (kgdb) print rt->rt_ifp > $4 = (struct ifnet *) 0xfffff8002be6c800 > (kgdb) print *rt->rt_ifp > $5 = {if_link = {tqe_next = 0xfffff800b0cfe050, tqe_prev = > 0xfffff800b0cfe0a0}, if_clones = {le_next = 0x0, > le_prev = 0x0}, if_groups = {tqh_first = 0x0, tqh_last = 0x0}, > if_alloctype = 0 '\0', if_softc = 0x0, > if_llsoftc = 0x0, if_l2com = 0x0, if_dname = 0x0, if_dunit = 0, if_index = > 0, if_index_reserved = 0, > if_xname = 0xfffff8002be6c860 "", if_description = 0x0, if_flags = 0, > if_drv_flags = 0, Ok, it seems all was correctly patched and the problem is because we have garbage in the ifnet pointer.
Please take a look. Maybe this is similar bug. #uname -a FreeBSD vpn01 11.3-RELEASE FreeBSD 11.3-RELEASE #0: Thu Sep 5 10:41:12 MSK 2019 root@vpn01:/usr/obj/usr/src/sys/VPN01 amd64 # diff /root/kernels/VPN01 /usr/src/sys/amd64/conf/GENERIC < ident VPN01 > ident GENERIC --- < #options INET6 # IPv6 communications protocols > options INET6 # IPv6 communications protocols --- < #device lpt # Printer > device lpt # Printer --- < #device snd_cmi # CMedia CMI8338/CMI8738 < #device snd_csa # Crystal Semiconductor CS461x/428x < #device snd_emu10kx # Creative SoundBlaster Live! and Audigy < #device snd_es137x # Ensoniq AudioPCI ES137x < #device snd_hda # Intel High Definition Audio < #device snd_ich # Intel, NVidia and other ICH AC'97 Audio < #device snd_via8233 # VIA VT8233x Audio --- > device snd_cmi # CMedia CMI8338/CMI8738 > device snd_csa # Crystal Semiconductor CS461x/428x > device snd_emu10kx # Creative SoundBlaster Live! and Audigy > device snd_es137x # Ensoniq AudioPCI ES137x > device snd_hda # Intel High Definition Audio > device snd_ich # Intel, NVidia and other ICH AC'97 Audio > device snd_via8233 # VIA VT8233x Audio --- < options INCLUDE_CONFIG_FILE # Include this file in kernel < options KDB # Kernel debugger related code < options KDB_TRACE # Print a stack trace for a panic --- < options IPFIREWALL < options IPFIREWALL_VERBOSE < options IPFIREWALL_VERBOSE_LIMIT=1000 < options IPFIREWALL_DEFAULT_TO_ACCEPT < options DUMMYNET < options IPDIVERT < options IPFILTER < options IPFILTER_LOG < options IPFILTER_LOOKUP < options IPSTEALTH --- < options NETGRAPH < options NETGRAPH_SOCKET < options NETGRAPH_IPFW < options NETGRAPH_ETHER < options NETGRAPH_BPF < options NETGRAPH_PPPOE # kgdb kernel.debug /var/crash/vmcore.2 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 25; apic id = 21 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80c21212 stack pointer = 0x28:0xfffffe085c59e3e0 frame pointer = 0x28:0xfffffe085c59e520 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 18740 (ppp) trap number = 12 panic: page fault cpuid = 25 KDB: stack backtrace: #0 0xffffffff80b51a07 at kdb_backtrace+0x67 #1 0xffffffff80b0aa1e at vpanic+0x17e #2 0xffffffff80b0a893 at panic+0x43 #3 0xffffffff80f854f9 at trap_fatal+0x369 #4 0xffffffff80f85559 at trap_pfault+0x49 #5 0xffffffff80f84bdd at trap+0x29d #6 0xffffffff80f649cc at calltrap+0x8 #7 0xffffffff80c1bf80 at rn_walktree+0x80 #8 0xffffffff80c20b83 at sysctl_rtsock+0x1f3 #9 0xffffffff80b17e8b at sysctl_root_handler_locked+0x8b #10 0xffffffff80b176e2 at sysctl_root+0x1f2 #11 0xffffffff80b17c06 at userland_sysctl+0x136 #12 0xffffffff80b17a8f at sys___sysctl+0x5f #13 0xffffffff80f865f6 at amd64_syscall+0xa86 #14 0xffffffff80f652ad at fast_syscall_common+0x101 Uptime: 7m55s Dumping 1351 out of 32604 MB:..2%..11%..21%..31%..41%..51%..61%..72%..81%..92% #0 doadump () at pcpu.h:234 234 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) list *0xffffffff80c21212 0xffffffff80c21212 is in sysctl_dumpentry (/usr/src/sys/net/rtsock.c:1566). 1561 info.rti_info[RTAX_GATEWAY] = rt->rt_gateway; 1562 info.rti_info[RTAX_NETMASK] = rtsock_fix_netmask(rt_key(rt), 1563 rt_mask(rt), &ss); 1564 info.rti_info[RTAX_GENMASK] = 0; 1565 if (rt->rt_ifp && !(rt->rt_ifp->if_flags & IFF_DYING)) { 1566 info.rti_info[RTAX_IFP] = rt->rt_ifp->if_addr->ifa_addr; 1567 info.rti_info[RTAX_IFA] = rt->rt_ifa->ifa_addr; 1568 if (rt->rt_ifp->if_flags & IFF_POINTOPOINT) 1569 info.rti_info[RTAX_BRD] = rt->rt_ifa->ifa_dstaddr; 1570 } Current language: auto; currently minimal (kgdb) backtrace #0 doadump () at pcpu.h:234 #1 0xffffffff80b0a638 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:388 #2 0xffffffff80b0aa58 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:781 #3 0xffffffff80b0a893 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:712 #4 0xffffffff80f854f9 in trap_fatal (frame=0xfffffe085c59e320, eva=0) at /usr/src/sys/amd64/amd64/trap.c:904 #5 0xffffffff80f85559 in trap_pfault (frame=0xfffffe085c59e320, usermode=0) at pcpu.h:234 #6 0xffffffff80f84bdd in trap (frame=0xfffffe085c59e320) at /usr/src/sys/amd64/amd64/trap.c:438 #7 0xffffffff80f649cc in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231 #8 0xffffffff80c21212 in sysctl_dumpentry (rn=0xfffff80188cef000, vw=0xfffffe085c59e690) at /usr/src/sys/net/rtsock.c:1566 #9 0xffffffff80c1bf80 in rn_walktree (h=<value optimized out>, f=0xffffffff80c21110 <sysctl_dumpentry>, w=0xfffffe085c59e690) at /usr/src/sys/net/radix.c:1094 #10 0xffffffff80c20b83 in sysctl_rtsock (oidp=<value optimized out>, arg1=<value optimized out>, arg2=<value optimized out>, req=<value optimized out>) at /usr/src/sys/net/rtsock.c:1931 #11 0xffffffff80b17e8b in sysctl_root_handler_locked (oid=0xffffffff81a51b28, arg1=0xfffffe085c59e908, arg2=4, req=0xfffffe085c59e840, tracker=0xfffffe085c59e7b8) at /usr/src/sys/kern/kern_sysctl.c:165 #12 0xffffffff80b176e2 in sysctl_root (arg1=0xfffffe085c59e908, arg2=4, req=0xfffffe085c59e840) at /usr/src/sys/kern/kern_sysctl.c:1915 #13 0xffffffff80b17c06 in userland_sysctl (td=<value optimized out>, name=0xfffffe085c59e900, namelen=6, old=0x0, oldlenp=<value optimized out>, inkernel=<value optimized out>, new=0x0, newlen=0, retval=0xfffffe085c59e968, flags=0) at /usr/src/sys/kern/kern_sysctl.c:2011 #14 0xffffffff80b17a8f in sys___sysctl (td=0xfffff80233776620, uap=0xfffff80233776b58) at /usr/src/sys/kern/kern_sysctl.c:1945 #15 0xffffffff80f865f6 in amd64_syscall (td=0xfffff80233776620, traced=0) at src/sys/amd64/amd64/../../kern/subr_syscall.c:132 #16 0xffffffff80f652ad in fast_syscall_common () at /usr/src/sys/amd64/amd64/exception.S:494 #17 0x0000000801de0bba in ?? () Previous frame inner to this frame (corrupt stack?)
Fixed with https://svnweb.freebsd.org/base?view=revision&revision=341677 and merged 11.3-RELEASE and 12.1-RELEASE.
*** Bug 230498 has been marked as a duplicate of this bug. ***