I have two routers running HEAD with IPSEC that have segmentation faults running 'netstat -s'. `netstat -s` output: .... 0 datagrams output pfkey: Segmentation fault `dmesg` output: FreeBSD xts-rtr 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r284194: Tue Jun 9 23:38:40 UTC 2015 root@xts-bsd:/usr/obj/nanobsd.soekris/i386.i386/usr/src/sys/IPSEC i386 FreeBSD xju-rtr 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r285668: Sat Jul 18 06:42:52 UTC 2015 root@xts-bsd:/usr/obj/nanobsd.soekris/i386.i386/usr/src/head/sys/IPSEC i386 /var/log/messages content: Jul 19 22:23:20 xju-rtr kernel: pid 49306 (netstat), uid 1986: exited on signal 11
Created attachment 158989 [details] patch There seem to be some bugs in the libxo-ification of pfkey.c. I don't get a segfault, but a couple of fields are mangled. Could you give the attached patch a try?
Created attachment 158990 [details] patch Oops, here's the actual patch this time.
(In reply to Mark Johnston from comment #2) Mark, Thanks again for the quick response! Your patch worked for me for what I opened the PR for. This was on the following `uname -a` with your patch FreeBSD xju-rtr 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r285706: Mon Jul 20 00:10:44 UTC 2015 root@xts-bsd:/usr/obj/nanobsd.soekris/i386.i386/usr/src/head/sys/IPSEC i386 And the resulting `netstat -s` output: ..... 0 datagrams output pfkey: 10 requests sent from userland 0 bytes sent from userland histogram by message type: getspi: 1 acquire: 2 #132: 1392 #133: 1 #134: 2 #136: 2 #139: 2 0 messages with invalid length field 0 messages with invalid version field 0 messages with invalid message type field 0 messages too short 0 messages with memory allocation failure 7 messages with duplicate extension 0 messages with invalid extension type 0 messages with invalid sa type 0 messages with invalid address extension 0 requests sent to userland 0 bytes sent to userland histogram by message type: expire: 32 x_spddump: 24 x_spdflush: 24 x_spdexpire: 592 #164: 14976973269188063411 0 messages toward single socket 0 messages toward all sockets 0 messages toward registered sockets 0 messages with memory allocation failure
(In reply to Jason Unovitch from comment #3) Regarding the output... does this look a sign of another issue with random memory being read? histogram by message type: ..... #164: 14976973269188063411
(In reply to Jason Unovitch from comment #4) That's quite interesting. I don't see that on my workstation (all counters are 0), but I tried booting up an i386 VM with IPSEC enabled in the kernel, and lo and behold: pfkey: 0 requests sent from userland 0 bytes sent from userland 0 messages with invalid length field 0 messages with invalid version field 0 messages with invalid message type field 0 messages too short 0 messages with memory allocation failure 0 messages with duplicate extension 0 messages with invalid extension type 0 messages with invalid sa type 0 messages with invalid address extension 0 requests sent to userland 0 bytes sent to userland histogram by message type: #164: 8513580559518937276 0 messages toward single socket 0 messages toward all sockets 0 messages toward registered sockets 0 messages with memory allocation failure I don't see any obvious problems on the userland side of things. Given that this is happening at the same index on both of our systems, it seems more likely that this is a kernel issue. I'll try and track it down.
(In reply to Mark Johnston from comment #5) Super! Well your patch does fix the segmentation fault on i386 so that looks good to commit. If you have any extra patches to test for this issue then let me know and I'll test it out for you.
A commit references this bug: Author: markj Date: Mon Jul 20 02:38:52 UTC 2015 New revision: 285708 URL: https://svnweb.freebsd.org/changeset/base/285708 Log: Fix some libxo format string errors in the pfkey stats code. PR: 201700 Changes: head/usr.bin/netstat/pfkey.c
Created attachment 159030 [details] patch This turned out to be another bug in netstat. Some changes were made to the way protocol counters were fetched from the kernel that broke on 32-bit systems. It only affected the IPsec counters since the rest are fetched using sysctls. Could you give the attached patch a try?
(In reply to Mark Johnston from comment #8) Looks good to me. Thanks again Mark! Tested on `uname -a` plus your patch: FreeBSD xju-rtr 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r285727: Mon Jul 20 23:37:39 UTC 2015 root@xts-bsd:/usr/obj/nanobsd.soekris/i386.i386/usr/src/head/sys/IPSEC i386 The resulting `netstat -s` output: ..... (truncated for brevity) ..... 0 datagrams output pfkey: 1pfkey: 10 requests sent from userland 1120 bytes sent from userland histogram by message type: getspi: 1 add: 2 delete: 1 register: 2 x_spdupdate: 2 x_spdadd: 2 0 messages with invalid length field 0 messages with invalid version field 0 messages with invalid message type field 0 messages too short 0 messages with memory allocation failure 0 messages with duplicate extension 0 messages with invalid extension type 0 messages with invalid sa type 0 messages with invalid address extension 10 requests sent to userland 1392 bytes sent to userland histogram by message type: getspi: 1 add: 2 delete: 1 register: 2 x_spdupdate: 2 x_spdadd: 2 1 message toward single socket 7 messages toward all sockets 2 messages toward registered sockets 0 messages with memory allocation failure
(In reply to Jason Unovitch from comment #9) The spacing didn't look quite right in the Bugzilla window so I repasted the output and munged the start of the output. It is correct so disregard the copy/paste mishap at the top of the original ..... (truncated for brevity) ..... 0 datagrams output pfkey: 10 requests sent from userland 1120 bytes sent from userland histogram by message type: getspi: 1 add: 2 delete: 1 register: 2 x_spdupdate: 2 x_spdadd: 2 0 messages with invalid length field 0 messages with invalid version field 0 messages with invalid message type field 0 messages too short 0 messages with memory allocation failure 0 messages with duplicate extension 0 messages with invalid extension type 0 messages with invalid sa type 0 messages with invalid address extension 10 requests sent to userland 1392 bytes sent to userland histogram by message type: getspi: 1 add: 2 delete: 1 register: 2 x_spdupdate: 2 x_spdadd: 2 1 message toward single socket 7 messages toward all sockets 2 messages toward registered sockets 0 messages with memory allocation failure
A commit references this bug: Author: markj Date: Tue Jul 21 23:57:39 UTC 2015 New revision: 285782 URL: https://svnweb.freebsd.org/changeset/base/285782 Log: Fix counter reads on platforms where sizeof(uint64_t) != sizeof(uint64_t *). In the kernel, structs such as tcpstat are manipulated as an array of counter_u64_t (uint64_t *), but made visible to userland as an array of uint64_t. kread_counters() was previously copying the counter array into user space and sequentially overwriting each counter with its value. This mostly affects IPsec counters, as other counters are exported via sysctl. PR: 201700 Tested by: Jason Unovitch MFC after: 1 week Changes: head/usr.bin/netstat/main.c
A commit references this bug: Author: markj Date: Fri Jul 31 00:21:41 UTC 2015 New revision: 286099 URL: https://svnweb.freebsd.org/changeset/base/286099 Log: MFC r285782: Fix counter reads on platforms where sizeof(uint64_t) != sizeof(uint64_t *). PR: 201700 Changes: _U stable/10/ stable/10/usr.bin/netstat/main.c
The counter issue is fixed in stable/10; the issue doesn't affect 9 or earlier. I'm trying to get that fix into 10.2, but it may be addressed by an EN after the release instead. Thanks for the report!
Also fixed in 10.2-RELEASE as of FreeBSD-EN-15:12.netstat.