Created attachment 248509 [details] ktrace pfctl -sr under jail Hi, I have an assertion error on pfctl inside the my jail. apparently, my pf is still working. However, I get the following error when I run: # pfctl -sr Assertion failed: (elems <= maxelems), function pf_nvuint_32_array, file /usr/src/lib/libpfctl/libpfctl.c, line 147. Tested with: # freebsd-version -kru 14.0-STABLE 14.0-STABLE 14.0-STABLE :/usr/src # git show commit 6a75d3b3fbe4c66bf9b8c18db55bba19ffb492e4 (HEAD -> stable/14, origin/stable/14) related loader.conf: tcp_bbr_load="YES" net.fibs="3" Jail configuration: # cat /etc/jail.conf.d/jail.conf path = "/usr/jails/${name}"; host.hostname = "${name}"; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.clean; somejail { mount.devfs; vnet; vnet.interface = "epair4b"; exec.prestart = "service netif cloneup epair4 || echo interfaces are already exists"; exec.prestop = "ifconfig epair4a destroy"; } My kernel conf: # cat /usr/src/sys/amd64/conf/RACK include GENERIC ident RACK device pf device pflog device cryptodev options ALTQ options ALTQ_HFSC makeoptions WITH_EXTRA_TCP_STACKS=1 options RATELIMIT options TCPHPTS options ZFS options NETGRAPH options NETGRAPH_ECHO options NETGRAPH_ETHER options NETGRAPH_IFACE options NETGRAPH_KSOCKET options NETGRAPH_TEE options NETGRAPH_SOCKET options NETGRAPH_NETFLOW options NETGRAPH_EIFACE options NETGRAPH_BRIDGE makeoptions DEBUG=-g makeoptions WITH_CTF=1 options KDTRACE_FRAME options KDTRACE_HOOKS options DDB_CTF `/etc/sysctl.conf` under jail: net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.cc.algorithm=htcp net.inet.tcp.functions_default=bbr net.inet.ip.redirect=0 This jail works as a network gateway for other jails. It was working, strangely that happens after I use the `py39-sshuttle` on it. FYI, `py39-sshuttle` has been tested on the other jails under similar configuration, and this error was not happened before (on the same host). For the context, AFAICS, sshuttle only adds an anchor to pf which only cantains two rules. https://github.com/sshuttle/sshuttle/blob/master/sshuttle/methods/pf.py Sample on other jails on the same time: # pfctl -a 'sshuttle-12300' -sr pass out route-to lo0 inet proto tcp all flags S/SA keep state pass out inet proto tcp from any to 127.0.0.1 flags S/SA keep state my `pf.conf` under that jail is fairly simple: multiple: nat pass from x.x.x.x/24 to any -> $SOMEIF pass all block from x.x.x.x/24 to any
Almost certainly fixed by this: commit 228ae54114e1efbe82686090bed9d2c8071ecea0 Author: Kristof Provost <kp@FreeBSD.org> Date: Fri Oct 27 14:13:57 2023 +0200 libpfctl: be more tolerant of kernel extensions Allow the kernel to supply more array elements than expected, but cut off when we hit what we think the maximum is. This will improve forward compatibility (i.e. old userspace with newer kernel). Reviewed by: zlei MFC after: 1 week Sponsored by: Orange Business Services Differential Revision: https://reviews.freebsd.org/D42392 (cherry picked from commit 2b1eb63fc9c6d6f64baaac59b7ea7c2a3228c03f) I suspect you're running a more recent kernel than userspace (which ought to work, but it the cause of this issue).
This bug is seen when running a 14.0-RELEASEp6 vnet jail on a 14.1-RELEASE host, during the execution of /etc/periodic/security/520.pfdenied. Updating the jail to 14.1-RELEASE fixes it, so I'm not sure any action is called for, but at least it's not an "affects only me".
Just reporting I see the same in a 13.2 VNET jail running on a 13.3-p1 host: [root@haproxy01 /home/spork]# pfctl -sr Assertion failed: (elems <= maxelems), function pf_nvuint_32_array, file /usr/src/lib/libpfctl/libpfctl.c, line 147. Abort trap (core dumped) [root@haproxy01 /home/spork]# I can provide more info if needed but it certainly looks like the same issue. I don't see any note of this in the -p2 update...
Seeing this on 14.1-p1 (running inside a bhyve VM hosted on 14.0-p7, FWIW) With freshly removed/installed libpfctl+pftop packages from latest: Assertion failed: (elems <= maxelems), function pf_nvuint_32_array, file libpfctl.c, line 153. Looking at the core dump: (lldb) bt * thread #1, name = 'pftop', stop reason = signal SIGABRT * frame #0: 0x00002a9ebc67b10a libc.so.7`__sys_thr_kill + 10 frame #1: 0x00002a9ebc5f4404 libc.so.7`__raise + 52 frame #2: 0x00002a9ebc6a79d9 libc.so.7`abort + 73 frame #3: 0x00002a9ebc5d75f1 libc.so.7`__assert + 81 frame #4: 0x00002a9ebb317189 libpfctl.so.0`pfctl_get_clear_rule + 3113 frame #5: 0x00002a9ebb31654e libpfctl.so.0`pfctl_get_rule + 14 frame #6: 0x00002a96985ba7f3 pftop`___lldb_unnamed_symbol217 + 291 frame #7: 0x00002a96985ba9fe pftop`___lldb_unnamed_symbol220 + 190 frame #8: 0x00002a96985b983a pftop`___lldb_unnamed_symbol202 + 106 frame #9: 0x00002a96985bf2ea pftop`___lldb_unnamed_symbol300 + 90 frame #10: 0x00002a96985bc1a8 pftop`___lldb_unnamed_symbol233 + 1176 frame #11: 0x00002a9ebc5c8a6a libc.so.7`__libc_start1 + 298 frame #12: 0x00002a96985b87ed pftop`___lldb_unnamed_symbol182 + 45 If I locally recompile / reinstall libpfctl+pftop with symbols, I get a different error: * thread #1, name = 'pftop', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x2d3) frame #0: 0x00002a6604b13f36 libc.so.7`strcasecmp_l + 118 libc.so.7`strcasecmp_l: -> 0x2a6604b13f36 <+118>: movzbl (%rax,%r13), %r12d 0x2a6604b13f3b <+123>: movq %rbx, %rdi 0x2a6604b13f3e <+126>: leaq -0x34(%rbp), %rsi 0x2a6604b13f42 <+130>: callq 0x2a6604b18550 ; symbol stub for: __runes_for_locale (lldb) bt * thread #1, name = 'pftop', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x2d3) * frame #0: 0x00002a6604b13f36 libc.so.7`strcasecmp_l + 118 frame #1: 0x00002a6604aaffd6 libc.so.7`bsearch + 70 frame #2: 0x00002a6604a384b0 libc.so.7`_nsdispatch + 1200 frame #3: 0x00002a6604a316f6 libc.so.7`getprotobynumber + 118 frame #4: 0x00002a5de143276c pftop`print_rule(pr=0x000045b37d7ab040) at pftop.c:1440:24 frame #5: 0x00002a5de142ff53 pftop`print_rules at pftop.c:1573:3 frame #6: 0x00002a5de1437e8c pftop`disp_update at engine.c:883:4 frame #7: 0x00002a5de14389eb pftop`engine_loop(countmax=0) at engine.c:1213:4 frame #8: 0x00002a5de1434929 pftop`main(argc=0, argv=0x00002a660178cb40) at pftop.c:2322:2 frame #9: 0x00002a66049d0a6a libc.so.7`__libc_start1 + 298 frame #10: 0x00002a5de142ee5d pftop`_start at crt1_s.S:83 (lldb) fr sel 4 frame #4: 0x00002a5de143276c pftop`print_rule(pr=0x000045b37d7ab040) at pftop.c:1440:24 1437 else print_fld_str(FLD_ACTION, actiontypes[pr->action]); 1438 1439 if (pr->proto) { -> 1440 struct protoent *p = getprotobynumber(pr->proto); 1441 1442 if (p != NULL) 1443 print_fld_str(FLD_PROTO, p->p_name); (lldb) print pr->proto (uint8_t) '\x11'
Possibly related: bug#275054
(In reply to eborisch+FreeBSD from comment #4) What version of the libpfctl package do you have installed, because current versions should have the relevant fix.
(In reply to Kristof Provost from comment #6) libpfctl/pftop are 0.11 and 0.10_1; built on the buildbots (FreeBSD_version: 1400097) on 6/20 and 6/21, respectively: # pkg info libpfctl libpfctl-0.11 Name : libpfctl Version : 0.11 Installed on : Thu Jun 27 10:39:39 2024 CDT Origin : net/libpfctl Architecture : FreeBSD:14:amd64 Prefix : /usr/local Categories : net Licenses : BSD2CLAUSE Maintainer : kp@FreeBSD.org WWW : https://cgit.freebsd.org/src/tree/lib/libpfctl Comment : Library for interaction with pf(4) Shared Libs provided: libpfctl.so.0 Annotations : FreeBSD_version: 1400097 build_timestamp: 2024-06-20T01:05:11+0000 built_by : poudriere-git-3.4.1-30-g79e3edcd port_checkout_unclean: no port_git_hash : a40e26254 ports_top_checkout_unclean: no ports_top_git_hash: f55872119 repo_type : binary repository : FreeBSD Flat size : 40.9KiB Description : Ports version of the base system pf(4) library, libpfctl. Ports should ideally use this interface rather than manually implementing the pf(4) ioctl() interface. # pkg info pftop pftop-0.10_1 Name : pftop Version : 0.10_1 Installed on : Thu Jun 27 10:39:39 2024 CDT Origin : sysutils/pftop Architecture : FreeBSD:14:amd64 Prefix : /usr/local Categories : sysutils net Licenses : BSD2CLAUSE Maintainer : grembo@FreeBSD.org WWW : https://github.com/grembo/pftop/ Comment : Utility for real-time display of statistics for pf Options : ALTQ : off Shared Libs required: libpfctl.so.0 Annotations : FreeBSD_version: 1400097 build_timestamp: 2024-06-21T09:03:54+0000 built_by : poudriere-git-3.4.1-30-g79e3edcd port_checkout_unclean: no port_git_hash : a40e26254 ports_top_checkout_unclean: no ports_top_git_hash: f55872119 repo_type : binary repository : FreeBSD Flat size : 151KiB Description : Pftop is a small, curses-based utility for real-time display of active states and rule statistics for pf, the packet filter. If I rebuild locally, I get the slightly different error; these have the 'FreeBSD_version: 1401000' annotation in pkg, but otherwise the same version.
On 14.0-p7 (with buildbot-provided pkgs libpfctl-0.11 and pftop-0.10_1) I'm getting the same error as I am on 14.1 with locally-rebuilt: # pftop pfTop: Up State 1-8/8, View: default, Order: none, Cache: 10000 12:20:53 [Pressing 5 or 6 gets the same result:] Segmentation fault (core dumped) # lldb -c pftop.core /usr/local/sbin/pftop (lldb) target create "/usr/local/sbin/pftop" --core "pftop.core" Core file '/root/pftop.core' (x86_64) was loaded. (lldb) bt * thread #1, name = 'pftop', stop reason = signal SIGSEGV * frame #0: 0x00002d8520600d96 libc.so.7`strcasecmp_l + 118 frame #1: 0x00002d852059c5c6 libc.so.7`bsearch + 70 frame #2: 0x00002d85205273e0 libc.so.7`_nsdispatch + 1200 frame #3: 0x00002d8520520639 libc.so.7`getprotobynumber + 121 frame #4: 0x00002d7cfcd07477 pftop`___lldb_unnamed_symbol228 + 391 frame #5: 0x00002d7cfcd059a7 pftop`___lldb_unnamed_symbol203 + 103 frame #6: 0x00002d7cfcd0aae1 pftop`___lldb_unnamed_symbol288 + 305 frame #7: 0x00002d7cfcd0b375 pftop`___lldb_unnamed_symbol300 + 229 frame #8: 0x00002d7cfcd081a8 pftop`___lldb_unnamed_symbol233 + 1176 frame #9: 0x00002d85204bfafa libc.so.7`__libc_start1 + 298 frame #10: 0x00002d7cfcd047ed pftop`___lldb_unnamed_symbol182 + 45 This is on some VMs; same result on with a VM hosted on bhyve and a VM hosted on VMWare.
For some reason the 14.0 version of the libpfctl tarball doesn't have the relevant fix. All other versions do. I'll push an update soon.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=3c34dd243da02758012c8a5669e23c06e14da168 commit 3c34dd243da02758012c8a5669e23c06e14da168 Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2024-06-27 17:21:34 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2024-06-28 08:30:15 +0000 net/libpfctl: add missing patch to 14.0 The 2b1eb63fc9c6d6f64baaac59b7ea7c2a3228c03f: "libpfctl: be more tolerant of kernel extensions" patch didn't make it into the 14.0 version of libpfctl. Spin the revision to add that. PR: 277093 Sponsored by: Rubicon Communications, LLC ("Netgate") net/libpfctl/Makefile | 4 ++-- net/libpfctl/distinfo | 16 ++++++++-------- 2 files changed, 10 insertions(+), 10 deletions(-)
This changes, but does not completely resolve things. Locally built libpfctl 0.12 on FreeBSD 14.1-p1 errors with: > Error Reading Rule (DIOCGETRULE): Device busy When I push 5 or 6 in pftop. (But doesn’t crash out.) This particular test is on a custom kernel; I can try GENERIC this weekend.
I'm getting a bus error doing that, which looks to be happening due to pftop's print_rule() doing this: > print_fld_str(FLD_ANCHOR, (char *)pr->anchor); That's just wrong. It'd be wrong if the first field of anchor was actually a string, but it's not so it's doubly wrong. The pftop maintainer needs to fix that.
I'll need to verify on some additional machines, but it appears with libpfctl (0.12) and pftop (0.10_1) locally re-built on 14.1, things are working. Yay! thanks for your help! With respect to that print command, elsewhere in pftop.c: > /* XXX overload pr.anchor, to store a pointer to > * anchor name */ > rule.anchor = (struct pfctl_anchor *) anchor; and above the print line there is a comment referring to the same: > /* XXX anchor field overloaded with anchor name */ > print_fld_str(FLD_ANCHOR, (char *)pr->anchor);
Is there anything left to do for this PR?
(In reply to Mark Johnston from comment #14) Not on the pf/libpfctl side of things. I don't know if pftop itself got fixed.