Created attachment 248509 [details]
ktrace pfctl -sr under jail

Hi,

I have an assertion error on pfctl inside the my jail. apparently, my pf is still working. However, I get the following error when I run:

# pfctl -sr
Assertion failed: (elems <= maxelems), function pf_nvuint_32_array, file /usr/src/lib/libpfctl/libpfctl.c, line 147.


Tested with:
# freebsd-version -kru
14.0-STABLE
14.0-STABLE
14.0-STABLE
:/usr/src # git show
commit 6a75d3b3fbe4c66bf9b8c18db55bba19ffb492e4 (HEAD -> stable/14, origin/stable/14)


related loader.conf:

tcp_bbr_load="YES"
net.fibs="3"


Jail configuration:

# cat /etc/jail.conf.d/jail.conf 
path = "/usr/jails/${name}";
host.hostname = "${name}";
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;

somejail {
  mount.devfs;
  vnet;
  vnet.interface = "epair4b";
  exec.prestart = "service netif cloneup epair4 || echo interfaces are already exists";
  exec.prestop  = "ifconfig epair4a destroy";
}


My kernel conf:

# cat /usr/src/sys/amd64/conf/RACK 
include GENERIC
ident RACK

device pf
device pflog
device cryptodev
options ALTQ
options ALTQ_HFSC
makeoptions WITH_EXTRA_TCP_STACKS=1
options RATELIMIT
options TCPHPTS
options ZFS
options NETGRAPH
options NETGRAPH_ECHO
options NETGRAPH_ETHER
options NETGRAPH_IFACE
options NETGRAPH_KSOCKET
options NETGRAPH_TEE
options NETGRAPH_SOCKET
options NETGRAPH_NETFLOW
options NETGRAPH_EIFACE
options NETGRAPH_BRIDGE

makeoptions     DEBUG=-g
makeoptions     WITH_CTF=1
options         KDTRACE_FRAME
options         KDTRACE_HOOKS
options         DDB_CTF


`/etc/sysctl.conf` under jail:

net.inet.tcp.sendbuf_max=16777216  
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384 
net.inet.tcp.recvbuf_inc=524288 
net.inet.tcp.cc.algorithm=htcp
net.inet.tcp.functions_default=bbr
net.inet.ip.redirect=0 


This jail works as a network gateway for other jails. It was working, strangely that happens after I use the `py39-sshuttle` on it. FYI, `py39-sshuttle` has been tested on the other jails under similar configuration, and this error was not happened before (on the same host).

For the context, AFAICS, sshuttle only adds an anchor to pf which only cantains two rules.
https://github.com/sshuttle/sshuttle/blob/master/sshuttle/methods/pf.py

Sample on other jails on the same time:

# pfctl -a 'sshuttle-12300' -sr 
pass out route-to lo0 inet proto tcp all flags S/SA keep state
pass out inet proto tcp from any to 127.0.0.1 flags S/SA keep state


my `pf.conf` under that jail is fairly simple:
multiple:
nat pass from x.x.x.x/24 to any -> $SOMEIF

pass all
block from x.x.x.x/24 to any