- Summary
With a policy-based IPSec tunnel over an in-kernel wireguard tunnel (if_wg.ko), it seems like the outer ip-ip header is not stripped at the correct time for state tracking to work.
This results in TCP sessions (UDP and ICMP are unaffected) dropping after an initial window of approximately 64k.

- Reproducing
Create a wireguard tunnel using the kernel module.
Create an IPSec tunnel in policy mode, with the outer layer routed over the wireguard tunnel.
Test TCP connectivity over the IPSec tunnel with some traffic (I've been using iperf, but any TCP-based protocol shows similar results)
Observe that the session drops after a small amount of traffic.

- Troubleshooting steps taken
A few different hints I've found that made me conclude this is likely a kernel issue:
1. Creating firewall rules for the enc0 interface with state tracking disabled fixes the issue.
2. Using the (old) go implementation of wireguard fixes the issue.
3. The following dtrace program:

    dtrace -x nolibs -qn '
    struct pf_addr { uint8_t addr8[16]; };
    struct pf_state_key_cmp { 
        struct pf_addr addr[2]; 
        uint16_t port[2];
        uint8_t af; 
        uint8_t proto; 
        uint8_t pad[2]; 
    };
    pf:ip:state:lookup
    /arg2 == 1 && stringof((char *)arg0) == "enc0"/
    {
      this->k = (struct pf_state_key_cmp *)arg1;
      printf("enc0 IN proto=%d  %d.%d.%d.%d <-> %d.%d.%d.%d  ports=%d,%d  %s\n",
        this->k->proto,
        this->k->addr[0].addr8[0], this->k->addr[0].addr8[1],
        this->k->addr[0].addr8[2], this->k->addr[0].addr8[3],
        this->k->addr[1].addr8[0], this->k->addr[1].addr8[1],
        this->k->addr[1].addr8[2], this->k->addr[1].addr8[3],
        ((this->k->port[0]&0xff)<<8)|((this->k->port[0]>>8)&0xff),
        ((this->k->port[1]&0xff)<<8)|((this->k->port[1]>>8)&0xff),
        arg4 ? "HIT" : "MISS");
    }'

produces the following log line when run with if_wg:

    enc0 IN proto=4  <wg peer IP> <-> <local wg IP>  ports=0,0  HIT

and the following when run with the old wireguard-go:

    enc0 IN proto=6 192.168.1.10 <-> 192.168.20.60 ports=5201,49970 HIT

4. While an iperf test is running, pf logs the following lines:

    <13><...> pf: loose state match: TCP out wire: 192.168.1.10:5201 192.168.20.60:42200 stack: - [lo=4067707952 high=4067644560 win=502 modulator=0] [lo=0 high=502 win=1 modulator=0] 2:0 A seq=4067707952 (4067707952) ack=0 len=1348 ackskew=0 pkts=50:0 dir=out,fwd
    <13><...> pf: loose state match: TCP out wire: 192.168.1.10:5201 192.168.20.60:42200 stack: - [lo=4067709300 high=4067644560 win=502 modulator=0] [lo=0 high=502 win=1 modulator=0] 2:0 A seq=4067709300 (4067709300) ack=0 len=1348 ackskew=0 pkts=51:0 dir=out,fwd
    <13><...> pf: BAD state: TCP out wire: 192.168.1.10:5201 192.168.20.60:42200 stack: - [lo=4067710648 high=4067644560 win=502 modulator=0] [lo=0 high=502 win=1 modulator=0] 2:0 A seq=4067710648 (4067710648) ack=0 len=1348 ackskew=0 pkts=52:0 dir=out,fwd
    <13><...> pf: State failure on: 1       | 5

5. No change in hardware, system version, firewall settings (except those that disable stateful filtering on enc0), tunnel settings, etc. has made any detectable difference to the issue.

- Environment
This issue has surfaced on an OPNSense installation, an issue with OPNSense unfortunately could not be solved (https://github.com/opnsense/core/issues/8680).
I have reproduced this bug on OPNSense, Version 23.7.12 (FreeBSD OPNsense.localdomain 13.2-RELEASE-p7 FreeBSD 13.2-RELEASE-p7 stable/23.7-n254871-d5ec322cffc SMP amd64) since that is the last version that also supported the wireguard-go implementation, which lets me swap between implementations quickly.
I have also reproduced the issue on OPSense Version 26.1 (FreeBSD OPNSense.localdomain 14.3-RELEASE-p15 FreeBSD 14.3-RELEASE-p15 stable/26.1-n272132-0101c21cac77 SMP amd64) with no change in behaviour at all.
I can try to reproduce the issue on vanilla FreeBSD 15.1 for example, though I don't expect OPNSense or the FreeBSD version to be the issue here, given the symptoms.

If you'd like me to try and produce any further debug logs or try a modified setup, let me know. I know the setup for reproduction is a bit special, and I have a working setup I can modify easily.