Example: interface host1 host2 lan1 inet 192.168.232.239 192.168.232.244 ipsec2 tunnel 192.168.232.239 192.168.232.244 inet 192.168.233.1 192.168.233.2 gif3 tunnel 192.168.233.1 192.168.233.2 inet 192.168.233.11 192.168.233.12 ifconfig from host1 lan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN options=4c03ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,LINKSTATE,TXCSUM> ether 52:54:00:7b:04:83 hwaddr 52:54:00:7b:04:83 inet 192.168.232.239 netmask 0xffffffe0 broadcast 192.168.232.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet 10Gbase-T <full-duplex> status: active ipsec2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 tunnel inet 192.168.232.239 --> 192.168.232.244 inet 192.168.233.1 --> 192.168.233.2 netmask 0xffffffff nd6 options=108<IFDISABLED,NO_DAD> reqid: 132 groups: ipsec gif3: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1480 options=80000<LINKSTATE> tunnel inet 192.168.233.1 --> 192.168.233.2 inet6 fe80::f7e8:fd66:5229:226d%gif3 prefixlen 64 scopeid 0x6 inet 192.168.233.11 --> 192.168.233.12 netmask 0xffffffff nd6 options=100<NO_DAD> groups: gif There is no firewall, GENERIC kernel only loaded. On the host2: route -4 add -host 10.10.10.10 127.0.0.1 -reject On the host1: route -4 add -host 10.10.10.10 192.168.232.244 (via LAN, icmp is OK) ping -n -t 1 10.10.10.10 PING 10.10.10.10 (10.10.10.10): 56 data bytes 36 bytes from 192.168.232.244: Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 aabb 0 0000 3f 01 1342 192.168.232.239 10.10.10.10 --- 10.10.10.10 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss route -4 change -host 10.10.10.10 192.168.233.2 (via if_ipsec over LAN, icmp is BAD) ping -n -t 1 10.10.10.10 PING 10.10.10.10 (10.10.10.10): 56 data bytes --- 10.10.10.10 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss route -4 change -host 10.10.10.10 192.168.233.12 (via gif over if_ipsec, icmp is OK) ping -n -t 1 10.10.10.10 PING 10.10.10.10 (10.10.10.10): 56 data bytes 36 bytes from 192.168.233.12: Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 aac3 0 0000 3f 01 131e 192.168.233.11 10.10.10.10 --- 10.10.10.10 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss
This is because the ICMP/ICMPv6 input code does check that message was not encrypted. If you apply IPsec transform to gif tunnel, I think you will get the same result.
And why ping is working ok? ping -c 1 -S 192.168.233.1 192.168.233.2 PING 192.168.233.2 (192.168.233.2) from 192.168.233.1: 56 data bytes 64 bytes from 192.168.233.2: icmp_seq=0 ttl=64 time=0.410 ms Is gif over if_ipsec the only reliable way to create encrypted ipv6 tunnel without need of install extra software? Pure ipv6 over if_ipsec is limited in use because of dropping 'packet_too_big' messages.
Created attachment 193277 [details] Proposed patch Can you try this patch? You need to rebuild the kernel.
Thank you! I was applied this patch and it seems to be ok. icmpv6&v4 errors via if_ipsec: 1240 bytes from 2a02:****:****:****::451a:5763: Packet too big mtu = 1480 36 bytes from 192.168.232.84 (192.168.232.84): Destination Net Unreachable Is it possible to fix this issue in future updates that I install via 'freebsd-update fetch install'?
A commit references this bug: Author: ae Date: Fri May 11 16:50:26 UTC 2018 New revision: 333497 URL: https://svnweb.freebsd.org/changeset/base/333497 Log: Apply the change from r272770 to if_ipsec(4) interface. It is guaranteed that if_ipsec(4) interface is used only for tunnel mode IPsec, i.e. decrypted and decapsultaed packet has its own IP header. Thus we can consider it as new packet and clear the protocols flags. This allows ICMP/ICMPv6 properly handle errors that may cause this packet. PR: 228108 MFC after: 1 week Changes: head/sys/net/if_ipsec.c
A commit references this bug: Author: ae Date: Fri May 18 10:17:13 UTC 2018 New revision: 333785 URL: https://svnweb.freebsd.org/changeset/base/333785 Log: MFC r333497: Apply the change from r272770 to if_ipsec(4) interface. It is guaranteed that if_ipsec(4) interface is used only for tunnel mode IPsec, i.e. decrypted and decapsulated packet has its own IP header. Thus we can consider it as new packet and clear the protocols flags. This allows ICMP/ICMPv6 properly handle errors that may cause this packet. PR: 228108 Approved by: re (kib) Changes: _U stable/11/ stable/11/sys/net/if_ipsec.c
Assign to committer resolving, track merge flags.
The fix committed into head/ and stable/11 branches. It will appear in 11.2-RELEASE, and then you can upgrade your system with freebsd-update to get it fixed. Thanks for the report!
This change breaks ICMP ECHO (pings) to the receiving end of peer to peer /30 of the IPsec tunnel between FreeBSD and Juniper JunOS on their SRX products. To JunOS 12.x, this seems to block both ICMP and BGP packets to the other end of the tunnel (being compared to 11.1-STABLE r331329), which works in this setup. To JunOS 17.x and an SRX, OSPF seems to work, but ICMP ECHO does not. (I don't yet have BGP in this setup). However, between 11.1-STABLE r331329 and 11.2-STABLE r335594, IPsec tunnels get established, pings work, and BGP does establish. In the case of 11.2 -> JunOS 17, the tunnels and OSPF did come up, and ICMP does work routed across the tunnel, just not to the tunnel's termination point. I can't seem to explain it, and seemly a little strange mix of OS and hardware, but reverting this one line seemed to fix all the issues I had.
(In reply to dpd from comment #9) > This change breaks ICMP ECHO (pings) to the receiving end of peer to peer > /30 of the IPsec tunnel between FreeBSD and Juniper JunOS on their SRX > products. > > To JunOS 12.x, this seems to block both ICMP and BGP packets to the other > end of the tunnel (being compared to 11.1-STABLE r331329), which works in > this setup. > > To JunOS 17.x and an SRX, OSPF seems to work, but ICMP ECHO does not. (I > don't yet have BGP in this setup). > > However, between 11.1-STABLE r331329 and 11.2-STABLE r335594, IPsec tunnels > get established, pings work, and BGP does establish. > > In the case of 11.2 -> JunOS 17, the tunnels and OSPF did come up, and ICMP > does work routed across the tunnel, just not to the tunnel's termination > point. > > I can't seem to explain it, and seemly a little strange mix of OS and > hardware, but reverting this one line seemed to fix all the issues I had. I have no idea how this commit can affect something. It should only affect inbound ICMP errors handling. Are you sure that described problems is not something related to misconfiguration across several booting?
(In reply to Andrey V. Elsukov from comment #10) Yes. Ironically ... I just reproduced this with bare metal just a few hours ago - but only between FreeBSD machines. Now that I have it "working" ... I'll revert the kernel in the morning and retest.
Created attachment 196035 [details] Shell Transcript of Issue.
I showed over here : https://lists.freebsd.org/pipermail/freebsd-net/2018-August/051301.html That it seems to work with this line removed. Attached is the shell transcript of my current observation. The Tunnel seems up, but no traffic is getting passed.
(In reply to dpd from comment #13) > I showed over here : > https://lists.freebsd.org/pipermail/freebsd-net/2018-August/051301.html > > That it seems to work with this line removed. > > Attached is the shell transcript of my current observation. The Tunnel > seems up, but no traffic is getting passed. You need to look at the output of `setkey -D`, `setkey -DP`, `netstat -sp esp`, `tcpdump -ni` on the network interface and check what actually is going on. Provided information is not enough for debugging. As I said this looks more like a problem with racoon, duplicate SAs, SPs or something like this. For example, in log you have two SAs with different SPI for the same direction. Before starting racoon, make sure that you have cleaned up from previous run on both sides.