Hello, I have setup some servers on the cloud provider Vultr and I have set up a floating IP for load balancing/high availability via BGP. Vultr's BGP system requires using an MD5 TCP signature which before r313330 in current and r315514 in stable/11 was not available as a module and required compiling a custom kernel with the TCP_SIGNATURE option enabled. I prefer to be able to just use freebsd-update so I found this quite inconvenient, but I am dealing with compiling and distributing a custom kernel anyways. However with this kernel my servers keep freezing with no useful error message which is incredibly frustrating. I figured that perhaps now that this functionality has been getting some work that whatever bug I'm hitting may be fixed in STABLE/11. So I tried using the kernel in the snapshot tarball for STABLE/11, but it's lacking the IPSEC_SUPPORT option, so I still have to compile my own kernel for the tcpmd5 module to load/work. I've done this, I have built the STABLE/11 kernel from r317316 and the module loads and bird doesn't complain about the TCP MD5 feature being missing. However BIRD isn't able to actually establish a connection to the other end, so it seems the TCP MD5 feature is now broken. I haven't upgraded my userland, it's still 11.0-RELEASE-p9 but I believe it should still work fine on an 11/STABLE kernel. Perhaps I'm doing something wrong here, but I can't figure out a working solution and I can't find any documentation. It seems this md5 tcp signature feature is rarely used and hard to even turn on. Please let me know what I can do to assist in debugging these issues. I'm glad that tcp md5 signatures will finally be easy to enable. I hope it won't be to hard to get this fixed. Issues: 1. IPSEC_SUPPORT still not enabled in GENERIC kernel, so I still have to compile my own kernel for the tcpmd5 kernel module to actually work 2. The tcp md5 signature feature doesn't seem to work, the other end rejects my server as if I had the wrong password. Vultr BGP Guide: https://www.vultr.com/docs/high-availability-on-vultr-with-floating-ip-and-bgp Bug tracking the splitting of ipsec and tcp md5 to seperate kernel modules. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212018 Bird output showing that BGP session can't be established. root@vps-vu-nj-1b:~ # birdc show proto all vultr BIRD 1.6.3 ready. name proto table state since info vultr BGP master start 05:14:24 Connect Socket: Connection refused Preference: 100 Input filter: REJECT Output filter: ACCEPT Routes: 0 imported, 0 exported, 0 preferred Route change stats: received rejected filtered ignored accepted Import updates: 0 0 0 0 0 Import withdraws: 0 0 --- 0 0 Export updates: 0 0 0 --- 0 Export withdraws: 0 --- --- --- 0 BGP state: Connect Neighbor address: 169.254.169.254 Neighbor AS: 64515 Last error: Socket: Connection refused
Check that your 11.0-STABLE kernel is later than r316697. If so, look at the output of "setkey -D" and "netstat -sp tcp | grep signature", but AFAIR, netstat has needed statistics only in STABLE branch, so you have to try copy it from 11.0-STABLE tarball too.
(In reply to Andrey V. Elsukov from comment #1) Thanks. I compiled the STABLE/11 kernel from r317316. Below is the output of setkey and netstat. I used the netstat out of the snapshot tarball as suggested. It seems bird isn't interacting the the kernel API/ABI correctly. Perhaps I need to recompile it for this kernel, or if I'm less lucky it needs to be patched. I've converted the actual output in the lines that contain "A: tcp-md5" since I think these might be my private key. root@vps-vu-nj-1b:~/base # ./usr/bin/netstat -sp tcp | grep signature 0 packets with matching signature received 0 packets with bad signature received 26059 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment root@vps-vu-nj-1b:~/base # setkey -D 2001:19f0:5:6a3::1 2001:19f0:ffff::1 tcp mode=any spi=264796535(0x0fc87977) reqid=0(0x00000000) A: tcp-md5 XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX seq=0x00000000 replay=0 flags=0x00000040 state=mature created: Apr 27 05:14:25 2017 current: Apr 28 04:20:09 2017 diff: 83144(s) hard: 0(s) soft: 0(s) last: Apr 27 05:14:29 2017 hard: 0(s) soft: 0(s) current: 957600(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 9576 hard: 0 soft: 0 sadb_seq=1 pid=87046 refcnt=1 45.76.6.230 169.254.169.254 tcp mode=any spi=4096(0x00001000) reqid=0(0x00000000) A: tcp-md5 XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX seq=0x00000000 replay=0 flags=0x00000040 state=mature created: Apr 27 05:14:25 2017 current: Apr 28 04:20:09 2017 diff: 83144(s) hard: 0(s) soft: 0(s) last: Apr 27 05:14:29 2017 hard: 0(s) soft: 0(s) current: 766000(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 9575 hard: 0 soft: 0 sadb_seq=0 pid=87046 refcnt=1
Created attachment 182147 [details] Proposed patch Can you try to build bird with this patch applied? You can put it into the port patches.
(In reply to Andrey V. Elsukov from comment #3) Thanks for the patch to bird. It's working for the IPv4 version of bird but I'm still having trouble with the IPv6 version, bird6. The bird6 package is a subordinate package so it should get the same patches, correct? I'm curious why this wasn't an issue with the old kernel code, but I'm glad it's working. Now that I'm getting a better understanding of how this works it seems I could have also used the setkey utility instead of having bird handle the md5 signature stuff. Thanks
(In reply to Joseph Mulloy from comment #4) > (In reply to Andrey V. Elsukov from comment #3) > Thanks for the patch to bird. It's working for the IPv4 version of bird but > I'm still having trouble with the IPv6 version, bird6. The bird6 package is > a subordinate package so it should get the same patches, correct? Yes, bird6 uses the same patches, but you still need to rebuild it too. > I'm curious why this wasn't an issue with the old kernel code, but I'm glad > it's working. Now that I'm getting a better understanding of how this works > it seems I could have also used the setkey utility instead of having bird > handle the md5 signature stuff. The behavior was unintended changed in the r313330, now TCP-MD5 requires correct SA for both direction, since both are used.
(In reply to Andrey V. Elsukov from comment #5) I am building all of my packages via poudriere and both packages were rebuilt with the patch you provided. Looking at "setkeys -D" it seems like the patch worked for bird6. Perhaps something isn't happening correctly on the kernel side in the ipv6 code path? If there is anything you would like me to do to debug this please let me know. Thanks
(In reply to Joseph Mulloy from comment #6) > If there is anything you would like me to do to debug this please let me > know. Can you use only IPv6 connections and see in the `netstat -sp tcp` statistic what wrong? Also can you show the output of `setkey -D`? I'll try to reproduce the problem with IPv6.
Created attachment 182826 [details] Proposed patch to net/bird port
Created attachment 182827 [details] Proposed patch to net/bird port
Olivier, I think the first patch is the most correct. Because it will try delete both SAs even if some error occurs.
Wrong PR number in my commit, then I reproduce the message manually. A commit references this bug: Author: olivier Date: Mon May 29 19:30:31 UTC 2017 New revision: 442031 URL: https://svnweb.freebsd.org/changeset/ports/442031 Log: Fix MD5 TCP signature usage with latest tcpmd5 kernel module PR: 218907 Submitted by: ae Reported by: Joseph Mulloy <freebsd-bugs@joe.mulloy.me> Approved by: melifaro (maintainer) Fix build with option FIREWALL PR: 217150 Submitted by: olivier Reported by: O. Hartmann <ohartmann@walstatt.org> Approved by: melifaro (maintainer) Sponsored by: Orange Changes: head/net/bird/Makefile head/net/bird/files/bird.in head/net/bird/files/bird6.in head/net/bird/files/firewall_support.patch head/net/bird/files/patch-sysdep-bsd-setkey.h
I figured out my problem with ipv6. I had the wrong source IP in my bird config. VUltr gives me a whole /64 of ipv6 IP addresses and I picked <prefix>::1 as my IP, but I needed to actually use the randomly generated IP they assign my VM by default. For some reason this wasn't needed on the old kernel, but that was probably a fluke. It's working great now, thanks.