Bug 218907 - tcpmd5 kernel module on STABLE/11 doesn't work with vultr bgp via bird
Summary: tcpmd5 kernel module on STABLE/11 doesn't work with vultr bgp via bird
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Olivier Cochard
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2017-04-27 05:56 UTC by Joseph Mulloy
Modified: 2017-06-30 03:44 UTC (History)
3 users (show)

See Also:


Attachments
Proposed patch (776 bytes, text/x-chdr)
2017-04-28 09:28 UTC, Andrey V. Elsukov
no flags Details
Proposed patch to net/bird port (812 bytes, patch)
2017-05-23 09:59 UTC, Andrey V. Elsukov
no flags Details | Diff
Proposed patch to net/bird port (812 bytes, patch)
2017-05-23 10:00 UTC, Andrey V. Elsukov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Joseph Mulloy 2017-04-27 05:56:57 UTC
Hello,

I have setup some servers on the cloud provider Vultr and I have set up a floating IP for load balancing/high availability via BGP. Vultr's BGP system requires using an MD5 TCP signature which before r313330 in current and r315514 in stable/11 was not available as a module and required compiling a custom kernel with the TCP_SIGNATURE option enabled. I prefer to be able to just use freebsd-update so I found this quite inconvenient, but I am dealing with compiling and distributing a custom kernel anyways. However with this kernel my servers keep freezing with no useful error message which is incredibly frustrating. I figured that perhaps now that this functionality has been getting some work that whatever bug I'm hitting may be fixed in STABLE/11. So I tried using the kernel in the snapshot tarball for STABLE/11, but it's lacking the IPSEC_SUPPORT option, so I still have to compile my own kernel for the tcpmd5 module to load/work. I've done this, I have built the STABLE/11 kernel from r317316 and the module loads and bird doesn't complain about the TCP MD5 feature being missing. However BIRD isn't able to actually establish a connection to the other end, so it seems the TCP MD5 feature is now broken. I haven't upgraded my userland, it's still 11.0-RELEASE-p9 but I believe it should still work fine on an 11/STABLE kernel.

Perhaps I'm doing something wrong here, but I can't figure out a working solution and I can't find any documentation. It seems this md5 tcp signature feature is rarely used and hard to even turn on. 

Please let me know what I can do to assist in debugging these issues. I'm glad that tcp md5 signatures will finally be easy to enable. I hope it won't be to hard to get this fixed.

Issues:

1. IPSEC_SUPPORT still not enabled in GENERIC kernel, so I still have to compile my own kernel for the tcpmd5 kernel module to actually work
2. The tcp md5 signature feature doesn't seem to work, the other end rejects my server as if I had the wrong password.

Vultr BGP Guide: https://www.vultr.com/docs/high-availability-on-vultr-with-floating-ip-and-bgp

Bug tracking the splitting of ipsec and tcp md5 to seperate kernel modules. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212018

Bird output showing that BGP session can't be established.
root@vps-vu-nj-1b:~ # birdc show proto all vultr
BIRD 1.6.3 ready.
name     proto    table    state  since       info
vultr    BGP      master   start  05:14:24    Connect       Socket: Connection refused
  Preference:     100
  Input filter:   REJECT
  Output filter:  ACCEPT
  Routes:         0 imported, 0 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              0          0          0          0          0
    Import withdraws:            0          0        ---          0          0
    Export updates:              0          0          0        ---          0
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Connect
    Neighbor address: 169.254.169.254
    Neighbor AS:      64515
    Last error:       Socket: Connection refused
Comment 1 Andrey V. Elsukov freebsd_committer 2017-04-27 12:12:27 UTC
Check that your 11.0-STABLE kernel is later than r316697.
If so, look at the output of "setkey -D" and "netstat -sp tcp | grep signature", but AFAIR, netstat has needed statistics only in STABLE branch, so you have to try copy it from 11.0-STABLE tarball too.
Comment 2 Joseph Mulloy 2017-04-28 04:25:08 UTC
(In reply to Andrey V. Elsukov from comment #1)

Thanks.

I compiled the STABLE/11 kernel from r317316.

Below is the output of setkey and netstat. I used the netstat out of the snapshot tarball as suggested. It seems bird isn't interacting the the kernel API/ABI correctly. Perhaps I need to recompile it for this kernel, or if I'm less lucky it needs to be patched. I've converted the actual output in the lines that contain "A: tcp-md5" since I think these might be my private key.

root@vps-vu-nj-1b:~/base # ./usr/bin/netstat -sp tcp | grep signature
        0 packets with matching signature received
        0 packets with bad signature received
        26059 times failed to make signature due to no SA
        0 times unexpected signature received
        0 times no signature provided by segment

root@vps-vu-nj-1b:~/base # setkey -D
2001:19f0:5:6a3::1 2001:19f0:ffff::1
        tcp mode=any spi=264796535(0x0fc87977) reqid=0(0x00000000)
        A: tcp-md5  XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
        seq=0x00000000 replay=0 flags=0x00000040 state=mature
        created: Apr 27 05:14:25 2017   current: Apr 28 04:20:09 2017
        diff: 83144(s)  hard: 0(s)      soft: 0(s)
        last: Apr 27 05:14:29 2017      hard: 0(s)      soft: 0(s)
        current: 957600(bytes)  hard: 0(bytes)  soft: 0(bytes)
        allocated: 9576 hard: 0 soft: 0
        sadb_seq=1 pid=87046 refcnt=1
45.76.6.230 169.254.169.254
        tcp mode=any spi=4096(0x00001000) reqid=0(0x00000000)
        A: tcp-md5  XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
        seq=0x00000000 replay=0 flags=0x00000040 state=mature
        created: Apr 27 05:14:25 2017   current: Apr 28 04:20:09 2017
        diff: 83144(s)  hard: 0(s)      soft: 0(s)
        last: Apr 27 05:14:29 2017      hard: 0(s)      soft: 0(s)
        current: 766000(bytes)  hard: 0(bytes)  soft: 0(bytes)
        allocated: 9575 hard: 0 soft: 0
        sadb_seq=0 pid=87046 refcnt=1
Comment 3 Andrey V. Elsukov freebsd_committer 2017-04-28 09:28:04 UTC
Created attachment 182147 [details]
Proposed patch

Can you try to build bird with this patch applied? You can put it into the port patches.
Comment 4 Joseph Mulloy 2017-04-29 04:05:59 UTC
(In reply to Andrey V. Elsukov from comment #3)
Thanks for the patch to bird. It's working for the IPv4 version of bird but I'm still having trouble with the IPv6 version, bird6. The bird6 package is a subordinate package so it should get the same patches, correct?

I'm curious why this wasn't an issue with the old kernel code, but I'm glad it's working. Now that I'm getting a better understanding of how this works it seems I could have also used the setkey utility instead of having bird handle the md5 signature stuff.

Thanks
Comment 5 Andrey V. Elsukov freebsd_committer 2017-05-02 00:43:11 UTC
(In reply to Joseph Mulloy from comment #4)
> (In reply to Andrey V. Elsukov from comment #3)
> Thanks for the patch to bird. It's working for the IPv4 version of bird but
> I'm still having trouble with the IPv6 version, bird6. The bird6 package is
> a subordinate package so it should get the same patches, correct?

Yes, bird6 uses the same patches, but you still need to rebuild it too.
 
> I'm curious why this wasn't an issue with the old kernel code, but I'm glad
> it's working. Now that I'm getting a better understanding of how this works
> it seems I could have also used the setkey utility instead of having bird
> handle the md5 signature stuff.

The behavior was unintended changed in the r313330, now TCP-MD5 requires correct SA for both direction, since both are used.
Comment 6 Joseph Mulloy 2017-05-11 02:43:07 UTC
(In reply to Andrey V. Elsukov from comment #5)

I am building all of my packages via poudriere and both packages were rebuilt with the patch you provided. Looking at "setkeys -D" it seems like the patch worked for bird6. Perhaps something isn't happening correctly on the kernel side in the ipv6 code path?

If there is anything you would like me to do to debug this please let me know.

Thanks
Comment 7 Andrey V. Elsukov freebsd_committer 2017-05-11 07:37:47 UTC
(In reply to Joseph Mulloy from comment #6)
> If there is anything you would like me to do to debug this please let me
> know.

Can you use only IPv6 connections and see in the `netstat -sp tcp` statistic what wrong? Also can you show the  output of `setkey -D`? 
I'll try to reproduce the problem with IPv6.
Comment 8 Andrey V. Elsukov freebsd_committer 2017-05-23 09:59:03 UTC
Created attachment 182826 [details]
Proposed patch to net/bird port
Comment 9 Andrey V. Elsukov freebsd_committer 2017-05-23 10:00:45 UTC
Created attachment 182827 [details]
Proposed patch to net/bird port
Comment 10 Andrey V. Elsukov freebsd_committer 2017-05-24 08:36:14 UTC
Olivier, I think the first patch is the most correct. Because it will try delete both SAs even if some error occurs.
Comment 11 Olivier Cochard freebsd_committer 2017-05-29 19:55:28 UTC
Wrong PR number in my commit, then I reproduce the message manually.

A commit references this bug:

Author: olivier
Date: Mon May 29 19:30:31 UTC 2017
New revision: 442031
URL: https://svnweb.freebsd.org/changeset/ports/442031

Log:
  Fix MD5 TCP signature usage with latest tcpmd5 kernel module

  PR:		218907
  Submitted by:	ae
  Reported by:	Joseph Mulloy <freebsd-bugs@joe.mulloy.me>
  Approved by:	melifaro (maintainer)

  Fix build with option FIREWALL

  PR:		217150
  Submitted by:	olivier
  Reported by:	O. Hartmann <ohartmann@walstatt.org>
  Approved by:	melifaro (maintainer)
  Sponsored by:	Orange

Changes:
  head/net/bird/Makefile
  head/net/bird/files/bird.in
  head/net/bird/files/bird6.in
  head/net/bird/files/firewall_support.patch
  head/net/bird/files/patch-sysdep-bsd-setkey.h
Comment 12 Joseph Mulloy 2017-06-30 03:44:21 UTC
I figured out my problem with ipv6. I had the wrong source IP in my bird config. VUltr gives me a whole /64 of ipv6 IP addresses and I picked <prefix>::1 as my IP, but I needed to actually use the randomly generated IP they assign my VM by default. For some reason this wasn't needed on the old kernel, but that was probably a fluke. It's working great now, thanks.