Bug 223835 - BGP session not established with md5 password via FRRouting
Summary: BGP session not established with md5 password via FRRouting
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.1-STABLE
Hardware: amd64 Any
: --- Affects Many People
Assignee: Andrey V. Elsukov
URL: https://github.com/FRRouting/frr/issu...
Keywords: IntelNetworking
Depends on:
Blocks:
 
Reported: 2017-11-24 07:38 UTC by Alexey
Modified: 2018-06-04 16:35 UTC (History)
10 users (show)

See Also:


Attachments
TCPDUMP (7.19 KB, text/plain)
2017-11-24 13:20 UTC, Alexey
no flags Details
TCPDUMP file for 185.1.62.69 (2.99 KB, application/vnd.tcpdump.pcap)
2017-11-24 13:37 UTC, Alexey
no flags Details
Proposed patch (379 bytes, patch)
2018-02-09 13:50 UTC, Andrey V. Elsukov
no flags Details | Diff
patch to net/openbgpd port (58.50 KB, patch)
2018-03-25 05:05 UTC, Andrey V. Elsukov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey 2017-11-24 07:38:36 UTC
Hello. 
I'm trying to establish BGP session with GOOGLE (AS15169).
Google use md5 password and BGP session always stay on `Connect`.
My configuration `FreeBSD 11.1-STABLE` and ` FRRouting 3.0.2`.

BGP configuration:
```
router bgp 29XXX
 neighbor 185.1.62.69 remote-as 15169
 neighbor 185.1.62.69 description -= GOOGLE =-
 neighbor 185.1.62.69 password some_password
 address-family ipv4 unicast
  neighbor 185.1.62.69 soft-reconfiguration inbound
  neighbor 185.1.62.69 route-map RM-4-BGP-GOOGLE-IN in
  neighbor 185.1.62.69 route-map RM-4-BGP-GOOGLE-OUT out

```
```
show ip bgp summary
BGP router identifier 192.XXX.XX.1, local AS number 29XXXvrf-id 0
BGP table version 1591061
RIB entries 1216045, using 158 MiB of memory
Peers 23, using 323 KiB of memory
Peer groups 4, using 288 bytes of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
185.1.62.69     4      15169       0       0        0    0    0    never      Connect

```
and debug log for this peer:
```
2017/11/23 16:23:58 BGP: 185.1.62.69 went from Active to Idle
2017/11/23 16:23:59 BGP: 185.1.62.69 [FSM] Timer (start timer expire).
2017/11/23 16:23:59 BGP: 185.1.62.69 [FSM] BGP_Start (Idle->Connect), fd -1
2017/11/23 16:23:59 BGP: 185.1.62.69 [Event] Connect start to 185.1.62.69 fd 36
2017/11/23 16:23:59 BGP: 185.1.62.69 [FSM] Non blocking connect waiting result, fd 36
2017/11/23 16:23:59 BGP: 185.1.62.69 went from Idle to Connect
2017/11/23 16:24:02 BGP: 185.1.62.69 [FSM] BGP_Stop (Connect->Idle), fd 36
2017/11/23 16:24:02 BGP: 185.1.62.69 went from Connect to Idle
2017/11/23 16:24:03 BGP: 185.1.62.69 [FSM] Timer (start timer expire).
2017/11/23 16:24:03 BGP: 185.1.62.69 [FSM] BGP_Start (Idle->Connect), fd -1
2017/11/23 16:24:03 BGP: 185.1.62.69 [Event] Connect start to 185.1.62.69 fd 36
2017/11/23 16:24:03 BGP: 185.1.62.69 [FSM] Non blocking connect waiting result, fd 36
2017/11/23 16:24:03 BGP: 185.1.62.69 went from Idle to Connect
```
I'm also have rebuild my kernel this support IPSEC:

```
options         IPSEC                   # IP (v4/v6) security
options         IPSEC_SUPPORT           # Allow kldload of ipsec and tcpmd5
options         TCP_OFFLOAD             # TCP offload
# The crypto framework is required by IPSEC
device          crypto                  # Required by IPSEC
device          cryptodev
options         TCP_SIGNATURE
```
setup ipsec key:
```
cat /etc/ipsec.conf
flush;
add 185.1.62.241 185.1.62.69 tcp 0x1000 -A tcp-md5 "some_password";
```
What is not right, help, please.
Comment 1 Andrey V. Elsukov freebsd_committer freebsd_triage 2017-11-24 08:49:33 UTC
You have security association only for one direction. Try to add the SA for backward and forward directions into ipsec.conf.
Comment 2 Alexey 2017-11-24 09:58:03 UTC
(In reply to Andrey V. Elsukov from comment #1)
I have add the SA for backward
Now my /etc/ipsec.conf have this:

flush;
add  185.1.62.241 185.1.62.69 tcp 0x1000 -A tcp-md5 "some_password";
add 185.1.62.69 185.1.62.241 tcp 0x1001 -A tcp-md5 "some_password";

I also do this /etc/rc.d/ipsec reload

[root@gate /home/pautina]# setkey -D
185.1.62.69 185.1.62.241
        tcp mode=any spi=4097(0x00001001) reqid=0(0x00000000)
        A: tcp-md5  6a757472 616e3764 45625577 72366339
        seq=0x00000000 replay=0 flags=0x00000040 state=mature
        created: Nov 24 11:53:32 2017   current: Nov 24 11:53:37 2017
        diff: 5(s)      hard: 0(s)      soft: 0(s)
        last:                           hard: 0(s)      soft: 0(s)
        current: 0(bytes)       hard: 0(bytes)  soft: 0(bytes)
        allocated: 0    hard: 0 soft: 0
        sadb_seq=1 pid=55761 refcnt=1
185.1.62.241 185.1.62.69
        tcp mode=any spi=4096(0x00001000) reqid=0(0x00000000)
        A: tcp-md5  6a757472 616e3764 45625577 72366339
        seq=0x00000000 replay=0 flags=0x00000040 state=mature
        created: Nov 24 11:53:32 2017   current: Nov 24 11:53:37 2017
        diff: 5(s)      hard: 0(s)      soft: 0(s)
        last:                           hard: 0(s)      soft: 0(s)
        current: 0(bytes)       hard: 0(bytes)  soft: 0(bytes)
        allocated: 0    hard: 0 soft: 0
        sadb_seq=0 pid=55761 refcnt=1
But BGP session state at Connect.
Comment 3 Andrey V. Elsukov freebsd_committer freebsd_triage 2017-11-24 10:26:48 UTC
(In reply to Alexey from comment #2)
> flush;
> add  185.1.62.241 185.1.62.69 tcp 0x1000 -A tcp-md5 "some_password";
> add 185.1.62.69 185.1.62.241 tcp 0x1001 -A tcp-md5 "some_password";
> 
> I also do this /etc/rc.d/ipsec reload
> 
> [root@gate /home/pautina]# setkey -D
> 185.1.62.69 185.1.62.241
>         tcp mode=any spi=4097(0x00001001) reqid=0(0x00000000)
>         A: tcp-md5  6a757472 616e3764 45625577 72366339
>         allocated: 0    hard: 0 soft: 0
>         sadb_seq=1 pid=55761 refcnt=1
> 185.1.62.241 185.1.62.69
>         tcp mode=any spi=4096(0x00001000) reqid=0(0x00000000)
>         A: tcp-md5  6a757472 616e3764 45625577 72366339
>         allocated: 0    hard: 0 soft: 0
>         sadb_seq=0 pid=55761 refcnt=1
> But BGP session state at Connect.

It seems SAs are not matched by any packets.
Check what addresses uses daemon and also check what shows `netstat -sp tcp | grep sig`
Comment 4 Alexey 2017-11-24 10:41:36 UTC
(In reply to Andrey V. Elsukov from comment #3)
[root@gate /home/pautina]# netstat -sp tcp | grep sig
        0 packets with matching signature received
        5261 packets with bad signature received
        5579 times failed to make signature due to no SA
        0 times unexpected signature received
        2 times no signature provided by segment

what are you mean about addresses daemon, what the daemon, frr, bgpd?

Maybe this?
frr      bgpd       41894 5  tcp6   *:179                 *:*
frr      bgpd       41894 6  tcp4   *:179                 *:*
frr      bgpd       41894 7  tcp6   *:2605                *:*
frr      bgpd       41894 8  tcp4   *:2605                *:*
Comment 5 Alexey 2017-11-24 10:43:28 UTC
I have repeat command and now have this.
[root@gate /home/pautina]# setkey -D
185.1.62.69 185.1.62.241
        tcp mode=any spi=4097(0x00001001) reqid=0(0x00000000)
        A: tcp-md5  6a757472 616e3764 45625577 72366339
        seq=0x00000000 replay=0 flags=0x00000040 state=mature
        created: Nov 24 11:53:32 2017   current: Nov 24 12:42:08 2017
        diff: 2916(s)   hard: 0(s)      soft: 0(s)
        last: Nov 24 11:56:34 2017      hard: 0(s)      soft: 0(s)
        current: 22336(bytes)   hard: 0(bytes)  soft: 0(bytes)
        allocated: 302  hard: 0 soft: 0
        sadb_seq=1 pid=57429 refcnt=1
185.1.62.241 185.1.62.69
        tcp mode=any spi=4096(0x00001000) reqid=0(0x00000000)
        A: tcp-md5  6a757472 616e3764 45625577 72366339
        seq=0x00000000 replay=0 flags=0x00000040 state=mature
        created: Nov 24 11:53:32 2017   current: Nov 24 12:42:08 2017
        diff: 2916(s)   hard: 0(s)      soft: 0(s)
        last: Nov 24 11:56:34 2017      hard: 0(s)      soft: 0(s)
        current: 10560(bytes)   hard: 0(bytes)  soft: 0(bytes)
        allocated: 132  hard: 0 soft: 0
        sadb_seq=0 pid=57429 refcnt=1

Previous statistics was only after reload ipsec.
Comment 6 Andrey V. Elsukov freebsd_committer freebsd_triage 2017-11-24 12:53:48 UTC
(In reply to Alexey from comment #4)
> (In reply to Andrey V. Elsukov from comment #3)
> [root@gate /home/pautina]# netstat -sp tcp | grep sig
>         0 packets with matching signature received
>         5261 packets with bad signature received

This means that socket is configured to receive and send TCP MD5 signatures, but TCP segments has wrong signatures. You need to make sure that used password is correct. 

>         5579 times failed to make signature due to no SA

This means that outbound or inbound TCP segments have been failed to find corresponding SA for given addresses and ports.

>         0 times unexpected signature received
>         2 times no signature provided by segment
> 
> what are you mean about addresses daemon, what the daemon, frr, bgpd?
> 
> Maybe this?
> frr      bgpd       41894 5  tcp6   *:179                 *:*
> frr      bgpd       41894 6  tcp4   *:179                 *:*
> frr      bgpd       41894 7  tcp6   *:2605                *:*
> frr      bgpd       41894 8  tcp4   *:2605                *:*

I would check the output of tcpdump for given TCP connections. Addresses in packets dump should match to addresses used in SAs. You also can use -M flag to specify used password and see that it is correct.
Comment 7 Alexey 2017-11-24 13:20:44 UTC
Created attachment 188239 [details]
TCPDUMP

(In reply to Andrey V. Elsukov from comment #6)

>This means that socket is configured to receive and send TCP MD5 signatures, but TCP segments has wrong signatures. You need to make sure that used password is correct.

I copied the password from the emails and use this password for tcpdump wtis -M

I run tcpdump: 'tcpdump -M some_password -i vlan62 -vvv -n host 185.1.62.69 and port bgp' go to vtysh and run: clear ip bgp 185.1.62.69 after this have results in attachment
Comment 8 Alexey 2017-11-24 13:21:01 UTC
(In reply to Andrey V. Elsukov from comment #6)

>This means that socket is configured to receive and send TCP MD5 signatures, but TCP segments has wrong signatures. You need to make sure that used password is correct.

I copied the password from the emails and use this password for tcpdump wtis -M

I run tcpdump: 'tcpdump -M some_password -i vlan62 -vvv -n host 185.1.62.69 and port bgp' go to vtysh and run: clear ip bgp 185.1.62.69 after this have results in attachment
Comment 9 Alexey 2017-11-24 13:37:39 UTC
Created attachment 188240 [details]
TCPDUMP file for 185.1.62.69

I'm repeat command: 'tcpdump -M some_password -i vlan62 -XXX -vvv -n host 185.1.62.69' and save data to file for next analized.
Comment 10 Alexey 2017-11-24 23:19:24 UTC
Good night everybody.
The problem is solved.
Many thanks to Marek Zarychta mailto:zarychtam@plan-b.pwste.edu.pl for the help.
He showed me a similar problem: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219453.
Аnd indeed, the problem was that on the interface TX/RX checksums was disabled.

Now everything works with the following settings:

On interface ix0 or ixl0 or other must be turn ON: rxcsum txcsum; (ifconfig ixl0 rxcsum txcsum)

At /etc/rc.conf:
ifconfig_ixl0="up -tso -lro -vlanhwtso" (I disabled only tso and lro)
ipsec_enable="YES"
ipsec_file="/etc/ipsec.conf"

At /etc/ipsec.conf:
flush;
add 185.1.62.241 185.1.62.69 tcp 0x1000 -A tcp-md5 "some_password";
add 185.1.62.69 185.1.62.241 tcp 0x1001 -A tcp-md5 "some_password";

On kernel you must add next:
options         IPSEC                   # IP (v4/v6) security
options         IPSEC_SUPPORT           # Allow kldload of ipsec and tcpmd5
# The crypto framework is required by IPSEC
device          crypto                  # Required by IPSEC
device          cryptodev
options         TCP_SIGNATURE

And need set password for neighbor on FRRouting, for example:
 neighbor 185.1.62.69 password some_password

I think it's necessary to describe all this in documentation. 
This would be good, as this problem arises for many. Or you can simply forget about it :)
Comment 11 Alexey 2017-11-25 13:27:49 UTC
Maybe it's still a bug.
Comment 12 Alexey 2017-11-27 16:03:36 UTC
I'm not sure if this is a bug. Maybe just for this faket must be mentioned in the documentation
Comment 13 Andrey V. Elsukov freebsd_committer freebsd_triage 2017-11-27 16:42:47 UTC
(In reply to Alexey from comment #12)
> I'm not sure if this is a bug. Maybe just for this faket must be mentioned
> in the documentation

I think it is driver's bug and driver should be fixed if it does not correctly support checksum offloading.
Comment 14 Alexey 2017-11-27 17:27:11 UTC
(In reply to Andrey V. Elsukov from comment #13)
What do I do in this case, reopen the issue?
Comment 15 Alexey 2017-11-27 17:36:58 UTC
Device that have bug: Intel® Ethernet X710-DA2

ixl0: <Intel(R) Ethernet Connection XL710/X722 Driver, Version - 1.7.12-k> mem 0xdd800000-0xddffffff,0xde808000-0xde80ffff irq 17 at device 0.0 on pci2
ixl0: Using MSIX interrupts with 5 vectors
ixl0: fw 4.40.35115 api 1.4 nvm 4.53 etid 8000226b oem 1.268.0
ixl0: PF-ID[0]: VFs 64, MSIX 129, VF MSIX 5, QPs 768, I2C
ixl0: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl0: Ethernet address: 68:05:ca:41:e8:18
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl0: SR-IOV ready
queues is 0xfffff800067b1000

At this night I weel try check for Intel® Ethernet X520-DA2 for this bug.
Comment 16 Marek Zarychta 2017-11-28 09:11:08 UTC
The bug is strictly related to disabling RXCSUMs on NIC. Other Intel drivers em (4) and igb (4) reveal the same behaviour. I have not tested NICs from other vendors.  
In fact, pcap dumps taken on interface shows that everything is fine, but counter of "packets with bad signature received" increases and TCP handshake couldn't be established when using TCP MD5 signed packets.
Comment 17 Olivier Cochard freebsd_committer freebsd_triage 2017-11-28 09:43:57 UTC
I've got the same problem regarding my inet4 BGP sessions using MD5 on all my bhyve and virtualbox virtual lab (using vtnet and em interface).
I don't have the problem regarding inet6 BGP session using MD5 on the same labs.
Can bug be related to my problems ?
Comment 18 Andrey V. Elsukov freebsd_committer freebsd_triage 2017-11-28 09:48:25 UTC
(In reply to Olivier Cochard from comment #17)
> I've got the same problem regarding my inet4 BGP sessions using MD5 on all
> my bhyve and virtualbox virtual lab (using vtnet and em interface).
> I don't have the problem regarding inet6 BGP session using MD5 on the same
> labs.
> Can bug be related to my problems ?

Probably it is not related to drivers, I'll check this a bit later.
Comment 19 Andrey V. Elsukov freebsd_committer freebsd_triage 2018-02-09 13:50:16 UTC
Created attachment 190461 [details]
Proposed patch

Can you try this patch?
Comment 20 Alexey 2018-02-09 14:09:36 UTC
(In reply to Andrey V. Elsukov from comment #19)
In the near future, I can not apply the patch. This is a fully working server.
Maybe someone has a laboratory, and he can test. I can only in March.
Comment 21 Olivier Cochard freebsd_committer freebsd_triage 2018-02-09 15:06:37 UTC
Thanks for this patch!

It fixes my problem  (applied on -stable):

[root@router]~# netstat -ss | grep sign
        40 packets with matching signature received

and now my inet4 TCP MD5 works:

[root@router]~# birdcl
BIRD 2.0.1 ready.
bird> show protocol R2inet4
Name       Proto      Table      State  Since         Info
R2inet4    BGP        ---        up     15:58:36.337  Established
Comment 22 Alexey 2018-02-09 15:57:25 UTC
(In reply to Olivier Cochard from comment #21)
Thank you for testing and positive result
Comment 23 Felipe N. Oliva 2018-02-09 16:10:10 UTC
(In reply to Olivier Cochard from comment #21)
I have the same problem, but with OpenBGPd.

I will try the patch!

Thanks,
Comment 24 Marek Zarychta 2018-02-09 18:05:51 UTC
Thank you for the patch. I also confirm that it fixes the issue on 11.1-STABLE for inet4. 
This with "options IPSEC_SUPPORT" default in GENERIC is going to make upcoming 11.2 the most TCP-MD5-friendly FreeBSD release ever.
Comment 25 commit-hook freebsd_committer freebsd_triage 2018-02-10 10:13:38 UTC
A commit references this bug:

Author: ae
Date: Sat Feb 10 10:13:18 UTC 2018
New revision: 329101
URL: https://svnweb.freebsd.org/changeset/base/329101

Log:
  Reinitialize IP header length after checksum calculation. It is used
  later by TCP-MD5 code.

  This fixes the problem with broken TCP-MD5 over IPv4 when NIC has
  disabled TCP checksum offloading.

  PR:		223835
  MFC after:	1 week

Changes:
  head/sys/netinet/tcp_input.c
Comment 26 Felipe N. Oliva 2018-02-16 13:59:55 UTC
(In reply to Olivier Cochard from comment #21)
Oliver, your problem is only without TSO/LRO?
I have the same problem, but TSO/LRO enabled in 11.1-p6 and 11.1-stable(r329156).
My environment is with openbgp.
Comment 27 commit-hook freebsd_committer freebsd_triage 2018-02-18 11:37:32 UTC
A commit references this bug:

Author: ae
Date: Sun Feb 18 11:36:46 UTC 2018
New revision: 329518
URL: https://svnweb.freebsd.org/changeset/base/329518

Log:
  MFC r329101:
    Reinitialize IP header length after checksum calculation. It is used
    later by TCP-MD5 code.

    This fixes the problem with broken TCP-MD5 over IPv4 when NIC has
    disabled TCP checksum offloading.

    PR:		223835

Changes:
_U  stable/11/
  stable/11/sys/netinet/tcp_input.c
Comment 28 Andrey V. Elsukov freebsd_committer freebsd_triage 2018-02-26 18:15:23 UTC
(In reply to Felipe N. Oliva from comment #26)
> (In reply to Olivier Cochard from comment #21)
> Oliver, your problem is only without TSO/LRO?
> I have the same problem, but TSO/LRO enabled in 11.1-p6 and
> 11.1-stable(r329156).
> My environment is with openbgp.

Felipe, is it still problem for you?
Comment 29 Felipe N. Oliva 2018-02-27 11:47:58 UTC
(In reply to Andrey V. Elsukov from comment #28)
Yes, here is 11.1-STABLE r329758.

interface:
vlan1977: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>

kernel:
options	IPSEC
options	TCP_SIGNATURE
device	crypto
Comment 30 Andrey V. Elsukov freebsd_committer freebsd_triage 2018-02-27 12:32:18 UTC
(In reply to Felipe N. Oliva from comment #29)
> (In reply to Andrey V. Elsukov from comment #28)
> Yes, here is 11.1-STABLE r329758.
> 
> interface:
> vlan1977: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> 	options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
> 
> kernel:
> options	IPSEC
> options	TCP_SIGNATURE
> device	crypto

Did you try to disable rx/txcsum? What interface is used as parent for vlans?
Does IPv6 work for you but IPv4 doesn't? 
What `netstat -sp tcp | grep sig` shows? 
Does setkey -D shows SAs for both directions?
Comment 31 Felipe N. Oliva 2018-02-27 12:46:22 UTC
(In reply to Andrey V. Elsukov from comment #30)
I will try with TSO/LRO.
Interface: ix (intel 10g)
Doesn't work with ipv4 and ipv6.

netstat -sp tcp | grep sig
	0 packets with matching signature received
	0 packets with bad signature received
	0 times failed to make signature due to no SA
	0 times unexpected signature received
	0 times no signature provided by segment

/etc/ipsec.conf:
add -4 <my_ip4> 187.16.216.55 tcp 0x1000 -A tcp-md5 "<password>"; # GOOGLE
add -4 187.16.216.55 <my_ip4> tcp 0x1000 -A tcp-md5 "<password>"; # GOOGLE
add -4 <my_ip4> 187.16.218.58 tcp 0x1000 -A tcp-md5 "<password>"; # GOOGLE
add -4 187.16.218.58 <my_ip4> tcp 0x1000 -A tcp-md5 "<password>"; # GOOGLE
add -6 <my_ip6> 2001:12f8::55 tcp 0x1000 -A tcp-md5 "<password>"; # GOOGLE
add -6 2001:12f8::55 <my_ip6> tcp 0x1000 -A tcp-md5 "<password>"; # GOOGLE
add -6 <my_ip6> 2001:12f8::218:58 tcp 0x1000 -A tcp-md5 "<password>"; # GOOGLE
add -6 2001:12f8::218:58 <my_ip6> tcp 0x1000 -A tcp-md5 "<password>"; # GOOGLE

setkey -D
2001:12f8::218:58 <my_ip6>
	tcp mode=any spi=130789163(0x07cbaf2b) reqid=0(0x00000000)
	A: tcp-md5  6a757472 616e3764 45625577 72366339
	seq=0x00000000 replay=0 flags=0x00000040 state=mature
	created: Feb 27 08:36:12 2018	current: Feb 27 08:41:32 2018
	diff: 320(s)	hard: 0(s)	soft: 0(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=7 pid=74097 refcnt=1
<my_ip6> 2001:12f8::218:58
	tcp mode=any spi=205209160(0x0c3b3e48) reqid=0(0x00000000)
	A: tcp-md5  6a757472 616e3764 45625577 72366339
	seq=0x00000000 replay=0 flags=0x00000040 state=mature
	created: Feb 27 08:36:12 2018	current: Feb 27 08:41:32 2018
	diff: 320(s)	hard: 0(s)	soft: 0(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=6 pid=74097 refcnt=1
2001:12f8::55 <my_ip6>
	tcp mode=any spi=17778168(0x010f45f8) reqid=0(0x00000000)
	A: tcp-md5  6a757472 616e3764 45625577 72366339
	seq=0x00000000 replay=0 flags=0x00000040 state=mature
	created: Feb 27 08:36:12 2018	current: Feb 27 08:41:32 2018
	diff: 320(s)	hard: 0(s)	soft: 0(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=5 pid=74097 refcnt=1
<my_ip6> 2001:12f8::55
	tcp mode=any spi=11511344(0x00afa630) reqid=0(0x00000000)
	A: tcp-md5  6a757472 616e3764 45625577 72366339
	seq=0x00000000 replay=0 flags=0x00000040 state=mature
	created: Feb 27 08:36:12 2018	current: Feb 27 08:41:32 2018
	diff: 320(s)	hard: 0(s)	soft: 0(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=4 pid=74097 refcnt=1
187.16.218.58 <my_ip4>
	tcp mode=any spi=49404247(0x02f1d957) reqid=0(0x00000000)
	A: tcp-md5  6a757472 616e3764 45625577 72366339
	seq=0x00000000 replay=0 flags=0x00000040 state=mature
	created: Feb 27 08:36:12 2018	current: Feb 27 08:41:32 2018
	diff: 320(s)	hard: 0(s)	soft: 0(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=3 pid=74097 refcnt=1
<my_ip4> 187.16.218.58
	tcp mode=any spi=209590058(0x0c7e172a) reqid=0(0x00000000)
	A: tcp-md5  6a757472 616e3764 45625577 72366339
	seq=0x00000000 replay=0 flags=0x00000040 state=mature
	created: Feb 27 08:36:12 2018	current: Feb 27 08:41:32 2018
	diff: 320(s)	hard: 0(s)	soft: 0(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=2 pid=74097 refcnt=1
187.16.216.55 <my_ip4>
	tcp mode=any spi=124856546(0x077128e2) reqid=0(0x00000000)
	A: tcp-md5  6a757472 616e3764 45625577 72366339
	seq=0x00000000 replay=0 flags=0x00000040 state=mature
	created: Feb 27 08:36:12 2018	current: Feb 27 08:41:32 2018
	diff: 320(s)	hard: 0(s)	soft: 0(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=1 pid=74097 refcnt=1
<my_ip4> 187.16.216.55
	tcp mode=any spi=4096(0x00001000) reqid=0(0x00000000)
	A: tcp-md5  6a757472 616e3764 45625577 72366339
	seq=0x00000000 replay=0 flags=0x00000040 state=mature
	created: Feb 27 08:36:12 2018	current: Feb 27 08:41:32 2018
	diff: 320(s)	hard: 0(s)	soft: 0(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=0 pid=74097 refcnt=1

FreeBSD 10.3 was working with only one direction. Did anything change?
Comment 32 Andrey V. Elsukov freebsd_committer freebsd_triage 2018-02-27 13:09:30 UTC
(In reply to Felipe N. Oliva from comment #31)
> (In reply to Andrey V. Elsukov from comment #30)
> I will try with TSO/LRO.
> Interface: ix (intel 10g)
> Doesn't work with ipv4 and ipv6.
> 
> netstat -sp tcp | grep sig
> 	0 packets with matching signature received
> 	0 packets with bad signature received
> 	0 times failed to make signature due to no SA
> 	0 times unexpected signature received
> 	0 times no signature provided by segment

It seems there were not any attempt to use TCP-MD5, probably you need to properly configure your BGP daemon. 

> FreeBSD 10.3 was working with only one direction. Did anything change?

Yes.
Comment 33 Felipe N. Oliva 2018-02-27 13:37:38 UTC
(In reply to Andrey V. Elsukov from comment #32)
pfSense with problem too.
https://redmine.pfsense.org/issues/7969
Comment 34 Andrey V. Elsukov freebsd_committer freebsd_triage 2018-02-28 17:21:16 UTC
> (In reply to Felipe N. Oliva from comment #31)
> > netstat -sp tcp | grep sig
> > 	0 packets with matching signature received
> > 	0 packets with bad signature received
> > 	0 times failed to make signature due to no SA
> > 	0 times unexpected signature received
> > 	0 times no signature provided by segment
> 
> It seems there were not any attempt to use TCP-MD5, probably you need to
> properly configure your BGP daemon. 
> 

When application wants to use TCP-MD5 signatures, it uses TCP_MD5SIG socket option to enable this feature. When socket has enabled this feature and no SAs are presents, or option is enabled and TCP segments have not such option there will be some counters incremented. In your case application did not enable this socket option, this is why I think about wrong configuration.
Comment 35 Andrey V. Elsukov freebsd_committer freebsd_triage 2018-03-25 05:05:33 UTC
Created attachment 191795 [details]
patch to net/openbgpd port

I looked at the openbgpd code from ports. Port has wrong patch, because of which openbgbd doesn't enable TCP_MD5SIG option for used sockets and thus MD5 signatures don't work.

Can you replace files/patch-bgpd_session.c with attached file and rebuild openbgpd, then test again?
Comment 36 Felipe N. Oliva 2018-03-26 11:59:26 UTC
(In reply to Andrey V. Elsukov from comment #35)

Of course and thank you.
Do I need to use the SA from IPSec too or with your patch I can use on openbgpd configuration?
Comment 37 Alexey 2018-03-30 08:00:57 UTC
(In reply to Andrey V. Elsukov from comment #19)
Hello.
Today I was able to update and test everything. Everything is working fine.