IPsec AH doesn't appear to work with Linux when using the SHA2 hmac. On the FreeBSD side, add -n SRC DST ah 5000 -m transport -A hmac-sha2-384 0x96HEXKEY; never works with the Linux, ip xfrm state add src SRC dst DST proto ah spi 5000 auth-trunc "hmac(sha384)" 0x96HEXKEY 192 mode transport I've tried a variety of truncation lengths (96, 128, 192) and hmac-sha2-256 but none worked. However by changing to "-A hmac-sha1 0x40HEXKEY" which has a default 96 bit truncation, AH worked between the two hosts.
A packet capture of an Authentication Header leaving FreeBSD shows an AH ICV that's not truncated at all (384 bits)
My mistake. What I did find is a problem with Linux AH truncation on values higher than about 160 bits, they are all off by 32 bits.
*** This bug has been marked as a duplicate of bug 222684 ***