Bug 26549

I am trying to establish several IPsec ESP tunnels from one system.
Each tunnel by itself works beautifully, but when I try using those
tunnels all together, the kernel suddenly understands no incoming
ESP message. The problem comes and goes with modifications in the 
SPD. If I delete all but one pair of SPD rules, the remaining 
tunnel works fine. As soon as I add a second pair of SPD rules 
the pair just added doesn't works. If I then delete the first
pair, the pair just added suddenly works. I can go round robin with
this and it is always the oldest pair that works and none of the
others.

This problem is critical for me as it puts my entire project 
into jeopardy. I will have to work around this in the next
couple of days.

Fix: 

Don't use IPsec :-( I will try reverting to gif tunnels with
transport mode, hoping that it will work better.
How-To-Repeat: You need three machines A and B, and C. We begin with A and B:

On machine A run:

if=ed0
aip=10.10.10.1
bip=10.10.10.2
aipsec=10.99.10
bipsec=10.99.20
ifconfig ${if} inet alias ${aip} netmask 0xffffff00
ifconfig lo0 inet alias ${aipsec}.1 netmask 0xffffff00
route add -net ${bipsec}.0/24 ${aipsec}.1
setkey -c <<END
spdflush;
add ${aip} ${bip} esp 1000 -E simple;
add ${bip} ${aip} esp 1001 -E simple;
spdadd ${aipsec}.0/24 ${bipsec}.0/24 any -P out ipsec
  esp/tunnel/${aip}-${bip}/require;
spdadd ${bipsec}.0/24 ${aipsec}.0/24 any -P in ipsec
  esp/tunnel/${bip}-${aip}/require;
END

On machine B run

if=sis0
aip=10.10.10.1
bip=10.10.10.2
aipsec=10.99.10
bipsec=10.99.20
ifconfig ${if} inet alias ${bip} netmask 0xffffff00
ifconfig lo0 inet alias ${bipsec}.1 netmask 0xffffff00
route add -net ${aipsec}.0/24 ${bipsec}.1
setkey -c <<END
spdflush;
add ${aip} ${bip} esp 1000 -E simple;
add ${bip} ${aip} esp 1001 -E simple;
spdadd ${aipsec}.0/24 ${bipsec}.0/24 any -P in ipsec
  esp/tunnel/${aip}-${bip}/require;
spdadd ${bipsec}.0/24 ${aipsec}.0/24 any -P out ipsec
  esp/tunnel/${bip}-${aip}/require;
END

ping ${aip}
ping ${aipsec}.1

should get responses, likewise, one machine A

ping ${bip}
ping ${bipsec}

should get responses just fine.

Now comes the problem, lets add another machine C into
the group. Configure machine C:

if=tl0
aip=10.10.10.1
cip=10.10.10.3
aipsec=10.99.10
cipsec=10.99.30
ifconfig ${if} inet alias ${cip} netmask 0xffffff00
ifconfig lo0 inet alias ${cipsec}.1 netmask 0xffffff00
route add -net ${aipsec}.0/24 ${cipsec}.1
setkey -c <<END
spdflush;
add ${aip} ${cip} esp 2000 -E simple;
add ${cip} ${aip} esp 2001 -E simple;
spdadd ${aipsec}.0/24 ${cipsec}.0/24 any -P in ipsec
  esp/tunnel/${aip}-${cip}/require;
spdadd ${cipsec}.0/24 ${aipsec}.0/24 any -P out ipsec
  esp/tunnel/${cip}-${aip}/require;
END

As you see, C only has a tunnel to A. Beause A is supposed to
be the logical gateway, the hub, the center of a star topology. 
Which means we add this to A:

cip=10.10.10.3
cipsec=10.99.30
route add -net ${cipsec}.0/24 ${aipsec}.1
setkey -c <<END
add ${aip} ${cip} esp 2000 -E simple;
add ${cip} ${aip} esp 2001 -E simple;
spdadd ${aipsec}.0/24 ${cipsec}.0/24 any -P out ipsec
  esp/tunnel/${aip}-${cip}/require;
spdadd ${cipsec}.0/24 ${aipsec}.0/24 any -P in ipsec
  esp/tunnel/${cip}-${aip}/require;
END

ping ${cip}
# works
ping ${cipsec}.1
# doesn't work
ping ${bipsec}.1
# works

Now delete the SPD entried for B:

setkey -c <<END
spddelete ${aipsec}.0/24 ${bipsec}.0/24 any -P out;
spddelete ${bipsec}.0/24 ${aipsec}.0/24 any -P in;
END

And try again:

ping ${cipsec}.1

This time it works. Now lets add the SPD for the B-tunnel 
back in:

setkey -c <<END
spdadd ${aipsec}.0/24 ${bipsec}.0/24 any -P out ipsec
  esp/tunnel/${aip}-${bip}/require;
spdadd ${bipsec}.0/24 ${aipsec}.0/24 any -P in ipsec
  esp/tunnel/${bip}-${aip}/require;
END

ping ${cipsec}.1

still works! BUT:

ping ${bipsec}.1

doesn't work now.

The problem is entirely on A's side as I can show you with
a tcpdump trace. On C I say:

tcpdump -n -i tl0 host ${aip}

And now I do

ping ${cipsec}.1

tcpdump shows:
19:51:30.945747 10.10.10.1 > 10.10.10.3: ESP(spi=2000,seq=0x14)
19:51:30.945915 10.10.10.3 > 10.10.10.1: ESP(spi=2001,seq=0x24)
19:51:31.953169 10.10.10.1 > 10.10.10.3: ESP(spi=2000,seq=0x15)
19:51:31.953300 10.10.10.3 > 10.10.10.1: ESP(spi=2001,seq=0x25)

as it should, and remember, the C-tunnel works now. Let's go
to B and 

ping ${aipsec}.1

tcpdump shows:
19:55:21.963950 10.10.10.2 > 10.10.10.1: ESP(spi=1001,seq=0x42)
19:55:22.975435 10.10.10.2 > 10.10.10.1: ESP(spi=1001,seq=0x43)

see how A never sends an icmp echo reply? It is because it 
never gets the icmp messages to the upper layer. Instead

netstat -s -p ip
ip:
        2313 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets reassembled ok
        2033 packets for this host