BACKUP node always installs on-interface route for vhid address, thus it stars to server the network requests. primary node: [root@taiga:local/etc]# ifconfig vlan1 vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1a:64:c6:a8:7c inet 192.168.3.22 netmask 0xffffff00 broadcast 192.168.3.255 inet6 fe80::21a:64ff:fec6:a87c%vlan1 prefixlen 64 scopeid 0x6 inet6 fd00::316 prefixlen 120 inet 192.168.3.13 netmask 0xffffffff broadcast 192.168.3.13 vhid 5 inet 192.168.3.1 netmask 0xffffffff broadcast 192.168.3.1 vhid 5 inet6 fd00::301 prefixlen 120 vhid 5 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 1 parent interface: bce0 carp: MASTER vhid 5 advbase 1 advskew 50 [root@taiga:local/etc]# netstat -rn | grep 192.168.3.1 192.168.3.1 link#6 UHS 0 114730 lo0 => 192.168.3.1/32 link#6 U 0 0 vlan1 192.168.3.13 link#6 UHS 0 16643 lo0 => 192.168.3.13/32 link#6 U 0 0 vlan1 backup node: [root@necromancer:local/etc]# ifconfig vlan1 vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 00:1a:64:67:a8:40 inet 192.168.3.4 netmask 0xffffff00 broadcast 192.168.3.255 inet6 fe80::21a:64ff:fe67:a840%vlan1 prefixlen 64 scopeid 0x6 inet6 fd00::304 prefixlen 120 inet 192.168.3.13 netmask 0xffffffff broadcast 192.168.3.13 vhid 5 inet 192.168.3.1 netmask 0xffffffff broadcast 192.168.3.1 vhid 5 inet 192.168.3.19 netmask 0xffffff00 broadcast 192.168.3.255 inet6 fd00::301 prefixlen 120 vhid 5 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 1 parent interface: bce0 carp: BACKUP vhid 5 advbase 1 advskew 250 [root@necromancer:local/etc]# netstat -rn | grep 192.168.3.1 192.168.3.1 link#6 UHS lo0 192.168.3.1/32 link#6 U vlan1 192.168.3.13 link#6 UHS lo0 192.168.3.13/32 link#6 U vlan1 192.168.3.19 link#6 UHS lo0 so all the hosts requesting 192.168.3.1 get to the backup node: (icmp inside the same broadcast segment) [root@wizard:~/.rtorrent]# ping 192.168.3.1 PING 192.168.3.1 (192.168.3.1): 56 data bytes 64 bytes from 192.168.3.1: icmp_seq=0 ttl=64 time=0.216 ms 64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=0.205 ms 64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=0.200 ms 64 bytes from 192.168.3.1: icmp_seq=3 ttl=64 time=0.209 ms 64 bytes from 192.168.3.1: icmp_seq=4 ttl=64 time=0.258 ms 64 bytes from 192.168.3.1: icmp_seq=5 ttl=64 time=0.275 ms 64 bytes from 192.168.3.1: icmp_seq=6 ttl=64 time=0.214 ms 64 bytes from 192.168.3.1: icmp_seq=7 ttl=64 time=0.194 ms ^C --- 192.168.3.1 ping statistics --- 8 packets transmitted, 8 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.194/0.221/0.275/0.027 ms tcpdump: 14:31:05.599130 00:1a:64:21:8e:80 > 00:1a:64:67:a8:40, ethertype IPv4 (0x0800), length 98: 192.168.3.24 > 192.168.3.1: ICMP echo request, id 8584, seq 3, length 64 14:31:05.599302 00:1a:64:67:a8:40 > 00:1a:64:21:8e:80, ethertype IPv4 (0x0800), length 98: 192.168.3.1 > 192.168.3.24: ICMP echo reply, id 8584, seq 3, length 64 14:31:06.600185 00:1a:64:21:8e:80 > 00:1a:64:67:a8:40, ethertype IPv4 (0x0800), length 98: 192.168.3.24 > 192.168.3.1: ICMP echo request, id 8584, seq 4, length 64 14:31:06.600411 00:1a:64:67:a8:40 > 00:1a:64:21:8e:80, ethertype IPv4 (0x0800), length 98: 192.168.3.1 > 192.168.3.24: ICMP echo reply, id 8584, seq 4, length 64 14:31:07.601117 00:1a:64:21:8e:80 > 00:1a:64:67:a8:40, ethertype IPv4 (0x0800), length 98: 192.168.3.24 > 192.168.3.1: ICMP echo request, id 8584, seq 5, length 64 14:31:07.601370 00:1a:64:67:a8:40 > 00:1a:64:21:8e:80, ethertype IPv4 (0x0800), length 98: 192.168.3.1 > 192.168.3.24: ICMP echo reply, id 8584, seq 5, length 64 ^C 1177 packets captured 485423 packets received by filter 0 packets dropped by kernel manual intervention to fix the routes fails: [root@necromancer:local/etc]# ifconfig vlan1 down [root@necromancer:local/etc]# route delete 192.168.3.1 delete host 192.168.3.1 [root@necromancer:local/etc]# netstat -rn | grep 192.168.3.1 192.168.3.13 link#6 UHS lo0 192.168.3.19 link#6 UHS lo0 [root@necromancer:local/etc]# ifconfig vlan1 up [root@necromancer:local/etc]# netstat -rn | grep 192.168.3.1 192.168.3.1 link#6 UHS lo0 192.168.3.1/32 link#6 U vlan1 192.168.3.13 link#6 UHS lo0 192.168.3.13/32 link#6 U vlan1 192.168.3.19 link#6 UHS lo0 Backup node info: [root@necromancer:~]# uname -a FreeBSD necromancer 10.0-STABLE FreeBSD 10.0-STABLE #0 r267040: Thu Jun 5 10:37:19 YEKT 2014 emz@necromancer:/usr/obj/usr/src/sys/NECROMANCER amd64 I have a pair of older machives, for example r263418 works as intended.
Weird, I just found that this has nothing to do with revision; this has something to do with an alias mask. Mine were /32 (for historical reasons), I changed them to /24 and everything went back to normal. In the same time 5 other vhid pairs between same pair of machines were working as intended. Not sure if this is a bug, probably some configuration trick, so I will close this.