Trying to follow the way that is recommended (man bridge) to do host IP connectivity with bridge members, I create vlan on top of bridge and assign my IPs there, and I notice that normal IP works, but not CARP: Host A: (client) ifconfig vmx1 inet 192.168.20.1/24 Host B: (server with CARP) ifconfig bridge1 create ifconfig bridge1 addm vmx1 untagged 3400 ifconfig bridge1.3400 create # main IP ifconfig bridge1.3400 inet 192.168.20.2/24 # CARP IP ifconfig bridge1.3400 vhid 99 pass 123 alias 192.168.20.3/24 Host A connectivity with Host B main IP works, but not with CARP IP: hosta:~# ping 192.168.20.2 PING 192.168.20.2 (192.168.20.2): 56 data bytes 64 bytes from 192.168.20.2: icmp_seq=0 ttl=64 time=1.150 ms 64 bytes from 192.168.20.2: icmp_seq=1 ttl=64 time=2.384 ms ^C --- 192.168.20.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.150/1.767/2.384/0.617 ms hosta:~# ping 192.168.20.3 PING 192.168.20.3 (192.168.20.3): 56 data bytes ^C --- 192.168.20.3 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss hosta:~# ifconfig vmx1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=4e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 00:50:56:a8:0b:e3 inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> hostb:~# ifconfig vmx1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=4e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 00:50:56:a8:55:0c media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> bridge1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=10<VLAN_HWTAGGING> ether 58:9c:fc:10:d8:fc id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 bridge flags=1<VLANFILTER> member: vmx1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> port 2 priority 128 path cost 2000 vlan protocol 802.1q untagged 3400 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> bridge1.3400: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1496 options=0 ether 58:9c:fc:10:d8:fc inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255 inet 192.168.20.3 netmask 0xffffff00 broadcast 192.168.20.255 vhid 99 groups: vlan carp: MASTER vhid 99 advbase 1 advskew 0 peer 224.0.0.18 peer6 ff02::12 vlan: 3400 vlanproto: 802.1q vlanpcp: 0 parent interface: bridge1 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Is CARP supposed to work when on top of bridge ?
You should probably disable TSO on the physical interface(s).
TSO disabled, no change to CARP issue.
Make 192.168.20.3 a /32 and try again. If you provide a tcpdump then that can/will confirm my suspicions. Not only would that confirm my suspicions, it would coincide how the documentation does it: https://docs.freebsd.org/en/books/handbook/advanced-networking/#carp
CARP IP mask changed to 32, no change to CARP issue: # ifconfig bridge1.3400 bridge1.3400: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1496 options=0 ether 58:9c:fc:10:d8:fc inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255 inet 192.168.20.3 netmask 0xffffffff broadcast 192.168.20.3 vhid 99 groups: vlan carp: MASTER vhid 99 advbase 1 advskew 0 peer 224.0.0.18 peer6 ff02::12 vlan: 3400 vlanproto: 802.1q vlanpcp: 0 parent interface: bridge1 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> simultaneous tcpdump on bridge and vlan interfaces while running ping 192.168.20.3 from 192.168.20.1 : # tcpdump -n -p -i bridge1 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on bridge1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 08:49:04.231548 IP 192.168.20.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 99, prio 0, authtype none, intvl 1s, length 36 08:49:05.243056 IP 192.168.20.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 99, prio 0, authtype none, intvl 1s, length 36 08:49:05.402873 IP 192.168.20.1 > 192.168.20.3: ICMP echo request, id 1459, seq 0, length 64 08:49:06.261547 IP 192.168.20.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 99, prio 0, authtype none, intvl 1s, length 36 08:49:06.472306 IP 192.168.20.1 > 192.168.20.3: ICMP echo request, id 1459, seq 1, length 64 08:49:07.271556 IP 192.168.20.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 99, prio 0, authtype none, intvl 1s, length 36 08:49:07.542819 IP 192.168.20.1 > 192.168.20.3: ICMP echo request, id 1459, seq 2, length 64 # tcpdump -n -p -i bridge1.3400 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on bridge1.3400, link-type EN10MB (Ethernet), snapshot length 262144 bytes 08:49:05.243051 IP 192.168.20.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 99, prio 0, authtype none, intvl 1s, length 36 08:49:06.261544 IP 192.168.20.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 99, prio 0, authtype none, intvl 1s, length 36 08:49:07.271553 IP 192.168.20.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 99, prio 0, authtype none, intvl 1s, length 36 08:49:08.331564 IP 192.168.20.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 99, prio 0, authtype none, intvl 1s, length 36 you can see ping packets on bridge interface, but not on vlan.
So one last thing .... vmx1 -- This implies you're on VMware? Have you checked the vsphere side for security related options regarding the interface? For example, by default, ESXi port groups (standard or distributed) have Forged Transmits set to Reject. What I'm guessing is happening here is carp is building a MAC based on the VHID and the interface is silently dropping those responses. You're setup looks pretty pristine otherwise.
Yes, Forged Transmits policy is "Accept", I'm aware that this is required on vmware, otherwise no CARP. Maybe I wasn't clear in the beginning, the issue is when CARP's interface parent is bridge, if parent is vmx1, then all is fine. Also I think no CARP replies are built at all, because I don't see with tcpdump incoming icmp packets on CARP interface, only on parent, bridge.
Forget carp altogether for a moment. Can you ping the alias ip at all? I have a similar setup at home (-carp and -esxi), but a simple ping to the alias ip should at the very least work: (My machine B) bridge0.10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1496 options=0 ether 58:9c:fc:10:6f:fa inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255 inet 192.168.10.9 netmask 0xffffffff broadcast 192.168.10.9 groups: vlan vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: bridge0 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> % ping 192.168.10.9 Pinging 192.168.10.9 with 32 bytes of data: Reply from 192.168.10.9: bytes=32 time<1ms TTL=64 Reply from 192.168.10.9: bytes=32 time<1ms TTL=64 If the ping doesn't work, then that points to esxi (whether that's the OS or driver) doing something here.
yes, deleted CARP alias, created simple alias IP on bridge vlan, and it works.
After a bit of debugging here, I can confirm this problem. The problem is that CARPs Virtual mac never gets associated with the bridge. This is because bridge1.xx isn't technically a member of the bridge and MACs sitting on it aren't learned. The primary bridge1.xxx interface works because that ip is simply associated with the main bridge's MAC. I'm not sure what the intended behavior is but a workaround is creating an epair, having the A inserted into the bridge while having the carp address sit on the b side interface. This would work because now you haven't something that is a member of the bridge in which the MAC should be learned.