Hello everyone. I met a problem with my Freebsd configuration. I used two fibs fib0 for management and fib1 for traffic routing. When i tried to connect to my freebsd my ssh session was closed by timeout. This session passed fib1 then it passed a switch and then this traffic came to mgmt interface in fib0. 1718370615708.png I checked pflog and found out that SYN was passed but SYN-ACK was blocked. 10:40:14.738757 rule 50/0(match): pass in on lagg0.3100: 192.168.1.10.39324 > 192.168.2.20.22: Flags [S , seq 3192491261, win 64240, options [mss 1460, [|tcp] 10:40:14.738823 rule 1/0(match): block in on lagg0.3101: 192.168.2.20.22 > 192.168.1.10.39324: Flags [S.], seq 3872911900, ack 3192491262, win 65535, options [mss 1460, [|tcp] 10:40:15.760558 rule 1/0(match): block in on lagg0.3101: 192.168.2.20.22 > 192.168.1.10.39324: Flags [S.], seq 3872911900, ack 3192491262, win 65535, options [mss 1460, [|tcp] 10:40:16.785316 rule 1/0(match): block in on lagg0.3101: 192.168.2.20.22 > 192.168.1.10.39324: Flags [S.], seq 3872911900, ack 3192491262, win 65535, options [mss 1460, [|tcp] 10:40:17.776546 rule 1/0(match): block in on lagg0.3101: 192.168.2.20.22 > 192.168.1.10.39324: Flags [S.], seq 3872911900, ack 3192491262, win 65535, options [mss 1460, [|tcp] 10:40:18.775315 rule 1/0(match): block in on lagg0.3101: 192.168.2.20.22 > 192.168.1.10.39324: Flags [S.], seq 3872911900, ack 3192491262, win 65535, options [mss 1460, [|tcp] 10:40:20.391522 rule 1/0(match): block in on lagg0.3101: 192.168.2.20.22 > 192.168.1.10.39324: Flags [S.], seq 3872911900, ack 3192491262, win 65535, options [mss 1460, [|tcp] 10:40:21.418648 rule 1/0(match): block in on lagg0.3101: 192.168.2.20.22 > 192.168.1.10.39324: Flags [S.], seq 3872911900, ack 3192491262, win 65535, options [mss 1460, [|tcp] Click to expand... Then i checked mgmt interface with tcpdump and there wasn't incoming traffic. The SYN packed was lost. admin@mypc:~ $ sudo tcpdump -nli mgmt host 192.168.2.20 and port 22 and host 192.168.1.10 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on mgmt, link-type EN10MB (Ethernet), snapshot length 262144 bytes 10:45:04.378916 IP 192.168.2.20.22 > 192.168.1.10.57788: Flags [S.], seq 1690518431, ack 2823437748, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 800362632 ecr 2768737784], length 0 10:45:05.382466 IP 192.168.2.20.22 > 192.168.1.10.57788: Flags [S.], seq 1690518431, ack 2823437748, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 800363638 ecr 2768737784], length 0 10:45:05.392406 IP 192.168.2.20.22 > 192.168.1.10.57788: Flags [S.], seq 1690518431, ack 2823437748, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 800363651 ecr 2768738798], length 0 10:45:06.390812 IP 192.168.2.20.22 > 192.168.1.10.57788: Flags [S.], seq 1690518431, ack 2823437748, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 800364642 ecr 2768738798], length 0 10:45:07.408389 IP 192.168.2.20.22 > 192.168.1.10.57788: Flags [S.], seq 1690518431, ack 2823437748, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 800365665 ecr 2768740814], length 0 10:45:08.425344 IP 192.168.2.20.22 > 192.168.1.10.57788: Flags [S.], seq 1690518431, ack 2823437748, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 800366684 ecr 2768740814], length 0 Click to expand... I checked interface out and there wasn't any SYN packet too. admin@mypc:~ $ sudo tcpdump -nli lagg0.3101 host 192.168.2.20 and port 22 and host 192.168.1.10 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on lagg0.3101, link-type EN10MB (Ethernet), snapshot length 262144 bytes 12:06:11.070143 IP 192.168.2.20.22 > 192.168.1.10.54686: Flags [S.], seq 3117832771, ack 2273301168, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 3221841358 ecr 2773605130], length 0 12:06:12.073943 IP 192.168.2.20.22 > 192.168.1.10.54686: Flags [S.], seq 3117832771, ack 2273301168, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 3221842359 ecr 2773606134], length 0 12:06:13.110800 IP 192.168.2.20.22 > 192.168.1.10.54686: Flags [S.], seq 3117832771, ack 2273301168, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 3221843399 ecr 2773606134], length 0 12:06:14.090184 IP 192.168.2.20.22 > 192.168.1.10.54686: Flags [S.], seq 3117832771, ack 2273301168, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 3221844378 ecr 2773608150], length 0 Click to expand...It looked like route leaking and i checked routing table but i didn't find any problem there. admin@mypc:~ $ sudo netstat -rn | grep 192.168.2 default 192.168.2.1 UGS mgmt 192.168.2.0/26 link#1 U mgmt 192.168.2.20 link#3 UHS lo0 admin@mypc:~ $ sudo setfib 1 netstat -rn | grep 192.168.2 192.168.2.0/24 10.222.253.101 UG1 lagg0.3101 192.168.2.0/24 10.222.253.102 UG1 lagg0.3101 My pf.conf # Port macros NET_MGMT = "192.168.2.0/26" JH_NOC = "192.168.1.10" # Tables table <DST_JH_NOC_TO_NET_MGMT> { $NET_MGMT } table <SRC_NET_MGMT_TO_JH_NOC> { $JH_NOC } # Config set skip on lo0 set skip on mgmt set skip on vtnet1 set skip on pfsync0 set limit states 6000000 set limit src-nodes 6000000 # Scrub scrub in all # Firewall policy pass out all block in log all rtable 1 pass in log quick proto icmp rtable 1 pass in log quick proto { tcp, udp } from <SRC_JH_NOC_TO_NET_MGMT> to <DST_JH_NOC_TO_NET_MGMT> rtable 1 pass in log quick proto { tcp, udp } from <DST_JH_NOC_TO_NET_MGMT> to <SRC_JH_NOC_TO_NET_MGMT> rtable 1 Click to expand... I supposed that traffic somehow leaked from fib1 to fib0. Please help me to fix it
Hi Egor, Cant see the attachment, I have a similar setup, without PF and it works. Also, are those dump from the same session? as the source port differs. Br. Santi
I've experimented a bit before. The interfaces also have a default fib. It is set as ifconfig ifname0 fib 1 . The packets it receives do not know the circumstances of this routing table, fibs... I don't know what it was, but I thought it was something like that :)
Created attachment 253636 [details] lab scheme
Created attachment 253637 [details] vm.conf
Created attachment 253638 [details] rc.conf
Created attachment 253639 [details] netif
(In reply to Tatsuki Makino from comment #2) I reproduced this problem in my lab. Config is: 1) Asus 750 hypervisor with proxmox 8.4: pve-fw01:~$ uname -a Linux pve-fw01 6.8.8-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-4 (2024-07-26T11:15Z) x86_64 GNU/Linux 2) Freebsd virtual machine with linked mellanox connectx6 pci card: test-fw01:~$ uname -a FreeBSD test-fw01 14.1-RELEASE-p4 FreeBSD 14.1-RELEASE-p4 GENERIC amd64 I catched traffic for all interfaces with enabled pf and with disabled pf. There is no difference. Tcpdump with enabled pf test-fw01:~$ sudo tcpdump -nei mce1.1280 host 172.16.188.194 and port 22 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on mce1.1280, link-type EN10MB (Ethernet), snapshot length 262144 bytes 16:50:53.753073 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222, win 64240, options [mss 1460,sackOK,TS val 3411125639 ecr 0,nop,wscale 7], length 0 16:50:54.816302 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222, win 64240, options [mss 1460,sackOK,TS val 3411126703 ecr 0,nop,wscale 7], length 0 16:50:55.840297 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222, win 64240, options [mss 1460,sackOK,TS val 3411127727 ecr 0,nop,wscale 7], length 0 16:50:56.864293 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222, win 64240, options [mss 1460,sackOK,TS val 3411128751 ecr 0,nop,wscale 7], length 0 16:50:57.888290 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222, win 64240, options [mss 1460,sackOK,TS val 3411129775 ecr 0,nop,wscale 7], length 0 test-fw01:~$ sudo tcpdump -nei mce1.3101 host 172.16.188.194 and port 22 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on mce1.3101, link-type EN10MB (Ethernet), snapshot length 262144 bytes 16:50:53.753130 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 4184867862 ecr 3411125639], length 0 16:50:54.760769 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 4184868868 ecr 3411125639], length 0 16:50:54.816333 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 4184868920 ecr 3411126703], length 0 16:50:55.820324 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 4184869929 ecr 3411126703], length 0 16:50:55.840332 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 4184869949 ecr 3411127727], length 0 16:50:56.841091 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 4184870950 ecr 3411127727], length 0 16:50:56.864323 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 4184870969 ecr 3411128751], length 0 test-fw01:~$ sudo tcpdump -ner /var/log/pflog host 172.16.188.194 and port 57836 reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file), snapshot length 116 16:50:53.753073 rule 5/0(match): pass in on mce1.1280: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222, win 64240, options [mss 1460, [|tcp] 16:50:53.753130 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] 16:50:54.760769 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] 16:50:54.816333 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] 16:50:55.820324 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] 16:50:55.840332 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] 16:50:56.841091 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] 16:50:56.864323 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] 16:50:57.863064 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] 16:50:57.888331 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535, options [mss 1460, [|tcp] Tcpdump with disabled pf test-fw01:~$ sudo tcpdump -nei mce1.1280 host 172.16.188.194 and port 22 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on mce1.1280, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:01:42.533010 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.179.42.34620 > 172.16.188.194.22: Flags [S], seq 215584557, win 64240, options [mss 1460,sackOK,TS val 3411775040 ecr 0,nop,wscale 7], length 0 17:01:42.533054 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.34620: Flags [S.], seq 1105243323, ack 215584558, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 2313997153 ecr 3411775040], length 0 17:01:42.533144 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 66: 172.16.179.42.34620 > 172.16.188.194.22: Flags [.], ack 1, win 502, options [nop,nop,TS val 3411775040 ecr 2313997153], length 0 17:01:42.533505 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 108: 172.16.179.42.34620 > 172.16.188.194.22: Flags [P.], seq 1:43, ack 1, win 502, options [nop,nop,TS val 3411775040 ecr 2313997153], length 42: SSH: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.5 17:01:42.566078 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800), length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 43, win 129, options [nop,nop,TS val 2313997190 ecr 3411775040], length 0 17:02:04.656826 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800), length 104: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 1:39, ack 43, win 129, options [nop,nop,TS val 2314019278 ecr 3411775040], length 38: SSH: SSH-2.0-OpenSSH_9.7 FreeBSD-20240806 17:02:04.656940 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 66: 172.16.179.42.34620 > 172.16.188.194.22: Flags [.], ack 39, win 502, options [nop,nop,TS val 3411797164 ecr 2314019278], length 0 17:02:04.657554 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 1514: 172.16.179.42.34620 > 172.16.188.194.22: Flags [.], seq 43:1491, ack 39, win 502, options [nop,nop,TS val 3411797165 ecr 2314019278], length 1448 17:02:04.657554 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 154: 172.16.179.42.34620 > 172.16.188.194.22: Flags [P.], seq 1491:1579, ack 39, win 502, options [nop,nop,TS val 3411797165 ecr 2314019278], length 88 17:02:04.657604 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800), length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 1579, win 126, options [nop,nop,TS val 2314019278 ecr 3411797165], length 0 17:02:04.657843 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800), length 1186: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 39:1159, ack 1579, win 126, options [nop,nop,TS val 2314019278 ecr 3411797165], length 1120 17:02:04.698103 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 66: 172.16.179.42.34620 > 172.16.188.194.22: Flags [.], ack 1159, win 501, options [nop,nop,TS val 3411797206 ecr 2314019278], length 0 17:02:04.792136 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 1274: 172.16.179.42.34620 > 172.16.188.194.22: Flags [P.], seq 1579:2787, ack 1159, win 501, options [nop,nop,TS val 3411797300 ecr 2314019278], length 1208 17:02:04.802961 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800), length 1514: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], seq 1159:2607, ack 2787, win 129, options [nop,nop,TS val 2314019418 ecr 3411797300], length 1448 17:02:04.802963 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800), length 150: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 2607:2691, ack 2787, win 129, options [nop,nop,TS val 2314019418 ecr 3411797300], length 84 test-fw01:~$ sudo tcpdump -nei mce1.3101 host 172.16.188.194 and port 22 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on mce1.3101, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:01:42.533054 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 74: 172.16.188.194.22 > 172.16.179.42.34620: Flags [S.], seq 1105243323, ack 215584558, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 2313997153 ecr 3411775040], length 0 17:01:42.566078 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 43, win 129, options [nop,nop,TS val 2313997190 ecr 3411775040], length 0 17:02:04.656826 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 104: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 1:39, ack 43, win 129, options [nop,nop,TS val 2314019278 ecr 3411775040], length 38: SSH: SSH-2.0-OpenSSH_9.7 FreeBSD-20240806 17:02:04.657604 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 1579, win 126, options [nop,nop,TS val 2314019278 ecr 3411797165], length 0 17:02:04.657843 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 1186: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 39:1159, ack 1579, win 126, options [nop,nop,TS val 2314019278 ecr 3411797165], length 1120 17:02:04.802961 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 1514: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], seq 1159:2607, ack 2787, win 129, options [nop,nop,TS val 2314019418 ecr 3411797300], length 1448 17:02:04.802963 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 150: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 2607:2691, ack 2787, win 129, options [nop,nop,TS val 2314019418 ecr 3411797300], length 84 17:02:05.514898 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 2871, win 129, options [nop,nop,TS val 2314019519 ecr 3411797357], length 0 17:02:05.515063 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800), length 110: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 2691:2735, ack 2915, win 129, options [nop,nop,TS val 2314019519 ecr 3411797408], length 44 routing tables info test-fw01:~$ sudo netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 172.16.188.193 UGS vtnet0 10.222.254.254 link#3 UHS lo0 10.222.254.254/31 link#2 U vtnet1 127.0.0.1 link#3 UH lo0 172.16.188.192/26 link#1 U vtnet0 172.16.188.194 link#3 UHS lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 link#3 URS lo0 ::1 link#3 UHS lo0 ::ffff:0.0.0.0/96 link#3 URS lo0 fe80::%lo0/10 link#3 URS lo0 fe80::%lo0/64 link#3 U lo0 fe80::1%lo0 link#3 UHS lo0 ff02::/16 link#3 URS lo0 test-fw01:~$ sudo setfib 1 netstat -rn Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire 127.0.0.1 link#3 UHS lo0 172.16.0.0/16 link#3 U1B lo0 172.16.176.0/26 192.168.255.0 UG1 mce0.124 172.16.176.0/26 192.168.255.2 UG1 mce1.124 172.16.176.64/26 192.168.255.0 UG1 mce0.124 172.16.176.64/26 192.168.255.2 UG1 mce1.124 172.16.176.128/26 192.168.255.0 UG1 mce0.124 172.16.176.128/26 192.168.255.2 UG1 mce1.124 172.16.176.192/27 192.168.255.0 UG1 mce0.124 172.16.176.192/27 192.168.255.2 UG1 mce1.124 172.16.177.0/26 192.168.255.16 UG1 mce0.125 172.16.177.0/26 192.168.255.18 UG1 mce1.125 172.16.177.64/27 192.168.255.16 UG1 mce0.125 172.16.177.64/27 192.168.255.18 UG1 mce1.125 172.16.177.96/27 192.168.255.16 UG1 mce0.125 172.16.177.96/27 192.168.255.18 UG1 mce1.125 172.16.177.128/27 192.168.255.16 UG1 mce0.125 172.16.177.128/27 192.168.255.18 UG1 mce1.125 172.16.178.0/28 192.168.255.32 UG1 mce0.126 172.16.178.0/28 192.168.255.34 UG1 mce1.126 172.16.178.32/27 192.168.255.32 UG1 mce0.126 172.16.178.32/27 192.168.255.34 UG1 mce1.126 172.16.179.0/28 192.168.255.64 UG1 mce0.128 172.16.179.0/28 192.168.255.66 UG1 mce1.128 172.16.179.16/29 192.168.255.64 UG1 mce0.128 172.16.179.16/29 192.168.255.66 UG1 mce1.128 172.16.179.24/30 192.168.255.64 UG1 mce0.128 172.16.179.24/30 192.168.255.66 UG1 mce1.128 172.16.179.28/30 192.168.255.64 UG1 mce0.128 172.16.179.28/30 192.168.255.66 UG1 mce1.128 172.16.179.32/30 192.168.255.64 UG1 mce0.128 172.16.179.32/30 192.168.255.66 UG1 mce1.128 172.16.179.36/30 192.168.255.64 UG1 mce0.128 172.16.179.36/30 192.168.255.66 UG1 mce1.128 172.16.179.40/30 192.168.255.64 UG1 mce0.128 172.16.179.40/30 192.168.255.66 UG1 mce1.128 172.16.179.48/30 192.168.255.64 UG1 mce0.128 172.16.179.48/30 192.168.255.66 UG1 mce1.128 172.16.179.64/28 192.168.255.64 UG1 mce0.128 172.16.179.64/28 192.168.255.66 UG1 mce1.128 172.16.180.0/27 192.168.255.48 UG1 mce0.127 172.16.180.0/27 192.168.255.50 UG1 mce1.127 172.16.180.32/27 192.168.255.48 UG1 mce0.127 172.16.180.32/27 192.168.255.50 UG1 mce1.127 172.16.181.0/28 192.168.255.80 UG1 mce0.131 172.16.181.0/28 192.168.255.82 UG1 mce1.131 172.17.0.0/16 link#3 U1B lo0 172.17.160.0/27 192.168.255.96 UG1 mce0.149 172.17.160.0/27 192.168.255.98 UG1 mce1.149 172.17.160.32/27 192.168.255.96 UG1 mce0.149 172.17.160.32/27 192.168.255.98 UG1 mce1.149 172.17.160.64/27 192.168.255.96 UG1 mce0.149 172.17.160.64/27 192.168.255.98 UG1 mce1.149 172.17.161.0/24 192.168.255.96 UG1 mce0.149 172.17.161.0/24 192.168.255.98 UG1 mce1.149 192.168.255.0/31 link#6 U mce0.124 192.168.255.1 link#3 UHS lo0 192.168.255.2/31 link#7 U mce1.124 192.168.255.3 link#3 UHS lo0 192.168.255.16/31 link#8 U mce0.125 192.168.255.17 link#3 UHS lo0 192.168.255.18/31 link#9 U mce1.125 192.168.255.19 link#3 UHS lo0 192.168.255.32/31 link#10 U mce0.126 192.168.255.33 link#3 UHS lo0 192.168.255.34/31 link#11 U mce1.126 192.168.255.35 link#3 UHS lo0 192.168.255.48/31 link#12 U mce0.127 192.168.255.49 link#3 UHS lo0 192.168.255.50/31 link#13 U mce1.127 192.168.255.51 link#3 UHS lo0 192.168.255.64/31 link#14 U mce0.128 192.168.255.65 link#3 UHS lo0 192.168.255.66/31 link#15 U mce1.128 192.168.255.67 link#3 UHS lo0 192.168.255.80/31 link#16 U mce0.131 192.168.255.81 link#3 UHS lo0 192.168.255.82/31 link#17 U mce1.131 192.168.255.83 link#3 UHS lo0 192.168.255.96/31 link#18 U mce0.149 192.168.255.97 link#3 UHS lo0 192.168.255.98/31 link#19 U mce1.149 192.168.255.99 link#3 UHS lo0 192.168.255.112/31 link#22 U mce0.310 192.168.255.113 link#3 UHS lo0 192.168.255.114/31 link#23 U mce1.310 192.168.255.115 link#3 UHS lo0 192.168.255.144/31 link#20 U mce0.310 192.168.255.145 link#3 UHS lo0 192.168.255.146/31 link#21 U mce1.310 192.168.255.147 link#3 UHS lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 link#3 URS lo0 ::1 link#3 UHS lo0 ::ffff:0.0.0.0/96 link#3 URS lo0 fe80::%lo0/10 link#3 URS lo0 ff02::/16 link#3 URS lo0
Created attachment 253640 [details] lab scheme with ip
Created attachment 253641 [details] pf.conf
Created attachment 253655 [details] frr config
Created attachment 253656 [details] rc.conf.d frr
It seemed i found the cause of the error. It frr i have routes from fib 0 although frr has to work in fib 1. Can it connected with this bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279662 ? I tried to change route table in zebra config but i failed. https://docs.frrouting.org/en/latest/zebra.html#cmdoption-zebra-routing-table test-fw01# sh ip route connected Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure fib 0 C>* 10.64.244.0/24 [0/1] is directly connected, vtnet0, 00:04:12 C>* 10.222.254.254/31 [0/1] is directly connected, vtnet1, 00:04:12 fib 1 C>* 192.168.255.0/31 [0/1] is directly connected, mce0.1240, 00:04:12 C>* 192.168.255.2/31 [0/1] is directly connected, mce1.1240, 00:04:12 C>* 192.168.255.16/31 [0/1] is directly connected, mce0.1250, 00:04:12 C>* 192.168.255.18/31 [0/1] is directly connected, mce1.1250, 00:04:12 C>* 192.168.255.32/31 [0/1] is directly connected, mce0.1260, 00:04:12 C>* 192.168.255.34/31 [0/1] is directly connected, mce1.1260, 00:04:12 C>* 192.168.255.48/31 [0/1] is directly connected, mce0.1270, 00:04:12 C>* 192.168.255.50/31 [0/1] is directly connected, mce1.1270, 00:04:12 C>* 192.168.255.64/31 [0/1] is directly connected, mce0.1280, 00:04:12 C>* 192.168.255.66/31 [0/1] is directly connected, mce1.1280, 00:04:12 C>* 192.168.255.80/31 [0/1] is directly connected, mce0.1310, 00:04:12 C>* 192.168.255.82/31 [0/1] is directly connected, mce1.1310, 00:04:12 C>* 192.168.255.96/31 [0/1] is directly connected, mce0.1490, 00:04:12 C>* 192.168.255.98/31 [0/1] is directly connected, mce1.1490, 00:04:12 C>* 192.168.255.112/31 [0/1] is directly connected, mce0.3101, 00:04:12 C>* 192.168.255.114/31 [0/1] is directly connected, mce1.3101, 00:04:12 C>* 192.168.255.144/31 [0/1] is directly connected, mce0.3100, 00:04:12 C>* 192.168.255.146/31 [0/1] is directly connected, mce1.3100, 00:04:12
I also tried to use bird and it didn't manage to get routes from fib 1.
Hmmm.... :) I don't know how far the fib that can be set on the network interface will be used, but if it cannot be set at least there, it will be difficult to route packet forwarding (which sysctl net.inet.ip.forwarding is set to 1). A process launched in an environment where the fib is 0 inherits that the fib is 0. If the process switches FIBs in the middle, setfib or setsockopt should be running somewhere, but in that case, there is a config for that somewhere. I don't think there is a function to recognize the FIB of the network interface where the packet arrived and select the FIB of the packet to be sent. [citation needed] ...Therefore, it seems that the service for packets passing through the route of fib 1 should be set to use fib 1. single-mindedly :) And since lo0 is also a route that can be passed through various ways, it is sometimes better to create lo1 or lo2 and separate them. I don't know if it really is. I think that's what it means, but I don't think it's on target :)
I set fib 1 for all frr processes in rc.conf.d Thats why i expected that it took only fib 1 routes from the kernel but it didn't. frr_fib="1" frr_enable="YES" frr_vtysh_boot="YES" frr_daemons="zebra bfdd bgpd mgmtd" frr_config="/usr/local/etc/frr/frr.conf" bgpd_fib="1" bfdd_fib="1" zebra_fib="1" mgmtd_fib="1"
(In reply to Egor from comment #15) >I set fib 1 for all frr processes in rc.conf.d Thats why i expected that it took only >fib 1 routes from the kernel but it didn't. When FRR runs under FIB 1 it means that all the connections with peers will be done from that FIB. Nothing more. If you want to import routes from non-standard fib (FIB ≠ 0) then use option "ip import table" instead, which should also work when you will run FRR under default FIB. I don't know if our FRR port supports multiple routing tables correctly, but definitely net/bird2 copes fine with them. That's probably not a bug though, but I have not investigated this PR extensively.
I tried to use bird but it didn't up bgp sessions in fib 1. bird_enable="YES" bird_fib="1" log "/var/log/bird.log" all; log stderr all; router id 10.64.244.139; protocol device { } protocol kernel { # Primary routing table learn; # Learn alien routes from the kernel persist; # Do not remove routes on bird shutdown scan time 10; # Scan kernel routing table every 10 seconds kernel table 1; ipv4 { import all; export all; }; } protocol bfd { interface "mce*" { interval 100 ms; min rx interval 100 ms; multiplier 5; }; } filter vxlan_fabric { if (net ~ [ 172.16.0.0/15{15,16}]) then { accept; } } protocol bgp { local as 4230040015; neighbor 192.168.255.0 as 4230041240; hold 180; keepalive 60; bfd on; ipv4 { export filter vxlan_fabric; import all; next hop self; }; } bgp1 BGP --- start 13:55:50.007 Connect Socket: Connection refused BGP state: Connect Neighbor address: 192.168.255.0 Neighbor AS: 4230041240 Local AS: 4230040015 Last error: Socket: Connection refused Channel ipv4 State: DOWN Table: master4 Preference: 100 Input filter: ACCEPT Output filter: vxlan_fabric
Created attachment 253684 [details] bird rd.d bird rc.d script that allowes to run int in non-standart fib
Created attachment 253685 [details] bird.conf config for bird daemon
I managed to run bird in non standart fib and at this time there wasn't leaked routes in routing table but it didn't change traffic behavior. I still had SYN-ACK drops in packet filter and i didn't see SYN packets on outgoing interface. 09:20:24.123696 rule 5/0(match): pass in on mce0.1280: 172.16.179.42.51806 > 172.16.188.194.22: Flags [S], seq 2399537834, win 64240, options [mss 1460,[|tcp]> 09:20:24.123742 rule 1/0(match): block in on mce0.3101: 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 1440191918, ack 2399537835, win 65535, options [mss 1460,[|tcp]> 09:20:25.130047 rule 1/0(match): block in on mce0.3101: 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 1440191918, ack 2399537835, win 65535, options [mss 1460,[|tcp]> 09:20:27.330596 rule 1/0(match): block in on mce0.3101: 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 1440191918, ack 2399537835, win 65535, options [mss 1460,[|tcp]> 09:20:31.531341 rule 1/0(match): block in on mce0.3101: 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 1440191918, ack 2399537835, win 65535, options [mss 1460,[|tcp]> test-fw01:~$ sudo birdc "show route table all" | grep 172.16.188 -A 3 172.16.188.192/26 unicast [bgp18 09:19:04.118] * (100) [AS4231000004i] via 192.168.255.114 on mce1.3101 unicast [bgp17 09:19:04.225] (100) [AS4231000004i] via 192.168.255.112 on mce0.3101 test-fw01:~$ netstat -nrF 1 | grep 172.16.188.192 172.16.188.192/26 192.168.255.114 UG1 mce1.310 test-fw01:~$ sudo tcpdump -nli mce0.3101 host 172.16.179.42 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on mce0.3101, link-type EN10MB (Ethernet), capture size 262144 bytes 09:59:40.753971 IP 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 3265751352, ack 2606051427, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 738593035 ecr 3656452229], length 0 09:59:41.748887 IP 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 3265751352, ack 2606051427, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 738594032 ecr 3656452229], length 0 09:59:41.755927 IP 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 3265751352, ack 2606051427, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 738594032 ecr 3656453231], length 0 09:59:42.779932 IP 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 3265751352, ack 2606051427, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 738595063 ecr 3656454255], length 0 09:59:43.788132 IP 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 3265751352, ack 2606051427, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 738596071 ecr 3656454255], length 0 09:59:43.803928 IP 172.16.188.194.22 > 172.16.179.42.51806: Flags [S.], seq 3265751352, ack 2606051427, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 738596083 ecr
(In reply to Egor from comment #20) >I managed to run bird in non standart fib and at this time there wasn't >leaked routes in routing table That's good to hear. Please consider closing this PR if the routes are no longer leaking for you.
My problem hasn't solved. Traffic is still leaking but now it's implicit. I am seeing it in tcpdump and pflog.
(In reply to Egor from comment #22) >Traffic is still leaking but now it's implicit. This is expected behaviour. I can no longer help troubleshoot this PR, so I am unsubscribing.
I have been testing it and i found configuration where the problem was gone. It's pve8.2.7 + FreeBSD 13.4 + intel e1000 driver + bird2. Other configurations had traffic leak. I tested this configs: pve8, freebsd 13.4, intel e1000, bird2 = working pve8, freebsd 14.1, intel e1000, bird2 = not working pve8, freebsd 14.1, intel e1000, frr9 = not working pve7, freebsd 13.4, intel e1000, frr9 = not working pve7, freebsd 13.4, intel e1000, bird2 = not working pve7, freebsd 14.1, intel e1000, frr9 = not working
Hi @Egor, Before this issue get addressed, I'd highly recommend you to employ vnet jails to isolate the mgmt traffic. See also my previous reply to mailing list [1]. 1. https://lists.freebsd.org/archives/freebsd-net/2024-February/004627.html
Hello, Zhenlei Huang. I want to separate my traffic for two different routing tables. Jails looks like overhead that will make maintain of the system more complicated.
(In reply to Egor from comment #0) > Hello everyone. I met a problem with my Freebsd configuration. I used two fibs fib0 > for management and fib1 for traffic routing. When i tried to connect to my freebsd > my ssh session was closed by timeout. This session passed fib1 then it passed a > switch and then this traffic came to mgmt interface in fib0. (In reply to Egor from comment #26) > Hello, Zhenlei Huang. I want to separate my traffic for two different routing > tables. Jails looks like overhead that will make maintain of the system more complicated. So you set fib0 for management, and fib1 for traffic routing, that is good. For jail setup, it is quite simple. Just leave the host (vnet0) as management, and spawn a dedicated vnet jail (say vnet1) for traffic routing, and move all the interfaces those participate the traffic routing and routing daemons to vnet1. The architecture is more clear rather than more complicated. You will benefit separated firewall rules, fine tuned ( per vnet sysctl knobs ), robust OOB management, etc. Yes, the overhead is one more vnet jail and some setup.