Bug 207877 - routing issue (some routes seem to stop being effective after some period)
Summary: routing issue (some routes seem to stop being effective after some period)
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.2-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-net mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-10 10:29 UTC by pg
Modified: 2016-08-08 08:24 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description pg 2016-03-10 10:29:29 UTC
Problem on 10.2-RELEASE-p12 with multiple network interfaces and PF (rules, NAT)
# ifconfig
<--cut-->
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
        ether 00:25:90:64:14:50
        inet 172.27.27.254 netmask 0xffffff00 broadcast 172.27.27.255
        inet 172.27.27.240 netmask 0xffffffff broadcast 172.27.27.240
        inet 172.27.27.252 netmask 0xffffffff broadcast 172.27.27.252
        inet 172.27.27.251 netmask 0xffffffff broadcast 172.27.27.251
        inet 172.27.27.250 netmask 0xffffffff broadcast 172.27.27.250
        inet 172.27.27.249 netmask 0xffffffff broadcast 172.27.27.249
        inet 172.27.27.248 netmask 0xffffffff broadcast 172.27.27.248
        inet 172.27.27.247 netmask 0xffffffff broadcast 172.27.27.247
        inet 172.27.27.246 netmask 0xffffffff broadcast 172.27.27.246
        inet 172.27.27.245 netmask 0xffffffff broadcast 172.27.27.245
        inet 172.27.27.244 netmask 0xffffffff broadcast 172.27.27.244
        inet 172.27.27.243 netmask 0xffffffff broadcast 172.27.27.243
        inet 172.27.27.242 netmask 0xffffffff broadcast 172.27.27.242
        inet 172.27.27.241 netmask 0xffffffff broadcast 172.27.27.241
        inet 172.27.27.239 netmask 0xffffffff broadcast 172.27.27.239
        inet 172.27.27.238 netmask 0xffffffff broadcast 172.27.27.238
        inet 172.27.27.237 netmask 0xffffffff broadcast 172.27.27.237
        inet 172.27.27.236 netmask 0xffffffff broadcast 172.27.27.236
        inet 172.27.27.235 netmask 0xffffffff broadcast 172.27.27.235
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
        ether 00:25:90:64:14:51
        inet 10.24.44.50 netmask 0xfffffc00 broadcast 10.24.47.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
<--cut-->

em0 - local network. Aliases on em0 for Jail's
em1 - connected to ISP

This configuration successfully operate a long time but recently (for no apparent reason) suddenly became manifest bug with routing.
Regardless from uptime (sometimes 12 hours, and maybe a couple of days) after a certain time the traffic running between jails begins to route via em1 (instead lo0 (net.link.ether.inet.useloopback=1)):
# tcpdump -n -e -ttt -i em1 host 172.27.27.252 or host 172.27.27.247 or host 172.27.27.248
00:00:00.054989 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161568673 ecr 0], length 0
00:00:00.318036 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.60176: Flags [.], ack 772660872, win 1275, options [nop,nop,TS val 1737287913 ecr 161568991], length 0
00:00:00.036971 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.33080: Flags [.], ack 1, win 1275, options [nop,nop,TS val 1701623130 ecr 161568238], length 0
00:00:00.050001 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.40200: Flags [.], ack 1, win 1275, options [nop,nop,TS val 1477098721 ecr 161568288], length 0
00:00:02.794992 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161571873 ecr 0], length 0
00:00:03.312026 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.29581 > 172.27.27.252.25: Flags [S], seq 1217253021, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161575185 ecr 0], length 0
00:00:02.887982 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161578073 ecr 0], length 0
00:00:00.111989 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.29581 > 172.27.27.252.25: Flags [S], seq 1217253021, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161578185 ecr 0], length 0
00:00:03.471007 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 108: 172.27.27.247.80 > 172.27.27.248.26056: Flags [FP.], seq 1448:1490, ack 1, win 1275, options [nop,nop,TS val 2142270801 ecr 161575507], length 42

In the same time:
# route -vn get -host
RTA_DST: inet 172.27.27.247; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
 172.27.27.247 link#0
   route to: 172.27.27.247
destination: 172.27.27.247
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight expire
       0         0         0         0     16384         1         0

locks:  inits:
sockaddrs: <DST,GATEWAY,IFP,IFA>
 172.27.27.247 link#4 lo0 127.0.0.1
RTA_DST: inet 172.27.27.250; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
 172.27.27.250 link#0
   route to: 172.27.27.250
destination: 172.27.27.250
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight expire
       0         0         0         0     16384         1         0

locks:  inits:
sockaddrs: <DST,GATEWAY,IFP,IFA>
 172.27.27.250 link#4 lo0 127.0.0.1
RTA_DST: inet 172.27.27.252; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
 172.27.27.252 link#0
   route to: 172.27.27.252
destination: 172.27.27.252
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight expire
       0         0         0         0     16384         1         0

locks:  inits:
sockaddrs: <DST,GATEWAY,IFP,IFA>
 172.27.27.252 link#4 lo0 127.0.0.1


# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            10.24.44.1         UGS         em1
10.24.44.0/22      link#5             U           em1
10.24.44.50        link#5             UHS         lo0
127.0.0.1          link#7             UH          lo0
172.27.27.0/24     link#4             U           em0
172.27.27.235      link#4             UHS         lo0
172.27.27.235/32   link#4             U           em0
172.27.27.236      link#4             UHS         lo0
172.27.27.236/32   link#4             U           em0
172.27.27.237      link#4             UHS         lo0
172.27.27.237/32   link#4             U           em0
172.27.27.238      link#4             UHS         lo0
172.27.27.238/32   link#4             U           em0
172.27.27.239      link#4             UHS         lo0
172.27.27.239/32   link#4             U           em0
172.27.27.240      link#4             UHS         lo0
172.27.27.240/32   link#4             U           em0
172.27.27.241      link#4             UHS         lo0
172.27.27.241/32   link#4             U           em0
172.27.27.242      link#4             UHS         lo0
172.27.27.242/32   link#4             U           em0
172.27.27.243      link#4             UHS         lo0
172.27.27.243/32   link#4             U           em0
172.27.27.244      link#4             UHS         lo0
172.27.27.244/32   link#4             U           em0
172.27.27.245      link#4             UHS         lo0
172.27.27.245/32   link#4             U           em0
172.27.27.246      link#4             UHS         lo0
172.27.27.246/32   link#4             U           em0
172.27.27.247      link#4             UHS         lo0
172.27.27.247/32   link#4             U           em0
172.27.27.248      link#4             UHS         lo0
172.27.27.248/32   link#4             U           em0
172.27.27.249      link#4             UHS         lo0
172.27.27.249/32   link#4             U           em0
172.27.27.250      link#4             UHS         lo0
172.27.27.250/32   link#4             U           em0
172.27.27.251      link#4             UHS         lo0
172.27.27.251/32   link#4             U           em0
172.27.27.252      link#4             UHS         lo0
172.27.27.252/32   link#4             U           em0
172.27.27.254      link#4             UHS         lo0
172.27.28.0/24     link#8             U       bridge0
172.27.28.254      link#8             UHS         lo0
172.27.30.0/24     172.27.30.2        UGS        tun0
172.27.30.1        link#12            UHS         lo0
172.27.30.2        link#12            UH         tun0

After the reboot everything works well for some time.