Bug 239371

Summary: epair(4) doesn't expose network traffic to bpf(4), ex. wireshark doesn't see it
Product: Base System Reporter: Yuri Victorovich <yuri>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: Closed Works As Intended    
Severity: Affects Only Me CC: kp
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   

Description Yuri Victorovich freebsd_committer freebsd_triage 2019-07-22 05:23:19 UTC
The epair instance is assigned IP addresses:
epair0a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:30:0a:fb:89:0a
	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:30:0a:fb:89:0b
	inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Jail is created with 10.0.0.2. Jail can connect to the host on ssh to 10.0.0.2, but wireshark shows zero packets on epair0b.
Comment 1 Kristof Provost freebsd_committer freebsd_triage 2019-07-22 07:32:21 UTC
Is this amd64 or i386?
Comment 2 Yuri Victorovich freebsd_committer freebsd_triage 2019-07-22 07:38:48 UTC
(In reply to Kristof Provost from comment #1)

amd64
Comment 3 Yuri Victorovich freebsd_committer freebsd_triage 2019-07-22 07:40:30 UTC
lo interface suffers from the same problem when it is used in a similar way.
Jail sends packets into lo1, because it is the only interface with the IP address in a jail, but wireshark never sees any packets on lo1.
Comment 4 Kristof Provost freebsd_committer freebsd_triage 2019-07-22 07:42:07 UTC
I asked because there seem to be issues with bpf on i386. Those cause the pf tests to fail there.

Are these old-style (i.e. non-vnet) jails?
In that case I wonder if you're not looking on the wrong interface. Looped back packets always turn up on lo0, even if from looking at the IP addresses you'd expect to see them on the interface. (Because there's a more specific route for the IP that tells it to use lo0.)
Comment 5 Yuri Victorovich freebsd_committer freebsd_triage 2019-07-22 07:51:49 UTC
(In reply to Kristof Provost from comment #4)

Yes, it's a non-vnet jail.
I'm looking at the interface that has the internal IP address of jail.
Comment 6 Kristof Provost freebsd_committer freebsd_triage 2019-07-22 08:41:29 UTC
Yes, that sounds like the expected behaviour.

Take a look at `netstat -r`. You'll almost certainly see something like 
'10.0.0.2         link#1             UHS         lo0'
Comment 7 Yuri Victorovich freebsd_committer freebsd_triage 2019-07-22 09:17:52 UTC
(In reply to Kristof Provost from comment #6)

Actually, it is on lo1:

$ netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
10.0.0.2           link#5             UH          lo1

But watching lo1 with wireshark shows nothing even when the connection to the host system succeeds (to 192.168.5.5).
Is this expected?
Comment 8 Kristof Provost freebsd_committer freebsd_triage 2019-07-22 09:26:06 UTC
(In reply to Yuri Victorovich from comment #7)
I believe so, but I'm not at home in the details of the loopback interface.
Do you see your packets on lo0?
Comment 9 Yuri Victorovich freebsd_committer freebsd_triage 2019-07-22 16:41:50 UTC
(In reply to Kristof Provost from comment #8)

The routing table only has 10.0.0.2 on lo1.

When I send something to the host's IP 192.168.5.5 (which is in a different net), it indeed shows on lo0.
But when I send something to some other IP 96.47.72.132, it doesn't show on lo0 as well.

This behavior is confusing: 1. Why lo0 gets involved when it isn't used from jail? 2. Why packets to host's IP appear on lo0 and the packets to the remote IP don't even though both of the IP addresses are equally unroutable based on the jail's routing table?

I guess I need to change to vnet-based jails.
Comment 10 Kristof Provost freebsd_committer freebsd_triage 2019-07-23 08:57:37 UTC
(In reply to Yuri Victorovich from comment #9)
lo0 is used for loopback traffic, and that includes connections from a public IP of your machine to a public IP from your machine.

So, if your machine has re0, with 10.0.0.1/24 assigned to it, connecting to 10.0.0.1:1234 will pass over lo0, even though you might expect to see that traffic on re0. The same thing applies when that connection is made from a (non-vnet) jail to the host machine. It still applies if you have IP addresses in different nets. Anything to yourself will pass over lo0.

If you connect to a different (i.e. non-local) IP, you will see the traffic on re0. That's all expected behaviour.
Comment 11 Yuri Victorovich freebsd_committer freebsd_triage 2019-07-23 09:08:02 UTC
Thank you, Kristof, for your explanation.