Bug 235342

Summary: em(4): em0 hangs when vnet jail starts while transferring files
Product: Base System Reporter: Yuichiro NAITO <naito.yuichiro>
Component: kernAssignee: Eric Joyner <erj>
Status: Closed FIXED    
Severity: Affects Some People CC: bz, d8zNeCFG, net, pi, pkubaj
Priority: --- Keywords: IntelNetworking
Version: CURRENTFlags: pkubaj: mfc-stable12+
koobs: mfc-stable11-
Hardware: amd64   
OS: Any   

Description Yuichiro NAITO 2019-01-31 06:46:30 UTC
I'm using "FreeBSD 13.0-CURRENT r343486 GENERIC-NODEBUG amd64" with following NIC.

em0@pci0:0:31:6:	class=0x020000 card=0x00008086 chip=0x15b78086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection (2) I219-LM'
    class      = network
    subclass   = ethernet

A following scenario causes em driver hangs.

1. start file transfer through em0.
   I used sftp or netatalk. Both of them causes same result.

2. start vnet jail while transferring the file.
   I used iocage with vnet=on dhcp=on bpf=yes.
   Without iocage using jail.conf causes same result.

3. "em0: TX(0) desc avail = 1024, pidx = 0" is shown on console repetitive and infinitely.

After 3., em0 hangs and no packets can be sent and/or received.
ifconfig up/down won't recover this hang up.

I've tried my jail with iocage set vnet=on dhcp=off bpf=no,
Nothing changed.

Upgrading to 'FreeBSD 13.0-CURRENT r343598 GENERIC-NODEBUG amd64'
won't fix this problem.

Workaround is installing net/intel-em-kmod for now.
Comment 1 Bjoern A. Zeeb freebsd_committer freebsd_triage 2019-01-31 10:19:04 UTC
How dos the vnet jail get connected to the outside world?  Does iocage also put in epairs and a bridge to em0?
Comment 2 Yuichiro NAITO 2019-01-31 14:25:25 UTC
(In reply to Bjoern A. Zeeb from comment #1)
Yes, iocage sets up bridge0 and epair0 as following.

 em0 ---- bridge0 ---- epair0a:epair0b
                              |
                        host  | vnet jail

Actually 'epair0a' is renamed to 'vnet0.1'.
bridge0 includes just two members.
Comment 3 Bjoern A. Zeeb freebsd_committer freebsd_triage 2019-01-31 14:28:58 UTC
In that case, on a system without jails (yet), does a (typing from the top of my head):

ifconfig bridge0 create
ifconfig bridge0 up
ifconfig bridge0 addm em0

have a similar effect already?
Comment 4 Yuichiro NAITO 2019-01-31 15:20:33 UTC
Yes, before starting vnet jail, bridge0 is already set up.
It is set up by iohyve that I configured in /etc/rc.conf as following.

```
iohyve_enable="YES"
iohyve_flags="kmod=1 net=em0"
```

ifconfig shows as follows. (I masked my IPv6 address)

```
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
	ether 80:ee:73:ba:35:5b
	inet6 fe80::xxxx:xxxx:xxxx:xxxx%em0 prefixlen 64 scopeid 0x1
	inet6 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx prefixlen 64 autoconf
	inet 192.168.3.3 netmask 0xffffff00 broadcast 192.168.3.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: iohyve-bridge-em0
	ether 02:7a:0d:e2:01:00
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 20000
	groups: bridge
	nd6 options=9<PERFORMNUD,IFDISABLED>
```
Comment 5 Martin Birgmeier 2019-01-31 15:30:31 UTC
Could this be related to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235031#c17 ?
Comment 6 Yuichiro NAITO 2019-02-01 08:03:05 UTC
(In reply to Martin Birgmeier from comment #5)
I'm not sure this problem is related to #235031.                                     
With `ifconfig em0 lro` and ` ifconfig em0 -lro`, nothing happens to me.
It seems that error messages are same.
Comment 7 Yuichiro NAITO 2020-01-24 02:09:28 UTC
r356310 has fixed this problem. Thanks!
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-24 03:19:18 UTC
^Triage: 

 - Assign to committer resolving (base r239240)
 - Mark no mfc-stable11 (assuming iflib conversions didnt make it there). Please reset mfc-stable11 flag (to ?) if it is indeed going to be merged there
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-24 03:20:40 UTC
^Triage: Re-open pending merge. @Eric could you please include this PR: reference in the merge so that the merge is tracked in this issue. Thanks!