Bug 242894 - bhyve: TCP / network disconnections between 12.0-RELEASE (guest) and 12.1-RELEASE (hypervisor)
Summary: bhyve: TCP / network disconnections between 12.0-RELEASE (guest) and 12.1-REL...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-virtualization (Nobody)
Keywords: needs-qa
Depends on:
Reported: 2019-12-26 15:18 UTC by IPTRACE
Modified: 2020-03-20 09:24 UTC (History)
2 users (show)

See Also:

dmesg, ifconfig, etc. (43.59 KB, text/plain)
2019-12-27 19:43 UTC, IPTRACE
no flags Details
dmesg, ifconfig, etc. (7.45 KB, text/plain)
2019-12-27 19:44 UTC, IPTRACE
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description IPTRACE 2019-12-26 15:18:32 UTC

I've got an issue with TCP/IP network connection (including ICMP).
I suppose there is something wrong between hypervisor and guest network stack.

Hypervisor: bhyve on FreeBSD 12.1-RELEASE just upgraded
Guest: FreeBSD 12.0-RELEASE before upgrade

Hypervisor: bridge and tap for guest
Guest: vtnet virtual netowrk card

So, I was trying tu upgrade guest to 12.1-RELEASE. After a few attempts where I was losing SSH connection, I saw there is an issue with network connection. Deep investigation gave me the information, that after some time of network traffic I was disconnected from network on TCP/IP stack, both bridge, tap and vtnet network cards were enabled and active. Then after some minutes traffic went well.
I was trying to download bigger file from Internet (100 MB), after 5 MB of downloading I lost network connection.

I have others 12.1-RELEASES and they work. I upgraded them before upgraded hypervisor. So, when hypervisor was as 12.0-RELEASE I hadn't got an issue.
I tried another guest (12.0-RELEASE) and the problem is the same.

To be more precise about my architecture, between hypervisor and that guest is another guest called firewall, it means the firewall pass the traffic between Internet and not upgraded guest yet I have an issue. The firewall (guest) is already upgraded to 12.1-RELEASE. Of course, I upgraded that guest before upgraded hypervisor.
Comment 1 IPTRACE 2019-12-26 20:34:35 UTC
% service netif restart
% service routing restart

commands helped to start the network again for a while but the issue still occurs.
Comment 2 Jason Tubnor 2019-12-26 23:57:19 UTC
What network interfaces (physical) are you using?

I also had issues with early versions of 12.0.  There were significant issues that I avoided the release until 12.1.  One of the issues were around IPv6.

As 12.0 is EoL, I'd step the guest up to 12.1.
Comment 3 IPTRACE 2019-12-27 00:17:42 UTC
1 GB/s of Intel card but it does not matter in this case.
There is an issue between 12.0-RELEASE-p10 and 12.1-RELEASE-p1.

I'm upgrading now but it's difficult during network access problem.
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-27 02:33:19 UTC
@IPTRACE, Could you please provide additional information, including:

- /var/run/dmesg.boot output (for guest and host separately) as attachments
- Complete network configuration (sanitized where necessary, for guest and host separately) as attachments
- Any relevant logs in dmesg / console when disconnects occur
- Ideally, tcpdump packet captures when the disconnections take place
Comment 5 Jason Tubnor 2019-12-27 03:13:47 UTC
(In reply to IPTRACE from comment #3)

If it is the igb interface and you are using VLANs, I had to disable VLAN Hardware Tagging (-vlanhwtag) on a 12.1 host to allow full use of the physical interface speed for the guest.  There appears to be a bug in the driver.
Comment 6 IPTRACE 2019-12-27 19:43:41 UTC
Created attachment 210261 [details]
dmesg, ifconfig, etc.
Comment 7 IPTRACE 2019-12-27 19:44:00 UTC
Created attachment 210262 [details]
dmesg, ifconfig, etc.
Comment 8 IPTRACE 2019-12-27 20:31:31 UTC
There isn't any VLANS.
Comment 9 IPTRACE 2019-12-27 22:05:24 UTC
Little workaround.

If I use the following commands, upgrade will continue without network disruption.

% service netif restart
% service routing restart

Restart OS does not help in such case like above.
Comment 10 Ole 2020-03-20 09:24:33 UTC
I ran into the same Problem with FreeBSD 12.0 and 12.1. It is documented here:


Solution was to configure the interface with -vlanhwtag on the host before starting the rest. (via rc.conf)

It occurred with both igb and em drivers for me.