Bug 234426 - SSH/SFTP connections from a 12.0-RELEASE VMware VM to the outside world are dropped with "ssh_packet_write_wait: broken pipe" errors
Summary: SSH/SFTP connections from a 12.0-RELEASE VMware VM to the outside world are d...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-26 20:49 UTC by Douglas Carmichael
Modified: 2019-10-02 18:00 UTC (History)
6 users (show)

See Also:


Attachments
tcpdump trace of an attempted SSH/SFTP session from the VM to the outside world. (14.71 KB, application/octet-stream)
2018-12-26 20:49 UTC, Douglas Carmichael
no flags Details
tcpdump trace of a successful SSH session to the VM from the outside world. (28.97 KB, application/octet-stream)
2018-12-26 20:49 UTC, Douglas Carmichael
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Douglas Carmichael 2018-12-26 20:49:06 UTC
Created attachment 200540 [details]
tcpdump trace of an attempted SSH/SFTP session from the VM to the outside world.

System: VMware Fusion 11.0.2 (build 10952296) running on macOS 10.14.2.

When I attempt to make SSH/SFTP connections from a 12.0-RELEASE VMware Fusion VM to the outside world, they are dropped with "ssh_packet_write_wait: Connection to (IP) port 22: broken pipe."

However, when I make SSH/SFTP connections from the outside world to the VM, they are successful.

I've attempted this both with OpenSSH in the base OS and OpenSSH from ports (security/openssh-server) and get the same issue.

I've even attempted changing the virtual Ethernet adapter from the default e1000/em driver to the vmxnet3/vmx driver, and still get the same issue.

Would this be a VMware problem, FreeBSD kernel problem, or both?
Comment 1 Douglas Carmichael 2018-12-26 20:49:47 UTC
Created attachment 200541 [details]
tcpdump trace of a successful SSH session to the VM from the outside world.
Comment 2 Yuri Pankov freebsd_committer 2018-12-26 23:27:23 UTC
See https://lists.freebsd.org/pipermail/freebsd-current/2018-December/072467.html.

Workaround here is adding the following to your ~/.ssh/config:

Host *
    IPQoS lowdelay throughput
Comment 3 Diego Linke 2018-12-31 19:49:38 UTC
I am facing this issue also in all of my 12.0-RELEASE VMs (upgraded from 11) running at AWS EC2 (kern.vm_guest: xen).

Unfortunately the proposed workaround by @Yuri Pankov didn't work. Just in case I also tried "ssh -o IPQoS=throughput ..." as mentioned at https://communities.vmware.com/thread/590825

Only SCP or any other non-interactive connection using remote commands and/or pipe are facing the broken packet_write_wait broken pipe error.
Comment 4 Dag-Erling Smørgrav freebsd_committer 2019-01-12 16:21:38 UTC
Are you saying that interactive ssh works fine, but scp doesn't?
Comment 5 Douglas Carmichael 2019-01-13 00:56:53 UTC
(In reply to Dag-Erling Smørgrav from comment #4)

Neither of them work from within the VM:

[dcarmich@dc-freebsd ~]$ ssh pfa3.x.rootbsd.net
Password for dcarmich@pfa3.x.rootbsd.net:
Fssh_packet_write_wait: Connection to 199.102.76.114 port 22: Broken pipe
[dcarmich@dc-freebsd ~]$ scp xorg.conf dcarmich@pfa3.x.rootbsd.net:.
Password for dcarmich@pfa3.x.rootbsd.net:
Fssh_packet_write_wait: Connection to 199.102.76.114 port 22: Broken pipe
lost connection
Comment 6 Diego Linke 2019-01-14 10:27:54 UTC
(In reply to Dag-Erling Smørgrav from comment #4)

@Dag-Erling maybe we are facing two very similar but different issue. I am wondering because this starts after 12.0-RELEASE and have the same symptom.

The SSH interactive sessions is working fine and looks like the keep alive was able to keep the connection working.

But bulk sessions (SCP, Rsync+SSH, SSH pipe dd) all of them soon or latter after the transmission starts will close with "ssh_packet_write_wait: Connection to (IP) port 22: broken pipe."

The SSH verbose mode, in both sides, are not being helpful. Each one blame another side to close the connection. PF and IPFW are disabled. 

Let me know if I can help with anything. 

Thanks,

Diego Linke