Bug 234426
| Summary: | SSH/SFTP connections from a 12.0-RELEASE VMware VM to the outside world are dropped with "ssh_packet_write_wait: broken pipe" errors | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Douglas Carmichael <dcarmich> | ||||||
| Component: | misc | Assignee: | freebsd-virtualization (Nobody) <virtualization> | ||||||
| Status: | Closed FIXED | ||||||||
| Severity: | Affects Some People | CC: | dcarmich, des, diego, dutchman01, mmpestorich, mp, yuripv | ||||||
| Priority: | --- | ||||||||
| Version: | 12.0-RELEASE | ||||||||
| Hardware: | amd64 | ||||||||
| OS: | Any | ||||||||
| Attachments: |
|
||||||||
Created attachment 200541 [details]
tcpdump trace of a successful SSH session to the VM from the outside world.
See https://lists.freebsd.org/pipermail/freebsd-current/2018-December/072467.html. Workaround here is adding the following to your ~/.ssh/config: Host * IPQoS lowdelay throughput I am facing this issue also in all of my 12.0-RELEASE VMs (upgraded from 11) running at AWS EC2 (kern.vm_guest: xen). Unfortunately the proposed workaround by @Yuri Pankov didn't work. Just in case I also tried "ssh -o IPQoS=throughput ..." as mentioned at https://communities.vmware.com/thread/590825 Only SCP or any other non-interactive connection using remote commands and/or pipe are facing the broken packet_write_wait broken pipe error. Are you saying that interactive ssh works fine, but scp doesn't? (In reply to Dag-Erling Smørgrav from comment #4) Neither of them work from within the VM: [dcarmich@dc-freebsd ~]$ ssh pfa3.x.rootbsd.net Password for dcarmich@pfa3.x.rootbsd.net: Fssh_packet_write_wait: Connection to 199.102.76.114 port 22: Broken pipe [dcarmich@dc-freebsd ~]$ scp xorg.conf dcarmich@pfa3.x.rootbsd.net:. Password for dcarmich@pfa3.x.rootbsd.net: Fssh_packet_write_wait: Connection to 199.102.76.114 port 22: Broken pipe lost connection (In reply to Dag-Erling Smørgrav from comment #4) @Dag-Erling maybe we are facing two very similar but different issue. I am wondering because this starts after 12.0-RELEASE and have the same symptom. The SSH interactive sessions is working fine and looks like the keep alive was able to keep the connection working. But bulk sessions (SCP, Rsync+SSH, SSH pipe dd) all of them soon or latter after the transmission starts will close with "ssh_packet_write_wait: Connection to (IP) port 22: broken pipe." The SSH verbose mode, in both sides, are not being helpful. Each one blame another side to close the connection. PF and IPFW are disabled. Let me know if I can help with anything. Thanks, Diego Linke The issue with the VMware NAT resetting ssh sessions (due to the openssh 7.8 IPQoS change) was fixed in Fusion 11.1.0 and Workstation 15.1.0. A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d55bf492f8f587e4a99f4dcb39a96159b4431782 commit d55bf492f8f587e4a99f4dcb39a96159b4431782 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2021-04-25 21:14:23 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2021-04-25 21:17:22 +0000 Revert "Add workaround for a QoS-related bug in VMWare Workstation." This reverts commit 77c2fe20df6a9a7c1a353e1a4ab2ba80fefab881. The VMware Workstation issue was fixed in 2019[1], and we'd rather not carry unnecessary local changes in OpenSSH. [1] https://communities.vmware.com/t5/VMware-Workstation-Pro/Regression-ssh-results-in-broken-pipe-upon-connecting-in-Vmware/m-p/486105/highlight/true#M25470 PR: 234426 Discussed with: yuripv Approved by: des MFC after: 2 weeks Sponsored by: The FreeBSD Foundation crypto/openssh/readconf.c | 22 ---------------------- secure/usr.bin/ssh/Makefile | 3 --- 2 files changed, 25 deletions(-) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6fd4891545c2a6d06dbc1927b2e0b375cd2b0b17 commit 6fd4891545c2a6d06dbc1927b2e0b375cd2b0b17 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2021-04-25 21:14:23 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2021-05-09 01:45:55 +0000 Revert "Add workaround for a QoS-related bug in VMWare Workstation." This reverts commit 77c2fe20df6a9a7c1a353e1a4ab2ba80fefab881. The VMware Workstation issue was fixed in 2019[1], and we'd rather not carry unnecessary local changes in OpenSSH. [1] https://communities.vmware.com/t5/VMware-Workstation-Pro/Regression-ssh-results-in-broken-pipe-upon-connecting-in-Vmware/m-p/486105/highlight/true#M25470 PR: 234426 Discussed with: yuripv Approved by: des MFC after: 2 weeks Sponsored by: The FreeBSD Foundation (cherry picked from commit d55bf492f8f587e4a99f4dcb39a96159b4431782) crypto/openssh/readconf.c | 22 ---------------------- secure/usr.bin/ssh/Makefile | 3 --- 2 files changed, 25 deletions(-) A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=afcad366c43a50f23871cd3fe97b1aad845124be commit afcad366c43a50f23871cd3fe97b1aad845124be Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2021-04-25 21:14:23 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2021-12-12 22:19:56 +0000 Revert "Add workaround for a QoS-related bug in VMWare Workstation." This reverts commit 77c2fe20df6a9a7c1a353e1a4ab2ba80fefab881. The VMware Workstation issue was fixed in 2019[1], and we'd rather not carry unnecessary local changes in OpenSSH. [1] https://communities.vmware.com/t5/VMware-Workstation-Pro/Regression-ssh-results-in-broken-pipe-upon-connecting-in-Vmware/m-p/486105/highlight/true#M25470 PR: 234426 Discussed with: yuripv Approved by: des MFC after: 2 weeks Sponsored by: The FreeBSD Foundation (cherry picked from commit d55bf492f8f587e4a99f4dcb39a96159b4431782) (cherry picked from commit 6fd4891545c2a6d06dbc1927b2e0b375cd2b0b17) crypto/openssh/readconf.c | 22 ---------------------- secure/usr.bin/ssh/Makefile | 3 --- 2 files changed, 25 deletions(-) |
Created attachment 200540 [details] tcpdump trace of an attempted SSH/SFTP session from the VM to the outside world. System: VMware Fusion 11.0.2 (build 10952296) running on macOS 10.14.2. When I attempt to make SSH/SFTP connections from a 12.0-RELEASE VMware Fusion VM to the outside world, they are dropped with "ssh_packet_write_wait: Connection to (IP) port 22: broken pipe." However, when I make SSH/SFTP connections from the outside world to the VM, they are successful. I've attempted this both with OpenSSH in the base OS and OpenSSH from ports (security/openssh-server) and get the same issue. I've even attempted changing the virtual Ethernet adapter from the default e1000/em driver to the vmxnet3/vmx driver, and still get the same issue. Would this be a VMware problem, FreeBSD kernel problem, or both?