Bug 255497 - net/wireguard-kmod: after update to wireguard-kmod-0.0.20210424_1 or 0.0.20210428 data going through vpn channel was dropped partly
Summary: net/wireguard-kmod: after update to wireguard-kmod-0.0.20210424_1 or 0.0.2021...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Bernhard Froehlich
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-04-29 18:48 UTC by Oleg Strizhak
Modified: 2021-05-05 22:29 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Oleg Strizhak 2021-04-29 18:48:44 UTC
Hi all,

I've to report a bug in wiregaurd-kmod. Here's my system:

FreeBSD `hostname` 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr  9 04:24:09 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

latest binary update, gitup'd latest release/13.0 src & ports.

After update to wireguard-kmod-0.0.20210424_1 data going through vpn channel was dropped partly. It happened if the size of session is larger than megabyte, so the ping works well - that's why I detected it after some time: my clients started to worry that theirs mail (smtp via wg0) are tampered while sending - the pictures got artifacts, archives came broken etc. At first, I've told'em to stop whinning and stop using m$ software as I several million times alreday recommend before =)) But in parallel I started to check things on the servers, just to asure myself, and the first simple test fails =(

[host] /home/user $ scp -p /tmp/k.zip user@192.168.x.y:/tmp/
Password for user@host:
k.zip			3%    0     5.5MB/s   00:10 ETAFssh_packet_write_wait: Connection to 192.168.x.y port 22: Broken pipe
lost connection

... again with verbose info:

[host] /home/user $ scp -vp /tmp/k.zip user@192.168.x.y:/tmp/
Executing: program /usr/bin/ssh host 192.168.x.y, user user, command scp -v -p -t /tmp/
OpenSSH_7.9p1, OpenSSL 1.1.1k-freebsd  25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 192.168.x.y [192.168.x.y] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type 3
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9 FreeBSD-20200214
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9 FreeBSD-20200214
debug1: match: OpenSSH_7.9 FreeBSD-20200214 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.x.y:22 as 'user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:nA1uA8Ii0BI3oMOZRjjrdJOp3Jo1voo7EGv6h45ZXJ8
debug1: skipped DNS lookup for numerical hostname
debug1: Host '192.168.x.y' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:10
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /root/.ssh/id_rsa 
debug1: Will attempt key: /root/.ssh/id_dsa 
debug1: Will attempt key: /root/.ssh/id_ecdsa 
debug1: Will attempt key: /root/.ssh/id_ed25519 ED25519 SHA256:..xxx...
debug1: Will attempt key: /root/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: Fssh_kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Offering public key: /root/.ssh/id_ed25519 ED25519 SHA256:..xxx...
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_xmss
debug1: Next authentication method: keyboard-interactive
Password for user@host:
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to 192.168.x.y ([192.168.x.y]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@o.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending command: scp -v -p -t /tmp/
File mtime 1619619978 atime 1619619968
Sending file timestamps: T1619619978 0 1619619968 0
Sink: T1619619978 0 1619619968 0
Sending file modes: C0644 59681372 k.zip
Sink: C0644 59681372 k.zip
k.zip			0%    0     0.0KB/s   --:-- ETAFssh_packet_write_wait: Connection to 192.168.x.y port 22: Broken pipe
lost connection

Re-installation & restart of wireguard-kmod-0.0.20210424_1

$ scp -p /tmp/k.zip user@192.168.x.y:/tmp/
Password for user@host:
k.zip			0%    0     0.0KB/s   --:-- ETAFssh_packet_write_wait: Connection to 192.168.x.y port 22: Broken pipe

$ scp -p /tmp/k.zip user@192.168.x.y:/tmp/
k.zip			18%   11MB   5.2MB/s   00:08 ETAFssh_packet_write_wait: Connection to 192.168.x.y port 22: Broken pipe
lost connection

Then I downgraded to previous build,

===>>> Upgrade of wireguard-kmod-0.0.20210424_1 to wireguard-kmod-0.0.20210415 complete

and since then everуthing was correct (I also checked it w/ checksum, copied different files, etc.):

$ scp -p /tmp/k.zip user@192.168.x.y:/tmp/
Password for user@host:
k.zip			100%   57MB   4.5MB/s   00:12

The other side is FreeBSD 12.2-RELEASE-p6 i386, latest binary update, gitup'd ports & src, wireguard-kmod-0.0.20210424_1 works good, as well as previous build - here everething seens to be ok on any build number

update: on the other size (12.2) there's a dovecot pop3 server and, in spite the fact that file scp'ing well, my clients had errors with wireguard-kmod-0.0.20210424_1: if mail is more than a couple of megabytes, some attachments are broken (see above). I downgraded it to wireguard-kmod-0.0.20210415 too and the problem seems to stop.
Comment 1 Bernhard Froehlich freebsd_committer 2021-04-29 18:54:32 UTC
Today a new snapshot was commited which includes many bugfixes. Could you please check if it also fixes your bug?
Comment 2 Oleg Strizhak 2021-04-29 19:00:48 UTC
(In reply to Bernhard Froehlich from comment #1)
I've already tried it on amd64 side (w/ backup 0.0.20210415) and after upgrade and again  couldn't scp files - it suddenly stops w/ the same error: Broken pipe. Lost connections

Maybe I'd install the same versions on both side simultaneously, but first I'd like to work out the trustworthy testing procedure
Comment 3 Jason A. Donenfeld 2021-04-30 05:09:06 UTC
Can you let me know whether the latest version fixes this? I'm not going to spend time looking into bugs with old snapshots.
Comment 4 Oleg Strizhak 2021-04-30 06:33:04 UTC
(In reply to Jason A. Donenfeld from comment #3)
Sure. Once I'd tried the latest build already but now I'm going to do a bullet-proof test - once again update the ports & source tree, compile it on both ends, and check it only after all these procedures. Moreover, the channel is used in offices work routine, and I can thoroughfully test it after working hours only. I'll send you my report asap.
Thank you for useful and cool soft and willingness to help =)
Comment 5 Jason A. Donenfeld 2021-04-30 07:05:49 UTC
I was able to transfer a 10G file using scp in both directions using the latest wireguard-kmod snapshot. So at least that's a good sign. But I'd be very interested to learn if your own setup poses problems.
Comment 6 Jason A. Donenfeld 2021-04-30 07:27:09 UTC
Unfortunately, I am also unable to produce this with the previous snapshot too. So I'll need much more information from you in order to make any progress with this bug. Otherwise it will likely remain unfixed.
Comment 7 Oleg Strizhak 2021-04-30 07:35:32 UTC
(In reply to Jason A. Donenfeld from comment #6)
You may see my update for the original post: scp works good, but there're errors while fetching rather big mail via pop3 =)) why? Plain old pop3.. how? And at the same time I can transfer big binary data in both direction w/o any errors.. I'm surprised and wanna clear the case before the schedule update as I use this setup on several servers
Comment 8 Jason A. Donenfeld 2021-04-30 07:42:07 UTC
I'm not sure I follow. I think it's probably best if you test using the latest snapshot, and if the problem persists there, describe the complete bug, along with reproduction steps.
Comment 9 Oleg Strizhak 2021-04-30 22:23:37 UTC
(In reply to Jason A. Donenfeld from comment #8)
no, it doesn't do scp =( How I tested:

At one end [FreeBSD 12.2-RELEASE-p6 GENERIC i386] I updated src & ports:
From https://git.FreeBSD.org/src
   eda28feb2e0..2f32a971b7f  main       -> freebsd/main
   e4419043d1d..51b2d043692  stable/13  -> freebsd/stable/13
Already up to date.
From https://git.FreeBSD.org/ports
   3036033a13db..bbd93cbbafce  main       -> freebsd/main
Updating 3036033a13db..bbd93cbbafce

built latest wireguard-kmod (build.log can supply if you need it):
...
ld -m elf_i386_fbsd -d -warn-common --build-id=sha1 -r -d -o if_wg.kld if_wg.o wg_noise.o wg_cookie.o crypto.o
:> export_syms
awk -f /usr/src/sys/conf/kmod_syms.awk if_wg.kld  export_syms | xargs -J% objcopy % if_wg.kld
--- if_wg.ko ---
ld -m elf_i386_fbsd -Bshareable -znotext -znorelro -d -warn-common --build-id=sha1  -o if_wg.ko if_wg.kld
objcopy --strip-debug if_wg.ko
===>  Staging for wireguard-kmod-0.0.20210428
===>   Generating temporary packing list
install -T release -o root -g wheel -m 555   if_wg.ko /usr/ports/net/wireguard-kmod/work/stage/boot/modules/
====> Compressing man pages (compress-man)

and upgraded:
...
===>>> Upgrade of wireguard-kmod-0.0.20210415 to wireguard-kmod-0.0.20210428 complete

service wireguard restart
$ wg
interface: wg0
  public key: aaa
  private key: (hidden)
  listening port: bbb

peer: bbb
  preshared key: (hidden)
  endpoint: x.y.z:aaa
  allowed ips: xx.yy.12.0/24, xx.yy.2.0/24, a.b.c.0/24
  latest handshake: 5 seconds ago
  transfer: 1.96 MiB received, 42.18 MiB sent

Then I did just the same prodcedures at the other end [FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr  9 04:24:09 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64]:
$ ( gitup release ; gitup ports )
# Host: git.freebsd.org
# Port: 443
# Repository Path: /src.git
# Target Directory: /usr/src
# Have: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
# Want: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
# Branch: releng/13.0
# Host: git.freebsd.org
# Port: 443
# Repository Path: /ports.git
# Target Directory: /usr/ports
# Have: 3036033a13db7f623e4f121eb7f70675e7c122e7
# Want: bbd93cbbafce73c647fd4b0c8bf34163e003f04a
# Branch: main
# Action: pull
...
===>>> Upgrade of wireguard-kmod-0.0.20210415 to wireguard-kmod-0.0.20210428 complete
...
$ wg
interface: wg0
  public key: bbb
  private key: (hidden)
  listening port: aaa

peer: aaa
  preshared key: (hidden)
  endpoint: z.y.x:bbb
  allowed ips: xx.yy.11.0/24, xx.yy.0.0/24, a.b.c.0/24
  latest handshake: 9 seconds ago
  transfer: 110.81 KiB received, 15.43 KiB sent

Then I trying to scp big (256Mb) file:
$ scp -p /var/db/mysql/ib_logfile1 user@xx.yy.0.11:/tmp/
ib_logfile1        0%    0     0.0KB/s   --:-- ETAFssh_packet_write_wait: Connection to xx.yy.0.11 port 22: Broken pipe
lost connection

trying to scp at another end (12.2 i386):
$ scp -p /boot/kernel/kernel user@xx.yy.2.1:/tmp/kern.tmp
kernel        0%    0     0.0KB/s   --:-- ETAFssh_packet_write_wait: Connection to xx.yy.2.1 port 22: Broken pipe
lost connection

I've tried several times - the same error just at start; as you see, this config is useless and I'd to downgrade, and for
testing purpose I downgrade wireguard-kmod at one end (13.0 amd64):
Installing wireguard-kmod-0.0.20210415...

then restart service wireguard (at 13.0 only) and tried scp:
$ scp -p /var/db/mysql/ib_logfile1 user@xx.yy.0.11:/tmp/
ib_logfile1       100%  256MB   4.8MB/s   00:52

yes, it copied file, md5 is equal at both ends.

Now I tried to scp at another end:
$ scp -p /boot/kernel/kernel user@xx.yy.2.1:/tmp/kern.tmp
kernel        0%    0     0.0KB/s   --:-- ETAFssh_packet_write_wait: Connection to xx.yy.2.1 port 22: Broken pipe

restarted wg, but got the same error.

Then I downgraded wireguard-kmod  at this end too, and now everything is ok: i can copy files in both directions, md5 is equal.
Comment 10 Jason A. Donenfeld 2021-05-03 08:03:07 UTC
Thanks for the details. I'm still not able to reproduce it. I'm curious: what kind of physical NIC do you have on those boxes?

Either way, I've now released a new snapshot, which might fix it: https://lists.zx2c4.com/pipermail/wireguard/2021-May/006694.html

Waiting for decke to bump the port.
Comment 11 Jason A. Donenfeld 2021-05-03 19:53:09 UTC
Oleg - it looks like decke has bumped the port to the newer version. Can you run that and let me know if the problem is fixed for you?
Comment 12 Oleg Strizhak 2021-05-03 20:29:41 UTC
(In reply to Jason A. Donenfeld from comment #10)
Hi Jason, first of all I want to thank you alot for your time and willingness to help!

I've tried new build, installed is at both ends and scp works perfect - I've transferred several megabytes files in both directions (scp-ing at the same time), then md5'em - everything was ok. But then I've tried pop3 - as you may remember that is the main problem, that occured at aour offices, and it's still here =( The pop3 testing procedure is quite simple: using nc, I get several mail (about 20Mb in total) on each end into file, and then compare those files. If I use wireguard-kmod-0.0.20210415 (and, obviously, any preceeding build) there's no differences. When wireguard-kmod-0.0.20210502 was installed I got the following diff:

...
@@ -nnn +nnn,26 @@
-AgA+AAAAAAAAAAEAIAAAABa20wBSZWNpZXZlciBNQVgyMTIxQl9BRFM1NDA0XzUuMDQ
\ No newline at end of file
+AgA+AAAAAAAAAAEAIAAAABa20wBSZWNpZXZlciBNQVgyMTIxQl9BRFM1NDA0XzUuMDQuMjAyMS+C
+IK/grqinoq6k4eKiri9wY2IyMTIxLkdCTFBLAQIUABQAAgAIAKQKhVLs3R4pHywAAC21AAA+AAAA
...
+IK/grqinoq6k4eKiri+RjyBNQVgyMTIxQl9BRFM1NDA0Lnhsc1BLBQYAAAAALgAuADITAACPrREB
+AAA=
+
+------eA2c4EfD16F07F418Bd21b1Cb8586596-DiIhk0KR2sG0lAaE-1619791567--

as you may see, if the message's got through the wg0 channel it's truncated.

Concerning you question about NIC, here you are: on 12.2 i386 it's 

sis0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=83808<VLAN_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,LINKSTATE>
	media: Ethernet autoselect (100baseTX <full-duplex>)
sis0@pci0:1:2:0:	class=0x020000 card=0xf3111385 chip=0x0020100b rev=0x00 hdr=0x00
    vendor     = 'National Semiconductor Corporation'
    device     = 'DP83815 (MacPhyter) Ethernet Controller'

and on 13.0 amd64 it's

em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
	media: Ethernet autoselect (100baseTX <full-duplex>)
em0@pci0:6:0:0:	class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'

Thank you one more time and wish it helps to catch the bug
Comment 13 Jason A. Donenfeld 2021-05-04 08:07:21 UTC
Can you show the exact commands used with nc? And which one needs to be running the new version to exhibit the bug? I'm interested in finding the most minimal reproduction case so I can repeat what you're doing.
Comment 14 Oleg Strizhak 2021-05-04 08:20:03 UTC
(In reply to Jason A. Donenfeld from comment #13)
That's quite simple command like this:

nc host_ip 110 < test.login > test.eml

where test.login is simple pop3 session:
user #username#
pass #password#
retr 23
retr 98
retr 147
quit

the main cause, as I can see, is the size of mail - it should be 10-20 Mb w/ attachments. First complains were about truncated attachments in my colleagues mails, and after some research I came to the fact that it occurs on specific wireguard-kmod build
Comment 15 Jason A. Donenfeld 2021-05-04 08:46:27 UTC
Can you provide even more detail?

Which end is required to have the broken wireguard for the bug to show itself?

Is the POP endpoint on a system with WireGuard, or is the WireGuard system forwarding packets?

Please - d e t a i l s are really important here.
Comment 16 Oleg Strizhak 2021-05-04 09:46:35 UTC
(In reply to Jason A. Donenfeld from comment #15)
I'll try to do my best =) pop3 session is going through wg0 channel; there're two servers, they're connected by wg channel and I do pop3 session from one server to another. E.g.

192.168.1.1 < wg0 10.0.0.1 <-> 10.0.0.2 > 192.168.2.1

surely there're
route -n add -net 192.168.2.0/24 10.0.0.2
route -n add -net 192.168.1.0/24 10.0.0.1
on the appropriate ends

And in order to get this mail truncation all I nave to do is simple pop3 session from 192.168.1.1 to 192.168.2.1
Comment 17 Oleg Strizhak 2021-05-04 09:50:50 UTC
(In reply to Oleg Strizhak from comment #16)
pop3 server _is_ on the wireguard endpoint, and the other side - pop3 seesion initiator - is on another wireguard endopint
There's no nat or forward interference, moreover - as I stated already - this in effect only since wireguard-kmod-0.0.20210424_1 build. I do tests after upgrade and, surely, after downdrage w/ the same data set
Comment 18 Jason A. Donenfeld 2021-05-04 09:54:23 UTC
Which end is required to have the broken wireguard for the bug to show itself?
Comment 19 Oleg Strizhak 2021-05-04 09:56:50 UTC
(In reply to Jason A. Donenfeld from comment #18)
I usually install the same build on both ends, if I got you question right
Comment 20 Jason A. Donenfeld 2021-05-04 11:12:41 UTC
Can you try installing it only on one end and let me know whether the issue is with the sending end, the receiving end, or both?
Comment 21 Oleg Strizhak 2021-05-04 11:18:44 UTC
(In reply to Jason A. Donenfeld from comment #20)
you mean the new build - installing it on one end? Sure.
I'll be able to do the test in the evening, at about 7-8 p.m. (MSK) - when the office is off.
Can you reproduce it on your hardware? or you simply don't have such a setup at hand?
Comment 22 Jason A. Donenfeld 2021-05-04 11:29:14 UTC
I can't reproduce it with my VMs and don't have the hardware you have to try it otherwise. So, I need as much narrowing information as possible.
Comment 23 Jason A. Donenfeld 2021-05-04 12:01:45 UTC
Can you reproduce this issue just using vanilla nc with a straight up file?


$ nc -l 1234 > bigfile
$ md5 bigfile

...

$ dd if=/dev/urandom count=262144 of=bigfile
$ md5 bigfile
$ nc 10.0.0.1 1234 < bigfile

Does this work, or are the md5 hashes different?
Comment 24 Oleg Strizhak 2021-05-04 23:14:49 UTC
(In reply to Jason A. Donenfeld from comment #23)

nc 128Mb test

Now wireguard-kmod-0.0.20210415 is on both endpoints.

@13.0 amd64 [10.10.10.1] 
dd if=/dev/urandom count=262144 of=/tmp/bigfile && md5 /tmp/bigfile
262144+0 records in
262144+0 records out
134217728 bytes transferred in 3.189628 secs (42079436 bytes/sec)
MD5 (/tmp/bigfile) = 79debec0f9a8deec69379fc522e2bec8
nc -N 10.10.10.2 1234 < /tmp/bigfile

@12.2 i386 [10.10.10.2]
nc -l 1234 > /tmp/bigfile
MD5 (/tmp/bigfile) = 79debec0f9a8deec69379fc522e2bec8

in reverse it's also OK

$ dd if=/dev/urandom count=262144 of=/tmp/bigfile && md5 /tmp/bigfile 
262144+0 records in
262144+0 records out
134217728 bytes transferred in 18.041686 secs (7439312 bytes/sec)
MD5 (/tmp/bigfile) = 0d7dfe27afd8f50b3fe347f8824b462b
nc -N 10.10.10.1 1234 < /tmp/bigfile

nc -l 1234 > /tmp/bigfile && md5 /tmp/bigfile
MD5 (/tmp/bigfile) = 0d7dfe27afd8f50b3fe347f8824b462b

Now git pull src & ports & compile latest wireguard-kmod (wireguard-kmod-0.0.20210502) on both endpoints.
Restart service wireguard restart

$ nc -N 10.10.10.1 1234 < /tmp/bigfile && md5 /tmp/bigfile 
MD5 (/tmp/bigfile) = 0d7dfe27afd8f50b3fe347f8824b462b

$ nc -l 1234 > /tmp/bigfile && md5 /tmp/bigfile
MD5 (/tmp/bigfile) = 0d7dfe27afd8f50b3fe347f8824b462b

in reverse direction:
$ dd if=/dev/urandom count=262144 of=/tmp/bigfile && md5 /tmp/bigfile && nc -N 10.10.10.2 1234 < /tmp/bigfile
262144+0 records in
262144+0 records out
134217728 bytes transferred in 2.926139 secs (45868544 bytes/sec)
MD5 (/tmp/bigfile) = 6675736b1c43eb79724935f744f0fb3f

$ nc -l 1234 > /tmp/bigfile && md5 /tmp/bigfile 
MD5 (/tmp/bigfile) = 6675736b1c43eb79724935f744f0fb3f

after all I've checked pop3 and this time there's no error on any-size mail.

Tomorrow I'll be able to finally check it in production environment and, hope there'll be no error, inform you about the case

Thank you
Comment 25 Jason A. Donenfeld 2021-05-05 07:54:54 UTC
Looks like the problem is no longer there?
Comment 26 Jason A. Donenfeld 2021-05-05 07:56:23 UTC
> And in order to get this mail truncation all I nave to do is simple pop3 session from 192.168.1.1 to 192.168.2.1

Should you have used 192.168 instead of 10.0 for those tests?
Comment 27 Oleg Strizhak 2021-05-05 08:06:00 UTC
(In reply to Jason A. Donenfeld from comment #26)
yes, it really looks so. I tested everything & still keeping an eye on maillog =)
If everything is ok for a day I'll close this bug

Thank you for your time and help!
Comment 28 Jason A. Donenfeld 2021-05-05 08:23:34 UTC
Was https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255497#c12 simply a testing mistake?
Comment 29 Oleg Strizhak 2021-05-05 10:14:08 UTC
(In reply to Jason A. Donenfeld from comment #28)
I don't think so - the effect was rather massive and quite repeatable, and it clearly disappeared when downgrading wireguard-kmod.
On the other hand, possibly it's something on provider's side. To clear the matter I can downgrade to specific `error' build and test once more, if you need this information. But later, and I can test these nics only in the evening or night time (MSK)
Comment 30 Jason A. Donenfeld 2021-05-05 10:25:31 UTC
Again, I'm unable to follow the logic of your message.

Does the latest snapshot fix it, or does it fail to fix it?

If if does fix it, please close this bug.

If it doesn't fix it, please provide ALL of the necessary steps to reproduce it with as minimal example as possible.

Alternatives to these options simply waste time.
Comment 31 Oleg Strizhak 2021-05-05 10:46:26 UTC
(In reply to Jason A. Donenfeld from comment #30)
I'm sorry but English is not me native language & I'm rather out of practice

So,
1. Now the latest build works perfect and if there's not an error for today I'll surely close the bug.
2. You asked whether it was a testing mistake - I don't think so: the testing procedure is the same all the time, error disappeared when I downgraded wg-kmod. If you need info on the case I can once more downgrade wg-kmod to the `error' build and test everything once more
3. I don't want to waste even a second of your time, and really grateful for all you've already done, indeed
Comment 32 Jason A. Donenfeld 2021-05-05 10:55:51 UTC
> 2. You asked whether it was a testing mistake - I don't think so: the testing procedure is the same all the time, error disappeared when I downgraded wg-kmod. If you need info on the case I can once more downgrade wg-kmod to the `error' build and test everything once more

This is the part I don't quite understand. It's well established that some old builds had problems. The question is whether the *new* builds have those problems.
Comment 33 Oleg Strizhak 2021-05-05 11:04:06 UTC
(In reply to Jason A. Donenfeld from comment #32)
The new build (wireguard-kmod-0.0.20210502) works perfect and I want to work it this way at least one working day long, to be completely sure
Comment 34 Jason A. Donenfeld 2021-05-05 12:53:44 UTC
Gotcha. Your latest message is inconsistent with your previous message in which you wrote:

> When wireguard-kmod-0.0.20210502 was installed I got the following diff:
> 
> ...
> @@ -nnn +nnn,26 @@
> -AgA+AAAAAAAAAAEAIAAAABa20wBSZWNpZXZlciBNQVgyMTIxQl9BRFM1NDA0XzUuMDQ
> \ No newline at end of file
> +AgA+AAAAAAAAAAEAIAAAABa20wBSZWNpZXZlciBNQVgyMTIxQl9BRFM1NDA0XzUuMDQuMjAyMS+C
> +IK/grqinoq6k4eKiri9wY2IyMTIxLkdCTFBLAQIUABQAAgAIAKQKhVLs3R4pHywAAC21AAA+AAAA
> ...
> +IK/grqinoq6k4eKiri+RjyBNQVgyMTIxQl9BRFM1NDA0Lnhsc1BLBQYAAAAALgAuADITAACPrREB
> +AAA=
> +
> +------eA2c4EfD16F07F418Bd21b1Cb8586596-DiIhk0KR2sG0lAaE-1619791567--
> 
> as you may see, if the message's got through the wg0 channel it's truncated.

But you now write:

> The new build (wireguard-kmod-0.0.20210502) works perfect

So which is it? Does 0.0.20210502 truncate messages, or does it work perfectly?
Comment 35 Oleg Strizhak 2021-05-05 13:22:01 UTC
(In reply to Jason A. Donenfeld from comment #34)
I confess that now is work perfectly and I've not a complain about truncated message from my local net users for almost a whole working day. In about an hour I'll explicitly ask'em on the case and, hope, will close the bug.

About that report inconsistency: possibly, being in a harry, that truncation occurred when there were different builds on endpoints. That's the only explanation of this imho.