Userland applications that use sendfile seem to hang on the network socket. Steps to reproduce: Enable sendfile in nginx: http { sendfile on; } curl -X GET http://localhost/ The HTML file will partially download and then stall indefinitely. I have also observed issues with samba that happen to use sendfile as well. It might be related to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264943 - on an intel nic using the em driver.
Please, do you have the same problem with 13.1-RELEASE? This may help to tell whether 270190 is a duplicate of 264943, which is tentatively associated with commits that were made (to STABLE) after the RELEASE.
(In reply to Graham Perrin from comment #1) releng/13.1 would be the latest 13.1-RELEASE with all associated security patches. To be specific, I was on 13.1-RELEASE-p7.
Hand over to sendfile(2) expert/maintainer.
Victor, did you have the same problem with 13.1-RELEASE-p7?
(In reply to Gleb Smirnoff from comment #4) No issues with 13.1-RELEASE-p[1-7] releases. I would also like to clarify when curling, I would do it from an external host and not local host. Curl stalls on both MacOS and Linux clients attempting to fetch a static file from nginx with sendfile on.
Also, I forgot to mention that pf is also enabled since I am using Bastille jails and Bhyve VMs. Nginx and samba are running natively on host, not jailed. PF config: ext_if="em0" vpnint = "tun0" vpnclients = "10.8.0.0/24" set skip on lo set skip on tun0 set skip on tap0 set skip on vm-public scrub in all table <jails> persist nat on $ext_if from <jails> to any -> ($ext_if:0) rdr-anchor "rdr/*" nat on $ext_if inet from $vpnclients to any -> ($ext_if) nat on $ext_if inet from {192.168.8.0/24} to any -> ($ext_if) block in pass out table <bruteforce> persist antispoof quick for $ext_if block quick from <bruteforce> anchor "f2b/*" pass in on $ext_if from 192.168.5.0/24 to any flags S/SA keep state pass in on $ext_if from 192.168.8.0/24 to any flags S/SA keep state pass in on $ext_if from 10.8.0.0/24 to any flags S/SA keep state
Recorded two videos between 13.1-release-p7 and 13.2-RC2 https://bender.lamer0.com/~victori/fbsd13.1-release-p7.mp4 https://bender.lamer0.com/~victori/fbsd13.2-RC2.mp4
Can you please try 'ifconfig em0 -rxcsum -txcsum'? Can you please record tcpdump file for a failed curl GET?
> ifconfig em0 -rxcsum -txcsum Not precisely sure what this command does. I assume it turns off checksums? So I listed two tcpdumps one with checksums off and another with it on. https://bender.lamer0.com/~victori/fbsd13.2-RC2-failed-curl.txt https://bender.lamer0.com/~victori/fbsd-13.2-RC2-failed-curl-chksum-off.txt First dump: 12:08:49.480182 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64) VictorsMBP2021.lan.59433 > bender.lan.http: Flags [SEW], cksum 0xeb9c (correct), seq 2684730209, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4236869021 ecr 0,sackOK,eol], length 0 12:08:49.480228 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) bender.lan.http > VictorsMBP2021.lan.59433: Flags [S.E], cksum 0x3c9c (correct), seq 3754760745, ack 2684730210, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 581411798 ecr 4236869021], length 0 12:08:49.480512 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.59433 > bender.lan.http: Flags [.], cksum 0x639d (correct), ack 1, win 2058, options [nop,nop,TS val 4236869021 ecr 581411798], length 0 12:08:49.480548 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0x5b9b (correct), ack 1, win 4107, options [nop,nop,TS val 581411799 ecr 4236869021], length 0 12:08:49.485002 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 126) VictorsMBP2021.lan.59433 > bender.lan.http: Flags [P.], cksum 0x8324 (correct), seq 1:75, ack 1, win 2058, options [nop,nop,TS val 4236869026 ecr 581411799], length 74: HTTP, length: 74 GET / HTTP/1.1 Host: bender.lan User-Agent: curl/7.88.1 Accept: */* 12:08:49.485124 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0x7b57 (correct), seq 1:1449, ack 75, win 4107, options [nop,nop,TS val 581411803 ecr 4236869026], length 1448: HTTP, length: 1448 HTTP/1.1 200 OK Server: nginx/1.22.1 Date: Thu, 16 Mar 2023 19:08:49 GMT Content-Type: text/html Content-Length: 5350 Last-Modified: Sun, 11 Sep 2022 04:28:27 GMT Connection: keep-alive ETag: "631d63eb-14e6" Accept-Ranges: bytes <!doctype html> <html lang="en" class="h-100"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta name="description" content=""> <meta name="author" content="Mark Otto, Jacob Thornton, and Bootstrap contributors"> <meta name="generator" content="Jekyll v3.8.5"> <title>Bender</title> <!-- Bootstrap core CSS --> <link href="bootstrap/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> <style> .bd-placeholder-img { font-size: 1.125rem; text-anchor: middle; -webkit-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; } @media (min-width: 768px) { .bd-placeholder-img-lg { font-size: 3.5rem; } } </style> <!-- Custom styles for this template --> <link href="sticky-footer-navbar.css" rel="stylesheet"> </head> <body class="d-flex flex-column h-100"> <header> <!-- Fixed navbar --> <nav class="navbar navbar-expand-md navbar-dark fixed-to[!http] 12:08:49.485132 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xc7b7 (incorrect -> 0xfdcb), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581411803 ecr 4236869026], length 1448: HTTP 12:08:49.485138 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0x042a (incorrect -> 0x9cc5), seq 2897:4345, ack 75, win 4107, options [nop,nop,TS val 581411803 ecr 4236869026], length 1448: HTTP 12:08:49.485143 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1298) bender.lan.http > VictorsMBP2021.lan.59433: Flags [P.], cksum 0x5ad9 (incorrect -> 0x79f1), seq 4345:5591, ack 75, win 4107, options [nop,nop,TS val 581411803 ecr 4236869026], length 1246: HTTP 12:08:49.485458 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.59433 > bender.lan.http: Flags [.], cksum 0x5db7 (correct), ack 1449, win 2036, options [nop,nop,TS val 4236869026 ecr 581411803], length 0 12:08:49.714556 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xc6d1 (incorrect -> 0xfce5), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581412033 ecr 4236869026], length 1448: HTTP 12:08:49.974553 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xc5cd (incorrect -> 0xfbe1), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581412293 ecr 4236869026], length 1448: HTTP 12:08:50.294551 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xc48d (incorrect -> 0xfaa1), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581412613 ecr 4236869026], length 1448: HTTP 12:08:50.737163 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xc2d3 (incorrect -> 0xf8e7), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581413055 ecr 4236869026], length 1448: HTTP 12:08:51.416558 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xc02b (incorrect -> 0xf63f), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581413735 ecr 4236869026], length 1448: HTTP 12:08:52.576553 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xbba3 (incorrect -> 0xf1b7), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581414895 ecr 4236869026], length 1448: HTTP 12:08:53.558557 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59418: Flags [.], cksum 0xfa42 (incorrect -> 0x3057), seq 1283370083:1283371531, ack 3857374885, win 4107, options [nop,nop,TS val 430886542 ecr 4053455418], length 1448: HTTP 12:08:54.696554 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xb35b (incorrect -> 0xe96f), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581417015 ecr 4236869026], length 1448: HTTP 12:08:58.736551 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xa393 (incorrect -> 0xd9a7), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581421055 ecr 4236869026], length 1448: HTTP 12:09:06.617558 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0x84ca (incorrect -> 0xbade), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 581428936 ecr 4236869026], length 1448: HTTP 12:09:07.198499 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) bender.lan.50798 > 169.254.169.254.http: Flags [S], cksum 0xce41 (correct), seq 745600524, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2852768900 ecr 0], length 0 12:09:08.197556 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) bender.lan.50798 > 169.254.169.254.http: Flags [S], cksum 0xca59 (correct), seq 745600524, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2852769900 ecr 0], length 0 12:09:08.469979 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.59433 > bender.lan.http: Flags [F.], cksum 0x1381 (correct), seq 75, ack 1449, win 2048, options [nop,nop,TS val 4236888011 ecr 581411803], length 0 12:09:08.470021 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.59433: Flags [.], cksum 0xb11e (correct), ack 76, win 4107, options [nop,nop,TS val 581430788 ecr 4236888011], length 0 12:09:09.118538 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59418: Flags [.], cksum 0xbd7a (incorrect -> 0xf38e), seq 0:1448, ack 1, win 4107, options [nop,nop,TS val 430902102 ecr 4053455418], length 1448: HTTP 12:09:10.404263 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) bender.lan.50798 > 169.254.169.254.http: Flags [S], cksum 0xc1bb (correct), seq 745600524, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2852772106 ecr 0], length 0 Checksum off dump: root@bender % tcpdump -i em0 -s0 -v port 80 tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:12:44.784331 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64) VictorsMBP2021.lan.59963 > bender.lan.http: Flags [SEW], cksum 0xe9c2 (correct), seq 3123453835, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1638701554 ecr 0,sackOK,eol], length 0 12:12:44.784370 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) bender.lan.http > VictorsMBP2021.lan.59963: Flags [S.E], cksum 0x331f (correct), seq 633199826, ack 3123453836, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 4071740117 ecr 1638701554], length 0 12:12:44.784566 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.59963 > bender.lan.http: Flags [.], cksum 0x5a1f (correct), ack 1, win 2058, options [nop,nop,TS val 1638701555 ecr 4071740117], length 0 12:12:44.784608 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0x521d (correct), ack 1, win 4107, options [nop,nop,TS val 4071740118 ecr 1638701555], length 0 12:12:44.786829 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 126) VictorsMBP2021.lan.59963 > bender.lan.http: Flags [P.], cksum 0x79a9 (correct), seq 1:75, ack 1, win 2058, options [nop,nop,TS val 1638701557 ecr 4071740118], length 74: HTTP, length: 74 GET / HTTP/1.1 Host: bender.lan User-Agent: curl/7.88.1 Accept: */* 12:12:44.786956 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0x77e2 (correct), seq 1:1449, ack 75, win 4107, options [nop,nop,TS val 4071740120 ecr 1638701557], length 1448: HTTP, length: 1448 HTTP/1.1 200 OK Server: nginx/1.22.1 Date: Thu, 16 Mar 2023 19:12:44 GMT Content-Type: text/html Content-Length: 5350 Last-Modified: Sun, 11 Sep 2022 04:28:27 GMT Connection: keep-alive ETag: "631d63eb-14e6" Accept-Ranges: bytes <!doctype html> <html lang="en" class="h-100"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta name="description" content=""> <meta name="author" content="Mark Otto, Jacob Thornton, and Bootstrap contributors"> <meta name="generator" content="Jekyll v3.8.5"> <title>Bender</title> <!-- Bootstrap core CSS --> <link href="bootstrap/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> <style> .bd-placeholder-img { font-size: 1.125rem; text-anchor: middle; -webkit-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; } @media (min-width: 768px) { .bd-placeholder-img-lg { font-size: 3.5rem; } } </style> <!-- Custom styles for this template --> <link href="sticky-footer-navbar.css" rel="stylesheet"> </head> <body class="d-flex flex-column h-100"> <header> <!-- Fixed navbar --> <nav class="navbar navbar-expand-md navbar-dark fixed-to[!http] 12:12:44.786971 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0xbe3e (incorrect -> 0xf452), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 4071740120 ecr 1638701557], length 1448: HTTP 12:12:44.786979 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0xfab0 (incorrect -> 0x934c), seq 2897:4345, ack 75, win 4107, options [nop,nop,TS val 4071740120 ecr 1638701557], length 1448: HTTP 12:12:44.786986 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1298) bender.lan.http > VictorsMBP2021.lan.59963: Flags [P.], cksum 0x5160 (incorrect -> 0x7078), seq 4345:5591, ack 75, win 4107, options [nop,nop,TS val 4071740120 ecr 1638701557], length 1246: HTTP 12:12:44.787262 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.59963 > bender.lan.http: Flags [.], cksum 0x543e (correct), ack 1449, win 2036, options [nop,nop,TS val 1638701557 ecr 4071740120], length 0 12:12:45.017220 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0xbd58 (incorrect -> 0xf36c), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 4071740350 ecr 1638701557], length 1448: HTTP 12:12:45.280388 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0xbc51 (incorrect -> 0xf265), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 4071740613 ecr 1638701557], length 1448: HTTP 12:12:45.600128 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0xbb11 (incorrect -> 0xf125), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 4071740933 ecr 1638701557], length 1448: HTTP 12:12:46.045701 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0xb953 (incorrect -> 0xef67), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 4071741379 ecr 1638701557], length 1448: HTTP 12:12:46.692849 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.59963 > bender.lan.http: Flags [F.], cksum 0x4cbf (correct), seq 75, ack 1449, win 2048, options [nop,nop,TS val 1638703463 ecr 4071740120], length 0 12:12:46.692917 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0x2d14 (correct), ack 76, win 4107, options [nop,nop,TS val 4071742026 ecr 1638703463], length 0 12:12:46.726122 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0xaf38 (incorrect -> 0xe54c), seq 1449:2897, ack 76, win 4107, options [nop,nop,TS val 4071742059 ecr 1638703463], length 1448: HTTP 12:12:47.887005 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59963: Flags [.], cksum 0xaaaf (incorrect -> 0xe0c3), seq 1449:2897, ack 76, win 4107, options [nop,nop,TS val 4071743220 ecr 1638703463], length 1448: HTTP 12:12:49.102309 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.59927: Flags [.], cksum 0x9ec5 (incorrect -> 0xd4d9), seq 2376174884:2376176332, ack 2841992953, win 4107, options [nop,nop,TS val 898200807 ecr 2061204116], length 1448: HTTP
Dump from FreeBSD 13.1-release-p7 tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:20:52.452752 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64) VictorsMBP2021.lan.61329 > bender.lan.http: Flags [SEW], cksum 0xe240 (correct), seq 2782022648, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 3561951592 ecr 0,sackOK,eol], length 0 12:20:52.452800 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) bender.lan.http > VictorsMBP2021.lan.61329: Flags [S.E], cksum 0xb7ce (correct), seq 3982759450, ack 2782022649, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3608895819 ecr 3561951592], length 0 12:20:52.453163 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.61329 > bender.lan.http: Flags [.], cksum 0xdecf (correct), ack 1, win 2058, options [nop,nop,TS val 3561951592 ecr 3608895819], length 0 12:20:52.453202 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.61329: Flags [.], cksum 0xd6ce (correct), ack 1, win 4107, options [nop,nop,TS val 3608895819 ecr 3561951592], length 0 12:20:52.458959 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 126) VictorsMBP2021.lan.61329 > bender.lan.http: Flags [P.], cksum 0xfe55 (correct), seq 1:75, ack 1, win 2058, options [nop,nop,TS val 3561951599 ecr 3608895819], length 74: HTTP, length: 74 GET / HTTP/1.1 Host: bender.lan User-Agent: curl/7.88.1 Accept: */* 12:20:52.459080 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.61329: Flags [.], cksum 0xfd8b (correct), seq 1:1449, ack 75, win 4107, options [nop,nop,TS val 3608895825 ecr 3561951599], length 1448: HTTP, length: 1448 HTTP/1.1 200 OK Server: nginx/1.22.1 Date: Thu, 16 Mar 2023 19:20:52 GMT Content-Type: text/html Content-Length: 5350 Last-Modified: Sun, 11 Sep 2022 04:28:27 GMT Connection: keep-alive ETag: "631d63eb-14e6" Accept-Ranges: bytes <!doctype html> <html lang="en" class="h-100"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta name="description" content=""> <meta name="author" content="Mark Otto, Jacob Thornton, and Bootstrap contributors"> <meta name="generator" content="Jekyll v3.8.5"> <title>Bender</title> <!-- Bootstrap core CSS --> <link href="bootstrap/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> <style> .bd-placeholder-img { font-size: 1.125rem; text-anchor: middle; -webkit-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; } @media (min-width: 768px) { .bd-placeholder-img-lg { font-size: 3.5rem; } } </style> <!-- Custom styles for this template --> <link href="sticky-footer-navbar.css" rel="stylesheet"> </head> <body class="d-flex flex-column h-100"> <header> <!-- Fixed navbar --> <nav class="navbar navbar-expand-md navbar-dark fixed-to[!http] 12:20:52.459092 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.61329: Flags [.], cksum 0x78fb (correct), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 3608895825 ecr 3561951599], length 1448: HTTP 12:20:52.459099 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.61329: Flags [.], cksum 0x17f5 (correct), seq 2897:4345, ack 75, win 4107, options [nop,nop,TS val 3608895825 ecr 3561951599], length 1448: HTTP 12:20:52.459106 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1298) bender.lan.http > VictorsMBP2021.lan.61329: Flags [P.], cksum 0xf520 (correct), seq 4345:5591, ack 75, win 4107, options [nop,nop,TS val 3608895825 ecr 3561951599], length 1246: HTTP 12:20:52.459518 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.61329 > bender.lan.http: Flags [.], cksum 0xc8f9 (correct), ack 5591, win 1971, options [nop,nop,TS val 3561951599 ecr 3608895825], length 0 12:20:52.459525 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.61329 > bender.lan.http: Flags [.], cksum 0xc8ac (correct), ack 5591, win 2048, options [nop,nop,TS val 3561951599 ecr 3608895825], length 0 12:20:52.460384 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.61329 > bender.lan.http: Flags [F.], cksum 0xc8aa (correct), seq 75, ack 5591, win 2048, options [nop,nop,TS val 3561951600 ecr 3608895825], length 0 12:20:52.460402 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.61329: Flags [.], cksum 0xc09d (correct), ack 76, win 4107, options [nop,nop,TS val 3608895827 ecr 3561951600], length 0 12:20:52.460422 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.61329: Flags [F.], cksum 0xc09c (correct), seq 5591, ack 76, win 4107, options [nop,nop,TS val 3608895827 ecr 3561951600], length 0 12:20:52.460688 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.61329 > bender.lan.http: Flags [.], cksum 0xc8a7 (correct), ack 5592, win 2048, options [nop,nop,TS val 3561951600 ecr 3608895827], length 0
Can you please tcpdump on the client side? Are you receiving all these retransmissions of the second packet and is the checksum correct?
(In reply to Gleb Smirnoff from comment #11) Here is the tcpdump from my OSX client to 13.2-RC2 Victors-MacBook-Pro-2021:~ root# tcpdump -s0 -v port 80 tcpdump: data link type PKTAP tcpdump: listening on pktap, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes 14:39:18.629508 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63550: Flags [.], cksum 0x152d (incorrect -> 0x4b41), seq 135699367:135700815, ack 110941672, win 4107, options [nop,nop,TS val 3315151788 ecr 2911311719], length 1448: HTTP 14:39:19.839510 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->ae21)!) 192.168.5.246.63551 > bender.local.http: Flags [SEW], cksum 0x8cc9 (incorrect -> 0xa0f2), seq 1059592423, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1016167298 ecr 0,sackOK,eol], length 0 14:39:19.839766 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) bender.local.http > 192.168.5.246.63551: Flags [S.E], cksum 0x2372 (correct), seq 1629066902, ack 1059592424, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1801464804 ecr 1016167298], length 0 14:39:19.839808 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->ae2d)!) 192.168.5.246.63551 > bender.local.http: Flags [.], cksum 0x8cbd (incorrect -> 0x4a72), ack 1, win 2058, options [nop,nop,TS val 1016167299 ecr 1801464804], length 0 14:39:19.840016 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0x4270 (correct), ack 1, win 4107, options [nop,nop,TS val 1801464805 ecr 1016167299], length 0 14:39:19.844338 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 126, bad cksum 0 (->ade1)!) 192.168.5.246.63551 > bender.local.http: Flags [P.], cksum 0x8d07 (incorrect -> 0x69f9), seq 1:75, ack 1, win 2058, options [nop,nop,TS val 1016167304 ecr 1801464805], length 74: HTTP, length: 74 GET / HTTP/1.1 Host: bender.lan User-Agent: curl/7.88.1 Accept: */* 14:39:19.850564 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0x632b (correct), seq 1:1449, ack 75, win 4107, options [nop,nop,TS val 1801464815 ecr 1016167304], length 1448: HTTP, length: 1448 HTTP/1.1 200 OK Server: nginx/1.22.1 Date: Thu, 16 Mar 2023 21:39:19 GMT Content-Type: text/html Content-Length: 5350 Last-Modified: Sun, 11 Sep 2022 04:28:27 GMT Connection: keep-alive ETag: "631d63eb-14e6" Accept-Ranges: bytes <!doctype html> <html lang="en" class="h-100"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta name="description" content=""> <meta name="author" content="Mark Otto, Jacob Thornton, and Bootstrap contributors"> <meta name="generator" content="Jekyll v3.8.5"> <title>Bender</title> <!-- Bootstrap core CSS --> <link href="bootstrap/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> <style> .bd-placeholder-img { font-size: 1.125rem; text-anchor: middle; -webkit-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; } @media (min-width: 768px) { .bd-placeholder-img-lg { font-size: 3.5rem; } } </style> <!-- Custom styles for this template --> <link href="sticky-footer-navbar.css" rel="stylesheet"> </head> <body class="d-flex flex-column h-100"> <header> <!-- Fixed navbar --> <nav class="navbar navbar-expand-md navbar-dark fixed-to [|http] 14:39:19.850569 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0xae86 (incorrect -> 0xe49a), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 1801464815 ecr 1016167304], length 1448: HTTP 14:39:19.850570 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0xeaf8 (incorrect -> 0x8394), seq 2897:4345, ack 75, win 4107, options [nop,nop,TS val 1801464815 ecr 1016167304], length 1448: HTTP 14:39:19.850571 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1298) bender.local.http > 192.168.5.246.63551: Flags [P.], cksum 0x41a8 (incorrect -> 0x60c0), seq 4345:5591, ack 75, win 4107, options [nop,nop,TS val 1801464815 ecr 1016167304], length 1246: HTTP 14:39:19.850652 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->ae2d)!) 192.168.5.246.63551 > bender.local.http: Flags [.], cksum 0x8cbd (incorrect -> 0x4480), ack 1449, win 2036, options [nop,nop,TS val 1016167310 ecr 1801464815], length 0 14:39:20.080657 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0xad9a (incorrect -> 0xe3ae), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 1801465045 ecr 1016167310], length 1448: HTTP 14:39:20.345000 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0xac91 (incorrect -> 0xe2a5), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 1801465310 ecr 1016167310], length 1448: HTTP 14:39:20.664998 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0xab51 (incorrect -> 0xe165), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 1801465630 ecr 1016167310], length 1448: HTTP 14:39:20.748963 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63550: Flags [.], cksum 0x0ce5 (incorrect -> 0x42f9), seq 0:1448, ack 1, win 4107, options [nop,nop,TS val 3315153908 ecr 2911311719], length 1448: HTTP 14:39:21.104856 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0xa999 (incorrect -> 0xdfad), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 1801466070 ecr 1016167310], length 1448: HTTP 14:39:21.784710 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0xa6f1 (incorrect -> 0xdd05), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 1801466750 ecr 1016167310], length 1448: HTTP 14:39:22.946090 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.local.http > 192.168.5.246.63551: Flags [.], cksum 0xa268 (incorrect -> 0xd87c), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 1801467911 ecr 1016167310], length 1448: HTTP
Will you be able to build several 13.1-STABLE kernels at different points to check what revision did broke it? The suspicious one for me is e31579b8558db508dfc3f8fc276611a7c3c93aa1, but that's just a guess. Could be a different one.
(In reply to Gleb Smirnoff from comment #13) Yeah, I suppose, but I will be traveling as of today, and will only be able to verify after the 20th. It would be a pity if this bug makes it into 13.2's release. It knocks out HTTP and samba/netatalk from functioning.
(In reply to Victor Igumnov from comment #14) Regarding testing, do you want me to branch it locally and revert e31579b8558db508dfc3f8fc276611a7c3c93aa1 to see if that addresses the issue?
(In reply to Victor Igumnov from comment #15) I just checked releng/13.1 and it does contain e31579b8558db508dfc3f8fc276611a7c3c93aa1 - so that can't be it?
I suspect that the problem could be caused by this revision. I could be wrong, though. What I suggest is to run 'git bisect' between stable/13 where 13.1 was branched and stable/13 where 13.2 was branched. I would recommend don't start with division by two and start with testing e31579b8558db508dfc3f8fc276611a7c3c93aa1 and then if it is bad, test one revision before it. If that fails, continue with regular bisection.
(In reply to Gleb Smirnoff from comment #17) Ran over 10 bisects, and came down to the following commit that might have broken it. > git bisect start > git bisect good releng/13.1 > git bisect bad relent/13.2 ... lots of recompiles and reboots later... root@bender % git bisect bad 131bad17ead4938179fb5446538324667987c349 is the first bad commit commit 131bad17ead4938179fb5446538324667987c349 Author: Corvin Köhne <CorvinK@beckhoff.com> Date: Wed Sep 7 09:07:03 2022 +0200 vmm: permit some IPIs to be handled by userspace Add VM_EXITCODE_IPI to permit returning unhandled IPIs to userland. INIT and STARTUP IPIs are now returned to userland. Due to backward compatibility reasons, a new capability is added for enabling VM _EXITCODE_IPI. Reviewed by: jhb Differential Revision: https://reviews.freebsd.org/D35623 Sponsored by: Beckhoff Automation GmbH & Co. KG (cherry picked from commit Obda8d3e9f7a5c04881219723436616b23041e5f) sys/amd64/include/vmm.h sys/amd64/vmm/amd/svm.c sys/amd64/vmm/intel/vmx.c sys/amd64/vmm/lo/vlapic.c sys/amd64/vmm/io/vlapic.h sys/amd64/vmm/io/vlapic_priv.h sys/amd64/vmm/vmm.c usr.sbin/bhyve/bhyverun. c usr.sbin/bhyve/spinup_ap.c 8 ++ 10 +++ 8 ++ 192 +++++++4 2 + 2 + 9 ++ 34 ++++++*++ 3 + 9 files changed, 198 insertions+), 70 deletions (-)
Verified in the morning the commit right before 131bad17ead4938179fb5446538324667987c349 works just fine. Must be something in the vmm subsystem that breaks it. I attempted to revert the commit but getting non-simple merge conflicts.
Looks like I figured it out, if bhyve/vm is disabled, everything works fine with that commit in place. If bhyve is running in the background network transmissions fail with sendfile with that commit.
(In reply to Victor Igumnov from comment #20) There must be something hide behind. I just can not repeat this on my old Intel i5-2500K box (FreeBSD-13.2-RC3). ``` # pkg install nginx; // sendfile is on by default # service nginx onestart # kldload vmm # bhyveload -m 1G -h / test-vm # bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,ahci-cd,/tmp/FreeBSD-13.1-RELEASE-amd64-disc1.iso -l com1,stdio -A -H -P -m 1G test-vm ``` In another session: ``` # fetch http://localhost/ ``` Test with RE(4) and CXGBE(4) ``` # fetch http://ip-address-of-re0-cxl0 ``` tcpdump on client or server both looks OK.
(In reply to Victor Igumnov from comment #19) Do you frequently have VMs booting, and/or sitting at a loader prompt? Commit 131bad17ead4938179fb5446538324667987c349 really only affects VM startup. Would you be able to try the patch here and confirm whether it fixes the problem when 131bad17ead4938179fb5446538324667987c349 is applied? https://reviews.freebsd.org/D39174
(In reply to Mark Johnston from comment #22) Hi, I was able to move up to commit 8f7ae0928ecffdb4427ab44270f42e9612e28cfa while reverting 131bad17ead4938179fb5446538324667987c349 semi-cleanly without any of the network hanging issues. commit 8f7ae0928ecffdb4427ab44270f42e9612e28cfa Author: Vladimir Kondratyev <wulf@FreeBSD.org> AuthorDate: Thu Mar 3 02:35:23 2022 +0300 Commit: Vladimir Kondratyev <wulf@FreeBSD.org> CommitDate: Sat Dec 24 11:33:26 2022 +0300 In terms of my use cases with bhyve, just a single linux VM that starts up once via rc.d. I am using bhyve via vm-bhyve port. Perhaps a bit more information concerning my setup. loader.conf kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" zfs_load="YES" pf_load="YES" pflog_load="YES" linux_load="YES" linux64_load="YES" linprocfs_load="YES" fdescfs_load="YES" tmpfs_load="YES" fuse_load="YES" cpu_microcode_load="YES" cpu_microcode_name="/boot/firmware/intel-ucode.bin" linsysfs_load="YES" kern.racct.enable=1 sysctl.conf net.inet.tcp.tso=0 <-- was needed when using re0 driver, otherwise would disconnect every few weeks. (Previous NAS box) net.inet.tcp.recvspace=262144 net.inet.tcp.sendspace=262144 kern.timecounter.hardware=ACPI-fast <-- Needed or extensive clock drift over time vfs.aio.max_buf_aio=8192 vfs.aio.max_aio_queue_per_proc=65536 vfs.aio.max_aio_per_proc=8192 vfs.aio.max_aio_queue=65536 root@bender % vm switch list NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS public standard vm-public 192.168.8.1/24 no - - em0 root@bender % ifconfig -a em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP> ether 1c:69:7a:a0:4e:4f inet 192.168.5.80 netmask 0xffffff00 broadcast 192.168.5.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=0<> metric 0 mtu 33160 groups: pflog bastille0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff groups: tun nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Opened by PID 1475 vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether e2:ef:1e:91:2e:e5 inet 192.168.8.1 netmask 0xffffff00 broadcast 192.168.8.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 20000 groups: bridge vm-switch viid-4c918@ nd6 options=9<PERFORMNUD,IFDISABLED> I wish I knew the exact bhyve command that is used to start the VM via vm-bhyve, it might have something specific that is triggering it? I might try my hand at reverting 131bad17ead4938179fb5446538324667987c349 against the latest releng/13.2 branch now that I am back home. In terms of https://reviews.freebsd.org/D39174 - do you want me to apply that usleep(1000); addition?
(In reply to Victor Igumnov from comment #23) > In terms of https://reviews.freebsd.org/D39174 - do you want me to apply that usleep(1000); addition? Yes please. With 131bad17ead4938179fb5446538324667987c349 un-reverted, so we can see if the problem is still there. > kern.timecounter.hardware=ACPI-fast <-- Needed or extensive clock drift over time Is this still a problem as of 13.1? Does your system use the TSC by default? If the answer to both questions is "yes", this would be worth a separate bug report, as we had fixed a number of problems with TSC calibration recently.
(In reply to Mark Johnston from comment #24) The usleep(1000); patch did not work. Here is a video - https://bender.lamer0.com/~victori/fbsd-13.2-rc3-bhyve.mp4 Concerning the clock drift, I will need to re-evaluate it. I have not bothered to revisit the issue since that sysctl property addresses the issue.
Can you run `top -SHz` after your VM has started to see what threads are running and actively using CPU (and maybe capture a snapshot of the output)? Can you leave it running while then starting curl?
(In reply to John Baldwin from comment #26) Found some additional information: root@bender % vm switch list NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS public standard vm-public 192.168.8.1/24 no - - em0 > vm switch remove public em0 root@bender % vm switch list NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS public standard vm-public 192.168.8.1/24 no - - - Curl will work just fine, but VM will not have a network. > vm switch add public em0 Curl will switch back to failing.
(In reply to Victor Igumnov from comment #27) I see, this looks like packets are getting forwarded to your VM instead of nginx. I wonder if https://cgit.freebsd.org/src/commit/?id=22893e584032f22f24cae8e8b1b77ea70e83bd69 is related. You could try disabling net.link.bridge.pfil_bridge and/or pfil_member to verify.
(In reply to Mark Johnston from comment #28) Tried, still hanging. root@bender % sysctl -a | grep pfil net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 1
(In reply to John Baldwin from comment #26) https://bender.lamer0.com/~victori/fbsd-13.2-rc3-top-curl.mp4
(In reply to Victor Igumnov from comment #29) Ok, but it looks like packets are somehow getting routed to the VM anyway. You could try running tcpdump -n on the VM's tap interface to confirm this.
(In reply to Mark Johnston from comment #31) > tcpdump -i vm-public -s0 -v -n port 80 I don't see any traffic on port 80 on the tap interface which the VM uses.
(In reply to Victor Igumnov from comment #32) Thank you for looking into this issue, I know this could be a bit frustrating in debugging an interaction between third-party port applications and core FreeBSD userland. Perhaps I will stick around on 13.1 a bit longer. Found the bhyve VM arguments passed: Mar 21 11:25:18: /usr/local/sbin/grub-bhyve -c /dev/nmdm-debian.1A -m /zroot/zVirtualMachines/debian/device.map -M 1G -r hd0,1 -d /boot/grub debian Mar 21 11:25:23: [bhyve options: -c 4 -m 1G -AHP -U 32901887-f5c2-11e9-945d-005043012398 -u] Mar 21 11:25:23: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/zroot/zVirtualMachines/debian/disk0.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:07:ff:a2] Mar 21 11:25:23: [bhyve console: -l com1,/dev/nmdm-debian.1A]
What does the 'vm switch remove public em0' command actually? There is no 'vm' command in vanilla FreeBSD.
(In reply to Gleb Smirnoff from comment #34) Modified the VM script to output the commands issued: > ifconfig vm-public deletem em0 vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 5a:b3:4f:70:0d:1e inet 192.168.8.1 netmask 0xffffff00 broadcast 192.168.8.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 groups: bridge vm-switch viid-4c918@ nd6 options=9<PERFORMNUD,IFDISABLED> tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff groups: tun nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Opened by PID 1846 Added back in: vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 5a:b3:4f:70:0d:1e inet 192.168.8.1 netmask 0xffffff00 broadcast 192.168.8.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 20000 groups: bridge vm-switch viid-4c918@ nd6 options=9<PERFORMNUD,IFDISABLED> tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff groups: tun nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Opened by PID 1846
(In reply to Victor Igumnov from comment #29) > net.link.bridge.pfil_member: 0 > net.link.bridge.pfil_bridge: 0 These are the new defaults. Did you try with old defaults (both values set to 1)?
(In reply to Gleb Smirnoff from comment #36) Hi, Sorry, I must have been toggling it off and on when I posted that. I have re-applied the settings to sysctl.conf and rebooted just in the case that property might not be dynamically applied. Sadly, it does not resolve the issue. root@bender % uname -a FreeBSD bender 13.2-RC3 FreeBSD 13.2-RC3 releng/13.2-n254602-87b5b46ce892 BENDER amd64 root@bender % vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE debian default grub 4 1G - Yes [1] Running (2022) k8s default grub 4 2G - No Stopped root@bender % cat /etc/sysctl.conf | grep pfil net.link.bridge.pfil_member=1 net.link.bridge.pfil_bridge=1 root@bender % sysctl -a | grep pfil net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 1 net.link.bridge.pfil_bridge: 1 net.link.bridge.pfil_onlyip: 1
(In reply to Victor Igumnov from comment #37) Gleb Smirnoff is helping me debug this issue, he has provided the following dtrace script that was invoked while attempting to fetch from nginx with sendfile on. dtrace -n 'fbt::vn_sendfile:entry / execname == "nginx" / { self->trace = 1 }' -n 'fbt::vn_sendfile:return / self->trace == 1 / { self->trace = 0 }' -n 'fbt::iflib_if_transmit:entry / self->trace == 1 / { stack(); }' Result: CPU ID FUNCTION:NAME 3 20251 iflib_if_transmit:entry if_bridge.ko`bridge_enqueue+0xba kernel`ether_output+0x619 kernel`ip_output+0x1316 kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 3 20251 iflib_if_transmit:entry if_bridge.ko`bridge_enqueue+0xba kernel`ether_output+0x619 kernel`ip_output+0x1316 kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 3 20251 iflib_if_transmit:entry if_bridge.ko`bridge_enqueue+0xba kernel`ether_output+0x619 kernel`ip_output+0x1316 kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 3 20251 iflib_if_transmit:entry if_bridge.ko`bridge_enqueue+0xba kernel`ether_output+0x619 kernel`ip_output+0x1316 kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb tso off: CPU ID FUNCTION:NAME 3 20251 iflib_if_transmit:entry if_bridge.ko`bridge_enqueue+0xba kernel`ether_output+0x619 kernel`ip_output+0x1316 kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 3 20251 iflib_if_transmit:entry if_bridge.ko`bridge_enqueue+0xba kernel`ether_output+0x619 kernel`ip_output+0x1316 kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 3 20251 iflib_if_transmit:entry if_bridge.ko`bridge_enqueue+0xba kernel`ether_output+0x619 kernel`ip_output+0x1316 kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 3 20251 iflib_if_transmit:entry if_bridge.ko`bridge_enqueue+0xba kernel`ether_output+0x619 kernel`ip_output+0x1316 kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb
Good news, I was able to pin point to the exact commit that broke sendfile networking for me while bhyve runs in the background. My initial bisect was flawed, I did not know that bhyve needs to be running in the background to invoke the issue. Here is my bisect log: git bisect start # status: waiting for both good and bad commits # bad: [5f7fc790c64fb120cd6918db84e985d40b874661] bhyve: create all vcpus on startup git bisect bad 5f7fc790c64fb120cd6918db84e985d40b874661 # status: waiting for good commit(s), bad commit known # good: [00935d2e533c82de250332b967127dfff4e44e8d] Add UPDATING entries and bump version. git bisect good 00935d2e533c82de250332b967127dfff4e44e8d # good: [8824cbace389c440394bb9ea6c127d0f8f85538b] cxgbe(4): dump_devlog should never fail silently. git bisect good 8824cbace389c440394bb9ea6c127d0f8f85538b # bad: [54f9ddf4ae5ad454e679436209c75f70957d29a8] rtw88: update Realtek's rtw88 driver git bisect bad 54f9ddf4ae5ad454e679436209c75f70957d29a8 # good: [382ffc87860cd09f9350179e59bdf88f07b55631] x86: Add a NT_X86_SEGBASES register set. git bisect good 382ffc87860cd09f9350179e59bdf88f07b55631 # skip: [d06a86d2025baa73a2ff1147dd348cc179313aef] umtx: Add new pi_futex type. git bisect skip d06a86d2025baa73a2ff1147dd348cc179313aef # skip: [b6b070eed27e67562e67db42d825638ad3f5d72a] linux(4): Replace copyin() by fueword32() in handle_futex_death(). git bisect skip b6b070eed27e67562e67db42d825638ad3f5d72a # skip: [3af6bc9d02cd65f65f8301927f1b1564b7aa1c31] linux(4): Implement CLONE_CLEAR_SIGHAND option bit. git bisect skip 3af6bc9d02cd65f65f8301927f1b1564b7aa1c31 # bad: [ba5cf9c09f933bdcdb0fd415f3f3582bdf7ca8ec] Apply libc++ fix for firefox visibility push/pop issue git bisect bad ba5cf9c09f933bdcdb0fd415f3f3582bdf7ca8ec # good: [180ab9d85b4a018198c454c33cdb8e1f8ff62e2b] asmc: Style git bisect good 180ab9d85b4a018198c454c33cdb8e1f8ff62e2b # bad: [4dfd3ffc4488e5e2662cdc40deec17d82432da0b] if: avoid interface destroy race git bisect bad 4dfd3ffc4488e5e2662cdc40deec17d82432da0b # good: [e760ac89f86586e287cc2b49fcf4a679e12140f3] powerpc: enable supported sanitizers on powerpc64* git bisect good e760ac89f86586e287cc2b49fcf4a679e12140f3 # bad: [bee87a73813d8599bdb926bd3dfb0704a35665c4] makefs: Fix warnings and reset WARNS to the default git bisect bad bee87a73813d8599bdb926bd3dfb0704a35665c4 # bad: [dee01da58a275b1cdc21b6211b26223968431449] Correctly measure system load averages > 1024 git bisect bad dee01da58a275b1cdc21b6211b26223968431449 # bad: [2737e6bca78c69280bf4d36e161941c88696c0a6] fusefs: use the fsname mount option if set git bisect bad 2737e6bca78c69280bf4d36e161941c88696c0a6 # bad: [7e6e22aab6b993e42328bafe0f64ee14a2b7c43c] genet: fix output packet corruption in uncommon case git bisect bad 7e6e22aab6b993e42328bafe0f64ee14a2b7c43c # bad: [fb231965f9b7b3930b19f95e454eb86b348e0640] mbuf: Properly set the default value for mb_use_ext_pgs git bisect bad fb231965f9b7b3930b19f95e454eb86b348e0640 # first bad commit: [fb231965f9b7b3930b19f95e454eb86b348e0640] mbuf: Properly set the default value for mb_use_ext_pgs The problematic commit is: https://cgit.freebsd.org/src/commit/?h=releng/13.2&id=fb231965f9b7b3930b19f95e454eb86b348e0640 mb_use_ext_pgs was swapped for a static integer, looking at the definition of the variable in question: bool mb_use_ext_pgs = false; /* use M_EXTPG mbufs for sendfile & TLS */ So now I am on the latest releng/13.2 branch with that one commit reverted and everything appears to be working! I must say, git bisect is magical. Gleb Smirnoff, thanks for reaching out - I thought this ticket was dropping in priority and might be some specific quirk to my setup. Post revert tcpdump from host (working): root@bender % tcpdump -i em0 -v -s0 port 80 tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes 00:38:27.071933 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64) VictorsMBP2021.lan.60907 > bender.lan.http: Flags [S], cksum 0xeb23 (correct), seq 2333296919, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 994099616 ecr 0,sackOK,eol], length 0 00:38:27.071988 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) bender.lan.http > VictorsMBP2021.lan.60907: Flags [S.], cksum 0x2db6 (correct), seq 2892280411, ack 2333296920, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2560874237 ecr 994099616], length 0 00:38:27.072263 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.60907 > bender.lan.http: Flags [.], cksum 0x5477 (correct), ack 1, win 2058, options [nop,nop,TS val 994099616 ecr 2560874237], length 0 00:38:27.072300 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.60907: Flags [.], cksum 0x4c75 (correct), ack 1, win 4107, options [nop,nop,TS val 2560874238 ecr 994099616], length 0 00:38:27.075219 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 126) VictorsMBP2021.lan.60907 > bender.lan.http: Flags [P.], cksum 0x7400 (correct), seq 1:75, ack 1, win 2058, options [nop,nop,TS val 994099619 ecr 2560874238], length 74: HTTP, length: 74 GET / HTTP/1.1 Host: bender.lan User-Agent: curl/7.88.1 Accept: */* 00:38:27.075384 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.60907: Flags [.], cksum 0x6e38 (correct), seq 1:1449, ack 75, win 4107, options [nop,nop,TS val 2560874241 ecr 994099619], length 1448: HTTP, length: 1448 HTTP/1.1 200 OK Server: nginx/1.22.1 Date: Thu, 23 Mar 2023 07:38:27 GMT Content-Type: text/html Content-Length: 5350 Last-Modified: Sun, 11 Sep 2022 04:28:27 GMT Connection: keep-alive ETag: "631d63eb-14e6" Accept-Ranges: bytes <!doctype html> <html lang="en" class="h-100"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta name="description" content=""> <meta name="author" content="Mark Otto, Jacob Thornton, and Bootstrap contributors"> <meta name="generator" content="Jekyll v3.8.5"> <title>Bender</title> <!-- Bootstrap core CSS --> <link href="bootstrap/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> <style> .bd-placeholder-img { font-size: 1.125rem; text-anchor: middle; -webkit-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; } @media (min-width: 768px) { .bd-placeholder-img-lg { font-size: 3.5rem; } } </style> <!-- Custom styles for this template --> <link href="sticky-footer-navbar.css" rel="stylesheet"> </head> <body class="d-flex flex-column h-100"> <header> <!-- Fixed navbar --> <nav class="navbar navbar-expand-md navbar-dark fixed-to[!http] 00:38:27.075399 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.60907: Flags [.], cksum 0xeea8 (correct), seq 1449:2897, ack 75, win 4107, options [nop,nop,TS val 2560874241 ecr 994099619], length 1448: HTTP 00:38:27.075410 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500) bender.lan.http > VictorsMBP2021.lan.60907: Flags [.], cksum 0x8da2 (correct), seq 2897:4345, ack 75, win 4107, options [nop,nop,TS val 2560874241 ecr 994099619], length 1448: HTTP 00:38:27.075418 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1298) bender.lan.http > VictorsMBP2021.lan.60907: Flags [P.], cksum 0x6ace (correct), seq 4345:5591, ack 75, win 4107, options [nop,nop,TS val 2560874241 ecr 994099619], length 1246: HTTP 00:38:27.075725 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.60907 > bender.lan.http: Flags [.], cksum 0x3ea6 (correct), ack 5591, win 1971, options [nop,nop,TS val 994099620 ecr 2560874241], length 0 00:38:27.075862 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.60907 > bender.lan.http: Flags [.], cksum 0x3e59 (correct), ack 5591, win 2048, options [nop,nop,TS val 994099620 ecr 2560874241], length 0 00:38:27.076435 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.60907 > bender.lan.http: Flags [F.], cksum 0x3e58 (correct), seq 75, ack 5591, win 2048, options [nop,nop,TS val 994099620 ecr 2560874241], length 0 00:38:27.076456 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.60907: Flags [.], cksum 0x364c (correct), ack 76, win 4107, options [nop,nop,TS val 2560874242 ecr 994099620], length 0 00:38:27.076481 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) bender.lan.http > VictorsMBP2021.lan.60907: Flags [F.], cksum 0x364b (correct), seq 5591, ack 76, win 4107, options [nop,nop,TS val 2560874242 ecr 994099620], length 0 00:38:27.076841 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) VictorsMBP2021.lan.60907 > bender.lan.http: Flags [.], cksum 0x3e55 (correct), ack 5592, win 2048, options [nop,nop,TS val 994099621 ecr 2560874242], length 0
(In reply to Victor Igumnov from comment #39) That's a useful data point. Unfortunately, I think the problem goes deeper. :) Commit fb231965f9b7b3930b19f95e454eb86b348e0640 fixed a regression from commit fcaa890c4469118255d463495b4044eef484fa3e. In between those two commits, sendfile does not use "extpg mbufs". So it sounds like extpg mbufs are related to this problem somehow. If you revert fcaa890c4469118255d463495b4044eef484fa3e as well, does the problem come back? A second experiment: with unmodified 13.2, try setting kern.ipc.mb_use_ext_pgs=0. Does the problem go away again?
(In reply to Mark Johnston from comment #40) Hi, > sysctl -w kern.ipc.mb_use_ext_pgs=0 That toggle does address the hanging behavior. Before my I rebooted into the unmodified kernel I noticed my reverted commit set that value to 0, while the unmodified kernel's default value is 1. Given that now everything works, will stick to the unmodified kernel. Do you need me to verify reverting fcaa890c4469118255d463495b4044eef484fa3e? Hardware details: vendor_id : GenuineIntel cpu family : 6 model : 142 model name : Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz stepping : 12 cpu MHz : 2112.00 cache size : 256 KB physical id : 0 siblings : 8 core id : 7 cpu cores : 8 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 6 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust sgx bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt constant_tsc md_clear ibrs stibp flush_l1d arch_capabilities ssbd xsaveopt xsavec xgetbv1 xsaves bugs : bogomips : 4224.00 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management:
(In reply to Victor Igumnov from comment #41) Ok. So to summarize so far: - the problem is unrelated to bhyve itself, but it only happens when em0 is a member of a bridge (maybe because this puts em0 in promiscuous mode?) - the problem doesn't occur in 13.1 - the problem doesn't occur when extpg mbufs are not used - some of the packets generated by the nginx host seem to have incorrect checksums(?) > Do you need me to verify reverting fcaa890c4469118255d463495b4044eef484fa3e? I think it is not necessary for now. Could you please try the following? 0. Enable extpg mbufs. 1. On the nginx host, run "dtrace -o /tmp/stacks.txt -n 'fbt::in_cksum_skip:entry {@[stack(), args[1]] = count();}'" and leave it running. 2. Trigger the hang. 3. ctrl-C the dtrace process, and attach /tmp/stacks.txt here.
(In reply to Mark Johnston from comment #42) Yes, you summarized it correctly; having the em0 interface on the bridge is causing the issue with ext_pgs. dtrace dump: kernel`nfsrvd_getcache+0x39c kernel`svc_run_internal+0xb4f kernel`svc_run+0x1ca kernel`nfsrvd_nfsd+0x2e4 kernel`nfssvc_nfsd+0x519 kernel`sys_nfssvc+0x9c kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 32 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`tunwrite+0x50a kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1321 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`tunwrite+0x50a kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1330 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`tunwrite+0x50a kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1359 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`tunwrite+0x50a kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1360 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`tunwrite+0x50a kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1368 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1335 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1338 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1343 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1344 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1351 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1356 1 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1362 1 kernel`ip_output+0x10f8 kernel`udp6_send+0x500 kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 1 kernel`ip_output+0x10f8 kernel`syncache_add+0xc47 kernel`tcp_input_with_port+0x1263 kernel`tcp_input+0xb kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 20 1 kernel`ip_output+0x10f8 kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`svc_vc_reply+0x144 kernel`svc_sendreply_common+0x97 kernel`svc_sendreply_mbuf+0x51 kernel`nfssvc_program+0x81b kernel`svc_run_internal+0xb4f kernel`svc_run+0x1ca kernel`nfsrvd_nfsd+0x2e4 kernel`nfssvc_nfsd+0x519 kernel`sys_nfssvc+0x9c kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 1 kernel`ip_output+0x10f8 kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`soo_write+0x49 kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 1 kernel`ip_output+0x10f8 kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`soo_write+0x49 kernel`dofilewrite+0x88 kernel`sys_write+0xbc kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 1 kernel`in_delayed_cksum+0x3d kernel`syncache_respond+0x699 kernel`syncache_add+0xc47 kernel`tcp_input_with_port+0x1263 kernel`tcp_input+0xb kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 60 1 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1298 1 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`svc_vc_reply+0x144 kernel`svc_sendreply_common+0x97 kernel`svc_sendreply_mbuf+0x51 kernel`nfssvc_program+0x81b kernel`svc_run_internal+0xb4f kernel`svc_run+0x1ca kernel`nfsrvd_nfsd+0x2e4 kernel`nfssvc_nfsd+0x519 kernel`sys_nfssvc+0x9c kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 224 1 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`soo_write+0x49 kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1009 1 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`soo_write+0x49 kernel`dofilewrite+0x88 kernel`sys_write+0xbc kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 88 1 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 58 1 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 373 1 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 506 1 kernel`in_delayed_cksum+0x88 kernel`udp_send+0xa90 kernel`udp6_send+0x500 kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 111 1 kernel`tcp_input_with_port+0x256 kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`tunwrite+0x50a kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 52 1 kernel`tcp_input_with_port+0x256 kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`tunwrite+0x50a kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 156 1 kernel`tcp_input_with_port+0x256 kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 52 1 kernel`tcp_input_with_port+0x256 kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`tunwrite+0x30e kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 660 1 kernel`ip_output+0xaee kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 2 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 48 2 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 113 2 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 283 2 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 328 2 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 331 2 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 337 2 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 397 2 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 508 2 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 1500 3 kernel`udp_input+0x2cf kernel`swi_net+0x12b kernel`ithread_loop+0x25a kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 49 4 kernel`ip_output+0x10f8 kernel`tcp_usr_send+0x17d kernel`vn_sendfile+0x12a4 kernel`sendfile+0x117 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 4 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_do_segment+0x278d kernel`tcp_input_with_port+0xb61 kernel`tcp_input+0xb kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 40 4 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 64 4 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 76 4 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 125 4 kernel`in_delayed_cksum+0x88 kernel`udp_send+0xa90 kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 129 4 kernel`ip_output+0x10f8 kernel`tcp_timer_delack+0xeb kernel`softclock_call_cc+0x133 kernel`softclock+0x79 kernel`ithread_loop+0x25a kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 20 5 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_timer_delack+0xeb kernel`softclock_call_cc+0x133 kernel`softclock+0x79 kernel`ithread_loop+0x25a kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 52 5 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_do_segment+0x278d kernel`tcp_input_with_port+0xb61 kernel`tcp_input+0xb kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 52 5 kernel`icmp_reflect+0x51d kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 576 5 kernel`in_delayed_cksum+0x88 kernel`udp_send+0xa90 kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendmsg+0x61 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 95 5 kernel`in_delayed_cksum+0x88 kernel`udp_send+0xa90 kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendmsg+0x61 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 169 5 kernel`ip_output+0x10f8 kernel`tcp_timer_rexmt+0x601 kernel`softclock_call_cc+0x133 kernel`softclock+0x79 kernel`ithread_loop+0x25a kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 20 6 kernel`in_delayed_cksum+0x3d kernel`tcp_output+0x1dbf kernel`tcp_timer_rexmt+0x601 kernel`softclock_call_cc+0x133 kernel`softclock+0x79 kernel`ithread_loop+0x25a kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 1500 6 kernel`in_delayed_cksum+0x88 kernel`udp_send+0xa90 kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 49 8 kernel`ip_output+0x10f8 kernel`tcp_do_segment+0x278d kernel`tcp_input_with_port+0xb61 kernel`tcp_input+0xb kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 20 9 kernel`udp_input+0x2cf kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`tunwrite+0x50a kernel`devfs_write_f+0xda kernel`dofilewrite+0x88 kernel`sys_writev+0x6e kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 79 10 kernel`ip_output+0x10f8 kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendmsg+0x61 kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 10 kernel`ip_output+0x10f8 kernel`sosend_dgram+0x347 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 10 kernel`ip_output+0x10f8 kernel`tcp_usr_send+0x17d kernel`sosend_generic+0x617 kernel`sosend+0x50 kernel`kern_sendit+0x20e kernel`sendit+0x1da kernel`sys_sendto+0x4d kernel`amd64_syscall+0x10c kernel`0xffffffff810896cb 20 17 kernel`ip_output+0x10f8 kernel`icmp_error+0x56c kernel`udp_input+0x8ca kernel`ip_input+0x118 kernel`netisr_dispatch_src+0xb9 kernel`ether_demux+0x144 kernel`ether_nh_input+0x346 kernel`netisr_dispatch_src+0xb9 kernel`ether_input+0x69 kernel`iflib_rxeof+0xbcb kernel`_task_fn_rx+0x72 kernel`gtaskqueue_run_locked+0x15d kernel`gtaskqueue_thread_loop+0xc3 kernel`fork_exit+0x7e kernel`0xffffffff81089e2e 20 19
Created attachment 241079 [details] patch Could you please test whether the attached patch fixes the problem? There is a missing MFC in 13.
(In reply to Mark Johnston from comment #44) Verified, the patch does, in fact fix the issue with kern.ipc.mb_use_ext_pgs=1
(In reply to Victor Igumnov from comment #45) Great! Thanks very much for your patience in getting this resolved. I'm working on getting this fixed in 13.2.
Mark, thanks a lot for nailing it down!
The patch will be in 13.2.