Setup: lighttpd 1.4.82 running on FreeBSD-15.0 with OpenSSL 3.5.4 from base or with OpenSSL 3.5.6 from ports Network driver: igb0 Serving static files with plain http and https using mod_openssl Problem: I got reports that a source packages fetched from my server where corrupted. I was able to reproduce this on multiple servers. Even with the default lighttpd config and adding: ssl.engine = "enable". Steps to reproduce: The file must be > 128 kbyte to become corrupted. Fetching the files and running sha256sum an the result showed: 6ef4e286... 20260601.tgz.curl.http 6ef4e286... 20260601.tgz.curl.https 6ef4e286... 20260601.tgz.fetch.http a717baa2... 20260601.tgz.fetch.https 6ef4e286... 20260601.tgz.wget.http 347d964a... 20260601.tgz.wget.https 6ef4e286... is the correct checksum The error log reports: lighttpd-1.4.82/src/mod_openssl.c.4808) SSL: addr:2***:****::*** ssl_err:5 ret:-1 errno:35: Resource temporarily unavailable I was unable to reproduce the problem on servers running FreeBSD-14.4. 14.4-RELEASE-p5 ssl=openssl35 mod_openssl OK 14.4-RELEASE-p5 ssl=openssl35 mod_openssl OK 15.0-RELEASE-p9 ssl=openssl35 mod_openssl broken 15.0-RELEASE-p9 ssl=base mod_openssl broken 15.0-RELEASE-p9 ssl=openssl35 mod_wolfssl OK sysctl kern.ipc.tls.stats | grep -v ': 0' FreeBSD-14: all values zero FreeBSD-15: kern.ipc.tls.stats.ocf.separate_output: 2114 kern.ipc.tls.stats.ocf.inplace: 12982513 kern.ipc.tls.stats.ocf.tls13_chacha20_encrypts: 232 kern.ipc.tls.stats.ocf.tls13_chacha20_decrypts: 191 kern.ipc.tls.stats.ocf.tls13_gcm_encrypts: 12984395 kern.ipc.tls.stats.ocf.tls13_gcm_decrypts: 6557803 kern.ipc.tls.stats.enable_calls: 3204921 kern.ipc.tls.stats.offload_total: 3204921 kern.ipc.tls.stats.threads: 4 Workaround: Disable KTLS in lighttpd.conf: ssl.openssl.ssl-conf-cmd += ("Options" => "-KTLS")
I tried to reproduce this. I installed 15R into a bhyve VM and selected e1000 as the nic (which is as close as I can get to igb, and they both use iflib, so I don't think that matters). I was unable to reproduce it. My conf file looks like: root@15R:/home/gallatin # egrep -v ^# /usr/local/etc/lighttpd/lighttpd.conf include "/usr/local/etc/lighttpd/lighttpd*annotated.conf" include "/usr/local/etc/lighttpd/conf-enabled/*.conf" server.modules += ( "mod_openssl" ) $SERVER["socket"] == ":443" { ssl.engine = "enable" ssl.privkey = "/usr/local/etc/lighttpd/ssl/server.key" ssl.pemfile = "/usr/local/etc/lighttpd/ssl/server.crt" } And after transferring a large file (/boot/kernel/kernel copied to the document root) many times, I don't see any issues. One thing I noticed is that I see separate_output elevated in my counters, which indicates the server is using sendfile (confirmed via truss). The reporters counters show inplace elevated: root@15R:/home/gallatin # sysctl kern.ipc.tls | grep -v ': 0' kern.ipc.tls.ifnet.permitted: 1 kern.ipc.tls.sw.gcm: 84 kern.ipc.tls.max_reclaim: 1024 kern.ipc.tls.sw_buffer_cache: 1 kern.ipc.tls.cbc_enable: 1 kern.ipc.tls.enable: 1 kern.ipc.tls.ifnet_max_rexmit_pct: 2 kern.ipc.tls.maxlen: 16384 kern.ipc.tls.stats.ocf.separate_output: 139557 kern.ipc.tls.stats.ocf.inplace: 312 kern.ipc.tls.stats.ocf.tls13_gcm_encrypts: 139869 kern.ipc.tls.stats.ocf.tls13_gcm_decrypts: 935 kern.ipc.tls.stats.active: 84 kern.ipc.tls.stats.enable_calls: 156 kern.ipc.tls.stats.offload_total: 156 kern.ipc.tls.stats.sw_tx_inqueue: 1 kern.ipc.tls.stats.threads: 8 Dirk: Are you serving from a ZFS or UFS fs? If you truss the web server while serving (and ktls enabled), do you see sendfile? Eg: root@15R:/home/gallatin # truss -cp `pgrep httpd` ^Csyscall seconds calls errors close 0.000044609 2 0 shutdown 0.003182343 2 0 sendfile 0.011392569 240 26 pread 0.000103809 6 0 fstatat 0.000041602 2 0 recvmsg 0.000379371 23 1 sendmsg 0.000223300 4 0 setsockopt 0.000085522 4 0 write 0.000330193 12 2 getpid 0.000484688 18 0 kevent 1.888839601 257 0 read 0.000713765 22 6 accept4 0.000378697 4 2 ------------- ------- ------- 1.906200069 596 37
I've been trying to reproduce without sendfile enabled, but I'm afraid I don't know how to disable it. I tried: server.network-backend = "writev" But truss shows it still using sendfile..
I see no difference serving from ZFS or UFS I can not reproduce the problem in the VM. 1) hardware 1, jail, serving from ZFS: server.event-handler = "freebsd-kqueue" server.network-backend = "writev" changes to this config lines did not effect the problem. CPU: Intel(R) Xeon(R) CPU E5506 @ 2.13GHz (2133.42-MHz K8-class CPU) Origin="GenuineIntel" Id=0x106a5 Family=0x6 Model=0x1a Stepping=5 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x1<LAHF> VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID TSC: P-state invariant, performance statistics igb0: <Intel(R) PRO/1000 82576> port 0xef80-0xef9f mem 0xfafe0000-0xfaffffff,0xfafc0000-0xfafdffff,0xfaf9c000-0xfaf9ffff irq 28 at device 0.0 on pci1 igb0: EEPROM V1.2-3 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: Using 4 RX queues 4 TX queues igb0: Using MSI-X interrupts with 5 vectors igb0: Ethernet address: 00:25:90:**:**:** igb0: netmap queues/slots: TX 4/1024, RX 4/1024 # sysctl kern.ipc.tls | grep -v ': 0' kern.ipc.tls.ifnet.permitted: 1 kern.ipc.tls.sw.gcm: 316 kern.ipc.tls.max_reclaim: 1024 kern.ipc.tls.sw_buffer_cache: 1 kern.ipc.tls.cbc_enable: 1 kern.ipc.tls.enable: 1 kern.ipc.tls.ifnet_max_rexmit_pct: 2 kern.ipc.tls.maxlen: 16384 kern.ipc.tls.stats.ocf.separate_output: 320188 kern.ipc.tls.stats.ocf.inplace: 305320885 kern.ipc.tls.stats.ocf.tls13_chacha20_encrypts: 108010 kern.ipc.tls.stats.ocf.tls13_chacha20_decrypts: 54048 kern.ipc.tls.stats.ocf.tls13_gcm_encrypts: 304731247 kern.ipc.tls.stats.ocf.tls13_gcm_decrypts: 152629728 kern.ipc.tls.stats.ocf.tls12_chacha20_encrypts: 17698 kern.ipc.tls.stats.ocf.tls12_chacha20_decrypts: 14882 kern.ipc.tls.stats.ocf.tls12_gcm_encrypts: 784118 kern.ipc.tls.stats.ocf.tls12_gcm_decrypts: 555362 kern.ipc.tls.stats.failed_crypto: 1332 kern.ipc.tls.stats.corrupted_records: 75 kern.ipc.tls.stats.active: 316 kern.ipc.tls.stats.enable_calls: 145561272 kern.ipc.tls.stats.offload_total: 145561272 kern.ipc.tls.stats.threads: 8 2) hardware 2, no jail, serving UFS on gmirror CPU: Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz (3400.20-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306c3 Family=0x6 Model=0x3c Stepping=3 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUS H,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDC M,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x21<LAHF,ABM> Structured Extended Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG> Structured Extended Features3=0x9c000600<MCUOPT,MD_CLEAR,IBPB,STIBP,L1DFL,SSBD> XSAVE Features=0x1<XSAVEOPT> VT-x: Basic Features=0xda0400<SMM,INS/OUTS,TRUE> Pin-Based Controls=0x7f<ExtINT,NMI,VNMI,PreTmr> Primary Processor Controls=0xfff9fffe<INTWIN,TSCOff,HLT,INVLPG,MWAIT,RDPMC,RDTSC,CR3-LD,CR3- ST,CR8-LD,CR8-ST,TPR,NMIWIN,MOV-DR,IO,IOmap,MTF,MSRmap,MONITOR,PAUSE> Secondary Processor Controls=0x7cff<APIC,EPT,DT,RDTSCP,x2APIC,VPID,WBINVD,UG,PAUSE-loop,RDRAND,INVPCID,VMFUNC,VMCS> Exit Controls=0xda0400<PAT-LD,EFER-SV,PTMR-SV> Entry Controls=0xda0400 EPT Features=0x6334141<XO,PW4,UC,WB,2M,1G,INVEPT,AD,single,all> VPID Features=0xf01<INVVPID,individual,single,all,single-globals> TSC: P-state invariant, performance statistics igb0: promiscuous mode disabled igb0: <Intel(R) I210 (Copper)> port 0xd000-0xd01f mem 0xf7100000-0xf717ffff,0xf7180000-0xf7183fff irq 17 at device 0.0 on pci3 igb0: attach_pre capping queues at 4 igb0: EEPROM V3.16-0 eTrack 0x800004d7 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: msix_init qsets capped at 4 igb0: intr CPUs: 4 queue msgs: 4 admincnt: 1 igb0: Using 4 RX queues 4 TX queues igb0: attempting to allocate 5 MSI-X vectors (5 supported) igb0: using IRQs 34-38 for MSI-X igb0: Using MSI-X interrupts with 5 vectors igb0: allocated for 4 tx_queues igb0: allocated for 4 rx_queues igb0: bpf attached igb0: Ethernet address: 00:25:90:**:**:** igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb0: Link is up 1000 Mbps Full Duplex igb0: link state changed to UP debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80001d50800), ignoring. igb0: link state changed to DOWN igb0: Link is up 1000 Mbps Full Duplex igb0: link state changed to UP # sysctl kern.ipc.tls | grep -v ': 0' kern.ipc.tls.ifnet.permitted: 1 kern.ipc.tls.max_reclaim: 1024 kern.ipc.tls.sw_buffer_cache: 1 kern.ipc.tls.cbc_enable: 1 kern.ipc.tls.enable: 1 kern.ipc.tls.ifnet_max_rexmit_pct: 2 kern.ipc.tls.maxlen: 16384 kern.ipc.tls.stats.ocf.separate_output: 2114 kern.ipc.tls.stats.ocf.inplace: 12982528 kern.ipc.tls.stats.ocf.tls13_chacha20_encrypts: 232 kern.ipc.tls.stats.ocf.tls13_chacha20_decrypts: 191 kern.ipc.tls.stats.ocf.tls13_gcm_encrypts: 12984410 kern.ipc.tls.stats.ocf.tls13_gcm_decrypts: 6557814 kern.ipc.tls.stats.enable_calls: 3204927 kern.ipc.tls.stats.offload_total: 3204927 kern.ipc.tls.stats.threads: 4 3) bhyve VM with vtnet0, no problems vtnet0: <VirtIO Networking Adapter> on virtio_pci5 vtnet0: Ethernet address: 00:a0:98:**:**:** vtnet0: netmap queues/slots: TX 1/1024, RX 1/512 vtnet0: link state changed to UP # sysctl kern.ipc.tls | grep -v ': 0' kern.ipc.tls.ifnet.permitted: 1 kern.ipc.tls.max_reclaim: 1024 kern.ipc.tls.sw_buffer_cache: 1 kern.ipc.tls.cbc_enable: 1 kern.ipc.tls.enable: 1 kern.ipc.tls.ifnet_max_rexmit_pct: 2 kern.ipc.tls.maxlen: 16384
Your VM was not using ktls, based on those counters. I switched my VM from vtnet to e1000 and still didn't hit an issue. I don't think igb vs vtnet should matter. vtnet does not handle EXTPG mbufs, but that just means the mbuf chain is re-written in ip_output, that does not impact the crypto. So far I've tried: - current, UFS, no repro, vtnet - 15R, UFS, no repro, vtnet - 15R, ZFS, no repro, em0 I'd really like to figure out the issue here. The big thing that stands out in the ktls stats is ocf.inplace vs ocf.separate_output. Our (Netflix) workload is nginx, with sendfile, and shows ~99% separate output, which is the path taken by sendfile without hdr/trl. I recently fixed a bug in main (which I need to MFC to 15) that fixes a problem with mixing sendfile headers with normal sendfile file data (https://cgit.freebsd.org/src/commit/?id=fd9af1e7084c550358d9598407f9ee7583e6a53f), but the failure mode there is very different. But I'm concerned there is some similar bug someplace that has gone undiscovered because we don't use that code path. The other thing you should be aware of is that the benefits of software ktls are mostly limited to the sendfile path, because it avoids having to copy the data into userspace, encrypt it, and copy it back into the kernel. If the data is warm in cache because its not static, then there is not much benefit to SW ktls (it may even cost more). So if I was you, I'd either disable ktls or try to figure out why your static content is not using sendfile. Please don't take this as me trying to sweep this bug under the rug; I really want to figure it out. If you could replicate it in a VM, that would be amazing.
I see: > kern.ipc.tls.stats.failed_crypto: 1332 > kern.ipc.tls.stats.corrupted_records: 75 Which crypto drivers are in use? Do you have qat.ko loaded? Please share "pciconf -lv" output.
(In reply to Andrew Gallatin from comment #4) I did more testing: 3) bhyve VM with vtnet0, no problems # sysctl kern.ipc.tls | grep -v ': 0' kern.ipc.tls.ifnet.permitted: 1 kern.ipc.tls.sw.gcm: 4 kern.ipc.tls.max_reclaim: 1024 kern.ipc.tls.sw_buffer_cache: 1 kern.ipc.tls.cbc_enable: 1 kern.ipc.tls.enable: 1 kern.ipc.tls.ifnet_max_rexmit_pct: 2 kern.ipc.tls.maxlen: 16384 kern.ipc.tls.stats.ocf.separate_output: 700 kern.ipc.tls.stats.ocf.inplace: 396 kern.ipc.tls.stats.ocf.tls13_gcm_encrypts: 1096 kern.ipc.tls.stats.ocf.tls13_gcm_decrypts: 36 kern.ipc.tls.stats.active: 4 kern.ipc.tls.stats.enable_calls: 16 kern.ipc.tls.stats.offload_total: 16 kern.ipc.tls.stats.threads: 4 Still unable to reproduce any checksum error in VM.
(In reply to Mark Johnston from comment #5) > kern.ipc.tls.stats.failed_crypto: 1332 > kern.ipc.tls.stats.corrupted_records: 75 Unrelated, this counters did not change when testing. Might be created from different jails and bad requests. > Which crypto drivers are in use? Hardware 1) # openssl engine (rdrand) Intel RDRAND engine (dynamic) Dynamic engine loading support Hardware 2) # openssl engine (dynamic) Dynamic engine loading support > Do you have qat.ko loaded? not loaded. Please share "pciconf -lv" output. Hardware 1) Active igb0: checksum failed em0@pci0:0:25:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x153a subvendor=0x15d9 subdevice=0x153a vendor = 'Intel Corporation' device = 'Ethernet Connection I217-LM' class = network subclass = ethernet igb0@pci0:3:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet Hardware 2) Active igb0: igb0@pci0:5:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10c9 subvendor=0x15d9 subdevice=0x10c9 vendor = 'Intel Corporation' device = '82576 Gigabit Network Connection' class = network subclass = ethernet igb1@pci0:5:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10c9 subvendor=0x15d9 subdevice=0x10c9 vendor = 'Intel Corporation' device = '82576 Gigabit Network Connection' class = network subclass = ethernet Hardware 3) Active igb0: checksum failed igb0@pci0:3:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x8086 subdevice=0x35bc vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet igb1@pci0:4:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x8086 subdevice=0x35bc vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet Hardware 4) Tower, all checksum ok re0@pci0:3:0:0: class=0x020000 rev=0x0c hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1043 subdevice=0x8554 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller' class = network subclass = ethernet Hardware 5) Laptop, all checksum ok re0@pci0:5:0:0: class=0x020000 rev=0x15 hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x103c subdevice=0x8978 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller' class = network subclass = ethernet Hardware 6) Active em0: checksum failed em0@pci0:0:31:6: class=0x020000 rev=0x20 hdr=0x00 vendor=0x8086 device=0x15fc subvendor=0x1458 subdevice=0x1000 vendor = 'Intel Corporation' device = 'Ethernet Connection (13) I219-V' class = network subclass = ethernet igc0@pci0:174:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x1458 subdevice=0x1000 vendor = 'Intel Corporation' device = 'Ethernet Controller I225-V' class = network subclass = ethernet Hardware 7) checksum OK em0@pci0:0:31:6: class=0x020000 rev=0x21 hdr=0x00 vendor=0x8086 device=0x156f subvendor=0x1458 subdevice=0xe000 vendor = 'Intel Corporation' device = 'Ethernet Connection I219-LM' class = network subclass = ethernet Test; Some machines have in /boot/loader.conf: hw.em.rx_process_limit="-1" But this does not have any effect on the problem here. Test with/without promisc mode makes no change. Test: with/without offloading makes no change. igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 otions=4e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> options=802038<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,WOL_MAGIC,HWSTATS>
Created attachment 271687 [details] Full pciconf -l -v on Hardware2