Enable option TCPHPTS in the kernel causes a large % of interruptions if there is no HPET eventtimer in the system. FreeBSD system running on VPS with 1 GB of RAM in the OVH data center # uname -a FreeBSD vps7XXXXX.ovh.net 12.1-STABLE FreeBSD 12.1-STABLE r354671 SUPPORT-12-1-3 amd64 #top -CHPS last pid: 54135; load averages: 1.13, 0.83, 0.83 up 0+00:40:41 01:17:27 370 threads: 2 running, 342 sleeping, 26 waiting CPU: 0.0% user, 0.0% nice, 0.4% system, 41.6% interrupt, 58.0% idle Mem: 87M Active, 23M Inact, 118M Wired, 1677M Free ARC: 40M Total, 5499K MFU, 32M MRU, 1220K Anon, 395K Header, 1405K Other 21M Compressed, 61M Uncompressed, 2.97:1 Ratio Swap: 1024M Total, 1024M Free PID USERNAME PRI NICE SIZE RES STATE TIME CPU COMMAND 11 root 155 ki31 0 16K RUN 27:09 57.51% idle 12 root -72 - 0 416K WAIT 13:12 42.04% intr{swi1: hpts} # sysctl -a kern.eventtimer kern.eventtimer.periodic: 0 kern.eventtimer.timer: LAPIC kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 4 kern.eventtimer.choice: LAPIC(600) i8254(100) RTC(0) kern.eventtimer.et.i8254.quality: 100 kern.eventtimer.et.i8254.frequency: 1193182 kern.eventtimer.et.i8254.flags: 1 kern.eventtimer.et.RTC.quality: 0 kern.eventtimer.et.RTC.frequency: 32768 kern.eventtimer.et.RTC.flags: 17 kern.eventtimer.et.LAPIC.quality: 600 kern.eventtimer.et.LAPIC.frequency: 500011170 kern.eventtimer.et.LAPIC.flags: 7 # sysctl -a net.inet.tcp.hpts net.inet.tcp.hpts.0.logsize: 3072 net.inet.tcp.hpts.0.curtick: 284411163 net.inet.tcp.hpts.0.curslot: 46360 net.inet.tcp.hpts.0.active: 0 net.inet.tcp.hpts.0.out_qcnt: 0 net.inet.tcp.hpts.0.in_qcnt: 0 net.inet.tcp.hpts.log: net.inet.tcp.hpts.skip_swi: 0 net.inet.tcp.hpts.minsleep: 250 net.inet.tcp.hpts.maxsleep: 102398 net.inet.tcp.hpts.out_tspercision: 0 net.inet.tcp.hpts.out_tsperpcb: 0 net.inet.tcp.hpts.in_tspercision: 0 net.inet.tcp.hpts.in_tsperpcb: 0 net.inet.tcp.hpts.no_tcbsfound: 0 net.inet.tcp.hpts.loops: 42998907 net.inet.tcp.hpts.logging: 0 net.inet.tcp.hpts.precision: 120 #dmesg FreeBSD 12.1-STABLE r354671 SUPPORT-12-1-0 amd64 FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1) sysctl: OID number(131) is already in use for 'me' VT(vga): resolution 640x480 CPU: Intel Core Processor (Haswell, no TSX) (2400.01-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1 Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x21<LAHF,ABM> Structured Extended Features=0x7a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID> XSAVE Features=0x1<XSAVEOPT> VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr Hypervisor: Origin = "KVMKVMKVM" ...
CC rrs
Is this a problem on HEAD as well? I'm not sure if some of the recent TCP changes might have fixed this.
rrs is working on it.
On VBox even with hpet - same strange CPU load, especially with 4+ cores.
I think I can reproduce this on 12-stable (r362275) With options TCPHPTS, I see a consistent interrupt load on a bhyve vm on a 12.1 host. CPU: 0.0% user, 0.0% nice, 0.0% system, 8.0% interrupt, 91.9% idle Without TCPHPTS the vm is almost completely idle and there is no interrupt load.
Meant to add: root@freebsd-stable:~ # sysctl kern.eventtimer kern.eventtimer.periodic: 0 kern.eventtimer.timer: LAPIC kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 4 kern.eventtimer.choice: LAPIC(600) HPET(550) i8254(100) RTC(0) kern.eventtimer.et.HPET.quality: 550 kern.eventtimer.et.HPET.frequency: 16777216 kern.eventtimer.et.HPET.flags: 7 kern.eventtimer.et.i8254.quality: 100 kern.eventtimer.et.i8254.frequency: 1193182 kern.eventtimer.et.i8254.flags: 1 kern.eventtimer.et.RTC.quality: 0 kern.eventtimer.et.RTC.frequency: 32768 kern.eventtimer.et.RTC.flags: 17 kern.eventtimer.et.LAPIC.quality: 600 kern.eventtimer.et.LAPIC.frequency: 67695695 kern.eventtimer.et.LAPIC.flags: 7
I could reproduce the problem using the settings of tjh@. More precisely, r362879, in a bhyve VM using a GENERIC kernel with TCPHPTS added, I observe a 5% to 6% interrupt load. Using r362880 this does not happen anymore. Also I don't observe an interrupt load on VBox with an up-to-date stable/12 machine. I think the issue is already solved by https://svnweb.freebsd.org/changeset/base/362880 and therefore I'm closing this issue. If the problem can still be reproduced on a system with r362880 or higher, please reopen.