Bug 241958 - Enable option TCPHPTS in the kernel causes a large % of interruptions if there is no HPET eventtimer in the system
Summary: Enable option TCPHPTS in the kernel causes a large % of interruptions if ther...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Michael Tuexen
URL:
Keywords: performance
Depends on:
Blocks:
 
Reported: 2019-11-13 23:27 UTC by Vladyslav V. Prodan
Modified: 2021-05-28 09:30 UTC (History)
9 users (show)

See Also:
koobs: mfc-stable12+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Vladyslav V. Prodan 2019-11-13 23:27:00 UTC
Enable option TCPHPTS in the kernel causes a large % of interruptions if there is no HPET eventtimer in the system.

FreeBSD system running on VPS with 1 GB of RAM in the OVH data center

# uname -a
FreeBSD vps7XXXXX.ovh.net 12.1-STABLE FreeBSD 12.1-STABLE r354671 SUPPORT-12-1-3  amd64

#top -CHPS
last pid: 54135;  load averages:  1.13,  0.83,  0.83    up 0+00:40:41  01:17:27
370 threads:   2 running, 342 sleeping, 26 waiting
CPU:  0.0% user,  0.0% nice,  0.4% system, 41.6% interrupt, 58.0% idle
Mem: 87M Active, 23M Inact, 118M Wired, 1677M Free
ARC: 40M Total, 5499K MFU, 32M MRU, 1220K Anon, 395K Header, 1405K Other
     21M Compressed, 61M Uncompressed, 2.97:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    TIME     CPU COMMAND
   11 root        155 ki31      0    16K RUN     27:09  57.51% idle
   12 root        -72    -      0   416K WAIT    13:12  42.04% intr{swi1: hpts}

# sysctl -a kern.eventtimer
kern.eventtimer.periodic: 0
kern.eventtimer.timer: LAPIC
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 4
kern.eventtimer.choice: LAPIC(600) i8254(100) RTC(0)
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 500011170
kern.eventtimer.et.LAPIC.flags: 7

# sysctl -a net.inet.tcp.hpts
net.inet.tcp.hpts.0.logsize: 3072
net.inet.tcp.hpts.0.curtick: 284411163
net.inet.tcp.hpts.0.curslot: 46360
net.inet.tcp.hpts.0.active: 0
net.inet.tcp.hpts.0.out_qcnt: 0
net.inet.tcp.hpts.0.in_qcnt: 0
net.inet.tcp.hpts.log:
net.inet.tcp.hpts.skip_swi: 0
net.inet.tcp.hpts.minsleep: 250
net.inet.tcp.hpts.maxsleep: 102398
net.inet.tcp.hpts.out_tspercision: 0
net.inet.tcp.hpts.out_tsperpcb: 0
net.inet.tcp.hpts.in_tspercision: 0
net.inet.tcp.hpts.in_tsperpcb: 0
net.inet.tcp.hpts.no_tcbsfound: 0
net.inet.tcp.hpts.loops: 42998907
net.inet.tcp.hpts.logging: 0
net.inet.tcp.hpts.precision: 120



#dmesg
FreeBSD 12.1-STABLE r354671 SUPPORT-12-1-0 amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
sysctl: OID number(131) is already in use for 'me'
VT(vga): resolution 640x480
CPU: Intel Core Processor (Haswell, no TSX) (2400.01-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306c1  Family=0x6  Model=0x3c  Stepping=1
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x7a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
...
Comment 1 Ed Maste freebsd_committer freebsd_triage 2019-11-14 21:21:05 UTC
CC rrs
Comment 2 Mark Johnston freebsd_committer freebsd_triage 2020-06-04 16:32:38 UTC
Is this a problem on HEAD as well?  I'm not sure if some of the recent TCP changes might have fixed this.
Comment 3 Michael Tuexen freebsd_committer freebsd_triage 2020-06-04 18:16:19 UTC
rrs is working on it.
Comment 4 Ivan Rozhuk 2020-06-10 16:31:45 UTC
On VBox even with hpet - same strange CPU load, especially with 4+ cores.
Comment 5 Tom Jones freebsd_committer freebsd_triage 2020-06-18 17:52:51 UTC
I think I can reproduce this on 12-stable (r362275)

With options TCPHPTS, I see a consistent interrupt load on a bhyve vm on a 12.1 host.

CPU:  0.0% user,  0.0% nice,  0.0% system,  8.0% interrupt, 91.9% idle

Without TCPHPTS the vm is almost completely idle and there is no interrupt load.
Comment 6 Tom Jones freebsd_committer freebsd_triage 2020-06-18 17:53:48 UTC
Meant to add:

root@freebsd-stable:~ # sysctl kern.eventtimer
kern.eventtimer.periodic: 0
kern.eventtimer.timer: LAPIC
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 4
kern.eventtimer.choice: LAPIC(600) HPET(550) i8254(100) RTC(0)
kern.eventtimer.et.HPET.quality: 550
kern.eventtimer.et.HPET.frequency: 16777216
kern.eventtimer.et.HPET.flags: 7
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 67695695
kern.eventtimer.et.LAPIC.flags: 7
Comment 7 Michael Tuexen freebsd_committer freebsd_triage 2020-08-22 21:15:08 UTC
I could reproduce the problem using the settings of tjh@. More precisely, r362879, in a bhyve VM using a GENERIC kernel with TCPHPTS added, I observe a 5% to 6% interrupt load. Using r362880 this does not happen anymore. Also I don't observe an interrupt load on VBox with an up-to-date stable/12 machine.

I think the issue is already solved by https://svnweb.freebsd.org/changeset/base/362880 and therefore I'm closing this issue. If the problem can still be reproduced on a system with r362880 or higher, please reopen.