Bug 192315 - high load average while idle, slow keyboard reaction
Summary: high load average while idle, slow keyboard reaction
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-08-01 18:53 UTC by Steve Wills
Modified: 2014-08-18 08:38 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Steve Wills freebsd_committer 2014-08-01 18:53:20 UTC
Was seeing load average of around 5, even with next to nothing running and the system was nearly completely idle. Keyboard was also slow to react and shutting down paused for longer than 60 seconds for bufdaemon. Touching the keyboard would make it continue. Setting kern.eventtimer.timer=HPET seemed to resolve the issue. See https://lists.freebsd.org/pipermail/freebsd-current/2014-July/051498.html for others who have experienced the same thing. Opening bug per Adrian's request.
Comment 1 Alexander Motin freebsd_committer 2014-08-01 19:05:57 UTC
Line "Event timer "LAPIC" quality 400" in verbose dmesg tells that LAPIC timer in your CPU (used by default) is not compatible with CPU C-states (deep idle states). Can you make sure that C-states are not used on your system by checking 'sysctl dev.cpu | grep cx_usage' and 'sysctl -a | grep cx_lowest'?

Timer code was not changing for quite some time already, while C-states were enabled by default less time ago, IIRC.
Comment 2 Alexander Motin freebsd_committer 2014-08-01 19:07:41 UTC
Theoretically C-states should be blocked automatically in such situation, but who knows... Also check whether your BIOS support C1E state, and if it does, try to disable it.
Comment 3 Steve Wills freebsd_committer 2014-08-01 20:24:18 UTC
(In reply to Alexander Motin from comment #1)
> Line "Event timer "LAPIC" quality 400" in verbose dmesg tells that LAPIC
> timer in your CPU (used by default) is not compatible with CPU C-states
> (deep idle states). Can you make sure that C-states are not used on your
> system by checking 'sysctl dev.cpu | grep cx_usage' and 'sysctl -a | grep
> cx_lowest'?
> 
> Timer code was not changing for quite some time already, while C-states were
> enabled by default less time ago, IIRC.

meatwad% sysctl dev.cpu | grep cx_usage
dev.cpu.0.cx_usage: 15.53% 82.84% 1.61% last 149us
dev.cpu.0.cx_usage_counters: 1509220 8046504 156453
dev.cpu.1.cx_usage: 15.21% 77.78% 6.99% last 1803us
dev.cpu.1.cx_usage_counters: 462080 2361986 212335
dev.cpu.2.cx_usage: 15.54% 78.02% 6.43% last 1601us
dev.cpu.2.cx_usage_counters: 542112 2721324 224459
dev.cpu.3.cx_usage: 14.02% 79.92% 6.05% last 1014us
dev.cpu.3.cx_usage_counters: 496716 2831459 214487
dev.cpu.4.cx_usage: 16.10% 77.77% 6.11% last 5971us
dev.cpu.4.cx_usage_counters: 467800 2258637 177471
dev.cpu.5.cx_usage: 15.24% 77.76% 6.99% last 612us
dev.cpu.5.cx_usage_counters: 462824 2361157 212369
dev.cpu.6.cx_usage: 19.44% 74.33% 6.21% last 872us
dev.cpu.6.cx_usage_counters: 640034 2446184 204690
dev.cpu.7.cx_usage: 15.40% 79.04% 5.55% last 51us
dev.cpu.7.cx_usage_counters: 464464 2382488 167297
meatwad% sysctl -a | grep cx_lowest
hw.acpi.cpu.cx_lowest: C8
dev.cpu.0.cx_lowest: C8
dev.cpu.1.cx_lowest: C8
dev.cpu.2.cx_lowest: C8
dev.cpu.3.cx_lowest: C8
dev.cpu.4.cx_lowest: C8
dev.cpu.5.cx_lowest: C8
dev.cpu.6.cx_lowest: C8
dev.cpu.7.cx_lowest: C8
meatwad% 

Note this is after setting kern.eventtimer.timer=HPET as mentioned previously. If it matters, I can undo that and run these again.
Comment 4 Steve Wills freebsd_committer 2014-08-01 20:24:54 UTC
(In reply to Alexander Motin from comment #2)
> Theoretically C-states should be blocked automatically in such situation,
> but who knows... Also check whether your BIOS support C1E state, and if it
> does, try to disable it.

How would I check that? Just look for something related to C1E in BIOS configuration?
Comment 5 Alexander Motin freebsd_committer 2014-08-01 20:34:21 UTC
(In reply to Steve Wills from comment #3)
> (In reply to Alexander Motin from comment #1)
> > Line "Event timer "LAPIC" quality 400" in verbose dmesg tells that LAPIC
> > timer in your CPU (used by default) is not compatible with CPU C-states
> > (deep idle states). Can you make sure that C-states are not used on your
> > system by checking 'sysctl dev.cpu | grep cx_usage' and 'sysctl -a | grep
> > cx_lowest'?
> > 
> > Timer code was not changing for quite some time already, while C-states were
> > enabled by default less time ago, IIRC.
> 
> meatwad% sysctl dev.cpu | grep cx_usage
> dev.cpu.0.cx_usage: 15.53% 82.84% 1.61% last 149us
> dev.cpu.0.cx_usage_counters: 1509220 8046504 156453
> dev.cpu.1.cx_usage: 15.21% 77.78% 6.99% last 1803us
> dev.cpu.1.cx_usage_counters: 462080 2361986 212335
> dev.cpu.2.cx_usage: 15.54% 78.02% 6.43% last 1601us
> dev.cpu.2.cx_usage_counters: 542112 2721324 224459
> dev.cpu.3.cx_usage: 14.02% 79.92% 6.05% last 1014us
> dev.cpu.3.cx_usage_counters: 496716 2831459 214487
> dev.cpu.4.cx_usage: 16.10% 77.77% 6.11% last 5971us
> dev.cpu.4.cx_usage_counters: 467800 2258637 177471
> dev.cpu.5.cx_usage: 15.24% 77.76% 6.99% last 612us
> dev.cpu.5.cx_usage_counters: 462824 2361157 212369
> dev.cpu.6.cx_usage: 19.44% 74.33% 6.21% last 872us
> dev.cpu.6.cx_usage_counters: 640034 2446184 204690
> dev.cpu.7.cx_usage: 15.40% 79.04% 5.55% last 51us
> dev.cpu.7.cx_usage_counters: 464464 2382488 167297
> meatwad% sysctl -a | grep cx_lowest
> hw.acpi.cpu.cx_lowest: C8
> dev.cpu.0.cx_lowest: C8
> dev.cpu.1.cx_lowest: C8
> dev.cpu.2.cx_lowest: C8
> dev.cpu.3.cx_lowest: C8
> dev.cpu.4.cx_lowest: C8
> dev.cpu.5.cx_lowest: C8
> dev.cpu.6.cx_lowest: C8
> dev.cpu.7.cx_lowest: C8
> meatwad% 
> 
> Note this is after setting kern.eventtimer.timer=HPET as mentioned
> previously. If it matters, I can undo that and run these again.

That does not tell much after you enabled HPET, except that your system does have C-states. Reboot without switching to HPET and check again. If C-states block working, you should not see states deeper then C1 used even if enabled.
Comment 6 Alexander Motin freebsd_committer 2014-08-01 20:35:00 UTC
(In reply to Steve Wills from comment #4)
> (In reply to Alexander Motin from comment #2)
> > Theoretically C-states should be blocked automatically in such situation,
> > but who knows... Also check whether your BIOS support C1E state, and if it
> > does, try to disable it.
> 
> How would I check that? Just look for something related to C1E in BIOS
> configuration?

Usually there is some option mentioning C1E state somewhere under the CPU options.
Comment 7 Alexander Motin freebsd_committer 2014-08-01 20:35:40 UTC
(In reply to Alexander Motin from comment #6)
> (In reply to Steve Wills from comment #4)
> > (In reply to Alexander Motin from comment #2)
> > > Theoretically C-states should be blocked automatically in such situation,
> > > but who knows... Also check whether your BIOS support C1E state, and if it
> > > does, try to disable it.
> > 
> > How would I check that? Just look for something related to C1E in BIOS
> > configuration?
> 
> Usually there is some option mentioning C1E state somewhere under the CPU
> options.

But that is after you make sure that normal C-states are not used by FreeBSD itself.
Comment 8 Steve Wills freebsd_committer 2014-08-01 21:06:13 UTC
(In reply to Alexander Motin from comment #5)
> That does not tell much after you enabled HPET, except that your system does
> have C-states. Reboot without switching to HPET and check again. If C-states
> block working, you should not see states deeper then C1 used even if enabled.

Ok, rebooted without setting HPET:

meatwad% sysctl kern.timecounter.hardware kern.eventtimer.timer
kern.timecounter.hardware: TSC-low
kern.eventtimer.timer: LAPIC
meatwad% sysctl dev.cpu | grep cx_usage
dev.cpu.0.cx_usage: 22.28% 77.71% 0.00% last 1359us
dev.cpu.0.cx_usage_counters: 17048 59466 0
dev.cpu.1.cx_usage: 21.54% 78.45% 0.00% last 939us
dev.cpu.1.cx_usage_counters: 17294 62983 0
dev.cpu.2.cx_usage: 31.10% 68.89% 0.00% last 1146us
dev.cpu.2.cx_usage_counters: 33021 73155 0
dev.cpu.3.cx_usage: 21.15% 78.84% 0.00% last 738us
dev.cpu.3.cx_usage_counters: 18109 67474 0
dev.cpu.4.cx_usage: 21.94% 78.05% 0.00% last 447us
dev.cpu.4.cx_usage_counters: 18022 64084 0
dev.cpu.5.cx_usage: 22.70% 77.29% 0.00% last 682us
dev.cpu.5.cx_usage_counters: 18103 61637 0
dev.cpu.6.cx_usage: 26.53% 73.46% 0.00% last 430us
dev.cpu.6.cx_usage_counters: 33056 91541 0
dev.cpu.7.cx_usage: 22.60% 77.39% 0.00% last 6us
dev.cpu.7.cx_usage_counters: 16724 57250 0
meatwad% sysctl -a | grep cx_lowest
hw.acpi.cpu.cx_lowest: C8
dev.cpu.0.cx_lowest: C8
dev.cpu.1.cx_lowest: C8
dev.cpu.2.cx_lowest: C8
dev.cpu.3.cx_lowest: C8
dev.cpu.4.cx_lowest: C8
dev.cpu.5.cx_lowest: C8
dev.cpu.6.cx_lowest: C8
dev.cpu.7.cx_lowest: C8
meatwad%
Comment 9 Alexander Motin freebsd_committer 2014-08-01 21:50:42 UTC
(In reply to Steve Wills from comment #8)
> (In reply to Alexander Motin from comment #5)
> > That does not tell much after you enabled HPET, except that your system does
> > have C-states. Reboot without switching to HPET and check again. If C-states
> > block working, you should not see states deeper then C1 used even if enabled.
> 
> Ok, rebooted without setting HPET:
> 
> meatwad% sysctl kern.timecounter.hardware kern.eventtimer.timer
> kern.timecounter.hardware: TSC-low
> kern.eventtimer.timer: LAPIC
> meatwad% sysctl dev.cpu | grep cx_usage
> dev.cpu.0.cx_usage: 22.28% 77.71% 0.00% last 1359us
> dev.cpu.0.cx_usage_counters: 17048 59466 0
> dev.cpu.1.cx_usage: 21.54% 78.45% 0.00% last 939us
> dev.cpu.1.cx_usage_counters: 17294 62983 0
> dev.cpu.2.cx_usage: 31.10% 68.89% 0.00% last 1146us
> dev.cpu.2.cx_usage_counters: 33021 73155 0
> dev.cpu.3.cx_usage: 21.15% 78.84% 0.00% last 738us
> dev.cpu.3.cx_usage_counters: 18109 67474 0
> dev.cpu.4.cx_usage: 21.94% 78.05% 0.00% last 447us
> dev.cpu.4.cx_usage_counters: 18022 64084 0
> dev.cpu.5.cx_usage: 22.70% 77.29% 0.00% last 682us
> dev.cpu.5.cx_usage_counters: 18103 61637 0
> dev.cpu.6.cx_usage: 26.53% 73.46% 0.00% last 430us
> dev.cpu.6.cx_usage_counters: 33056 91541 0
> dev.cpu.7.cx_usage: 22.60% 77.39% 0.00% last 6us
> dev.cpu.7.cx_usage_counters: 16724 57250 0
> meatwad% sysctl -a | grep cx_lowest
> hw.acpi.cpu.cx_lowest: C8
> dev.cpu.0.cx_lowest: C8
> dev.cpu.1.cx_lowest: C8
> dev.cpu.2.cx_lowest: C8
> dev.cpu.3.cx_lowest: C8
> dev.cpu.4.cx_lowest: C8
> dev.cpu.5.cx_lowest: C8
> dev.cpu.6.cx_lowest: C8
> dev.cpu.7.cx_lowest: C8
> meatwad%

Hmm. It looks like the deepest C-state was actually disabled automatically. Then it is surprising to have any problem with LAPIC. But, just to be sure that BIOS does not fool us, could you disable all C-states completely with something like this in rc.conf:
performance_cx_lowest="C1"
economy_cx_lowest="C1"
, and if it won't help, also disable C1E in BIOS.
Comment 10 Steve Wills freebsd_committer 2014-08-02 00:55:16 UTC
(In reply to Alexander Motin from comment #9)
> 
> Hmm. It looks like the deepest C-state was actually disabled automatically.
> Then it is surprising to have any problem with LAPIC. But, just to be sure
> that BIOS does not fool us, could you disable all C-states completely with
> something like this in rc.conf:
> performance_cx_lowest="C1"
> economy_cx_lowest="C1"
> , and if it won't help, also disable C1E in BIOS.

Setting performance_cx_lowest="C1" and economy_cx_lowest="C1" in rc.conf did seem to help.
Comment 11 Alexander Motin freebsd_committer 2014-08-02 07:12:17 UTC
(In reply to Steve Wills from comment #10)
> Setting performance_cx_lowest="C1" and economy_cx_lowest="C1" in rc.conf did
> seem to help.

Then I guess your ACPI BIOS lies to OS about C2 state semantics -- it should not affect LAPIC timer, but seems somehow does.
Comment 12 Gavin Atkinson freebsd_committer freebsd_triage 2014-08-03 16:09:43 UTC
Can you attach a verbose dmesg please?  Can you also try the patch attached to bug 192316 and see if that makes any difference?