Bug 202409 - vm_thread_new: kstack allocation failed on FreeBSD 11.2
Summary: vm_thread_new: kstack allocation failed on FreeBSD 11.2
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-08-18 06:36 UTC by ota
Modified: 2018-10-12 22:56 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description ota 2015-08-18 06:36:40 UTC
After creating some processes/threads, I encounter following to errors and the are printed to dmesg.

pmap_mapdev: Couldn't alloc kernel virtual memory
vm_thread_new: kstack allocation failed

System becomes unstable thereafter most of times although it depends on what failed to allocate memory.

This happens on any 10.x releases but not 11 current.

Easiest way to reproduce is use benchmarks/forkbomb

% forkbomb -f

brings the above errors in few seconds.

I recall a special fix applied to 9.x stable a while ago but the change didn't seem to apply to 10.x releases.
Comment 1 Mikhail Teterin freebsd_committer 2018-10-11 22:35:14 UTC
I'm having this problem now, with 11.2-STABLE r338668. My machine is, actually, a guest VM. Memory is 8GB, but the OS i386 using PAE.

The role of forkbomb in my case is played by ansible -- which forks off multiple processes for each of the hosts in the inventory.

Once the message starts appearing, the only "fix" is to reboot -- the problem remains even after ansible exits. The relevant dmesg is:

Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
[...]

There is no swap configured, but top reports ample free RAM:

Mem: 536M Active, 828M Inact, 141M Wired, 60M Buf, 6529M Free
Comment 2 ota 2018-10-12 02:06:19 UTC
(In reply to Mikhail Teterin from comment #1)

How much kmem_size do you set?

My machine has 2GB and I had vm.kmem_size originally had set 768MB for the purpose of ZFS.

I looked into the cases and tried to find how to recover.  I noticed that after seeing the error message, resources aren't recovered after process ends.  However, I couldn't figure out why resources are left leaked.

After few months of watching, I suspected too much kernel memory allocation and reduced it down to 512MB.  Since then, system became more stable and I stopped seeing this error happening.

#vm.kmem_size="768m" # bad
vm.kmem_size="512m" # okay

It looks over allocation made system unstable.
Comment 3 Mikhail Teterin freebsd_committer 2018-10-12 11:38:16 UTC
(In reply to ota from comment #2)
I have not tried to make any adjustments to vm-settings by hand -- leaving it all to be auto-calculated.

I suspect, the problem is due to some miscalulations, which only affects the relatively rare PAE-cases.

And, as you point out as well, something else is wrong if the problem remains even after the "offending" processes exit...
Comment 4 Mikhail Teterin freebsd_committer 2018-10-12 22:55:43 UTC
FWIW, still a problem with 11.2-STABLE r339319...