After process exits, kernel memory isn't released in some cases and that results failures on new process creations. The problem seems to be also hardware dependent as I only see this happening on only one of three machines I tested.
This started happening since 10.0-RELEASE and see it on 10.1-RELEASE as well. 11-CURRENT does not have this issue.
When the problem occurs, I see one of these 2 errors printed:
pmap_mapdev: Couldn't alloc kernel virtual memory
vm_thread_new: kstack allocation failed
One of the easiest way to reproduce the problem is by benchmarks/forkbomb.
% forkbomb -f
https://svnweb.freebsd.org/base?view=revision&revision=272667 seemed somewhat similar problem but code base is quite different on 10-stalbe and 11-current.
I have this problem NOW on 10.1-RELEASE.
I am using memtest to test the RAM. It seems to me that I had this problem a long time ago, and there was bad RAM, and replacing that made the problem go away.
That was a different machine from this one. I've made 2 passes of memtest without errors, however I'll let it run for a few more days to try to duplicate the temperature extremes.
Anyhow, I have a program that I wrote that copies a lot of files to multiple disks, like 17,000 files. I ran it several days ago, and system crashes with the error in the original poster's post.
I ran the program yesterday, and it ran to completion, no problem. Could be a temperature thing + RAM.
I also put in some diagnostics on each loop of the program:
char *delete = malloc(100);
fprintf(stdout, "%p\n", delete);
system("sysctl vm.kvm_free vm.kvm_size");
The loop enters for each directory on the disk, not for each file, so there might be 900 loop entries.
Now the vm.kvm_free and vm.kvm_size remain mostly constant.
BUT the pointer to the delete is malloc()ed at 1000 (hex?) bytes greater each time the loop enters !!!
I don't do any malloc()s after main(). This is probably not a reliable way to test things, but it looks suspicious. !!!
Is there a better way to get the amount of free memory at any instant, probably another sysctl parameter.
Anyhow this is just very frustrating and hard to track down.
I had another issue that never got sorted out after years, on 9.1: