Bug 20393

rwatson asked me to send a pr for this, so I am trying to help out.

Processes hang the system by getting stuck in vmwait instead of nanslp'ing like they should.  If you have MAXUSERS set to over 128ish and try to run at least 2000 processes, some will get stuck in vmwait causing the system to become almost unresponsive until they decide to exit.  If the number of processes wanting to run is much larger than 2000, the cycle continues.  console switching works, typing at a login prompt to log in yeilds no characters on the screen.  Any processes running on other terminals (including a rtptio 0 top) freeze.  ps in DDB shows a small percent of the children in vmwait instead of whatever they should be doing.

Fix: 

Let me know if I can help out further with the issue.  I have time to test things but I do not know C let alone the kernel.  I can provide a scratchbox to demonstrate the problem and any level of access to it if it would be helpful in resolving the issue.
How-To-Repeat: compile kernel on 4.1 with maxusers=200
run a progam(http://www.looksharp.net/~user1/test.c) which tries to fork x children which do the following:
  print starting time, sleep 20 seconds, print ending time, exit.
Note that if x > some number z around 1500, processes stop forking way before maxproc in shell or kernel is reached, and processes hang as described.  After a period of time several times longer than 20 seconds, some processes exit and more start.  If x > ~2000, the cycle repeats several times.   
Problem 2: repeat above with maxusers = 600 (> ~512). 
kernel panics from running out of kernel memory instead of the freezing behavior.  (one person I showed this part of the issue to, after inspecting sourcecode, was unsure if it should panic or wait for free mem in fork1 here:  MALLOC(p2->p_cred, struct pcred *, sizeof(struct pcred), \n       M_SUBPROC, M_WAITOK);