Java, is the JVM responsible for: OS Unable to fork: Cannot allocate memory ?

CREDITS: I would like to thank my old friend and now again coworker Josu Feijoo for his help on getting to the conclusions below.

This error means that no more processes or threads can be open due to memory starvation. Most likely a hard reset will be needed so certainly it is not a nice problem to have.

Clearly its resolution depends on what you are running but most likely there are too many threads been run in your system so the first step for troubleshooting would be to get all threads and identify the culprit:

$ top -H -b -n 1 

After you identify the culprit (tomcat in our case) take a look at how threads are going up indicating a leak:

$ top -H -b -n 1 | grep tomcat -c

In Java the JVM memory requirements are the sum of maximum heap memory, the maximum "perm space" memory and the number of threads multiplied by the thread stack size:

Xmx + MaxPermSize + (Xss * number of threads)

That is the reason Virtual memory consumption is so important. I see a lot of miss-leading comments and posts on the web claiming that you should only worry about resident memory (top RES) and not much for virtual memory (top VIRT) and under those assumptions some engineers might think the below is simply OK. Note how this JVM is using 2.3GB physical memory but 18GB virtual memory:

$ ps -e -o rss,vsize,cmd | grep tomcat
2298588 18019720 /opt/jdk/bin/java -Xmx2048m -XX:MaxPermSize=512m

This should not be OK for most web applications. Unless on purpose you need to maintain a lot of threads you should be suspecting of thread leaking. At that point you need to use jstack, jvisualvm or any other tool that allows you to connect to the debug port of the JVM to further troubleshoot which is the responsible leaker in the Java code. Or perhaps you can quickly inspect your code for threads you open and simply do not stop. You might be using some library or middleware responsible for it like the Camel ProducerTemplate. Housekeeping is necessary, resources are not infinite and those of us ignoring that rule will pay with a resource starvation surprise. stack

Needless to say that if you really need to spawn so many threads then do not forget to run the JVM with enough memory that accounts also for the thread stack consumption.