Recovering from unexpected JVM OutOfMemory with -XX:OnOutOfMemoryError

You should never have an OutOfMemoryError in your project but usually problems happen in production and it takes a while to solve them. There is where the -XX:OnOutOfMemoryError JVM Option comes handy. It just works supplying a script/command you want to run in the case the JVM reports a java.lang.OnOutOfMemoryError. Here is how to provide the flag, how to build the script that will put Tomcat down and how to test it.

1. Generate a scaffold war project. I named it oom-app (The OutOfMemory App):

   mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DarchetypeArtifactId=maven-archetype-webapp -DgroupId=com.nestorurquiza.test -DartifactId=oom-app -Dversion=1.0.0-SNAPSHOT -Dpackage=com.nestorurquiza.test
   

2. Edit web.xml

   $ cd oom-app/
   $ vi src/main/webapp/WEB-INF/web.xml 
   
   
     Archetype Created Web Application
     
           com.nestorurquiza.test.OomListener
     
   
   

3. Create the listener that will cause the OutOfmemory error:

   $ mkdir -p src/main/java/com/nestorurquiza/test
   $ vi src/main/java/com/nestorurquiza/test/OomListener.java
   package com.nestorurquiza.test;
   
   import javax.servlet.ServletContextEvent;
   import javax.servlet.ServletContextListener;
   
   
   public class OomListener implements
           ServletContextListener {
       public void contextInitialized(ServletContextEvent event) {
           System.out.println("******************Originating an OutOfMemoryError Exception*************************");
     StringBuilder sb = new StringBuilder();
     while(true) {
              sb.append(this.toString());
           }
   
       }
   
       public void contextDestroyed(ServletContextEvent event) {
           System.out.println("Will I ever run?");
       }
   }
   

4. Configure the JVM to run a script once the OutOfMemoryerror is gotten. For tomcat you do that in setenv.sh:

   $ vi /opt/tomcat/bin/setenv.sh
   ...
   JAVA_OPTS="$JAVA_OPTS -XX:OnOutOfMemoryError=/usr/sbin/killTomcat"
   ...
   

5. Make sure you have the script ready. Note we are just killing tomcat. In production you are supposed to automatically restart any service when it dies. There are multiple solutions for that but allow me to recommend "Monit" for Unix/Linux servers.

   $ sudo vi /usr/sbin/killTomcat
   ps -ef|grep tomcat|grep -v grep|awk '{print $2}'|xargs kill -9
   $ sudo chmod +x /usr/sbin/killTomcat
   

6. Restart tomcat.
7. Build and deploy (Using a simple tomcat webapps deployment):

   $ mvn clean install
   $ cp target/oom-app.war /opt/tomcat/webapps/
   

8. Once the war is deployed you will get an output like:

   ******************Originating an OutOfMemoryError Exception*************************
   java.lang.OutOfMemoryError: Java heap space
   #
   # java.lang.OutOfMemoryError: Java heap space
   # -XX:OnOutOfMemoryError="/usr/sbin/killTomcat"
   #   Executing "/usr/sbin/killTomcat"...
   Killed