Lab: JVM Production Debugging 101

Editor's Notes

• #3: Picture source: CSI Las Vegas (http://flowtv.org/wp-content/uploads/2007/11/csi3.jpg)
• #7: Image source: http://www.about-larnaca.info/2012/06/thief-is-caught-red-handed-in-kiti.html
• #8: Invite discussion. Ask audience to point out different data that is (a) useful and (b) readily accessible. Limit to 3 minutes.Image source: http://lets-rap.com/wp-content/uploads/2011/05/house-md-d-house-md-1048019_1152_864.jpg (copyright Fox)
• #9: Expound a bit on anything that hasn’t been raised in the earlier discussion. Limit to 2 minutes, less if possible.
• #14: “All this and more” sales pitch. Mention the profiler.
• #16: Actual scenario details: pathological regular expression in a service (http://swtch.com/~rsc/regexp/regexp1.html).Exhibited behavior: very high single-core CPU utilization. Little or no GC activity. Possible StackOverflow if left long enoughAnalysis: stack trace will exhibit very deep, repetitive call stack to java.util.* classes. Package and class names will indicate a regex issue.Bonus points to whomever recognizes the pathological regex scenario without looking at the code, beyond the “problematic Regex” axiom.Workaround: depends on the details, in some cases the offensive input can be deleted or routed to a dead letter queue. In this case, none – this has to be resolved at the algorithmic level.Image source: http://blog.rogersbroadcasting.com/billhart/2010/01/19/tuesday-january-19th-2010-dialing-miscue-avatar-can-kill-you-look-where-kellie-flew/
• #17: Actual scenario details: GC storm as a result of exponentially-growing HashSet. Adding small elements to a java.collection.HashSet (or HashMap) rapidly without specifying an appropriate capacity is a recipe for disaster.Exhibited behavior: ~100% single-core CPU utilization. Very high GC activity – eden generation fills up rapidly, overflowing to old gen with FGCs in increasing frequency. Eventual OutOfMemoryException.Analysis: jstat –gcutil or tracking the graphs in VisualVM clearly exhibit high GC pressure and inability to clear up enough RAM on each GC cycle. Stack trace is hit-or-miss (may exhibit a thread very clearly working on HashMap.resize, but as likely to point at code generating the strings added to the HashSet). Heap dump will likely exhibit a HashSet with very high load factor and an appropriately high count of items in the map.Bonus points to whomever recognizes the exponential expansion scenario without looking at the code.Workaround: restart the service, and possibly set up a cron task to restart periodically. VM flags that watchdog OOM situations are useless because OOME fires way too late, if at all.Image source: http://pathogenomics.bham.ac.uk/blog/wp-content/uploads/You-Cant-Handle-the-Truth.jpg (A Few Good Men)
• #18: Actual scenario details: Memory leak as a result of saving substrings from web calls. Substring actually references the original string, or in practical terms the resulting HTML from each website is kept in active memory until a large-enough allocation fires OOME. GC isn’t immediately overwhelmed because it mostly has large, long-lived objects, so once memory is defragmented (a single FGC cycle) it just doesn’t have much it can do about the memory pressure.Exhibited behavior: Service process dead. OOME clearly marked in the logs.Analysis: Save logs aside; restart application to collect more information. jstat –gcutil output and a pre-failure thread dump should be saved aside as a matter of good course, but are not useful in the analysis of this scenario. Either saving pre-failure heap dumps or turning on the -XX:+HeapDumpOnOutOfMemoryError JVM option is necessary to resolve this problem.Bonus points for legitimate scenario suggestions at this point, but cut it off very quickly as a good lesson in “not jumping ahead of our data.”Workaround: restart the service, and possibly set up a cron task to restart periodically. VM flags that watchdog OOM situations can help.
• #19: 5-10 minutesImage source: http://www.penny-arcade.com/comic/2002/02/22
• #20: Image source: http://www.forensicinnovations.com/blog/wp-includes/images/mobilekit.jpg
• #23: Image source: http://etc.usf.edu/clipart/28300/28384/brain_28384.htmH. Newell Martin, The Human Body (New York: Henry Holt and Company, 1917) 145
• #24: Image source: http://www.ign.com/articles/2008/03/28/guitar-hero-aerosmith-first-look?page=2 (Guitar Hero: Aerosmith @ IGN)
• #25: Follow through on evidence in the form of discussion (who used…? What did you find?) – don’t discount useful alternative theories!Forensic analysis: read through stack trace. Identify likely culprit. Bonus points to whomever yells “pathological regex” first.
• #26: Heap dump actually has useful data (hashmap capacity, size and load factor) but it’s not likely anyone will notice that.Forensics: jstat –gcutil clearly shows GC storm. Stack trace to figure out the where, and common sense to figure out the why.
• #27: Forensics: Analyzing the heap dump will clearly evidence an abundance of suspiciously-large strings; drilling into a couple of these will let us see that the strings kept in the heap are in fact full-blown HTML responses.Bonus points to whomever recognizes the substring scenario without looking at the code (nBA employees don’t count ).Evidently this behavior was changed in Java 7u6: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4513622