Java Tuning - Software Development, Java, and some more.
Posts tagged OOME
ConcurrentHashMap fat memory footprint
Mar 23rd
While running product sizing tests, we’ve found that an over enthusiastic usage of ConcurrentHashMap (CHM) had evaporated a good ~170MB of much needed heap space (we ran with a 1.5GB heap).
As it turns out, a empty CHM weighs around 1700B. Yes, I’m talking about a map with no entries at all, just the plumbing!
We used a CHM to store user session attributes, having 100,000 user sessions generated 100K CHM instances worth 170MB of heap (100K times 1.7KB).
We took measurements using the super Eclipse MAT.
The obvious solution for saving these scares 170MB, was to switch from a CHM to a Hashtable. A Hashtable cost only around 150B per instance (8% of a CHM).
Other possible solutions could have been: moving to a list structure (seek time is not an issue as we rarely have more than 4-5 attributes per session), or resorting to a an array of Objects.
Change implications:
1. Performance - The product doesn’t have any user scenario that cause multiple threads to concurrently access the same session attributes map, so we don’t expect any performance loss, on the contrary, I’m expecting a hashtable to prove faster for single thread access, over a CHM.
2. Thread safety is a low risk aspect, as both CHM and HT provide the same basic guarantees for a single API operation (e.g., map.get(key)).
To conclude, a CHM is a good idea when you have a shared map structure suffering from a high R/W thread access contention. But dragging behind itself such a large memory footprint, CHM is not ideal to use in masses, or when concurrency performance is not the focus.
P.S
A CHM automatically allocates 16 segments, each with a 16-element array – one best practice is to measure the average map population during your product’s sizing tests, and initialize the CHM with the minimum initialCapcity and loadFactor, required to contain your usage.
Why catch Throwable is evil – A real life story
Feb 28th
Disclaimer: Now I know that this is an old idiom, I’m just presenting my own real life incident taken straight away from the bloody Java trenches.
Exceptions can be threads assassins
when running on top of Websphere thread pool, any Runtime exception that isn’t caught by the applicative code, will bubble up in the stack, ending up killing the specific thread. WAS helps here, by automatically creating a new thread that will take the place of the murdered one, but still, killing and immediately creating a thread is everything but the thread pool rational.
Hiring a thread bodyguard
A simple way to avoid thread death is wrapping the first applicative layer (e.g., Run() method) with a try block that catches and swallows any Exception that’s thrown from anywhere in the application code.
Our project’s code also used this concept, but instead of catch (Exception e), it had a catch (Throwable t), When I noticed that I didn’t rushed to fix it, just in case someone before me had done funky stuff with dynamic class loading that might throw ClassNotFoundError (although this should be caught at a very localized resolution), or maybe it’s there for some other historical reason that not being one the code’s forefathers I’m just not aware of. In any case, I did promise myself that I’ll revisit this piece of code in the future.
Getting some bulls to do correct things
today I finally got the excuse I needed in order to change the catch Throwable in a catch Exception:
We were running stress tests, when the server had an OOME (out of memory error). Since the catch Throwable caught and swallowed the OOME (as OOME is a subclass of Error which is a subclass of Throwable), the thread that generated the OMME kept on living, instead of dieing right there, and so, the JVM continued running, crippled and limping, instead of turning to an honorable solution like hara-kiri. Choosing the quick death route would have been rewarded with a quick resurrection to be provided by the gracious NodeAgent and its watchdog mechanism, and the end result would have been a newly born healthy server ready to get back in business. A retreat in order to attack, you might put it.
Instead, the server had to limp for long minutes, suffering from a series of consecutive strokes (OOME), until the OOME was so bad that the JVM just had to exit.
Conclusions
The Catch Throwable was causing down time, by preventing an imminent restart of the JVM due to an OOME.
Open Questions
- I know that an uncaught exception kills only the specific thread does the JVM treats an error differently? Put other words, if the OOME is not caught, will the entire JVM die or only the specific thread? I assume that the answer is the entire JVM, maybe this is implemented by the JVM itself, or maybe it’s implemented somewhere in the WAS bedrock. If for some reason it’s not the case, one could catch an Error and then execute System.exit(1); in order to hasten the process imminent death.
Via rss
Via e-mail
