JVM Heap Dump Analysis - OpenJPA memory leak

hello my name is Pierre Charbonneau I'm a senior IT consultant and the other on July support patterns at blogspot.com today's deal will basically show you how to perform a GVM heap dumb analysis and it will precisely show you are actually how we're able to pinpoint the root cause of an apache open GPU memory leak if you are interested the full article is also available from the blog link that you will see also from this video so now let's see how this whole analysis process is working so what you see here is basically the result of the heap dominologist loading process from the IBM support assistant workbench we're actually using memory analyzer - from that workbench but you could also be using the standalone version of math as well so what we did basically was to generate JVM head down following the discovery of a memory leak from our production environment and we used a GM opportunity to do that and it did generate a heap file at about 1.5 gig that's why you see this file heap that VI N and then we loaded the heap dump file from the IBM area diagnostic tool which pretty much the same version as map so the first thing I will notice is a summary of your heap done right because the heap dump is basically a snapshot of your Java heap at a specific time so it's giving you detail of all the active objects at the time of the capture so let's look at our heap dump so as you can see this the size of the heap dump is 1.5 you got a Java heap itself because we have a Java heap setup at - gate and the time of the problem why the capture we had an old gen space at the ball 1.5 gig Plus on the object from the young gent space so that's why you see about 1.5 gig of footprint so mat will initially show you a summary such as the size number of classes objects right class order as well and then typically the first thing we want to do is go to the league saw spec report okay that report is pretty useful that's what will generate right now is basically going to show you a summary of the top contributors of their memory footprint so let's see what we got you see so from the 1.5 gig we had two problems a spec or leak suspect that Matt was able to identify okay one with a footprint as you can see around this and around this about six hundred Meg's for this one and two forty three Meg's for the other one then if you scroll down you will see from the analysis that we did from our production environment what we found is the problem suspect number one as you can see we had about 600 Meg's of footprint or almost 40% of the entire Java Java heap or do jet space accumulated in one instance only of a link list so that one is quite suspicious right 600 Meg only one instance of a linked list right so since why the leaks aspect is good at to highlight UD stub contributors now how can you deep dive further into an analysis well look the refresh saying here in the application class holder right in the event you're familiar with class solderless you have multiple Java application you will have their own class loader so that's why you will see that like this because to pick your class or loaded to a class loader so man will be able to detect the referencing classes to a given class holder so here's the memory address of this objects we're going to do is basically copy-paste this address and there's a function in the tool called fine object by address as you can see here click on this you basically put the address of the class loader you see and it is going to show you here's an another time and then your you will be able to drill down into the classes and referencing and everything so in this case as you can see this class loader add this a memory address is referencing about 600 Meg's of heat right which was basically was the leak suspect so let's drill down just outgoing references they were looking at or not outgoing not incoming outgoing means or basically you're drilling down from one object to its reference references like let's see that object has a field or a vector or let's say a hashmap of object right sure so data outgoing allows you to drill down right so outgoing drill down and the object dependencies incoming is when you want to go back in the tree and see okay this object who is the reference or of this guy right so right now we're going to look at the outgoing dependencies so we can identify the culprit alarms record the leaking objects so first you want to filter by the top right so as you can see the class or obviously will have some structure of classes as you can see here now you don't need to expand that and look let's see what we found here you see so that was kind of a surprising finding when it is now is this so we can see a mix of WebLogic object in there because we're using a Horikawa WebLogic for our application it's a porthole application what we found is one instance of PC it's called PC registry you see from comm Apache open GPA right remember the title of this whole numeric analysis GP which stands for Java persistence API well that's what we use in our application to do the database transaction I will use that open jpg for the mapping in database mapping so we use that framework to do that type of sequel queries against our database so it was quite surprising when we found one instance of PC version said forget since almost using six on under a mega memory so using magic and again continue to drill down and what we found is this you see this this class in open PG has a field called listeners it's pretty much linkless you see so now we get a good correlation right from the report link list as you can see now we're about to find that linked list right and it linked list is basically when you drill down further you'll be able to see next next next right so at some point what we're able to see I'm not going to show you all the code because it's proprietary but at this point we were able to drill down and to the dependencies and basically what this structure is is keeping basically all the detail or cache of all your queries I made a data definition on it especially if you're from here with open PGA but the data is static in nature right like the meta data is static in nature so why are we leaking meta data right why is this data structure growing over time so the heap dominance did allow us to pinpoint that the actual leak or the primary leak is is due to that data structure accumulating in the PC registry from the Apache open PG work so that's what we're about to achieve with the actual healer - as you can see very powerful right it's all you - and then assuming it's the application code you'll be able to pinpoint exactly or what type of object right and then but sometimes hume Damasus is not enough by itself to pinpoint the ultimate root cause in this case what it's telling us is that we know which object is leaking that was the primary objective now question is why is our application or framework leaking at the first place right why did he know and this one so they give you all the details will give you the object type but then you will need to do with your own due diligence to pinpoint the root cause so let me show you what we did next so after we found the leak in that structure since open PJ is an open source framework we had to look at the source you see what you see is the open source is the source code of this class PC registry of that version and here's the field you see register class listener that basically Phil is used to reduce your class listeners when using PG right so you want to create entity entity manager instance you first have to initialize a factory and then create entity manager to run your business sequel right well this class is keeping a static cache of such listeners listeners you see that feel and you see that's the one you see Matt is quite good at showing you exactly the field name so when you have access to the source you can just trade correlation this is very powerful right now why is this data structure leaking well there is a remove register obviously as you can see here to unregister so what we did next was to review the documentation you see that's what the he done won't give you sir by removing the documentation what we realize is that the when we're using the factory basically the entity manager factory to create instances our application code was actually not calling the close method see if you're familiar with a GP specification you should know that when you create a factory instance unless you're caching this in a static manner we have singleton pattern you need to call the close method right the close method is going to dispose these object metadata otherwise you'll be leaking this data structure which is exactly what the heap dump is showing us we're leaking the data structure so after we found this we look at the code this code this code then we went back to the application code and then we realize that our application was creating new instances of the factory entity manager factory over and over and over without invoking the close method which is responsible to remove the listener object right from this data structure so that's the root cause the solution was to basically create a single item or static instance of the factory ensuring that we get only one copy of it right as opposed to leaking that data so that's how we implemented and after we did that we saw pretty major improvement right no no longer any leak and no longer footprint even we took a hit dump after the defects and we notice not that data structure lessening was like down to even like one Meg or two right so and was not growing over time so that's the root cause and and I hope you did appreciate right because you can see that investigation it did not take that much time once you're getting familiar with the to leak suspect some of the concept like outgoing and coming references you will see it very powerful it allows you to deep dive into the object obviously you need to know your framework code you need to know your application code in others so for you to draw some correlation but the tool by itself is very powerful to allow you to the type of profiling and memory footprint analysis to eventually pinpoint the root cause of your problem so I hope you did appreciate that tutorial regarding the mat and GPA memory leak I will get back don't worry with a lot more videos and he dumped had I have a lot more tutorials on the same and I'll also be providing some tutor on how to investigate me moral econ a Java heap side as well so I hope you appreciate this video and have a good day thank you

Channel: Pierre-Hugues Charbonneau

Views: 68,175

Keywords: Apache, JPA, JVM, Heap Dump, Java (programming Language), Hosting, Software, Technology, Computer, Tutorial, Web Application Monitoring, Network Monitoring, Java Monitoring

Id: 5joejuE2rEM

Length: 12min 26sec (746 seconds)

Published: Wed Mar 13 2013