Thread Dump Analysis Generation Techniques - Part 1

hi everyone my name is phi1 Oh so today's video will start with the generation techniques for the tread them analysis so if you have not done already a success that you first go to our first video with which are covering the fundamentals entre Dom analysis so let's have a look at the agenda for today so essentially we're going to learn how to generate tread them using our echo WebLogic 12c so that's our scope for today the future video will include the other technologies like Apache Tomcat and also the IBM WebSphere and J bus as well so technology stack that we're going to use today so essentially the lab specification are going to use Java hotspot 1.8 just a century service spec 60 64-bit or accuser of course Oracle WebLogic 12c 12.1 3 and also the Ubuntu 1504 Linux kernel 3.19 ok so this is what we're going to use in technology for our lamb the tools were going to use for TRADOC generation essentially for tools we'll see today the OS native command of course which is kill - tree with the java PID the GDK j stack utility which i found very useful especially on windows environment java visual vm of course which is one of my preferred tool for typical troubleshooting there on a java technology and of course the oracle logic admin console which you will see can be useful for certain scenarios ok so let's get started alright so as you can see this is a word lab setup here so we're using the ubuntu of course as I mentioned 15.0 for 64-bit so to save some time I did of course pre pre stored most of the processes here so we're going to focus on the trade damage generation techniques here okay so so what you see is actually I have a WebLogic server running as I mentioned I'm using the WebLogic 12c so I did essentially create a standalone or Blahnik 12 C Java process so and this is a typical environment of course this is the dista of ubuntu so kind of allowing to do a couple of the their desktop and of course getting access to terminal Linux kernel there so so let's say you're supporting a job aids and environments and so essentially the the typical thing you want to get is the Java process right PID process which is a typical thing you are doing to generate the tread dump so typical commander uses the PS grab Java as I just did it's going to list you all the Java processes running on your server now the one that we need here is the one your when your WebLogic environment I said depending of your setup at WebLogic of course you may have one more than one instances right let's in your production environment so essentially in our case the PID is to 683 so that's the first technique to generate read them so you need to the PID of the Java process and the command to use a scale minus three and then you use the PID of the Java process right in kill - tree of course don't do kill - 9 obviously right otherwise you'll be killing a process so kill manage the tree is a native command essentially it's sending a signal and you could see sending a signal of the process which will initiate the hotspot JVM - Jarrett read them now when using that technique it will generate the output the standard output log so depending on the technology in this case we're using web logic it's going to print it to the standard output log of the server itself right so we can see here write the actual trade down put all the detail about the treads of course as we saw in the fundamentals and of course at the very end you get some the garbage collection treads and the heat breakdown now this technique is of course the most common one use right by most production support individuals of course it's very useful it's also the most responsive right so even if your servers and is in big trouble you may have sometimes some issues using other - like other tools like Java shah oviya many things sometimes when the the instance is running close to all the memory and other issues sometimes you may have you may struggle just attach to the process by itself so kill - tree at some point it's I found still very quite responsive there even if your JVM isn't trouble it will just generate the Treader most of the time there okay so now the limitation with the technique of course is it will generate the data from that log right so of course innings it means yet you gotta extract the data from the log itself and be able to read it like as we can see here so of course if you start generating many many snapshotting can become a bit a lot of noise in the log right so what I would recommend of course make sure that you're splitting the log in a regular basis so we can regularly extract read them easily from from the data right that's kind of a limitation of the technique here okay so kill - tree very basic as I mentioned of course you can run this command multiple times you see generate just data it's it's very fast it's non intrusive for the JVM it's it just takes a few milliseconds of generation so the technique number two so let's say you wanna have the data straight from that view right and you may not want you to get a tread Dom directly here so the other technique that I use is as I mentioned was the G stack utility right so G stack is different right it will do a remote attached to the PID process it's going to print the tread dam here at that location right so we're going to use a G stack if you use a J stock command from the version that you won't write your news if explicitly specify which G dkj stack you want to use now keep in mind Jay stack it's part of JDK itself not the GRE okay so you need to go under the GDK bin folder and then you can put the PID even more - to get the data and as you can see that the G stack is very useful because it's going to end attach and then going to generate the full tread um in your console as opposed to negative command will will print it in the log that G stack will free to directly in the in the actual console that you want so now that I've found something very useful right so let's say I want to do a trade um live or if you're dealing with a very large file on the server something can be very useful just for G stack and you can see the beginning to trade them here in our environment Department doesn't have too much traffic right now of course so you'll see the basic treads our Blodgett only for now so we see just you know those are the treads from the socket marks are right or essentially way for incoming requests or they're not doing anything right now so he sees very useful right you can have a look quickly at a trader from G stack in that date view and it's again it's it's very fast as well plus J stock and can print some native frames as well so if you need to go to that level but most of the time you don't need to go there you just need to stick at the Java level of the stack trace okay so we can definitely see the big tree down here and again it's very fast you can done that pretty non-intrusive we just did the one right so and you will see in the future session and we start to generate a load and simulate problems or we'll go back to these techniques of course and you'll be able to compare and and decide which one you would prefer to use but again as I said I would recommend that you learn each of them because depending on the scenario you may have to fall back on one or another so there's not in certainly one best way right so it's always a depending on certain SS scenarios okay so very basic so for today div command now like I mentioned these are the two main techniques now the if you're using Java visual VM so that was a swing scope our session right so here our Java VM running now of course I have the distal version right as I mention I have a bone to so in this case it did auto detect locally right from the Linux kernel it did you take my robotic process running so jovi very very useful for that spur see if you have running in there on Windows but even you can do remote attach as well you don't need a distal read of course I even my site I work on a lot of environments but we'll just do remote a touch of Java visual via and you get the same value as as well so in this case we can see the WebLogic process and so a couple of things you can do so you see in this case you want to generate that tread down right you can go to the tread view and you get first you get a view of all they're running treads and then from that you can get a tread dump either about right-clicking here trump or we can from that view as well that i found interesting because first you look of all the treads so drivers will give you at least a view on on the treads the water in running state even some of the trade in the block state and then you can get at read them right so very as we just did and and of course you have the value again is you you don't get to to extract the data from the output log as opposed to negative comment in this case you get the data directly in Java visual VM right similar to G stack but it's already pre formatted with some color and everything so that I've found very useful of course if you had the chance to have Java visual VM running that's what I would recommend it's very easy now again like I mentioned if your environment is is facing performance problem like a low memory a lot of pass time you may find a stack and Josh I'm struggling a bit okay that that's some of the issues that I've seen in production so please learn how to use a native command right as we just explained because that's your ultimate way of generating shred um especially when your environment is in trouble and the reality is that most of the time the environment is in trouble right we need it read them of course trouble me the JVM could still be healthy so you can fall back on these two techniques but a JVM is struggling and the garbage collection then you may have to fall back on a native okay so very as I said it can generate multiple snaps all right it's very facile so fangio visual vm depending of the spec of your hardware and environment so that's a very useful technique as well now we're not going to go too far with the java vm today so I'm going to shoot a four technique right now okay so so far native that I just explained g stack java vm ultra dump command now there's another option that you mean not may or may not have realized so far right it's from the admin console blonde check itself yes and sometimes I find it very practical to and I will show you why right so this is our web logic console so I created as I said a standalone a server and right now there's there's not much running in that environment and as I mentioned later on we're going to go back with some other lives to simulate problems so essentially an under server you'll see there's a monitoring tab right they were Blodgett and then you have other types the one that we care about is at red stop okay so in this case you get the breakdown of all the threads right I've web logic and there's a this option called dump tread stack from the WebLogic console right so that find useful is that you can refresh that view right and then you'll be able to see the state of the treads and then when you see for instance for example some hiking trail hugging shred typically will means a tread executing the same request let's in more than five seconds or so to oblige ik will identify it as a hugging shred or a hugger so in this case when you start to see these a buildup of hugging treads you can get a tread down by just clicking the dump stack now you can see from that view right essentially we get the same data or you take you get the same data right from we look so what is missing here do you see anything missing right so what is missing you should be able to see something is missing so you'll notice that the stack trace is the same but let's look at the format right if you go at your visual VM you see you get all the native full tread dump get even the state of the objet like in mixed mode and you get all the native information on the treads right including the lock the object monitor lock IDs as well so this is unfortunately what is missing from that view right so from WebLogic that's a limitation so you will not study be getting all the all the information so that's what I found dependable type of analysis that you want to do this technique is very useful if you want to look at the let's see a problem slowness condition and you can monitor the treads okay like I said earlier the hugging tread and then you get quickly a tread down and focus on the stack trace okay to see what the application is doing where it's waiting its waiting from a database a remote system or maybe there's some lock contention issue now depending on the type of issue this technique will be useful enough by its by itself but as soon as you need to do to do more deep dive like lock contention issue that locks this is why you need to fall back on the native commands or Java so VM or G's type because they have much more information on the ID and everything is this is not going to show you all the details but I would still recommend that you use it in your day-to-day when you're doing some troubleshooting okay now another limitation of this technique or so is the admin server needs to be healthy so sometimes depending on the environments may have a buildup of treads and won't be able to login to the server so of course in this case you want to fall back on the J stack or the native mode again as as we saw which is always the ultimate way to get read them okay but I still recommend they use it because again certain lightweight scenarios you don't you don't need to go at the server level so instead spending too much time get a native tread dumb extracted data just go quickly to the console get a peak of the treads see what they are doing even when the server is healthy it's a good practice right so you can generate tread them as a baseline so you won't give you on their health issue how many treads are running what they are doing and it can compare that view when when you're starting to get let's see a response and a gradation so it's always good practice to do some baseline compression as well okay so that's essentially it for techniques that we saw today for the part two as I mentioned we're going to see similar techniques but this time we'll be using Apache Tomcat to do that since obviously Tomcat is very popular server containers we're going to use it with a different tool so we can show you this technique also across other technologies okay well I hope you appreciated this session and please feel free to comment if you have any question on these techniques thank you

Channel: Pierre-Hugues Charbonneau

Views: 17,442

Keywords: Java (Programming Language), Java Virtual Machine (Video Game Platform), Thread Dump, Programming Language (Software Genre), Oracle Weblogic, Java VisualVM, jstack, Ubuntu (Operating System), Linux Kernel (Software)

Id: -AtwHKXdo2A

Length: 14min 31sec (871 seconds)

Published: Mon Aug 24 2015