Understanding and Debugging Memory Leaks in Your Node.js Applications [I]

all right so I think it's time to get started all right so hi everybody my name is Adi I work at Google and I get to work on node core and I'm a member of the node technical training committee and I also work on Google cloud and I get to make one of the best places to run no applications so and today I'm gonna talk to you about memory leaks so I'm assuming you have somebody of what a memory leak is if you don't this is what it looks like memory up to the right so so if this was your stock you'd be really really happy but but yeah that's a memory leak this one is a slow one you can see over several days something is leaking so it's not a not a nice one sometimes they're really fast and brace on your application just goes kaboom and then bad things happen maybe your manager beats down your neck and says hmm fix this all right so let's just first dig a bit deeper into what what am I really leaked really is and this is a definition that Wikipedia has and I don't like it that much because it's it's confusing it memory that is no longer needed it's not released but what does release mean because well we have garbage collection we don't have to free memory ourselves anyway so what gives so the contract the garbage collection has with you is that anything that is actually garbage something stuff that you're not using stuff that's not tied to the ground is is garbage and anything like yeah okay I like this video because that mailbox was not garbage okay so digging this metaphor and looking at your JavaScript objects if you're holding on to a JavaScript object and what what do I mean about holding on to a JavaScript object if you have an object that points to another object the first object is holding on to the second one if the first object is not garbage the second one is not garbage so if you have a global variable that has an object inside it that object is not garbage that's the contract GC has with you and anything that's garbage it will seek sweep of it so it's like having a having a Roomba one of those robotic vacuum cleaners these throw stuff on the ground it was sweep it up but if it's actually not on the ground it can't sweep it up alright so how many people have actually had the opportunity to go debug a memory leak was it fun so okay so I mean it would be better if you don't write memory leaks to begin with and I have three steps to do that first I don't write code don't go from any use code from anybody else yeah okay I'm half joking so I think it's important to understand what memory leaks are and how they manifest in your code I think if the goal is never to have memory leaks that's that's hard I'm not saying it's impossible it's very very hard and chances are the application you're running with node may be a small part of it is code that you actually wrote there's probably 5,000 NPM modules you're using that who knows who wrote they may have memory leaks and then there's other stuff that may have memory leaks so I think what's more important is understanding why memory leaks happen and even more important being able to debug having the tools that you have experience with that will let you develop those memory leaks so I'll go through some examples in this example this is a like a very simple express request handler the hello world app you do with Express but I'm allocating a really big array inside this handler in its garbage because I never do anything with it but this is not a memory leak why is it not a memory leak because GC can see that you're not using this anywhere you're not holding on to this object that you've allocated right there's nothing that would prevent it from being garbage collected so you run this it's it's not gonna be fast for you to allocate this object unnecessarily but it it's not a remedy okay so better example so I'm actually holding on to the garbage now so I put the garbage in the requester object so this surely should leak memory right okay this isn't a memory leak either because you have to look at the request object when your request is done it comes in two Express Express gives you information about this is what the user wanted and you send a response back and when express note that you had done with it it throws away all references to the request object so the request object is garbage anything that it only it refers to is also garbage if on the other hand if you take in this array and store it somewhere else as well in addition to request that garbage then it would be different then we'd have to look at where did he store it is that thing garbage or not so this is still not quite quite a memory week all right so the key insight or one of the necessary conditions is about expected lifetimes right so when you have an object and you think this should only live for a short amount of time so this is an object I have allocated and I think it's short-lived and then you go store it in draw into an object that's really long-lived then that might be a memory leak and another thing to note here is it's about expectations right so that's why memory leaks become a bit fuzzy because you're working with a bunch of api's and what was the contract you had with that API or that module that you're using whether if you give it an object are they expecting it to be like art is the contract that they're going to hold on to it for for a real long time or is the contract that they'll just use it and never keep a reference to it so that's why it becomes tricky right that's why it's hard to avoid memory leaks because you have to really understand how your objects are going to be used okay so one more example this is a bit contrived but I wanted to actually show you our memory leak this is most definitely a memory leak because I have a module global map so there's an object on the outside request and I every time a single request comes into my server I throw more stuff into that object the outermost object is going to live forever so now you're actually leaking all the requests that come in so this will quickly run out of memory so on my laptop I can I can handle probably 15,000 requests per second in a simple Express server so so this is 15,000 requests will be la will be leaked every second so this will run out of memory really fast which is good because then you'll know right away that something is wrong so okay so this is a slightly less contrived example this is actually taken from stock a roofer so there's a memory leak in here there might be some object allocations there are some object allocations in here as well the obvious one is the object literal that we have content type JSON that object is not leaking right because that object is passed on to response dot right head and we have a contract with that function that that is not going to hold on to that after the request is done what is it actually leaking is actually the same problem as before the client is a top-level object client dot on what is the contract of the on function the contract of the on function this is an event emitted by the way is that it you give it a function closure closure is the right term but you give it a function that you've defined online that basically makes a closure that's an allocation and it may refer it it may have storage associated with it and the contract it has is that whenever you give me a function I'll put it in a list and whenever that particular event app happens I will call this function so you're doing it every single time a request hits your server right so you clear educating a new closure every single time this is going to leak memory as well so this one this one looks like a typo the this code shouldn't be inside the request handler seems like it should have been out at the top level that's my guess but oh yeah so this would leak all right so now things are getting tricky so this small program you can run this in node today this program is the only thing it's doing is looking at how much memory it's consuming you know it's doing nothing else and this program will increase its memory footprint as you run it it will run out of memory eventually so this is not quite obvious because I actually don't see an allocation right there right so so the so and and actually there's an asterisk on the on the leaf this is it's debatable whether this is a leak this is a thing that JavaScript doesn't like anyway so maybe what happens here is that we're tying down the event loop this is a sync right this is not letting this is going all the way to run on the same thing what console dot log does and me the covers is it's going to allocate some closures and say on the next stick do something right but the next thing take is never going to come okay it will eventually come but after very long time so this will leak memory and as well so so thing is functions that you call may leak memory and you have to understand what they do so so but so this is the interesting example but leaks can be totally unexpected so this is so take this code run with the latest version no date might X by the don't panic this is getting fixed to really soon in eight point seven on Tuesday this will be fixed and also this is this is a very specialized condition that most likely it's not happening in relapse so this will leak memory but not right away it runs stable for about two three minutes and then it just goes like a hockey stick and sneaks memory so what's when I point here my point here is that the code you might have that's running correctly today might leak memory in the future because something else has changed or the conditions under which your application is running change that now you suddenly have a memory leak so it's great to understand that object lifetimes and when I have a long length logic and I store a shortly Bob the inside it that's a memory leak but it's even more important to be able to debug memory leaks alright so for debugging memory leaks the let's talk about some of the tools that you have available and so whenever I look at these tools there's a four node there's a few questions that I'm asking myself can it what kind of leaks can it find can I use this in production because that's the the thing I fear the most write leaks that I can catch in development are are usually the easier ones the things that make it to production are usually the harder ones and if I tool that I'm familiar with doesn't really work that well in production that's a problem so say so those other questions I'm asking and I'm actually going to focus more in this talk I will cover native leaks and and Java simply leaks at all but my main focus is JavaScript leaks because I'm guessing most people are interested in JavaScript most people are writing JavaScript and that's what you care most about so first question when you leaking memory where is it actually leaking if you let me go back to slides so in this case this was this is actually quite an ironic this is a leak in the v8 garbage collector so the garbage collector itself is leaking memory so so so so the problem was that it's the v8 garbage collector is quite awesome actually it it is quite adaptive it's trying to figure out what your app is doing and it's coming up with a heuristic that this would give you the best performance for your garbage collector we won't be interrupting your we won't be incurring as much latency if we work this way and there was a bug that was existed in that heuristic and that's why this leaked memory so this is a native leak right this is not a JavaScript leak but your application may still experience this so so that's what I mean by a native leak and there's some tools available for debugging native leaks as well and then I mean the ultra most level the most basic level you need to understand I quit you'd I would guess you'd have some monitoring in your production applications that mythili's is showing you a chart that that basically shows how much memory you're actually consuming so or or maybe it's just the top command you have on your on the next machine or the task manager it just shows you memory usage those are I mean those are the first set of tools to look at when your application is behaving really so if you are not going to use any other tools this is the most basic thing that you can have this this has zero dependencies okay I'm like there's one dependency but that that's just for formatting numbers so you you don't need anything you just put this in your application and we'll just keep spinning out how much memory it's actually consuming and you can have a running take off what's actually happening in your application from a memory point of view so this is nice actually there's there's some modules one is called memory usage by by Thomas Wharton right here that actually is Rena is actually plots it so it makes it even nicer to use it the challenge might be that that one may not be suitable to run in production because it opens a browser page and shows you the data so but if your so this stuff is the most basic thing you could have that tells you gives you an idea of how applications behaving so let me cover a few topics like let me cover what this actually tells you it tells you what the resident said size of your application is the resident set size is how much ram your application is consuming this includes anything that v8 is using this includes anything that note core itself is using it includes anything that any other library that node uses or any other native module that's running as part of your application or if the Postgres for example first quest module that you're using if it's allocating objects that are C++ objects those would all be in the RSS so if your RSS is going up you probably have a memory leak okay I should define that and say if the resident sets is going up and you don't expect it to go up that's when you're likely to have a memory leak heap total is the total amount of memory that you have available for JavaScript objects the way the garbage collector works is that at any given time you have some JavaScript objects but it grows eventually until he reaches the heap total and at that time it decides whether maybe it was too short a time in which I've was able to deplete all allocated heap space so maybe I need to go the the total heap size so this is this is a really good metric of how much space GC thinks you need for JavaScript objects so if you have a memory leak in JavaScript you would probably see this number going up as well heap used is all it's a bit more variable because it goes up and down as garbage collector works so you can look at that as well but he total is probably a good enough metric to look at and then there's external which is interesting so this is any memory the node core is allocating for for for objects that it needs to connect between JavaScript and native site so if you have let's say a buffer this would go into the external memory Soviet reports this as well so so this is interesting as well and later on I'll come back to how so if your overall external memory is going up as well that means maybe you're leaking a buffer or some of the wrapper object that note core is using okay so now let's say you now you know that you have a memory leak what do you what are you going to do about it so the this dual heap snapshots has been available for a long time it there's um almost any memory leak tutorial that you look for on the internet for node it's probably going to talk about heap snapshots so I'll give you a fairly fast overview for this thing is and and then I move on to some some brand new things that are now available so I'll actually do a demo for this so so heap snapshots so this is your chrome dev tools you can run no - inspect and you can attach dev tools to it and it's so basically normally you probably see interface like like console or sources but this is the memory tab and you can actually take a heap snapshot of the current running application that you have I'm not going to do that right now I actually have taken a heap snapshot that I saved earlier and that you interesting thing is you can save these snapshots and as files on disk and then you can load them back up and you can view them so I'll cover a few things that are being shown here it's telling you right now like right now all the objects that existed in the jaws repeat every single one of them and what they look like who was holding on to that object remember earlier I said that something is not garbage if something is holding on to them right so if you expected something to be garbage this will tell you why that thing is who's holding on to that thing making it not garbage so and and just looking at this let me fin crease the font size one once so just looking at this you can notice that there's lots of objects and the constructor that allocated that object is the first column and this is the number of objects of that constructor that exist so looking at this you can right away tell if there's a memory leak and that's then this object thing looks very common if I sort it by count seems like something that's just an object literal is leaking shallow size is the amount of memory that specific object is and retain sizes all the memory that objects reachable from that object have so let me drill down a bit a little bit so each of these entries is an individual object that existed on a JavaScript heap right these are right now live in memory in your node.js application now let's just look at one of them so when you click on one of them or expand them this actually shows you what the object has it's actually showing you the internals of the object it has a map it has the proto and it has one property called URL so so and looking at some of these other objects in here you can see most of these objects are this one so right away you know this is the leaking object so can anybody tell me which object like where the leak might be well you know what the leaking object looks like you don't know very in code this object was allocated and if you're really familiar with you could write if you just by looking at the object it has a special property name that you know only one place and code adds then you have a pretty good idea where the leak is but that may not be the case in all cases so in this example I have no idea I have no idea where this object was allocated because map is huge and I didn't write all of it and and yeah I know there's a leak and the leaking object looks like this but I don't know where it is in more real-world examples you won't see such a stark contrast right so you won't see like this one object type is so so predominantly common I have lot of other object types so what do you do in those cases well there's a really nice comparison view you can say compare this log to this other log that I captured let's say 30 seconds later and it actually tells you the Delta like we have this many fewer objects of this type and this many few more objects of that type right so so this can give you an idea of over time this is what's increasing right so this gives you a better signal for what what actually is is maybe leaking if there's too much noise and not enough signal all right so let me also show you this retainers section so let me click on an object let's go find one let's say let's say this one so it's telling me who's ordering or holding onto this object so so the output says object number seven two seven five three nine has a property named chunk that is holding on to this object okay who's holding on to chunk next next it just keeps on so it looks like there's a linked list of next pointers that just is growing and boundedly so you already are getting some hints about where the leak might be okay so so this is all good this is what a heap snapshot is let me I have a summary slide right over here there's a detailed tutorial that's linked over here and and I'll at the end I have some links to this to these slides so you can actually find all the all the speaker notes as well so you'll find all these links so so this is great but it has a the challenge might be that it tells you what is like what the leaking object looks like but isn't necessarily tell you where it might be and there's some other practical considerations for production as well so I showed you dev tools it's fairly unlikely that you're going to attach a live debugger to your production server but if you're doing that you think about it carefully because you can do damage to your production server other considerations keep taking heap snapshots are inexpensive we have to do a complete GC and then we have to walk your entire heap and can internalize all the objects that existed in the heap so if you're doing this in production with live traffic you might have experienced a latency spike when you actually take the snapshot so you definitely don't want to be taking these snapshots like once every second because that probably won't even have to finish there's a really nice module called heap dump that lets you take heap snapshots and writes them to disk so if you and that would be a much better way of doing this in production right so install that module and maybe run it in an interval that every R it takes the snapshot and writes it to disk that may be a valid way of doing it but be aware that you're going to witness that latency spike so other consideration there they they may be sensitive data in fact this is into your entire heap your customer data will be in here so be careful who you share with this data with okay so the next one the next tool I'm going to show you is allocation timeline so allocation timeline is right here so it's actually pretty much the same thing as allocation snapshots the difference is it basically takes a sequence of snapshots effectively and it the really good thing about it is actually you can focus on a really small part it gives it can give you a really good understanding of in this small time pin interval what was getting allocated I'm not going to spend too much more time on this because this is so this is nice but the thing I really want to show you is there's a detailed tutorial linked here so the things I really want to show you is something brand-new sampling a profiler there's a new in actually it's it has been there for a year for a few people know about this and this is what it looks like it's the same profile that I showed you before but visualize same app same memory leak but if anybody's familiar with CPU CPU profiler it's very similar to the it's telling you these functions are allocating memory and the width of the of the bar chart width of the chart tells you how much memory they're allocating so it is built for production it has very lower overhead and it's based on sampling so at any given time what it does is this these are the objects that are live on the heap and these this is the function that allocated them so in this case you can actually right away tell that the biggest bar in here is AB get 4 like so you can actually see the call stack and this pinpoints to you exactly which our function was it that actually allocated the memory so so I think this is great and the reason this is great is because I implemented them so this all right so let's go back here resent ok so this actually doesn't capture all your objects on the heap that's why this can be fast for production right so you can actually do this like get the gather these snapshots every 10 minutes or 20 minutes and this wouldn't slow and things down substantially it is it takes a representative sample of your heap which what that means is that it is it uses a Poisson process for for sampling objects as they're being allocated so on average it takes one sample every 512 kilobytes that's why it can be low overhead but it it does have some caveats the caveats are that right now anything that's in lined into the function so if you have a function and you inline lots of things into it allocation will be attributed to the function like the containing function so if you see app dot get is hot then you should look really look at all the functions it called synchronously because allocation may actually be inside those but that's something we are actively working on making better and then there's some edge cases the via optimizer is is really smart and it optimizes things quite a bit so there's some types of memory some types of allocations that it cannot observe but it's already pretty good to use and I would definitely welcome feedback on this all right so let me quickly cover native leaks sooner or later I mean if you're writing native modules you probably care about this or if you work on note 4 you care about this but let's say you have an app and it uses comes some components that are native how can you debug them well you can use valgrind but the challenge I have with valgrind is that again it's not the best for production because you slow down your application quite a bit so guess what there's a sampling dprofiler the same thing I showed you available for native code as well it's there's a TC malach native alligator that has the same feature in fact that was the inspiration that's the thing to use for production services at Google that's how we debug memory leaks at Google for all the things we run in production and that was a motivation behind why we added sampling a profiler to to do v8 so so there's some links there they tell you how to do it I won't go into too much detail about that because but it basically gives you the same kind of output but one thing I will maybe try to inspire some people here that it seems like we have a sampling profiler for JavaScript another one for native code would it be nice if somebody wrote a module that basically give you a gave you a consolidated view of all possible leaks so if anybody wants to go write that that would be awesome so the slides are right there memory usage I mentioned already that that gives you a plot of memory user for application yang in the previous talk showed you that there is a new module inspector that's built into node core that can give you access to the JavaScript API for inspector all of these tools that I showed you all the snapshots allocation profiler and the sample heap profiler are available through that so you can just write JavaScript code and get all of this functional so that's all there heap dump and he profile I just noticed that there was no npm module for he profile so i just published this i think earlier today so brand-new and and if you really want to do low-level memory debugging ln node an MDB are also good there's a talk link at the bottom debugging issues in production it's by Yunnan from Netflix it's it's a really great talk it talks about the considerations for using tools and production and how you can debug memory leaks in production that I definitely recommend checking that out so the thing you should walk away with is leaks are going to happen right so what's the message I want you to take away that there are tools available try them out because you want to be prepared when the leak happens in production and you don't want to you won't want to be using these tools for the first time when you actually have a production service that's actually leaking memory so try these I would give them feedback so I'll be at the Google cloud platform booth so if anybody has questions I can answer questions there actually we have a really awesome demo for debugging and production that we can show you once once you drop by right that's it [Applause]

Channel: node.js

Views: 34,750

Rating: 4.9299998 out of 5

Keywords:

Id: hliOMEQRqf8

Length: 30min 48sec (1848 seconds)

Published: Mon Oct 16 2017