Node's Event Loop From the Inside Out by Sam Roberts, IBM

okay my name is Sam Roberts I used to work for strongloop but now I work for IBM since we were acquired and working with node for about four four and a half years and before that I've done a lot of UNIX systems programming my background is first embedded systems and then like UNIX and real-time operating systems so I've worked at the system layer a lot and oh oh so I'm that's what I'm going to talk about so Doc's called the event loop from the inside out and it was partly inspired by a series of interviews I got pulled in to I'd sometimes get cold and to end people saying you know can you check out this guy you know he or she looks like they know a lot about node but can you ask them some questions see what they know about node core you know you're lower than Express and loop back and stuff and one of my go-to questions is um what's a node event loop and my secondary question is is Note single threaded or multi-threaded and a surprisingly number of people had I either unclear wrong or muddy answer to those questions so the part of the purpose of this talk is to clarify what is an event loop what is the node event loop so you can speak confidently about it and also it impacts how node works and you you will care someday when things go wrong so I will so yeah here's the questions what's an event loop is node single or multi-threaded we hope to answer these questions so I'm going to jump right into it there's going to be a bunch of pseudo C code here please don't criticize my my pseudo C code it's not always some tactically accurate but it's hopefully enough to give you a sense of what's going on and if you really want real C code I've got a guess somewhere you can play with so we're going to jump right into it so networking on UNIX is is based around sockets it was socket the socket is what's returned by the socket system call and it's a number it's an integer is integers are usually they're sometimes called socket descriptors they're more often called file descriptors this is very confusing it's not my fault they don't have anything to do with files necessarily they're just called that their files also have file descriptors but anyhow so let's just be clear the all of the system call documentation api's discuss things in terms of file descriptors I'll use that as well so file descriptor can refer to a socket or two other things and it's kind of a it's a system way of doing object oriented abstraction a file descriptor is an integer offset into an array that's kept in the kernel every process has an array of file descriptors that array then has a pointer to an object which is the open control block for whichever resource is attached to the file descriptor and that has a pointer to a virtual function table so you basically it allows different things like sockets files pipes to all support read but read does different things depending on what the underlying resource is so this is you know okay fundamentals of UNIX programming one file descriptors are not really files so I'm going to jump into a kind of a classic way of doing TCP servers and it's I call it thread per thread per socket so basically what you do is you is you create a socket so server equals socket you bind that socket to a port so you know 80 for example and then you call listen on it until you call listen that socket could be used potentially for accepting connections or making connections it could be used for either side once you call listen it will accept connections so this is a your standard three part stands I call socket bind it to a port call listen you're now ready to accept connections so and the accepts you accept connections by calling except it's in a system call so the thing about system calls is they can block block system calls are effectively our library calls into the system and if you if you ask the kernel to do something for you that it can't do right now or that it's going to take a little bit of time it's going to D schedule your process you're not going to take any more CPU you're not going to get anymore see pou can't do anything and sometime later when whatever you asked for is as complete or ready your process will get scheduled again this is great when when you don't want to use CPU and you've got nothing to do but it's not good if you actually wanted to do something else so in particular it's not good if you want to handle multiple connections at the same time so we're going to sew the thread per connection model you call accept it blocks TCP connection arrives it unblocks you now have a new socket this socket is specifically for the TCP client at that point you can you could start reading and writing on that socket but if you're doing that you couldn't accept any more connections so you kick it off into a thread you call pthread create you pass it a function pointer you pass it your socket descriptor file descriptor call well you will and that that thread function can now start for this is an echo server it can read data from the socket write data back out the socket and it just does that in a loop in its own thread the main thread goes back to its while loop calls except so this works fine a lot of Java servers work like this but back in the bad old days systems would often croak if you had more than you know four or five hundred a thousand thousand silent threads at the same time would take systems down so operating systems are much better than that now you can easily handle a few thousand threads on Linux there's no problem at all windows same thing but the fact remains a thread is heavy heavy weight relatively heavy weight to the amount of data we actually need per connection all we really need to know to talk to a client is its socket descriptor and we have to remember what we intended to do with that descriptor that's it is you know a dozen bytes of memory that's all we take so to spin up an entirely new operating system resource a thread just to handle a single socket doesn't doesn't scale well doesn't scale to tens of thousands of connections so there's a better way now I originally had a kind of a simpler piece of code here that used a traditional set of UNIX system calls called pull its pull and select they're two different system calls basically do the same thing but I didn't have time so I'm going to jump right into equal so equal equal is what Linux has BSD systems use KQ Windows systems do things that are completely different but have the same end effect but I'm going to focus on equal and Linux so free pull you basically do the same thing as before you set up your server so you call socket called by and called listen at an equal and then you create an e pull descriptor and what we're going to do is we're going to use this descriptor to tell the colonel about the things that were interested in and the kernel is going to use it to tell us when the things when things happen that we that we asked about so we excuse me so to start off here we have a server we only have one socket the server socket the only thing we can be interested in is is there an incoming TCP connection so so we ask that we create a file to script they pull descriptor we say hey we're introduced to scripture to the e pull loop and we're interested in when there's incoming TCP sockets so we set it up then then we go into the loop so we're in the loop we're going to we're going to once again block on the kernel we're going to say we're going to call a pull wait at which point we're going to get D scheduled no CPU will take will happen we're not going to do anything and when the kernel sees something interesting happen interesting as in we were interested in it it'll wake us up so what kinds of things can it tell us about well so far we've only told it we're interested in the server socket so the only thing it can tell us about is the server socket so that's that's one of the two things this really simple piece of code to tune do so when the server socket is ready you pull return what can we know it's available we know there's a CP connection what do we do we accept the connection and once we've accepted the connection we have a socket descriptor this one for the the TCP client but there's not necessarily need data just because it's connected to us doesn't mean it sent us anything so we can't do anything else with it yeah we can't read from it we can't write to it so what we're going to do is we're going to add it to the loop we're going to say hey this client socket I'd like to know when there's data available so once again I call I pull control add I pass the the connection socket to the leepo loop and I go back to the loop waiting so this point two things can happen the next time the loop wakes up it will because of either there's a new TCP connection or there's data available on one of on one specific TCP connection so and that so that's the the other part of the loop so if there's data available in a specific TCP connection time to start echoing so I can read up I can read or date some data out of the out of the connection and then I can write it back out to the client some you might be choking at this point it's actually much more complicated than that because just because I can read data from a socket doesn't mean I can write to it you know so what you actually have to do is read the data stick it in and stick it in memory somewhere add that socket back in saying hey I'm interested in it but when it's writable wait until it's writable pull the data back out there's a lot of a lot of bookkeeping involved and this is why people use libuv it's the there's a lot of race conditions there's a lot of subtleties there's a lot of things you can do wrong so you really shouldn't be doing this stuff yourself use UV it does it really well but this is what it's doing at its core so this this is that is the node event loop the answer the question is what is the node event loop is it's in a semi infinite while loop calling either a pull wait or pull or k-q wait whatever you know the system call is but it's a semi infinite loop calling on the server sorry on the kernel I'm kidding blocking until there's something interesting when something interesting happens you know what happens is node is it transforms out into a JavaScript API so something interesting it might manifest as an event it might end you know as a callback and the FS thing I think sorry the most precise of language here we have s module so that that's that's that now that you've seen the lo loop you can ask another question is what when at semi-infinite so when does it exit well if think about it work we've got any poll Sat there's a set of events were interested in we call wait so that's too interesting there's an interesting state what if there are no events well if there are no events April wait will never give you an event so you're either going to it's always either going to always return immediately so you busy loop or it's going to block forever which is silly and not very useful so what node does is it exits so that's you know when does node exit if you ever get asked that or you know have bugs and don't understand why node is or is not exiting the answer is no two exits when there's nothing to wait on in the e-poll loop anymore there's a bit of a caveat there's also an unwrapped call on a lot of node objects so if you happen to create a row something that it is being e-poll waited on time a network socket something but it's it's not core to what you're doing and you don't want to keep node alive just because that resource is there you can unwrap it so UV has a way of saying okay yeah I am waiting for those sockets but I'm not going to count them if that's the only sockets I have left or whatever resource it is I'm going to call it quits and exit the loop okay so no it has a you know not a huge API but it's a reasonable wide API there's a lot of modules a bunch of things so is everything in the node API eople able an answer is kind of yes and no some things are directly eople some things you have to fudge around a little bit to make them available then there's basically three kinds of things so Pola will file descriptors these are directly pulled well that's that's kind of what he pulled as a file descriptor in the loop boom pull for it time I'm going to talk about time in a second time at least one time out is directly Pola ball and everything else well everything else you have to do some jiggery-pokery to make it work and you know it's also very difficult if you're not really familiar if the underlying system calls right now it's not documented in the node API what how things are implemented and it met and it will matter to you eventually so I'll run through these things so first of all socket so sockets include net diagram HTTP TLS HTTP child processes namely their pipes standard out standard in standard error okay about five minutes all of those things are directly probable the timeouts you get one timeout there's one timeout argument to the 2kq in Ipoh so all timeouts anode are sorted from you know most the one that's going to happen next to the one that's furthest away and every time a poll is called it looks at you know when the next time out is and it takes that time and sticks it in e poll so that's that's dealt with by a node for you file systems everything everything in the file system layer is not it can't be it can't be selected on I don't have time to talk about it but even though it is a file descriptor you cannot effectively pull on them so that always happens in a thread pool so UV had UV has a thread pool and so basically and you can't you can't wait on threads these are in any pol loop so there's a thing called there's this ancient UNIX programming trick called the self pipe so basically create a pipe it's got two ends when the threads done it writes a bite into the pipe the other end of the pipe is being waited on in the e pol loop and the e pol loop wakes up so this technique shows up over and over again when you've got something that's asynchronous or something off the main loop and you want to wake the loop up you'll use a pipe to do it there's also event FTS there's other mechanisms Linux that are better so title is a fret so DNS DNS is an interesting case and if you read the docs that goes so the docks are better now but there's all these back bending loops it goes to try and describe it but the basic deal is that there's one the system resolver for host names does not necessarily use DNS a lot of you might know there's a hosts file you can write directly write things in there's multicast DNS there's Sun Yellow Pages NIST there's lots of ways of resolving host names so if you want to resolve hostnames and have the system do it in any of the many possible ways you have to call get a get host by outer as or get a tour infos a more recent one so that's blocking so it has to happen on a thread pool everything else in DNS is asynchronous and so it's safe if you want to stay stay out of the thread pool but however get hosts by outer is the default so when you do an HTTP GET or TCP or net connect and you give it a host name it's going to go through the thread pool so some quick notes about the thread pool so who uses it FS uses it DNS uses it but only for DNS lookup everything else is fine crypto uses it but only for crypto random bytes and crypto PB pbkdf2 password-based key derivation for algorithm HTTP GET and request doesn't use it unless you pass a hostname in which case it has to pass it through the thread pool so if you don't like that if you know that your host name is DNS look able do the lookup yourself using the other DNS functions that will happen entirely on the main loop and out of the thread pool and C++ add-ons can use a thread pool now there's nothing wrong with the thread pool it exists there it's very useful it does things but there's only four threads by default so if you have a lot of a few of thousands of concurrent connections and each of them is doing a DNS lookup or doing some file system activity or doing anything that uses a thread pool they hit contention for the thread pool and you can find your performance dropping through the floor so you might need to increase the size of the thread pool or if you can get out of the thread pool so you're those of you two things so if you want to increase the thread pool size UV thread pool size is your environment variable of choice the only one and if you can get off the thread pool maybe you can for some cases I'll keep those signals I'm not going to talk about more it doesn't happen on a thread pool they're completely async to use self pipes child processes don't use the thread pool there so they're safe for you know high concurrent use you don't have to worry about that C++ add-ons C++ add-ons are a common source of problems here if they're they may not use the thread pool at all because they don't need to that's fine they may make blocking system calls and they should have used the thread pool that's the problem so you should review your C++ add-ons and make sure they're not blocking your loop and then if they do use a thread pool which is good they didn't block you but now you might have thread pool contention so if you're doing a lot of file system activities and you have a C++ add-on that does some magical stuff for you that's non-blocking well maybe that's all going through your for thread thread pool and you need to increase your thread pool size it so it's good to keep a keep an eye on that and actually for that I would say there's a really useful metric for applications is the loop time strong agent did that APMEX tricks did it I think lots of other monitoring solutions do it you you want to watch your loop time because problems will arise you'll see them often in your loop times a loop should finish in like 8 to 12 14 milliseconds spending in your system if you have stunk and hardware and stuff but if you if you see your your loop times in the hundreds of milliseconds you're either doing something incredibly CPU can intensive or you're blocking and blocking that's a problem you need to find it so that's I sorry for a fast side to talk so hopefully that this makes you know all of you everybody knows now who didn't before or already knew what the event loop is when notes multi-threaded and the basic reason we say that it scales well which is low low resource cost for connection and that's that's kind of ended my talk like said I actually wrote real see code like compiles and runs it's buggy it's not as good as UV but it's good to play with if you want to there's a it's in a guest and also burp elder who's a much funnier and charming person than me did a great talk it note confidant you about the event loop he talked about it from the what is called this inside out he talked about it from the outside in what it looks like more at the node API surface and how it interacts with the famously miss named process next tick inside immediate which do the opposite of each other I won't go on about it okay am I supposed to answer questions do we have time for that is there any questions this is at least one question yeah in the slides it's correct so I think the my slides are in the whatever the application is so and if you can't find it find me I can dig it dig it out yes well it uses a thread yes yeah so again because it uses a thread it does not block let us be clear um it runs the stuff through the thread pool so the thread blocks but who cares the thread is only doing one thing so so the main event loop is not going to get blocked and it won't get blocked because this so-called blocking call is being kicked out into the thread pool but if there's no thread available it's going to take longer than you wanted then that will only happen if you have a hostname to HTTP GET or Ornette dog connect if you use IP addresses in fine guy calls out to different service no it would be imprudent to cash your own DNS you should use a caching DNS proxy and most Linux absolutely has one but or you can use you can do the lookup using the other DNS calls yourself if you if you do the lookup that will happen with pure DNS and will happen on the main event loop and in that end it would also be cached like if you use localhost as your what's a cook stop okay after we done you could ask me afterwards but there's there is a classic there is a classic Linux proxying DNS server and I would you should use that and not do caching yourself otherwise you're going to screw up the timeouts and everything else okay [Applause]

Channel: node.js

Views: 84,931

Keywords:

Id: P9csgxBgaZ8

Length: 23min 1sec (1381 seconds)

Published: Thu Dec 15 2016