Prime Reacts: I like this Backend

all right so I think this is going to be a really interesting one because he's going to go over the Netflix back end as he sees it and I've done some work in this area so I know a good deal at least of what's sort of happening so this is going to be like a fun moment of like what do I know versus what can he figure out and I think this is actually really funny I think this is going to be a really really funny like just watching you know what I mean so let's do this thing okay let's do this thing I feel good about it welcome to another episode of Dev tool them all today we're gonna do this tool Netflix I need to be very careful I'm gonna blur any video I'm gonna mute any audio so I don't get copyrighted good obviously I'm not gonna stream Twitter we're gonna look portion of this and if you if you guys don't know this is the series where we pick popular website and look at the requests sent from the browser by the animations to understand I hate long like because that wasn't a Ken Burns so if you know what a Ken Burns effect is is that there's a point in which you focus in and you expand that point but like just zooming at points out of screen makes me feel like I'm gonna vomit and how the back end Works how the streaming is done is in particular at least Ken Burns doesn't feel as bad the content is delivered what kind of protocol is being used and all sorts of cool things how about we jump into it right as usual we always go to netflix.com let's go to netflix.com I've got a thing and we try to hit enter immediately I'm gonna pause this I need to be logged in to view anything in Netflix so I am logged in with my account I'm using a VPN so that to hide any information about me obviously this I created a new account in my Netflix just by the way you know this video right here that should be playing um guess what I was the person that originally made it play with volume on I would just like to let you know that I'm sorry um but here's the deal here's the deal uh there was nine cells in the test nine cells okay nine a B test cells and guess what of those nine a B test cells test cell eight I fought hard for okay I fought for cell eight I spent my almost entire time implementing cell a you know what cell eight is cell eight was a muted video Plus like location specific subtitles such that you wouldn't have that and they are up in the right place so they wouldn't actually interfere with the video and be below the yellow gradient there all that stuff I did that for you and guess what subtitles performed the worst I muted the video for you and I made it so that their subtitles and you picked the one with the audio on you guys did you voted with your eyeballs you voted with the play button you are to blame I showed you the door I did that you sons of is coming in here tell me that trying to dunk on me ridiculous ridiculous it was like 50 million people okay that went through this and people had just went right in the just went right in for this viewing so that my behavior is not uh reflected here or I wanted I want to try as much as possible to get a brand new get out of here Behavior I also clicked on some of the shows and played some of them all right it's not L plus reported plus and subplus unfollowed okay it's l plus you kept watching stuff because it started playing and your eyeballs were like oh this show is actually cool last chance you I could watch last chance you I could watch last chance you oh man I forget his name I like last chance he was a good show you guys did this you did this I have never had a Netflix account so don't lump me in with these other losers that's right Beast girl you're not a loser you're awesome Nightshade dudes over there like I've I love it but I hate it just a test uh to make sure everything is working you started this man and also to test the resume portion of all right one more thing one more thing by the way there is a feature on the website you could actually turn it off for all of Netflix I don't know if you know this but if you go into your your users right here you can actually just turn off auto play universally so you'll no longer have that experience so just so you know it's on the website uh shut the up okay just so you know hey hey you shut up stop stop it's been there for years stop complaining it's right there stop being such a lazy apple and go and do it okay I don't know why you're always yelling at me the whole time okay everyone always brings it on to me not only did I build it and try to save you guys you voted for it and then blame me for it okay hey you Twitter hate you you read it this app so the first thing we notice what do we notice what do we notice we're sending a request to www.netflix.com we're getting redirected that's fine one redirect is absolutely fine right I don't even notice but I type netflix.com but we went to www.net2lex.com that tells me that the browser saved that fact uh prior that means if I visit netflix.com immediately go to www.netflower.com and always use it I should slash browse that seems like such a oh so we are being redirected to slash browse so this is a decision made by Netflix to so they can create content for your behavior so if we click on the browse the request being sent here took 800 milliseconds I believe back in the day 450 milliseconds was the SLA requirement of map map of course talks to GPS GPS of course talks to subscriber of course subscriber talks to you know Rex it talks to uh billboard talks all that kind of stuff builds up everything and boom outcomes you know out comes your lolamo list of list of movies duh and I noticed this fluctuate took 1.5 seconds sometimes 800 but this tells me that there is server side logic being run to yes there's a shiton of server side logic uh yeah so if you switch profiles that should cause at least you get a few cash you know get a few cash misses as you do that because you know we're not going to just generate every list of list of movies for you ahead of time everybody knows this okay first time you come in we got map do you know map does map Auto generates uh you know because GPS like here's here's the thing that's kind of unique is that at Netflix we generate what we think you're gonna like but not everything's all filled in some of it's very Dynamic right so you can imagine that a movie could be going out of window within an hour and then you come on in an hour and a half later so think about it we can't just show that movie anymore so therefore we have to you know move it out so there's some dynamism to the whole package as well you know to create this particular content for me let's see this so if I click on preview look at this the content that I get from that request alone I get the image I get these and look at this I want you to pay attention to this this is just right this is not clickable yet this is just HTML with a bunch of images and I wanted to notice this so they did the calculation to look at how far I wash Cobra Kai and they showed this progress bar so Cobra Kai man that's called acting there is some sort of a calculation that is happening on the back end to generate this HTML uh what kind of server do they use if I look at the server you can't trust the server property sometimes it's overridden NQ stands for node quark I worked a lot on node Quark I helped prevent some dos attacks just saying that's node quark right because you can easily change this I really said don't use node this but look at this it's like core prod release 1.3 Baseline I have no idea what this server is obviously of course not that tells me that so uh Netflix actually write this thing uh arbitrary or maybe mean something to their client um they're using hdb2 just for netflix.com we're going to come to the rest but look at this guys the moment you start seeing these white lines this is a bad indication this tells me that they are using to retrieve certain content there are requests that being starved and stalled right the white line basically means hey I don't have enough resources to send this request to this particular domain I'm full so 300 millisecond weight right 330 milliseconds 300 what what is this uh this looks like a CDN see my guess is what I'm looking at is that's actually downloading video which would be the right move the 30s so all of these guys are waiting right they can't send any requests why all of these requests are just downloading images yeah thumbnails literally and where are they downloading them all of them from the same domain and what do we know with the studio on one you can only open six connections six six TCP connection per domain that's it is what Chrome does it's just the limit that Chrome has so that means you can only send six requests at concurrently the probably this requests are being sent previously here right and now we're done so there's some sort of stalling that can be solved with HTTP 2 here yeah how bad it is I don't know you you might feel it when you open Netflix at the first time but they probably they don't want to do anything about it I'm alone I'm a little bit surprised that we don't have better steering here we have a lot of steering that goes on so I'm a little bit surprised that there's not more steering because if I'm not mistaken we send down like three separate steering URLs per image I I'm a little bit surprised that that's there you guys I don't I don't just tell I'd love to text you but I don't want to open mine up and expose myself everywhere you know you you watched my hacker nose video there is no reason to move to http 2 or 3. if uh that's not true there's definitely reason to move to http 2 especially in a situation like this obviously you can alleviate that by having more steering but HTTP 2 also it really is fantastic for a ton of images because there is a cost to moving to those protocols yeah because the CPU memory Etc while we're using reactors no such thing as CPU or memory well hdb11 is this very simple direct protocol there is no you know system activities going to you know what we're gonna do be upset that we're I mean it's using a gigabyte okay we're not getting out of this I I what are we gonna be upset for an extra 20 Megs because you're using HTTP 2. yeah we don't use stream priority okay no one uses stream priority that's crazy talk so HTTP 2 if you're just using it for hpac plus you know header that's the header compression and just like uh being able to make multiple calls you oh man you get a huge benefit there you're gonna use the streams and and parse the packets too I wasn't being defensive I was just on Netflix for using react don't use react Netflix find what you're looking for right there's none of that should be one is simpler but there is some bottlenecks that you can reach right so you only move when you absolutely need it and it sounds like Netflix needed here I've just I mean again CDN steering seems like a better route right look at this there's there is a lot of stalling going on here so okay uh we're viewing this you see the word angular which is that is how's stalling going on here I swear I just saw the word angular or was that in chat okay so again uh we're viewing all these images that is that's not oh my goodness I can see the request that I've I've helped craft this right here is my code this is like our whole caching mechanism right here oh my goodness is that still oh my goodness it's still there's that fell core is that Costco oh my goodness I love you core is still there oh my goodness that's trpc before trpc was trpc oh my goodness oh my goodness it's still there right I want you to now take a look at this I'm gonna resume recording again and I am going to make sure to scroll down all right let's go to COBRA Kai but I'm going to clear everything I'm going to resume recording hopefully I don't you're going to get a bunch of requests yeah oh cl2s Consolidated hover over Cobra call and just let It capture some some requests I'm gonna pause it again let's see what happened now okay oh no look at those path evaluators a title Netflix will effectively bring some metadata we'll play some content which I'm gonna blur here uh showing you the content like a preview of the content and retrieve some interesting information that we're going to talk about here so the first request is a post request you have Consolidated and I'm not sure what is what this does but it looks like exactly what it's posting it doesn't return anything right but if posting my current state to the server right so it's not really very informative but the first request the second request is downloading the same image which we actually have so I think this is a wasted I know that request but we're doing it anyway maybe they're doing that because they're for me disabled maybe they're just hey ignore everything and Gadget I really doubt it but they're sending this request again plus that downloading that image caused another connection to be established effectively we did the initial connection we did this TLS as well so let's go to the next request pass evaluator thing is very interesting let's go to the preview look what we got I know the metadata that's an atom I got to this let's zoom in so you guys that means it's a hard value summary Bob summary yeah that means look box art see I'm not crazy box art you can see that this is Cobra Kai I call it other information about like the ratings and stuff like that and I think you can get there user rating from here so it's like being exposed is what yeah sometimes it's hard that means basically it wasn't released probably but sometimes this gets populated and here's my favorite thing the router the Netflix router is one of the I don't know in design that I've seen in a while I heard this in a podcast cadmium way back maybe seven eight years ago yeah but I never seen it until today right when I don't care for this video when you click on result look at this the router the Netflix router actually tells the client which server has this content right oh does your obviously client IP address that's wrong that's not my IP address right I'm connected to another place but plus it's 192 right a year old is a very sexy 92.109 is I think it's a public iPad it's one nine two one six eight is a private uh local network but yeah if I yeah so there's the license uh limited downloadable license whatever this property is actually all the servers that have this content the IP address of that server the domain of that server it's so that way in case like a good reason for this is like if if you were to start downloading uh they have different routes they have different items they go different places they take different routes right and so if you start downloading and things aren't coming in fast enough you can switch see if you have better bit rate right that these things can go down uh licenses on those servers can expire things can get a little bit Goofy on the server you know like there's a lot of things that can happen like if you really think about everything that happens when it comes to video streaming it's an extremely complicated thing and one of them is just having all of this you know you need all of this there to be able to do it protocol it uses C50 lax I'm using a vpnla so it gives you this it's actually it's not La it's uh clearly uh the airport LAX okay we have our little box clearly put in the airport at LAX DNS information it gives you some key information like hi I guess the the bandwidth on this thing uh there's another server right all the other servers here you can see Server 50. it's a red box right like 78 72 35 37 33 32 yeah so these are a pool of servers to give to the client to connect this is as opposed this is what I refer to as a client load Bank client side load balancing where you give the client a bunch of servers and the client decides which one to hit okay versus the server-side load balancer which is what we all know we use you use a reverse proxy which acts like a load balancer that's not that's not what it's for routing and efficiency and it transparently that load balancer uh shifts you to any server right so you take the load from the reverse proxy and puts it on the client side and I always been fascinating I really wanted to see something uh like this in the in the wild and this is the first time I actually seen it so this is basically how they do client side load balancing by letting the client choose which server obviously you see trick play down here I don't know if you can even see twerk play right there let's uh see this thing right here that's trick play yeah you have all that you have all the information in there timed text right so you have all the time Tech stuff uh that's great obviously you have you lose certain metadata here because you don't have the knowledge as if it's if you have a sentence over but the beauty is you don't you no longer have a a one uh central point of failure which is the notice that we also rank them right I think that's an important point which is uh the ranking system allows you I understand use the vrrp protocol to have the same IP addresses on multiple machines and uh you can use DNS also to do the Geo load balancing all sorts of Tricks but still right this is interesting regardless so I just wanted to talk about that I love it a lot and once we once we give these servers you'll notice that they will get they will stop being used here so uh so the we're downloading content from 50 78 50 50 78 so now we're gonna start establishing HTTP one connections again I think this is a bottleneck right but doesn't seem like we have this bottleneck here with the videos because we're downloading a tiny video but look at this we're downloading all this content now right through different uh Avenues effectively right and this is basically what is this this is octet stream so they're not using the hls the HTTP live streaming they're using their own maybe prop proprietary they're using a probably a proprietary protocol here for doing it because it's an octet stream and we you can you can actually take that right and then copy right and then you can paste it and it will get downloaded right the reason it get downloaded because there is a token here that represents me now if I send you this link you can download it obviously because you have my token but if I change this to something else you get forbidden right that means the server is doing some sort of an authentication as well and here we know what they are using look at this you can tell so much information the backend it's like a stripping it naked I love it and you can also tell that the engineering he doesn't even know about Falcor coming there getting all naked like that okay didn't even buy me a dinner stripping my request naked fixes the server here and I love what they did here I don't know if you noticed with the previous uh div tooling sessions if you don't do anything to this header it will show the nginx version which I believe personally I think it's bad you never release what version you're using to the client because this could be exploited right because there might be a bug in certain nginx version and you can easily scan the web for bad nginx versions and exploit that yeah that is fine if if you don't have this information hide that information because that client doesn't really need to know the nginx version any header with uh X in it these are usually like transient headers that's right unless there is a feature that is explicitly in certain in nginx version you don't really need to that so any information about hackers that gives hacker any tip remove it effectively that's just a best security practice and uh we're just downloading the video again I really think HTTP 114 video is good for serial like window in one video maybe that's fine but HTTP 2 I don't recommend I think but HTTP 3. oh look at uh you gotta also look right here you gotta also look at how much data was downloaded and also the surface in which the video is being played we're probably downloading like what my guess a 480p stream maybe for a little box art right it's going to be a pretty thin stream and so when something's coming down we have 84k of a 30 you know of that much and it's taking us you know 30 milliseconds to download that we're effectively downloading what would be one video frame in a 30 by 480p and we're able in one video frame probably able to download 150 of them so it's just like do you need a lot of space do you need this to be super timely um I think you know some of these arguments he's making makes way more sense if you're trying to do something like real time right real time you want the connection to be as like close to the metal as possible right if you can use UDP use udps RTP you go through all that crap make all the different connections blasting down stuff but this is just like when you're doing these longer forms play ones where you can aggregate a lot of data or you can keep on to a lot of data like look at this you can see right here we already have over 30 seconds worth of data right here right and so it just doesn't it doesn't really matter if you're downloading a lot of concurrent requests sent at the same time definitely that will save you connection times definitely that will sing save you a number of connections because you don't need to open these oldest number of connections to send concurrent requests rather hdb3 you get TLS for free for the first handshake and I feel like a Salesman I don't know why you get all this stuff for free with so much car but yeah I believe hdb3 is a good choice for hey I'm pretty sure nginx has considered that pretty sure they did right and they choose the HTTP 2 I don't think it's going to give them a lot because of the TCP head of line blocking but with the concurrent request like sending all this thing predicting the future hey the user gonna play all these segments let's download all this stuff right so let's go and clear this and then actually click on COBRA Kai and hopefully we don't I'm gonna now click on it so we can go to that page right let's let's let it let it let it rip let it rip let it rip I'm gonna blow this you're not gonna see anything so no spoilers you're not gonna see anything hopefully and then I'm gonna let this download and now I'm gonna pause the video so here's here's what I'm gonna do now I want you to notice this I don't I'm not interested to look at it because we kind of looked at how Netflix downloads their content but I'm gonna now clear this and I'm gonna show you something I absolutely love here look at this when I scroll over this we see we see the thumbnails right so look how G ingenious this thing is this is genius what's the difference between ingenious and ingenious I know there's a difference but I I'm not quite sure this is genius we're immediately getting the thumbnails if you ask someone to build something like this they would obviously fetch that content immediately so there will be requests going to the server but no none of these requests are going to the server look at this there's blob at the middle of them and I made a video maybe three years back about how genius this thing is this is called create object URL you would basically download the entire sets of thumbnails and save them in memory and reform reference them by ID so this is a unique identifier for that image so now all these images are in memory for that particular browser very smart right and that's why they are blazing fast when you hover let's go blazingly fast he knew it knew it and immediately see that and this is another genius design you might say Hassan isn't that dumb to download or how would you download all possible how many episodes how many minutes 30 minutes worth of thumbnail no don't look at this if you notice let's zoom in they won't show us that there you go one two three four five ten one two three four five ten so every 10 seconds they change the thumbnail it's a 10 second thing so it is easy to break this like I mean you don't need all that stuff people think that you have to have so much information in trick play but you don't actually need that much information right you just need enough that people can identify what they're looking for like you don't actually like I'm sure if you look here you can see almost the exact same thing going on here look at that yeah it's about every 10 seconds I mean it makes perfect sense I bet if you looked at the same stuff in YouTube you'd see you'd see the exact same stuff is that it only makes sense to do this whole video into a 10 second segment and down and literally request everything and these these blobs are small right so you don't need to encode them in anything like Beyond 40 you know 40p or whatever so you can just like as you ingest the movies you have all the final versions and all the encodings and so then we encode everything and as we encode it we also generate you know the view plane of what how you would go through it it totally makes sense like when you think about it the thing to be downloaded for the 10 seconds that's how much that's not much right for so for a hundred second video that's just 10 minute images for a thousand second video that's just 100 images you can download them up front to give the users this beautiful scrolling immediately fast design right yeah so if I do this now again there'd be no other way to generate those images think about if we on if we onboard 100 videos a day onto Netflix and their average length is 30 minutes and are producing one every 10 seconds I mean that's six per minutes times 30 that's 180 right so 180 times 100 that'd be like eight you know what what is that 18 000 images per day we'd have to generate you have to do it with some sort of mix of AI and all that let's go here like we do some level of AI on stuff look at this so we're downloading these images obviously but now we're sending the request I wanted to pay attention to the request now here okay let's take a look at this now makes sense the requests are these requests do you see these little ranges it's pretty downloaded the actual segments of the videos and and Netflix tries to add as much as possible to hide this information right or at least encode it look at the E 38 this is I believe this is the event ID or the video itself the title so that's always 38 no matter what it's always 38. so that tells me this is the Cobra Kai title but I want you to pay attention to the Token the token sometimes the same sometimes it changes why because sometimes you're hitting a different server which gives you which hands you a different token obviously right there you go we are hitting some different server there you go so I had to refresh multiple times to see the client-side load balancing here we're hitting server 68 and that's over 5 and then 6868 and then yeah what we notice here is every time we hit the same server we get the same token right but at the moment we hit another server we get a different token right that kind of makes sense right and not necessarily so that means those servers are completely isolated right they're not using the same uh authentication mechanism each server actually hands out a unique token for that server we're not would that make sense because or else you'd have to gate every single server you have that's delivering content especially parts of content you'd have to gate it based on a response from another server right you'd actually have to go jump in there and like what are you gonna go do go talk to authentication on Amazon and then come back to your thing that's located in Ohio you know in some small ISP are you going to really want to do that every single time someone requests however many milliseconds right it would make no sense so it makes sense to have some sort of authorized token to be delivered per content you know Network in each one of these things talking to the to some sort of another backend that authenticates that the server itself authenticates itself another kind of an ingenious decentralized you notice that Netflix is is all about this decentralization as much as much as possible obviously the whole thing is in AWS so if AWS went down Netflix will go down got him I don't know if they moved out uh from to another cloud provider but yeah so yeah guys this is all what I wanted to discuss I think I think some of the designs are fascinating some of them I think really the um the server knows you're authenticated because it knows when you're not authenticated and it knows that you're not authenticated by taking the difference between the position of where you're authenticated at and where you're not authenticated at taking this difference it's able to tell your authentication status through authentication differences everybody knows that the the hdb11 they might have thought about this probably and they want to move to hdb2 or three so they might have thought about it then they made some experimentals okay does it buy us any value to move to hdb2 how much concurrency are we gonna getting how much performance are we gonna getting uh maybe they didn't see that much value they saw they went back to a simple HTTP one one that's probably it but a quick will definitely give them more I believe uh concurrency and high throughput if uh if I'm not mistaken um that being said remember guys quick doesn't come cheap there will be an extra CPU cycle that maybe Netflix are not willing to pay yet right because there should be one does again we use react you see this little thing right here let's react okay so whatever cycle you're saving it can't be that that can't be the reason you know what I mean it doesn't have this comp there's a lot of problems with HTTP 3. complexity at the server at the back end uh to to kind of parse these packets and kind of reassemble them together hope you enjoyed this video I'm going to see it I didn't enjoy that stay awesome really good job this is a great video this is a great video I am very happy that they were uh able to do all this you know to get everything done this is beautiful really enjoyed it um lots of collapse good job uh is it Hussein yeah great job is great great video great job very happy about it to see people try to like look at stuff I'm very you know the funny part is just looking at the path evaluator and seeing Adam still there he didn't even see a ref so he'd even see how he didn't really look at the data long enough to be able to see that it's actually a tree I want to look at it but I also don't want to actually docs myself

Channel: ThePrimeTime

Views: 227,550

Keywords: programming, computer, software, software engineer, software engineering, program, development, developing, developer, developers, web design, web developer, web development, programmer humor, humor, memes, software memes, engineer, engineering, Regex, regexs, regexes, netflix, vscode, vscode engineer, vscode plugins, Lenovo, customer service

Id: zQSwPEqBiLU

Length: 34min 7sec (2047 seconds)

Published: Thu Feb 16 2023