Sometimes it feels like I am a god. Hi everyone, today we'll be talking about how NETFLIX onboards new content onboards new content onto their platform So if you have a TV series or a movie and you want to get it uploaded on NETFLIX, apart from the legal challenges there's also engineering challenges that NETFLIX solves. I'll be keeping this video as simple as possible so that the maximum number of viewers can understand what's going on. But there will be some technical details when it comes to video encoding and other technical processes. Firstly what kind of challenges will we face when we are uploading new content ? Well, we need to store it in different formats sometimes you might be knowing about MP4, AVI and other formats The reason they have this is because different people have different internet connection speeds. So if you have a really good internet connection speed and you can deal with a really difficult format for example a detailed one where the data loss is minimum and you want to see like maximum video quality And then you'll have something like medium quality and low quality too. So, all of these are nothing but codecs. A codec is a way in which you compress video. So originally, like this video right now, It's going to be taking a lot of detail, but when I edit this video I'll make sure that the size of the file is not huge. I'll try to keep it within 1GB. So that is one type of codec. If I reduce the quality more then the size of the file reduces because it is lossy compression. I'm losing some data to keep the file size smaller. And the second thing that NETFLIX does is play with different resolutions. If you are watching on a cell phone, then the resolution that you need is much lesser than the resolution you need on your "What's it called" TV or even on your laptop. In this way you're seeing that a single video has multiple formats and multiple resolutions and each of these formats and resolutions are creating tuples like they are creating pairs. You have high quality 720p The number of formats lets call that F into(multiplied by) the number of resolutions R are the number of videos that you'll end up processing. If the engineers in the Netflix come up with much better technique of storing data. Let's say you had high quality requiring you 6GB Now its just requiring you 1GB. Then you take the older movies that you had encoded which are 6GB big. You run them through the new process and it becomes 1 GB. But the thing is this process is going to take some time.So you don't want to give all this responsibility to a single computer because it's going to take time and it has a chance of failing. (what if the computer shuts down?) So what netflix does is really interesting and very smart. It takes the original video and breaks it into chunks. Now what you can do with each of these chunks is to run them through different resolutions and different formats. At the end of it, you will have this chunk lets say chunk A .mp4 So that's a format In resolution 1020.Then you will have A in avi may be 480 and so on and so forth. Effectively you have taken a really big video and broken it into small parts, so that you can deal with it effectively per processor One resolution, one format, one chunk That's one task. The story of processing these chunks is pretty interesting. Initially what used to happen is, you would have this video file and you would break it into chunks of 3 minutes each.So that's equal size it looks good because every processor is doing the equal amount of work and you can actually quantify it. But the thing is imagine an action movie and at the 3rd minute the two cars-the villain's car is just about to overtake the hero's- and then you have a new chunk. If that's the case and someone makes an API call for this chunk it's going to take time. Like initially you are watching this video you come to this point, you get an API call and there is a lag. The user experience is bad because you wanted to see that seamlessly. What they ended up doing is breaking the chunks not based on time stamps but based on scenes. So you can make this instead of 3 minute thing, you can make it much more fine grained 4 secs each. It's called a shot one shot 4 seconds and you can collate shots, put them all together to create a scene So that's the car scene you can think about. Instead of having it arbitrarily stop at 3 minutes you collate them into scenes and each scene has a lot of chunks. 4 second long chunks. Right. Now if a person is watching a video and they click on some point.The video suggestion algorithm will take this as one scene. And the user experience will be much better because you get the entire block fetched together. In fact this algorithm is much more complicated. What happens is netflix sees the entire movie and treats it like a set of chunks. If you arbitrarily go to points then netflix assumes that this movie is a sparse movie, in the sense that you go one point and you see a scene and then you head to next point and then you see a scene and so on and so forth So its recommendation algorithm, its prediction algorithm is going to say that this is a sparse movie or sparsely seen movie and what we should be doing is not trying to be too smart not trying to be sending a lot of data, instead just send the data that the user has asked for because they are probably clicking on different points in that buffer that you get. On the other hand if it's a very engaging movie lets say, I don't know whats an engaging movie but something that is dense movie meaning that people are watching it for a continuous period of time and you can easily say that you know linearly that this part is going to be picked up next. Then this is called a dense movie. Instead of sending just the part that you have asked for it predicatively, proactively fetches the future parts, gets it onto your computer and shows it to you. If you are wondering where netflix stores all this data, then its like google drive called Amazon S3 Something that nearly all the engineers know. This is where people store their static data meaning that you don't change that data, you can go and store stuff. It's extremely cheap compared to a database because a database has updates and gives you other guarantees also. So Amazon S3 is what netflix uses to store that video content. The most interesting thing about netflix is that they were able to bring up an innovative solution to something that was there in the internet space for ages. You know about internet service providers. If you go on your browser right now and type facebook.com. What's going to happen is that you will talk to your internet service provider. They have a list of addresses.They map that to IP addresses. So if you facebook.com,its mapped to an IP address: they have a table over here, which maps it. And this IP address is, you can assume it to be physical place. Its actually a computer some where on the internet which is giving you Facebook. So you are literally talking to Facebook when you say facebook.com. So that's, let's say, over here. Very similarly when you say Netflix, it is an IP address. It's going to be taking you to a computer which gives you Netflix or is Netflix basically. So you can actually, end up chatting with it maybe. But Netflix exists somewhere and every time you ask your internet service provider to talk to netflix, it goes and talks to that computer and then returns you the response. These servers are usually in the U.S which means they are geographically concentrated. In a place like India which is really far its going to take a lot of time to send a signal and then receive it especially if its video because there is a lot data which is going to be coming in and its going to be slow. So to improve on user experience,one of the principle things you do as an engineer is to cache information. which means you pre-compute and store it in some place. Let's say sacred games comes out in India You want to watch that, you put in in a cache. Now Netflix extended the concept and applied it to ISP's. So what the ISP does is that when ever it gets a request from India, let's say and its a movie which is from Bollywood, they won't go and hit the Netflix U.S server just like that. They are going to be asking a cache which has been placed by netflix.This is called a Open Connect box. In this box, you are going to have a ton of movies. You can assume this to be something like a hard drive and if you find the movie here, that's well and good you just return it quickly. So that's a lot of bandwidth which was saved hitting the netflix server, that's a lot time which was saved that's much better user experience and also this is localized. So for India you can keep separate movies for Britain you can have different movies, for U.S you can have different movies. This is a brilliant concept because what you have done is reduced the load on not just you but also the ISP's. So they really want to have these boxes. Every time you hit netflix and get a really quick response,you end up assuming that your ISP guy is a really nice guy. Its gone upto such an extent that around 90% of netflix trafic is taken care of by these ISP boxes that they provide. They are called open connect and this technology is revolutionary not so much who knows but youtube is also doing this. I think youtube red boxes come up with ISP again saving a lot of bandwidth for them and really improving user experience in a lot of places. in that case what you can do is, around 4 am in the night is a good time:The load on boxes is minimum. So you can have a lot of write operations being sent in from the U.S server, so it will suggest you what to copy. 1) You register your movie on netflix, 2) netflix processes them the same way that we talked about. 3) After it has been brought down to chunks 4) It sends them to your ISP or maybe it can directly send it over here and populate this box with these new movie chunks. That way this box has the latest content and the users are happy. So its the innovative menthods on the video processing and the video serving side which keep netflix running at scale.If you think about 90% of your requests are being taken care of by this box. So that is a superb gain and its a really innovative solution.We will be having a lot more videos like this which is system designing in the real world.This is the interesting bit and of course if you have any doubts or suggestions,you can leave them in the comments below. If you like the video then make sure to hit the like button and if you want notifications for further videos like this, hit the subscribe button I'll see you next time :)
