HLS Streaming Server with NGINX

hello everyone I'm Jacob Kauffman I'm the nerd on the screen and today we are making an HLS streaming server using nginx alright everyone so a few years ago I made a video about restreaming your live streams using the rtmp module of nginx it's a fairly straightforward process and on the comments of that video I had quite a few people asking me how they could use that module to embed a live stream on their website but that wasn't the point of that video that video was about restreaming so taking a live stream out of an encoder on your computer like OBS sending it to your restreamer and then from there going to several different live streaming web sites like twitch mixer YouTube that kind of thing now whenever people have asked me about setting up their own streaming servers I've always told people it's not really cost-effective you want to go and use one of the big websites out there like YouTube or dailymotion or twitch or mixer smash cast Picardo there are quite a few out there because those websites their entire business model relies upon keeping those streaming servers going they've got the bandwidth for it and it's just much simpler and easier to stream to those rather than setting your own server up however tonight is a Friday night and my favorite live stream that I watch every Friday night has been postponed because mixer which is owned by Microsoft for those of you who don't pay attention to the streaming websites mixer implemented a new Community Safety policy earlier this week this new policy required all creators who were not mixer partners to log into their accounts watch a video about the community safety policies of mixer wait a waiting period of 24 hours to make sure that video really sets in your mind and then sign a pledge to follow the Community Guidelines before your stream key will be regenerated and you can stream again so they implemented that earlier this week now yesterday they already had a major outage because when they were migrating their data store presumably to set up this new system that they were doing they messed that up and they took their own website down for a couple of hours yesterday so tonight low and behold the websites down again and the streamers that I normally watch hadn't done that verification thing yet anyway so it would be at least twenty four hours until they can stream and this is all in the name of enforcing their community conduct now they use buzzwords like making sure they're a welcoming and positive community but honestly this is something I would have expected from Google and YouTube it is not something I would have expected from mixer because so far since Microsoft had bought mixer they hadn't pulled a whole lot of crap like this yet and now they have so now it's on and now nerd in the street is going to offer our viewers a way to watch our streams without going through any of the big major West Coast tech companies if I can help it because screw Google and screw Microsoft and screw all of these other companies who are doing all of this honestly dailymotion is based in France and they haven't done anything like this yet so if somebody asked me what my favorite video or what my recommended video streaming website is right now it would honestly be dailymotion Ustream has disappeared in the past couple of years live stream comm got bought up by Vimeo Ustream was bought by IBM so there's a need for what I'm about to show you especially for people who aren't streaming games you know if you're streaming actual live content podcasts talk shows things like that you might very well want one of these setups anyway and if you are streaming video games maybe consider setting yourself up a website and use one of the multi streaming servers that I already showed how to make to stream both to whatever big-name website you are streaming to before and then also streams your own website that way if the big website goes down all of your loyal viewers who go to your website will still be able to watch you so that's why I'm doing this video and this was not a planned video I have never attempted this setup before but how hard can it freaking be you know I found some guides online for it so we're gonna try running through it right now just a heads up some of my videos are very planned out and practiced some of them are not this is one of the ones that is not so without further ado we're going to cut the desktop and gonna set ourselves up a streaming server ok everybody and here we are on the desktop so the first thing we're gonna do is head on over to digitalocean comm they're spamming us with all sorts of ads about their new features that I don't use and I've got some more credit on here just so you guys know you can go to digital oceanauts code to get a few free bucks on digital ocean and it will also help me out a little bit as well I greatly appreciate everyone who's used that referral link so far but we are going to get started with a droplet here we're gonna use Ubuntu 18.04 for this video and let's go with let's go with a $10 a month droplet for this video I'll only be running this for a few hours I think so I can pay the extra a few cents for that slightly more upgraded droplet we'll go with one located in New York we don't need any of these options here I'm gonna go with the super insecure one-time password for my particular server and we will create this droplet here we've encountered an error creating your droplet something went wrong acknowledging so it looks like digitalocean might be having some issues today - let me turn off my ad blocker and see if that helps alright maybe I just had to relock in since I have moved my computer from Saint Louis to Denver they probably detected that and were wanting me to login again ok can I dismiss these notifications now yes great create a droplet off to a great start having trouble making our server just let me spin up a droplet all right so if you've never used it as a lotion before it's a virtual private server provider right now they're spinning up a Linux virtual machine for me to use in their data center as soon as that's done spinning up we will open up a terminal to log into it here and we will copy our IP address we will SSH into route at that IP address I'll turn up my font size a little bit for the video might turn it back down later depending on how much text is on the screen all right and here we can log in here we go all right so I am in my Nia digitalocean droplet as usual the first thing we'll do is run an apt update in an apt upgrade because you always want to start with an up-to-date package base whenever you are about to dive into any project so app to upgrade all right we are just going to reboot our machine to get into that newer kernel well that's rebooting I am going to mention this is a guide that I am going to attempt to follow if we run into any issues then I will solve them but I'll link this guy down in the description below for if you guys don't want to pause the video every two seconds to read my commands now back when I made my video about restringing I'm pretty sure this is the engine X rtmp module that I used this was the original one now you can see it hasn't been updated for a couple of years so we are going to be using this other Fork of it which is still being updated fairly actively looks like our server is done rebooting and we are all up to date now we are going to need to compile Engine X from a source with this rtmp module because currently the version of the rtmp module on the version of engine X in the Ubuntu repository don't work with each other so that's no big deal we are currently in a clean directory we are going to clone that git repository that I just showed you we are going to install a few dependencies so this is build essential Lib pcre which is the Perl compatible regular expressions library we're getting the development package for that pcre library as well and we're also getting lib ssl development package so we will install all of those alright so right now we've got a single folder and that is our engine X our TMP module folder I should have just done LS minus L there's our single engine X our TMP module folder we are also going to download the latest version of engine X with W get and at this time the latest version of engine X is 1 point 17.3 if you're following that website I mentioned earlier you're going to get 1 dot 10.1 we're going to attempt this with the latest version of engine X we will use tar to extract that engine X archive right there so LS minus L we can get rid of our tar ball now okay so we've got two folders we are going to CD into the engine X folder we are going to run the configure script we are going to pass in the options with HTTP SSL module and add dash module equals dot slash so this is going to be pointing into our engine X our TMP module folder looks like we do have to type it out there tab-completion doesn't work in this case but we will attempt to run that the HTTP gzip module requires the gzip library alright looks like that tutorial we were following missed one of the dependencies so we are going to sudo apt install Z Lib 1g and z Lib 1g dev okay and as you can see we already had Zeeland 1g but now we have the developer package as well which we need to compile this alright we did not get an error that time so we will run make and we will let that go alright and that is finished compiling so now we will run sudo make install cool looks like that's done and running sudo make install did install this to our actual system you can see we've got a user local Engine X directory now so we are going to make our engine X configuration file we will use Nano for that and we are going to go into slash user slash local Engine X conf and nginx kampf and as you can see there is a little bit of a default file here I'm actually going to move that current one from nginx kampf to user local nginx conf nginx kampf dot original and let me just move into that directory alright so as you can see we've currently got this directory with our comp default and comp dot original which are probably identical if we run an md5 sum on nginx comp default and we do the same thing on dot original yes those are identical so we just backed up that config file for no reason since it was already backed up no matter we are going to go into engine x.com making sure I spelled that correctly since I keep misspelling engine X now I'm gonna paste this example config in from the website hidden on principal characters we will remove now let's go up to the top of this and walk through it so we actually understand what we're setting up here so work our processes Auto this is basically saying it will automatically decide how many threads we want to use worker connections is how many connections a single thread can run at the same time now for our rtmp configuration this should look fairly familiar we are still listening on the regular rtmp port of 1935 the application is called show in this case but we could rename this to whatever we wanted for the restreaming server we used live for this one I'll leave it at show we're turning our live capability on and here is where we're turning on HLS now when I made my restreaming video back in I think it was 2016 HLS was not a feature in the rtmp module yet but it is built-in now so we turn HLS on this is the path that HLS is going to use to store its fragments we'll actually customize this because I don't think this directory exists here I'm going to call this path slash Engine X slash HLS and we want a trailing slash there according to our example at least HLS fragment is setting how many seconds each fragment will last so this is saying that every HLS fragment will contain three seconds of content we'll leave that for now the playlist length is how many fragments are going to be located in our playlist so 60 so we'll have three minutes where the content and the playlist to the given time now this deny play all as the comment says it turns off the ability for people to consume our stream in the rtmp format but that's fine because they're going to be watching in the HLS format rtmp real time media protocol web browsers have never been able to read this this has only ever been used to go from streaming encoders to streaming servers now back in the day before HTTP Live Streaming websites like Ustream and live stream required viewers to have Flash installed Adobe Flash because they used flash video players they converted this rtmp video into FLV video and sent it chunk at a time to the flash video player on the client now more recently in the last few years it's become much more common for the streaming server to instead convert the video from rtmp to HLS and it sends that HLS playlist to the web browser the web browser pulls the fragments one at a time and plays them back which of course is what's going to use most of our bandwidth but hopefully digital oceans data center can handle it now if we come down here this is an HTTP server I'm not so sure we even need this enabled because I don't want to use this server as the web server I only want to use it as the media server we are going to change this root directory to slash engine X and if we have trouble getting this to play at nerd in the street calm then we'll come back and revisit the HTTP server section if we don't have any trouble then we'll come back and disable this in a bit for right now we're gonna save that configuration file and we're going to start engine X and that for us is going to be slash user local engine X s been engine X and let's go ahead and normally this would just start in the background but I actually want to start it in the foreground for now so we will run - G daemon off and we've got an error a IO on is unsupported on this platform I think that was one of the web server configuration items a IO yes so AI o stands for asynchronous input and output once again this is under the HTTP server this is not part of our actual streaming server so I am going to comment it out and let's try starting engine X again and of course we did not make our engine X and nginx hls directories so we will make their engine X and make their engine X slash HLS and we will CH own engine X do we have a user called engine X that's cat out our etc' password file and grep for nginx that's not what the users called looks like we are using WWF or nginx so we will CH own - our wwww data engine X and so now if we do an LS - al on nginx you can see that both the slash and directory and the internet / HLS directory are both owned by WWD which is what nginx will run under on Ubuntu let's try running it again okay and we're not getting any errors so nginx should be running right now so definitely a few steps we went through there that we're not covered in the tutorial but that's why I'm making the video now let's open up OBS studio here I have not streamed on this computer since the nerd in the street 7-year anniversary stream earlier this summer let me turn studio mode off because we are working in a 1080p window here and that may change this to the webcam so you can see me now and boy is my microphone causing my shirt to Sag down there alright the next thing I'm gonna do is fix the focus on my webcam let me turn that autofocus off that was zoom not focus there we go and now that we've taken care of that we are going to stream to the IP address of our server so let me pull up digitalocean again and copy that we will go into the OBS settings and we are streaming here and our application name was show if you'll recall so I'm streaming to the IP address of our server / show and let's apply okay now I'm actually gonna open up a second terminal here because I do want to keep Engine X running in the foreground here I'm curious what we'll see when we start streaming there if we see anything at all but let me just ssh into that IP address one more time and we're going to go into that engine x / h HLS directory so right now if we look in here there are no files in here at all so let's see what happens in both of these terminals when we start streaming if we're able to start streaming i'm gonna click start streaming we have connected which is a good sign looks like we are currently sending data and nothing happened in our terminal here which is understandable however if we run LS al here you can see we now have a dot m3u8 playlist for HLS which is very cool now if I open up VLC right now and I attempt to playback I'll go media open network stream and I'm going to type in HTTP colon slash slash and this is gonna be 157 dot 245 dot 135 don't 171 / HL s / 3 m3u8 if I just click play I'm able to open and now VLC is crapping out Thank You VLC kill that let's take a look at our config one more time user local engine X config engine X comp so the HTTP server I guess is what we're actually trying to access the HLS media through so that's why we need that HTTP server included in the engine X config now if we just try and navigate to that path you'll see we get a 404 not found ah now as you can see this is not called stream m3u8 it's called test m3u8 did I write that somewhere in here no I didn't interesting but we will replace this path with test and 308 so now that's a valid file so let's open up VLC again I'll have to find out where we set that name of the playlist open network stream so HTTP 157 245 135 171 colon 8080 /h LS slash test dot m3u8 and if I start playing that Hey look at that took a little just a half a second to buffer there but at this point the framerate looks pretty good the lag is kind of up there and of course we don't have any audio because OBS isn't recording any audio but we do have a working stream going on right now so the final step would be to get this into an actual web site so to do that we're gonna head on over to live dot not stock Co and we're gonna go ahead and test right here on our live web site it's not like I'm streaming right now so even though this is a production web site I'm only editing this one page not like I'm going to mess anything up so I'm going to type in here my normal video player and we need to login to the nerd of the street web server and just check what version of video j/s we're using video Jas is the open source video player that you use if you're watching any of my videos on none of the street comm let me mute this this video right here in the webpage is running in video j/s with the logo at the top right and the quality and speed controls in the bottom right this is all powered by video j/s so here I am in the nerd in the street webserver and we're going to go to VAR w W HTML player and let me cat out video J s dots I think it's just video J s actually yes alright let's just cut out the top of video J s okay so I'm running version six point two point seven so we're just going to upload this file here this is video J s HTTP streaming j s for production you'd probably want to use Minjae s save a little bit of bandwidth there alright if I do another LS here you can see I have just uploaded video j s dash HTTP - streaming J s we already had HLS source selector actually because I use that for the automatic quality selection so maybe I didn't need this streaming plug-in at all alright let's do this first of all let's just try using the plugins I already had setup yeah cuz I've already got HLS built in I think I'm just over complicating things so let's set the source to HTTP colon slash slash the IP address of our digitalocean droplet paste that there colon 8080 /h LS slash test m3u8 and let's hit save on that the midi could not be loaded okay just double check one more time our address that was the correct address our type that we're declaring is correct so let's go ahead and include an additional script here and that is going to be our HTTP - streaming dot j s library there so we will click Save and the media could not be loaded let's take a look what's going on in our console here blocked loading mixed active content is this course oh no it's because we don't have HTTP on that webserver okay so yeah that this is an issue because we're currently on an HTTPS website I think Firefox is blocking the HTTP content so let's come back over to our nginx server we're just gonna set up cert bot real quick and for that we're going to native the main name so as you can see this project is ballooning a little bit but we can make this work so I'm going to log into my DNS provider which happens to be Linode and another two-factor authentication page and here is my Linode control page so we'll go to our domains here we'll go to nerd on the street comm here it is and we're going to add a record called we'll call it live stream dot nerd in the street comm and the IP address it should be the last time we have to type this anywhere is going to be our digitalocean droplet size ress and i am going to edit that and set it for five minutes TTL just in case we need to edit it and at this point i am just waiting for Linode to propagate that i have to sail in hodes dns servers are very slow at propagation I switched over to them because I thought they would be a little bit more robust than hovers built in DNS servers but yeah hover if I made a new entry like this and then I pinged it I would pretty much immediately get the IP address that I added Linode you can see I'm still pinging it and I'm still getting the main nerd of the street web server and I am not getting the actual IP address for that new entry yet so I'll be back in a couple of minutes after that propagates alright looks like that has finally gone through let me just confirm here with a ping and looks like I need to clear my DNS cache system CTO restart DNS mask there we go alright so as you can see live stream nerd the street comm now points to this new IP address which means we can come over here and we can install cert bot so we'll do a studio app install software properties common I have to spell properties rights and it was already installed to do a DAT repository universe I'll tell you guys I'm working with Ubuntu a lot in my current day job now which is great but I find I keep typing in apt commands into my arch install here which is kind of it's embarrassing even though nobody is around to see it we will add apt repository PPA cert bot cert bot enter and we are going to sudo apt install cert bot Python cert thought engine X so what we need to do is generate an SSL certificate which I've showed how to do before it's super easy with let's encrypt so we are going to stop streaming at this point and we are going to run sudo cert bot - - engine X and actually this isn't going to work I already know we'll type in our email address we'll agree to the terms we don't want to subscribe to the newsletter right now so as you can see no names are found in your configuration file it's actually going to let us continue on anyway so let's try to so live stream nerd on the street calm and waiting for verification cleaning up ok well that seems to have worked now if I come back over here and I stop engine X and I started again I don't know if the configuration file that cert bot put in our system is actually going to be read by our custom compiled version of engine X or not oh and I see it's absolutely not because cert bot pulled in Ubuntu's version of engine X so we're going to studio Apps purge Python cert bot engine X and we'll do sudo apt Auto remove I'm wondering so you have to remove engine X all right so yeah we don't want all of that that bun - engine X stuff we made our own engine X all right so let's edit our configuration file again nano engine X Kampf alright so we're going to make this server actually listen on port 443 SSL our server name is going to be live-streamed honored on the street comm our SSL certificate now I want to use the full pass for those I think so let me go to e TC let's encrypt live and live streamed into the street comm all right so our SSL certificate is going to be Etsy let's encrypt live livestream nerd on the street comm slash cert 1 dot PIM and then our SSL Certificates he is going to be the same thing except Prive key 1 dot pim and you know instead of just the certificate let's go ahead and point to our full chain 1 dot pim file and let's save this config and let's try starting engine x no such file or directory is this not a location oh it's just full chain dot him I see what I did this trips me up every time I try to configure let's encrypt manually these are the names of our links this is where they're pointing to there is no number at the name of the link okay so we'll come down here and we will take away the one from the end there both of those files we'll try running engine X and engine X is running ok now we're gonna come into OBS and can I make it so oh hey there we go so OBS can actually share the sound because we've got pulseaudio running on this computer so we are going to change this stream to point to live stream nerd on the street calm and course just like I said in the restraining video you'd want to change your application name to some long complicated string and use that as a sort of password because the Internet's rtmp module does not support authentication per se with stream keys but it's basically the same thing if you just make your application name something very complex so we will start streaming and that looks like it is going now if we come back into nerd on the street let's edit that source and it is going to now point to HTTP live stream nerd on the street comm / HLS slash test hey and look at this look at this it is going to now point to that is actually not as bad of lag as I thought it would have and I've got other plugins here on my industry my website slash h LS that how good is this so we've now got a live stream going look this that is actually not as bad of lag as I thought it was so we do have a maybe 30 seconds of lag I'd say it's it's not as bad as like twitch there it's about the same as twitch it's not as good as mixers FTL streaming obviously but compared to something like YouTube streaming or dailymotion it's certainly acceptable I mean look at how good quality that is I can why I can't really zoom in because my website is is designed not to do that but yeah I mean I can I can full stream the video here obviously I'm on a 4k screen and you're watching a 1080p clip out of it so you can't see the entire screen right now but this quality is pretty darn good the quality is affected by you know exactly what I'm streaming out of OBS but yeah so at this point I'd be interested to see how many people can watch this at the same time before it starts to throttle out my network connectivity here if I take a look at the digital ocean panel here our bandwidth our public bandwidth we are currently writing between one and a half and three megabits per second which isn't bad and then what are we reading is when I'm more interested in as soon as that updates looks like the the digital ocean panel is a little bit behind we'll wait for that to catch up in the meantime we can install if' top interface top and we can run that and as you can see this is currently this this is me I've got Comcast internet so it looks like we're sending about you know the the normal 2 megabits per second from the stream server to the person watching my stream interesting and then we've also got 2 megabits per second coming from that same IP address so this is me streaming in and then this is the stream getting sent out very interesting so yeah I would be very interested to see obviously nirn of the street doesn't do live streaming a whole lot and even if I try to do a stream I probably wouldn't be able to get that many people to watch a test stream anyway but yeah I'd be interested to see how many people we could push that up to digital ocean is in a data center and they've got great network connectivity Linode would probably have just as good if not better connectivity to ISPs in the United States so yeah with my gigabit internet I'm having no trouble at all both streaming and watching the stream obviously other people need to be able to watch the stream too so I'd be interested to see if somebody on a slower connection would be able to watch this and if it would still be smooth but for now that was really easy that way you know all things considered it took me about an hour and 15 minutes to get my own streaming server set up and you know now that I've done this once I've never done this before ever I've always said it wasn't worth the time to set that up but now that I've done it I think none of the street might want to start streaming to our web site with our own video player like this because we already have our own chatroom I don't know where the chatroom is that right now I think I broke it with my might changed embed here I'll have to play around with my website to see you know what I need to do that didn't do it that just made the video take up the full width yeah what I need to do to get the chat to show up but yeah seriously guys it is getting a little ridiculous all of these things that streaming websites are making people jump through and they say it's for safety and they say it's for preventing harassment and they say it's for all this different stuff but really why would you not want to have a server that you control yourself no the technology is there we have these open-source tools that we can set up in less than an hour if you know what you're doing the technology is here and it allows us to have servers that we controlled ourselves as individuals we don't have to be pandering to these West Coast's big technology companies you know we don't have to care about what Google thinks of us we don't have to care about what Microsoft thinks of us and this is coming from a relatively family-friendly you know content creator but obviously I care a lot about the individual power to express yourself on the Internet so seriously why would you not set this up at the very least set this up and use it with the restreamer that I showed people how to make you can keep streaming to mixer and twitch and whatever else you want to stream to but those websites are going to screw you over and over again we see them screwing people over and over again and even when it's not on purpose like mixers Terms of Service thing which is absolutely on purpose you know it's it's it's these websites making stupid mistakes and you know oh we tried to do a database migration and we crashed a website for a day and it's affecting your content you know if nerd of the street comm goes down it's nobody's fault but my own or possibly Lynn ODEs but that hasn't happened yet and I would much rather put my faith in Linode or digitalocean than in twitch or mixer so if you're a streamer make a streaming server grab a domain name give your viewers the option to watch your content without being on anybody else's platform it will benefit you in the long run for now that's everything I had to talk about if any of you have questions about anything I talked about feel free to go to the forums at nerd the street comm I've answered a lot of questions about using the rtmp module before and I'd be happy to answer more now and also if you want more tutorials like this showing people how to do things on Linux themselves please consider joining the nerd Club at nerd club not schoo it's just three dollars a month and it goes a long way but for now that's it I hope this was helpful I'm Jacob califor and the end of the street and I'll see you guys in the next one bye [Music]

Channel: Nerd on the Street

Views: 118,707

Keywords: nerdonthestreet, jacob, kauffmann, jacobgkau, HLS, streaming server, RTMP, NGINX, server, streaming, live stream, video, Linux, setup, how to, set up, configure, website, embed, tutorial, guide, live, flash, http, html, html5

Id: Y-9kVF6bWr4

Length: 34min 49sec (2089 seconds)

Published: Fri Sep 06 2019