DevOps Lab Project - Learn to be a DevOps Engineer through this Practical Lab Project

today i'm going to take you through a practical devops project that's going to teach you the tools that you need to know as a devops engineer this video is not only going to show you the tools but give you valuable insight on the tasks that a devops engineer takes on on a day-to-day basis by the end of this video you're going to learn how to use vagrant ansible docker and docker swarm to create your own devops lab we first start our project by provisioning four linux servers using vagrant once the servers are provisioned we are going to use ansible to create and run a playbook that will automate the configuration of the servers after the servers are configured we will containerize a simple web application finally we will convert our servers to docker swarm nodes and bring up the application on the swarm this is the perfect starter project for learning devops and it's something you can take further and expand on as well let's go ahead and get started [Music] all right let's go ahead and get started with the first practical section of this project which is setting up vagrant virtualbox and then using vagrant to bring up our servers now you don't have to follow along and do this yourself but i have all the commands available in github in the description below so if you want to use that and follow along that's great but if you want to just watch and see what a devops engineer does that's fine too either way you're going to learn a lot so let's go ahead and get started and the first thing we want to do is install vagrant so again i have all the instructions in the github below with the links to these websites so the first thing you want to do is go on over to the vagrant installation website and then go to the vagrant downloads page once you do that just select the operating system that you're interested in for windows you just download a binary and install it and for linux there's a couple commands that you're going to put in once you have vagrant installed the next part is to install your provider so there's a lot of different hypervisor providers that you can choose from but the one we're going to use in this lab is virtualbox so under the vagrant documentation you can see all the different providers that vagrant allows you to use you can see that we have vmware docker hyper-v and virtualbox so we're going to use virtualbox for this lab so just go on to the top and then go to this list and it's going to show you the compatible versions of virtualbox that you can use so you want to make sure that you're using the right version of virtualbox to get all this working so we'll go here and you'll want to download virtualbox 6.1 or whatever the latest version is this lab should work the same no matter which version of virtualbox that you use so once you have vagrant and virtualbox installed you can check virtualbox by going up here and opening it up and i have a clean slate environment here so let's go ahead and get started with vagrant to provision some virtual machines so we're going to have four different servers the first one is going to be our control workstation server which we're going to install ansible on and we're going to use that server to push out configurations to our three other servers which are going to be the nodes in our docker swarm cluster so let's go ahead and get started i'm going to pull up a terminal here and we're going to clone the repository and get started with vagrant okay we're back in github here and we want to grab this url and go to our terminal and we want to clone it down so clone it and then we're going to hop into that directory and let's open up our code editor i'm using visual studio code and you can see that there's a couple files here we have a readme which i will fill out later for you guys so it's going to have some instructions there for you and then we have a host file this is the host to ip address mappings we're going to copy this over to our control workstation just so it can have name resolution to the other servers and the last file we have here is our vagrant file this is your first introduction to infrastructure as code and you can see here that it's pretty easy to read and sort of understand what's going on here basically i'm provisioning three servers in this configuration and then setting some settings like uh host name uh the image name ip address and then the port that we're going to ssh on you can see that it's not using 22 you can see that i'm doing 2 2 0 0 and then incrementing it up this is because they're all going to sit on the same box so it's nice to have them on different ports underneath here we have some additional configuration this is basically just a for loop saying for each of these servers and then it goes in and it sets some additional settings you can see that i'm setting the number of cpus and the amount of memory so if you want to you can just make some changes to these settings to customize the servers as you need to now having a look at this you can see that there's only three servers but we want four we want the control server and then three nodes so if i want to make an additional server all i need to do is take this copy it and paste it down there make sure to include this comma and that should be good now we need to just make the settings unique so let's make it have a unique ssh port and then the ip address will go 53. so one thing about these ip addresses i didn't just pick them randomly i went into virtualbox and i went to file host network manager and it pulls up all the virtualbox ethernet addresses and i just made sure that they are in one of these subnets so you can see that i'm going to be using virtual adapter 6 for this project and i'm making sure that they're in this 172.16.1.1 subnet so if you want to customize this you can go ahead and do that and use your own specific ip address range so i just wanted to show that quickly let's head on back into vagrant okay so we're back in the vagrant file and i can see that i didn't change this hostname so let's change that to node 3. and now we're ready to begin provisioning so i'm gonna go into our shell here and let's clear the screen and do an ls and you can see the vagrant file right there so now i'm ready to bring up my virtual machines using vagrant so the command to do that is vagrant up so that should be enough to bring up our virtual machines as they're coming up we should see them being created in virtualbox here it's going to take a few minutes so i'm going to speed it up but before i do that i'm going to show you guys a couple commands that you can reference in the github i have a link to this in the github so let's go here and this is the link that i'm going to give you guys it's a vagrant cheat sheet and it's basically all the commands you're going to need to use vagrant so this cheat sheet shows you basically everything from how to create vms starting them up logging into them stopping and cleaning them up so it's a great reference if you haven't used vagrant for some time you just pull up this cheat sheet and it gives you all the commands that you need i also have a good playlist on vagrant so if you want to learn more about using vagrant and all the ins and outs all the commands and all the nuances of it go ahead and check out the video alright so let's head on back in you can see that we got about three virtual machines coming up here and uh note 2 is being provisioned so it's probably going to be another minute or so one thing that i'll mention here is if you run into any errors just leave them in the comments below or join our discord and we'll try to help you out but usually just going out to google and copy and pasting the error there you're going to find a solution for it there might be a few things specific to your workstation so just google there and you'll probably be able to find an article that can help you out all right so vagrant has finished provisioning our virtual machines let's hop into virtualbox here and you can see that the four virtual machines have been created if we go into visual studio code you can see this hidden directory basically this is vagrant storing the state of the virtual machine so it's good to know that it's there another thing to mention is everything in this folder where the vagrant file is is going to be shared to the vagrant hosts and i'll show you how to access that directory because it's going to be important to know how to access these files the first thing we're going to do once we get into our control workstation is copy this host file over to the right location so let's go ahead and do that right now so back in the terminal let's clear this screen and to ssh into our machine we just go vagrant ssh and since we have multiple vagrant hosts running within this directory we need to specify the name so we're going to go control which is the name of our virtual machine and it took a second there but it's up if it asks you for a password the default password for the vagrant account is vagrant so make sure to use that if it's prompting you for a password now that we're in let's hop on over to the directory called slash vagrant and if we do an ls-la here we can see that all the files match up exactly with what's in our host directory so the files are shared between your host and your vagrant servers [Music] okay so now that all our servers are up let's use ansible to push out configuration to them so the first thing to get ansible working is we need to be able to connect to those workstations from our control station and to make ssh work we need ipconnectivity and it's nice to have name resolution as well so if i were to go ping node 1 you can see that name resolution isn't working and that's because the act slash host file doesn't have any information on these hosts so i created this host file right here that has all the ip information of the host so let's copy it over to our control workstations host file so i'll just go copy hosts to it posts and i probably need to do a sudo to do this as root so let's do that and now when i go ping node one you can see that name resolution is working and ipconnectivity is there okay so once you have ipconnectivity and name resolution working the next thing you want to do is make sure that ssh connectivity works so i think in our setup that ssh connectivity should be there by default but i'm also going to show you the commands you can run to create your own ssh key and push it out to the servers as this is probably going to be a likely scenario that you run into so first of all let's just test to see if ssh works so go ssh vagrant at node one vagrant for the password and you can see you can get in so let's exit out uh clear the screen and head on over to github and in here underneath the solution underneath ansible i have a couple commands to make the host ssh accessible so you can run these commands and i'll just run through them as well so the first one is to create an ssh key pair and you can see that it's saving it at this location and then the next step is to actually copy over that ssh key to the servers so we do this hit yes put in the password for each one and now when we go ssh vagrant at node 1 you can see it logs in directly and it doesn't request a password so that's the benefit of using an ssh key file it's not going to prompt you for a password when you try to ssh into the workstation which would be very inconvenient when you're running an ansible playbook you don't really want to be putting in passwords when you run the playbook you want to just run it and it goes out and pushes everything out okay so once you have ssh connectivity to your other servers we can start using ansible so let's go into visual studio code and get started with ansible so i'm going to create a new directory here and i will call it ansible and then within that folder i'm going to create some new files so i'm going to call one my hosts and this is going to be our host inventory file and then i'm going to create another one called playbook one now let's go ahead and start populating these files so let's go into my hosts and i'm going to go to my github and you can see i have this file right here and i'm just going to copy it and paste it in so this is a very simple ansible host file so just to provide a little more information on this ansible has two different formats you can have inventory files in you can have it in this basic way that i'm doing it here or you can have a yaml format so let's just start off in this basic way and when you want to sort of learn more about ansible and how you can change this to a yaml file to get more customization you can start to customize it later but this is enough to get us started and push out configuration from the control station over to the nodes so just to describe quickly how i have this set up i have this section here called control and then this one called nodes there's also magic words in ansible that would refer to everything and that would be like the keyword all and you could push configuration to everything within this file so this one's good because i'm either going to push out configuration to just the control station or i'm going to push out information to just the nodes so this is a pretty simple project with only four servers so there's not much to this but you could organize this any way that your enterprise organization wants it to be right so you might do something like web servers and do like webo1 web002 and then like database servers and like db01 db02 there's also ways to combine groups within groups but that's sort of out of scope of what we're doing here if you want to learn more about the inventory file i have a link up above that you can go out and check out my ansible course and it'll teach you everything that you need to know about the inventory file so let's go ahead and clear this out and move into our playbook so the playbook file in ansible is a pretty special file it's basically everything ansible is going to do when you run the ansible playbook command so let's go ahead and take this and go into our playbook and i'm just going to copy this over because i think just explaining it is going to make it pretty obvious what's happening here so in this playbook you can see that it's referencing the nodes group so all the servers within this nodes group node 1 2 and 3 are going to do these tasks so let's have a look at these tasks here we have three different tasks you can see each of these tasks have a name and these are just the names that i put in there so i said for the first task ensure that docker is installed so how would you install docker on a workstation by itself well if i were to install docker on a station i would go sudo apt update and then once my cache is updated i would do a sudo apt install docker dot io and then after that i would do a pseudo docker install docker dash compose and then after that i would add a user to the docker group so instead of doing that manually on all our hosts we're going to use ansible to do it so let's have a look back here so we have the docker.io we have docker compose and then we have the user vagrant being added to the group docker now one more keyword that i like to add to these app commands is this update cache yes and basically this is equivalent to doing a pseudo apt update so we have these three commands let's go ahead and run them but before i run them let's make sure that basic ansible connectivity is working okay so before we run the playbook i actually want to test just basic ansible connectivity without running the playbook and just make sure that all the hosts can respond to what we call an ad hoc ansible command so to run an ansible ad hoc command just go ansible and then specify your group now our groups here are control or nodes and that group is specified in the inventory file so we'll go nodes and then we're going to go dash i to specify the inventory file that we're using so we'll go my hosts and the reason we specified this is you could have multiple inventory files like you could have one for qa production staging whatever you need right so that's why you might have multiple inventory files if we go dash m this means module and we want to run the command module and then we're just going to go hostname so basically what this command is going to do is it's going to talk to all the hosts within the group nodes using this inventory file and it's going to run the command hostname so all the host should return their host name and you can see that it doesn't run unless you install ansible so let's go ahead and install ansible so i'll go with sudo apt install ansible and hit yes to that and this will just take one minute okay so it looks like ads will installed let's rerun this command and we got an error and couple things wrong here so let's clear the screen we're in the wrong directory we need to be in the ansible directory and the reason for that is that's where our host inventory file is so in the command that we ran it's specifying my hosts so if you're doing that you need to specify the qualified path so it's just easiest to be within the ansible directory so you can just specify it like that the next thing that we got wrong here is i actually need to specify dash a and this is for argument so it's the command module and it's specifying the argument of hostname so let's go ahead and try this and that looks a lot better so you can see node three came back node three node one node one and then node two node two the next thing you wanna do is run this other command from my github which installs python and this just gives your ansible hosts a lot more functionality so let's copy this and paste it in and i didn't copy that so let's try it again paste that in and this is installing python simple.json again this is just going to give your ansible hosts more functionality for your playbooks and it's just something i do whenever i'm working with ansible for the first time so all these hosts have successfully installed python simple.json let's go ahead and run our playbook so clear the screen once more and then to run it we run the ansible playbook command followed by specifying our hosts inventory file and then our playbook so let's take this put it in and it wants the sudoers password and the reason it's asking for this is in this playbook it says has this argument become yes so basically what this argument says is become the root user and if you want to become the root user you got to give the root user password so we'll go ahead and put that in and now you can see that it is running through the ansible playbook and the first thing that you'll notice here is it's gathering facts facts are exactly what they sound like they're just facts about the hosts so you can use facts on the host to make logical decisions of which software should be installed things like that it's sort of a little more advanced topic when getting into ansible and i explain it all in my andable course so if you haven't checked that out already go ahead and check that out so that's our first play in our playbook is ensure docker is installed after that it's going to ensure docker is installed let's call this docker compose that makes a little more sense okay so next thing is it's going to ensure docker composing installed and then it's going to add the user to the docker group so i'm going to go ahead and speed this up and have a look at the end results okay so it looks like our playbook has successfully ran here all the tasks have finished if you're following along i hope your playbook has ran successfully as well if you ran into any errors i would just take your errors and pop them into google and see if anything comes up a lot of the time the issue is connectivity or maybe the playbook you have has some differences than the one i have so just make sure to check your syntax and make sure your playbook looks sound so now that we have docker installed let's just go on to one of these nodes and just verify that docker is working so let's ssh into node one and uh let's go docker and you can see that docker is working it's giving us back a list of commands so that's how easy it is to push out software using ansible and you could take this playbook and add as much logic as you want to you could basically set up anything that you could manually using ansible it just takes practice and work to get these playbooks working there's also things like ansible roles that really help you out so make sure to look into all of that if you want to learn more about configuration management for this video i just sort of wanted to introduce it as a concept so you'd be familiar enough to go out and take it further to get your servers configured the way you need them to [Music] so that brings us to the third section of this video which is docker and docker is a way to containerize applications and it's one of the most important things that you need to learn in devops or just it in general basically every service or application is getting containerized now it's very infrequent that you go into a server and manually install software on it there's usually some sort of containerization so knowing about containers is really a must for devops engineers so to get started let's create a new folder and we'll call it docker so once we have this docker folder created let's uh add a new file here and we'll just call it docker file and uh one more thing i'd like to show just to test out docker and make sure it's working you can go docker run hello dash world and this is going to go out and download the hello world image from docker and then uh run it and return hello world so that's another way to test docker just thought i would show it it's just a good thing to do to verify docker is working so going back into visual studio code let's create a docker file so i have one in my github here so basically what a docker file is is it's a way to create a docker file this is sort of the infrastructure as code portion of docker so let's just copy this over and paste it in and i'll explain how it works so you can look up all this santax on the docker website but just to break it down simply the from section is the image that we're using so we're using a python image running on alpine alpine's just a lightweight linux os and then we're setting the work directory to slash code and then we're using flask flask is a lightweight web framework it's setting some environment variables that flask is going to use it's making sure that some software is going to be installed it's going to copy over this requirements.txt file and then it's going to do a pip install requirements.txt so requirements.txt is a file used in python applications to specify the modules that they need so basically this is taking the file copying it over to the container and then using pip to install those requirements the last few things here is expose 5000 it's exposing the port 5000 to the host container so you'll be able to access the flask web server if you do like a curl on this port and then this is just saying copy all the files from the current directory to uh i think this code directory and then the command is to run flask run so this is all we need in our docker file the next thing we should do is uh create this app.pi file and the requirements.txt let's actually have a look at the github and you can just copy over these files or create them yourself but we want this one and then we'll create it in our darker folder and then the requirements.txt we actually only need flask so i'll just create that go new file requirements.txt and we'll go flask so really simple flask application here if you don't know python that's fine basically it's saying import flask import socket and it's just running a super lightweight web server that returns hello world and then the host name so now that we have our docker file which specifies to run our python application let's go ahead and use docker compose and docker compose is just a convenient way to specify how our container should be built and run so go new file docker compose.yaml and you can see it's a very simple file here basically docker compose files specify services so you could have multiple services here our application is really simple it just contains a web service and this is basically everything about the web service it's saying build the web service based on the docker file and expose port 5000 if your application required multiple services in it you could add them here and do something like it would look like something like this right you would have like database and then i think you would use image and you go like mongodb right and that that would be like sort of uh oh the gotta make sure the alignment's right here so this would be like sort of like a traditional web database application uh you have your web server and then your database server using mongodb and the two could talk to each other but we're just going to keep it simple for now uh we're going to take this out and just make sure that our basic web server works so to test that out we're going to open up our terminal clear the screen and check out our commands so the first command is to do docker compose up and we need to do that in the vagrant slash docker directory because this is where our docker file is and our docker compose file so i'll run that and it's first it's doing the build of the image so it's building up our image and then it's going to run the container and again if you're new to docker just try to follow along if you can and if you want to learn more information about it i have a course up above so it looks like our web server came up you can see that it is running on this ip address this port let's uh pull up a new terminal and i need to go to my code uh projects devops project and then this is where my vagrant file is so i'll go vagrant control and i'm just oh i need to ssh into control and the reason i'm doing this is i want a separate connection to test my web server so if i clear this and go curl node 1 5000 and you can see it says hello world i am and then it gives you a long string so let's think about this for a second so we have a container running on node one you can see the request came in right here saying get http 200 okay so it all looks good but what you might be wondering is why the host name came up as this now to explain that i will go into node one and if we do a docker ps and resize that you can see that is the container name so i hope that makes sense basically the application is running flask within the container and then when the flask application asks for the host name the flask application is running in the container so returns the container name [Music] now let's go ahead and get this container running on all three servers now the best way to do that is to use an orchestrator so we're going to be using docker swarm but there's other orchestrators that you can use just the same and the most popular one out there you've all probably heard of is kubernetes so you can do this just the same on kubernetes kubernetes is a little more difficult to get into but surprise surprise i do have a course on it that is about an hour and a half two hours long so if you want to watch that right now or after it's a good way to get into kubernetes and you can take that and move that application to kubernetes or you can follow along here and use docker swarm and docker swarm is a really good way to learn orchestration uh it's really simple so let's go ahead and get darker swarm set up on all three nodes so if we want to get docker swarms set up we could go into each of our nodes manually and install it and configure it but instead of doing that let's use ansible to set up and install docker swarm so to get docker swarm installed we're going to go out to github and grab a playbook for it so i have one here which is devops journey answers form playbook and you can see that i forked this directory from someone else's that i found out on the internet so one thing as a devops engineer is you're not going to be writing everything yourself you're going to be using the open source community to find other projects and find what other people did to solve problems and use their work when possible it's going to save you a lot of time so we fork this directory and you can see that there's quite a bit to this playbook but if you have a look at it basically it sets up ansible swarm it sets up a master and then it sets up the workers so we'll probably have to do just a little bit of customization which is probably just going to be the inventory file we probably need a group named worker and then one named manager and that should be it so let's take this go back into our control station and go get clone and then throw in the directory it is cloned it out if we go in there do an ls you can see all the files there let's uh nano my hosts and i am going to grab my one from ansible here paste it in and then there was those two groups that we had which was a worker and manager so we'll go manager worker and so manager will make node 1 and then worker will make node 2 and node three so go to node one node two oops and node three and i think this should be enough let's go ahead and try to run the playbook and if we run into any issues we'll just troubleshoot them and see what we can do so let's go ansible playbook and then specify our inventory file which is my host and then uh it was called swarm.eml put in the password so it looks like we have an error here node one installed okay node two and node three did not let's have a look at their playbook again and see it does mention see this i p address doesn't look great we go ifconfig 10 so this is the ip address we want to use right what we want to use is subnet we want to use this subnet on for node 1 node 2 node 3. i guess this is the first interface we don't really care about this ip so there's something about the playbook using f0 we want it to be using ethernet one so let's go here and just look so you can see it's hard-coded s0s0s0 so let's change that in the playbook and see if it fixes our issue so i'll go in here docker or it was ah i don't have it in here i was running it from the root directory so i don't have access to it so let's actually take this go to vagrant this way i'll be able to access it in my visual studio code so get clone reclone this oh take that stuff out there we go cloned over swarm and then i have to recreate that host file actually let's uh copy it over we've already created it so let's just copy it over my hosts to my house okay there it is and do we have that old oh before we before we run the command let's actually go into visual studio code and make that change that i was mentioning ansible swarm playbook let's go into swarm.yaml and then f.0 you can do a find replace at all but i'm just going to do this there's only four of them okay so changing this it's going to be using f1 which is uh our 172 subnet which uh swarm is installed on so rerun this vagrant and this should go out and rerun it and we'll see if we get the same error all right so it looks like it finished okay and that last step is successful so it joined the cluster so we have a full docker swarm set up here let's go into one of the nodes and just verify that docker swarm is set up so i'll go ssh node one and to verify it we'll go docker node ls and there you go so we have our three nodes in the cluster node one node two node three all the statuses are ready and this one's the leader so the leader is basically the manager and then the workers are node two and node three so that's pretty convenient and imagine if you had like a twenty node cluster how easy that is to just get docker swarm installed on all the nodes and set up and everyone joined to the cluster so that's that just shows you the power of ansible and how great it is okay so let's move forward here i'm going to go back into that vagrant docker directory and if i do docker ps you can see i'm still running this container and this is running through docker compose created a container i want to take down this container and have it running on the swarm now so if i just go docker compose down this should see the docker compose file and then just kill all the containers related to it so make sure you're in this specific directory when you run this command otherwise it's not going to work so you can see that it removed everything about that container to docker ps there's no containers running so now we're ready to move forward so one convenient part of docker swarm is it can read your docker compose files and launch the container and have it managed by docker swarm using that compose file so to do that we just need to go docker stack deploy and then we specify a compose file and then uh we'll say docker compose and then we need to give this service a name so i'm just going to call it my app and then we get an error and it's saying the image reference must be provided so there's one change that we actually need to make in our docker compose file to make this work and i'll show that here if we go into docker docker compose we have this build parameter so docker swarm doesn't want to build the images it wants you to specify an already built image so i'm going to go like this image and then i'm gonna specify this now to explain this this is my actual docker hub so this is going out to docker hub and it's pulling down my image so you guys can use this same image name and run this as well and it should work for you this is a public repository so it's going to go out and uh grab my image that i've already built oh and make sure to take that out and go back into the command prompt and we should be ready to bring our service up so let's rerun this command and it looks like it is working so let's do a couple verification steps so if we go docker stack ls you can see that my app is running the orchestrator is swarm and it's just a single service application if we want more information about this service we go docker stack services my app and you can see that it's running it's replicated and it's a single replica so you can have multiple replicas and that's what we're going to do here we're going to scale up our application to three nodes so if we want to scale up our application we can do that very easily with docker swarm by going docker service scale and then specifying this specific name which is my app and then the service name or my app underscore the service name and then uh equals and the amount that we want to scale it to so let's uh let's do it to six actually because we're not limited to just three we could have as many instances of this application running on our nodes as we want to it's you're only really limited to the amount of cpu and ram that you have so now that we have that let's rerun these commands and you can see replicas is six of six so we have six instances of my app underscore web running and if we want specifics of this we go docker service ps my app underscore web and there you go you can see two instances on node two two instances on node one and two instances on node three so it's getting load bounce nicely there now if you want to you can go back to your control station and then do curl node 1 0 0 0 or port 5000 i mean right and you can see that you're getting the two different host names right because it's load balancing on node one between the two different containers uh this one and this one and if you change the name node two node two does the same the load balancing between the two containers within it and then node three it's load balancing there as well now if you wanted to load balance between all three nodes you could do that with a load balancer you could use something like engine x you could use dns load balancing or you could use a docker swarm service load balancer there's quite a few different options for that so in this video we learned how to provision configure and orchestrate multiple servers to run a containerized application in future videos i will show you how you can use terraform to move this lab onto your favorite cloud provider but until then if you're interested in learning more about devops check out the other videos on my channel thanks again for watching and i'll see you all in the next video

Channel: DevOps Journey

Views: 5,232

Rating: 5 out of 5

Keywords:

Id: YuZ002YrvUA

Length: 48min 32sec (2912 seconds)

Published: Fri Sep 17 2021