Real Time DevOps Project | Use Terraform Jenkins AWS to deploy REST API

in this session of a devops we will be building a devops project from the scratch using terraform genkins on AWS Cloud let's break down this whole project into three category and the first category I'm talking about is the tools so we will start by using the terraform second we'll also use the genkins for building our cicd Pipeline and aw is going to be our preferred Cloud option the second category I'm talking about is the application and for the application we will be building a rest based web application where we will be having some rest end points and those rest end point application will be able to store the data onto the database that will be our RDS instance using MySQL and this whole application is going to be our python flask application the third category I'm talking about is the AWS component and the services but before we set up our devops project we need to know the flow of our whole project on a high level so for that we will be using a domain which is my own custom domain which I have already purchased which is j.org and this domain we'll be using to access our rest API Services which is deployed onto our ec2 instance and that ec2 instance will be able to store the data onto our RDS instance which in our case is MySQL so this is going to be our flow for this particular project but to implement this particular request flow of this particular DeVos project we need to enable certain component and the services of AWS so let's take a look onto the components individually so first we'll also work on the Route 53 which is a AWS manage service for managing the domain request second we need to set up the whole networking for this particular project and for that we will be using the VPC then we'll be using the subnet then we'll also try to set up our internet gateway then we need to set up our route table and at last we need to set up our application load balancer for our ec2 instances one side note for you for this particular project this whole implementation is very generic and although we are using the python based flask rest API but you can just replace that whole component with nodejs or any other application and rest of the infrastructure set of C ICD pipeline will still remain the same let's jump to the demo and before we begin I have divided the whole project into three different GitHub repository so the first GitHub repository I'm talking about is the python application which is our flas based thr API so that is going to be our first GitHub repository second GitHub repository I'm talking about is the terraform and AWS so our whole infrastructure provisioning will happen with the terraform so all the infrastructure setup we will be doing using our terraform as our preferred tool and the third GitHub repository I'm talking about is the Chan kins where we will be setting up our cicd pipeline using the chanin I will also share the GitHub repository link into the description section but here this is my first GitHub repository for my python application which is a restbed application using the flask this is the second repository where I'll be storing my terraform code for provisioning the whole AWS infrastructure here you can see all the certificate manager e to hosted Zone all the networking and the infrar related modules third is our terraform genkins GitHub repository which will help us to set up our Jenkins onto our AWS Cloud if it sounds overwhelming at this point of a time then I would highly suggest to pause this video at this point of a time and take a look onto this particular slide where I have I highlighted like what are the different component I will be working in these three particular repositories all right let's move ahead and the first component we will be setting is our Jenkins and if you move to the next slide then this is the jenin setup we need to do and for any setup we will be performing uh throughout this devops project we will be setting up our VPC so for Jenkins the first thing we need to set up is the VPC so here you can see uh this is the first boundary where you can see our AWS account but in inside that AWS account you will see a VPC and in the VPC you will see a IP range so VPC is going to be my virtual private Data Center and in in this VPC we will be setting up all the components which relates to our genkins now we need to set up the VPC so I would recommend you to just clone this particular GitHub repository and here you can see onto my uh ID I have already cloned that particular repository and in this repository there are certain modules which I have already created but these modules we will go one by one and we'll we are just start enabling those modules one by one so the first thing which you need to take a look onto this particular GitHub repository is the main.tf file which I have highlighted over here so just open this main.tf file over here and here you will see a quite a lot of module there is a networking there is a security group uh there is a Jenkins module which will install the Jenkins there is a load balancer Target group then there will be an application load balancer and at last you will see a hosted zone for domain as well as for the certificate manager for managing our certificates but we are just going to Target each module one by one so the first first thing is we need to focus on our networking so first thing which you need to set up is the VPC so for that what I'll do I'll just disable all these modules so I'll just comment uh all rest of the modules so that we only know like how to set up our VPC for Jenkins so I have commented all this section so let's take a look onto this networking module and I would highly recommend to install the terraform onto your local machine while working with this particular repository so here uh this is the networking module as I have uh named it so in the same GitHub repository you'll find a networking uh folder over here so this is the networking folder where I have placed the all the code relates to my networking so just open this one and there will you will again find a main.tf file uh that is going to be our modules main file so just open this one and here uh you will find a resource which is like a AWS VPC so if you take a look onto our slide so here you can see this is the VPC which we are trying to set up and when you're using a terraform then this is the code block which you need to use for creating your VPC so the resource name is AWS VPC and this is a custom name which I have kept it which is like a Dev project one and VPC EU Central I think it's not a EU Central I have uh put it a wrong name over here but it is a EU West one so just just keep in mind so this is the resource and this is the custom name it doesn't matter just keep any name of your choice and here this is the CER block this is going to be my IP range for my particular VPC so if you take a look onto this slide once again so the IP range is 11.0.0 do6 so this is going to be the IP range of my VPC and here I'm passing it as a variable so here you can see VPC cider and if you take a look so this is the variable which I have been passing to this particular module and if you take a look onto the main.tf then here we will be passing that particular value from here but here we have declared it as a variable and where to find this value so if you go into the terraform tf. Wars then here you will find the VPC CER and this is the IP range of my particular VPC so this is the first resource block for creating your VPC now we have seen how to set up the VPC and the terraform code but before we provision the VPC there are two more components which we need to take a look so one is public subnet and one is the private subnet so why we need a public subnet and private subnet so so any resource which need a internet access or which can be accessed from a internet that will reside inside our public subnet and any resource which doesn't need an internet access and that can only be accessed from internally from public subnet then that will go into our private subnet so what's the example of our private subnet resources so for example databases so databases are only accessed from our application so our application will resite into our public subnet and those will be able to communicate to over private subnet so whenever you work with the networking so you need to set up your public as well as the private subnet within the VPC let's take a look onto the public and the private subnet infrastructure code inside our networking module so like we have previously seen the VPC code so here you can see the subnet code also so this is the block which I have created for creating the subnet and this is going to be my public subnet and similarly this is going to be my private subnet so here uh there are certain things which you need to keep in mind so here this is the count so here I have created a account because into the terraform variables if you'll see the value then this is the public subnet IP range so here I have specified the IP range which Falls within the VPC IP range so here you will see this is a 11.0.0 do0 and here you will find the public uh subnet IP range is like 11. 0.1.0 and here the second IP range for the second public subnet is the 11.0.2 do0 and the subnets are for the the providing the maximum availability so this is the IP range which I'm using for public subnet and this is the IP range I'm using it for the private subnet and these values are in a array or you can say a list so which contains two IP ranges so that's why if you go to the networking main.tf file once again so this is the count is for that particular reason so it will iterate on those values uh based on the public subnet and the private subnet IP ranges and then it will just create those subnet but as you have seen in the diagram these subnets like the uh sorry the public subnet and the private subnet reside within the VPC so we need to make a reference to our VPC so first of all here I have chosen the VPC ID and that VPC ID which you can see this is the VPC which we have just created so first we have created the VPC then we are going to create a subnet which is going to be a public subnet but we need to make a reference that in which VPC it is going to be created cre it so here we are making a reference to that particular VPC ID and here you will see the name so which is pointing to the same VPC ID all right now we know the uh VPC where it belongs then we need to specify the IP range so here you can see uh here I'm itating and fetching the value of my uh IP range for the public subnet all right then second here is the availability Zone over here so here I'm passing the value since I'm working into for Jenkins I'll be working inside my Europe west region so here I will be specifying the availability Zone as a variable so if you go to the main.tf so from where we'll we are sending the Valu so here you can see the EU availability Zone and in the TF tfws file you will find the availability Zone uh here you can see uh line number seven so EU West 1 a and EU West 1B so these are the availability zone for my Jenkins now we have seen the public subnet code then let's take a look onto the private subnet code as well so here the code is pretty much exactly similar the only thing which I'm changing over here is the private subnet IP ranges uh the uh IP range which I have assigned for my private subnet and rest of the details are exactly same for my private subnet let's try to provision or let's try to set up at least this infra which includes VPC and the subnet and here you will see there are other resources which I need to create but I will for the time being I'll just comment out all those resources because we are just at the moment we are looking into VPC and the subnet so I will just comment all this section over here all right so now uh our in the networking we are setting up the VPC and the subnet and we have our rest of the code commented inside this particular GitHub repository also before we uh proceed there is also one more important thing which I need to show which I would like to show to you is the provider. DF since we are working with the AWS so we have the provider section where I am using the region which is like a West one region and here I'm using the credentials for my a WS account so I have stored this credential at this particular location which is my local uh MEC OS directory which is like a user Rahul and. AWS so I would highly recommend to create the AWS credential for your own AWS account and if you can if you don't know like how to create it then just go to your AWS account over here uh sign in to console and go to your root account go to security credentials and here you as you can see I have created some access key for myself and here you can also create it a new access key for you so so that you will have a access key and the secret ID for your AWS account and those uh you can store inside the credential file that's the AWS credential file which you can store it onto your local laptop so that your terraform code can use those credential to provision the infrastructure so that's a key important thing which you need to keep in mind all right so let's open the terminal and here I'll just open the terminal I'll just clear the screen I have been working previously so I'm just going to increase a bit over here I'll clear it once again and just check the path so I'm into the terraform Jenkins directory which is correct and then I would like to run the command like a terraform init command so terraform init command is really essential to initialize your modules as well as also download the AWS required dependencies so that terraform can provision the infrastructure so let's hit enter over here and here you can see our terraform has been initialized successfully all right so let's clear this screen the next command which I need to run is the terraform plan command so that we know like how many resources we are going to provision and here you can see it's saying me like five resources is going to create which include uh one VPC two subnet uh the two public subnet and the two private subnet which equates to five resources I'll clear the screen and I'll show you uh like this is my AWS account go to the homepage and we are provisioning this whole infra into the EU west region I can show you one more slide this is our Target architecture into the different region so our Jenkin setup will reside into the EU west region so that our infrastructure code or application infrastructure is separate from our Jenkins so that's why I'm working into the EU West one availability reason and that's the availability Zone I have set up while using the VPC and the subnet so that's the key difference so just keep in mind we are right now working into the EU West one region and since I mentioned like EU West one reason so go to the drop down into the AWS console for availability zone or the region and here you will find the option for the reason that is EU West one so click on it and here you I'll Zoom a bit over here and if you go to the VPC section then you will not find any VPC which has been which I have not which I have yet not created so go to the VPC here you can see this is the default VPC uh but we need to create our own VPC over here let's run the terraform apply command so that we can create those uh vpcs and subnet so the command is terraform [Music] apply and it is asking are you sure you want to create those resources type yes and hit enter and here you can see it has created the five resources so let's go back to our AWS console once again and and refresh this page over here and here you can see our first VPC has been created and this is the VPC ID which you can see over here and the IP range which you can see that is 11. 0.0.0 so that's the IP range which I have assigned so we have verified the VPC the next thing which we need to verify is the subnet so again go to the left navigation menu over here and go click on subnets over here and here you can see there are four subnets which we have created two public and two private subnet so let's open uh the public subnet over here and let's open one private subnet over here so to to uh verify the details so here uh first let's check the IP ranges so here you can see the public subnet 11.0.1 do0 and the private subnet should start from 11.0.3 do0 so 3.0 is and 4.0 is for private subnet and 1.0 and 2.0 is for public subnet and also you can see uh the VPC uh uh this is our private subnet but this private subnet should come inside our VPC uh which we have just created so click on this VPC and here you can see this is the VPC which we have just created so this this is the way you can verify uh your VPC and the subnets now after creating the VPC and the subnet the next resource which we need to create is the internet gateway and we need to create an internet gateway for our public subnet so that any resource which we will be creating inside the the public subnet will have access to the internet and as well as anyone from outside world can access the resource present inside our public subnet through internet gateway so let's jump to the code and I'll show you like how to create your internet gateway so this is my uh terraform Jenkins repository go to the networking and open the main.tf file and again we need to enable few codes over here so next resource which we have commented previously we need to enable is the internet gateway so here I'll just enable the internet gateway and I'll comment the of the code so this is the resource uh block which will be creating a internet gateway and again if you take a look onto this diagram this internet gateway uh is still present inside this particular VPC so here we will make a reference to the VPC ID so that uh we will be able to create the internet gateway within that particular VPC so that VPC is there and this is the name of my internet gateway so let's create this internet gateway uh by using the terraform plan and apply command so go to terminal over here I'll just clear the screen once again then run the terraform plan command and here you can see it is planning to add a resource one resource I'll clear the screen once again and I'll run the trra form apply and I'll Supply the auto approve so that I don't need to type uh yes over here but in production just try to uh be careful don't uh use Auto approve flag I'll hit enter and it might take a minute or so to create a internet gateway and here you can see our internet gateway has been created so let's go back to our AWS console and click on this internet gateway on the left navigation menu inside our VPC section so click over here and here you can see our VPC our internet gateway has been created and this is the internet gateway and this is the VPC which it has been attached already moving ahead the next resources which we need to create for this full infrastructure is the route table and we need to create a two route table one for the public subnet and one is for our private subnet so these are the route table which we will be creating and for that let's go back to our ID and here we are going to enable next line of code so here you will see uh previously we have set up the internet gateway the next thing which we need to set up is the route tables and Route tables we will be creating it for uh public subnet as well as for private subnet so I'll first enable this code for Route table and this is going to be our public route table and this is uh I'll show you like how to create our association but let's first take a look this is our route table for our public subnet and if you go down uh here then this is our route table for our private subnet all right and I'll just uh remove this code also so I think that's it all right okay so this is the code for our public route table and here you can see again we need to mention the VPC ID so that uh we create those route table within the VPC so that's important thing secondly uh we need to create a internet gateway route since as you have seen over here into the diagram uh this route table is responsible for routing the all the requests so if anyone who is reiding outside of this particular AWS uh environment then that uh request will come and that request will come via internet gateway and this route table will be responsible for forwarding that request from internet gateway uh to our subnet so we need to create a route for this internet gateway for our public subnet but in private subnet we don't create a internet gateway so here in this section of when we are creating a route table inside our terraform so I have created a section for a route and here we need to specify the IP address which means 0.0.0 that indicates that it is a internet request so anyone from internet can access this particular resource so whenever you specify this address then you're making it quite open to uh internet so that's the first thing and this is the Gateway ID since we have already created our internet gateway which you can see over here this is the internet gateway and this is the name of that particular resource so which you will eventually find over here into the gateway ID so if you click on this one then you can see this is the same internet gateway it is pointing to so here we have created a route table we have assigned the VPC and here we have created a route for internet and here we have assigned the internet gateway ID to that particular route table so that is going to be my public route table so similarly if you go to the private route table then you will not find any internet gateway because we have not assigned any route or we have not created any uh route between the internet gateway and our private subnet so that's the key difference between the public and the private subnet and the route tables uh let's try to create the route table and for the time being I'm just going to disable this Association I'll show you after creating our route table so I'll just going to disable this Association of our public route table as well as private route table so I have disable dis able both of it go to terminal and I'll clear the screen and I'm just going to run the command terraform apply Auto approve you can also run the terraform plan command but since I have already know this code so that's why I'm running the terraform apply with auto approve and here you can see two resource added which means we have created our route table which is public and the private let's go back to our AWS console over here and on the left navigation menu click on the route table and here you can see our private route table and RT is for route table abbreviation and this is our private and public route table so let's click on this public route private as well as public route table so here you will find uh that this is the VPC which it has been attached properly same over here so it has been attached properly but if you take a look uh onto the subnet Association then you will find it's completely empty there are no subnet Associated similarly onto our public route table if you go to subnet Association then you will find this is completely empty because we have not created any association these route tables are still independent and not attached with any subnet but uh for public route table you will find a route for internet gateway so if you go to routes over here then here you can see this is a route with the IP address 0.0.0.0 for internet and this is the internet gateway so if you click on this internet gateway then it will point to the same internet gateway which we have just created previously so this is our public route table uh with our route to the internet gateway and this is our internet gateway but similarly if you go to our private route table which is our private route table so here you will not find any route so go on Route so here you can see this is the default route there is no there's no IP address like a 0.0.0.0 for internet and there is no connection to Target to our internet gateway so that's how uh we create our route table the next thing which we will do we'll focus on the subnet Association so we will fill this entries over here okay so let's go back to the code once again into the networking so first of all we are going to enable the association between the public route table and our public subnet so I'm just going to enable this block of code over here and here you can see this is the AWS route table Association resource and here we need to specify the subnet and here you can see this is our public subnet by the name which we have previously created and this is our route table which is our public route table and which you can see over here so this is the highlighted name for our private route table and this is the private route table I'm using and this is the association between our subnet and similarly if you go to our private route table so Association then here you will find the association between our private route table and private subnet so here it's the private route table and this is our private subnet let's apply those changes so go to terminal over here I'll clear the screen and I will just run the terraform apply Auto approve so it took a couple of minute to create the association so let's go back to our AWS console to verify the association so this is our private route table so I'm just going to refresh this section over here and go to subnet Association so here you can see our private subnet has been associated with our private route table and similarly if you go to our public route table and click on subnet Association this was our previous one so if I go over here and I just try to refresh this section then go to subnet Association and here you can see our subnet has been Associated properly uh one thing over here if you are interested in learning the AWS networking from the AWS console UI site without terraform and if it feels like very overwhelming at this point of a time then just go and check my one session I'll post the link into the description section where I have shown like how to create your whole networking in an easy way using the AWS console so I would highly recommend to go and check those sections so that it will be easy to understand all these Concepts let's uh move ahead with our project moving ahead the next resource which I have highlighted over here is the net Gateway but since we are working into the public subnet so we will not be creating a net Gateway uh inside our AWS infra for Jenkins and finally we will be creating an ec2 instance where we will be installing the genkin inside our public subnet but keep in mind that we have already set up our public subnet and private subnet but since we will mostly work on our public subnet so we will not be using private subnet that much for our Jenkins installation but how the ec2 setup would look like so here you can see this ec2 instance is again present inside the VPC and inside of public subnet so first of all whenever we create a ec2 instance then we need to point to the correct VPC along with our proper subnet so that we can provision the ec2 instance inside our VPC or the networking which we have created so far so let's jump back to our ID and start looking into the ec2 code of our terraform all right so let's open the main.tf where we have mentioned our module so as you have seen previously we have created all the networking related module and we have provision all the infr related to networking the next thing which we we need to set up is our ec2 instance and for creating an ec2 instance there are certain things which we need to create beforehand so the first thing is the security group so here is the security group which we will be creating and I'm just going to disable other modules over here so there are certain Security Group which we need to enable Security Group are just the firewalls for our Linux machine which is in case our ec2 installs so on a high level whenever we create a security group then it is we are just allowing and disallowing certain ports onto in our firewall so uh on High level we will be enabling the port 443 for SSH we will be enabling Port 22 for doing the SSH into our ec2 instance and also the port 8080 so that we can access the genkins running on our port 8080 here you can see this is the module which I have created for security group and this Security Group is responsible for enabling the port uh like a 22 uh 443 8080 and the port 80 so if you go to the security group module uh over here uh under this uh terraform Jenkins directory there you'll find a security group directory and there is a main.tf file so here I have created a couple of Security Group in the First Security Group I'm just enabling the port uh 22 as a inbound Rule and then Port 80 and then Port 443 and the outgoing request I'm allowing all the request to go out from my ec2 in instance I'm not blocking any of the request and also there is a one more Security Group I'm enabling is for Jenkins on a port 8080 so here you can see this is a inbound request or inbound rule for port 8080 so these are the security groups which will be created once I'm run once I will run the terraform apply command let's open the terminal and run the terraform apply command over here so I'm just going to run terraform apply Auto approve and it should take a couple of minute or maybe even shorter to create those Security Group and here you can see our security group has been created let's go back to our AWS console to verify that I'm just going to close I'll just go to the homepage and in the search box type ec2 go to ec2 and in the left navigation menu you will find an option for a security group so here you can see there are couple of Security Group which is this one and the this one although I have kept all both the names same uh that's one mistake but it's not going to create any problem we are just worried about the security group ID that should be different so here you can see I'll open this Security Group and I will open the security group so I will change the name later anyway so this is the security group to allow Port 880 so if you go and check the inbound rule over here then you will find that Port range is 8080 so we are allowing the port 8080 uh on this Security Group for ec2 instance and another Security Group rule if you go over here then here is the description which you can see like SSH 22 Port 443 and although I have mentioned 80 but 880 we have already created over here here so I just need to rename this one also and also the port 80 but here if you check the inbound rules over here then you will find the ports like 22 80 and 443 so we are enabling these ports uh using the security group for our ec2 instance okay let's take a look onto the diagram once again and we are still not there yet we are still not created any ec2 instance and genkins so we are just completing the prerequisits which is needed to uh facilitate the genkins installation onto our ec2 machine so now we are going to set see the ec2 uh terraform code how we can set up our ec2 instance and also we can install the genkins at the same time so let's go back to our Cod IDE over here I'm just going to close all the tips over here and I'll just open the man terraform file again so we have set up the networking we have set up the security group and the next thing is we are just going to enable the chanin module this one and here I'm just going to comment the other one the load balancer and other modules we are going to see it later so here this is the Jenkins module and here you can see the source is available under the Jenkins directory so just go here and open the Jenkins directory as well all right so here are a couple of things which I'm forwarding uh to that particular module first one is the Ami ID which is like a machine ID here I'm using the Ubuntu so uh I can show you that one also over here so Ami ID so this is the Ami ID which you generally get from your AWS console so if you take a look over here onto browser I'll just going to open this one so when you go to ec2 dashboard if you click on launch instances and if you choose like this then you will get a Ami ID so I just copied this Ami ID from here so you can just choose suitable uh operating system Linux operating system and just copy the Ami ID if you don't want to use the uban to and if you want to use so UB to then you can just uh continue with this same Ami ID okay so that's that's about the Ami ID again let's go back over here the second is instance type so I I'm just going with the t2. medium because it's a jankins and we will be creating a certain pipelines onto the janin so t2. micro the small one will not be sufficient enough that's why I'm just going with the uh medium instance tag name so it is just optional thing so just keep whatever uh tag name you want to associate with your ec2 instance public key so this is important because whenever you're working with uh ec2 then you need to create a public and the private key for your ec2 instance so that you can later on SSH into your ec2 instance so before I show you the remaining code of terraform let's create the public and the private key so that we can associate the public part of the key with our ec2 instance and we can keep the private key with us for generating the SSH key pair the command which you need to use is SSH keyen and this command can vary based on the uh like what kind of operating system you're using if you're using Windows then just try to install this utility and if you're using Mac OS or Linux then this command will work uh by default so I'm just going to run the SSS key genen command and this will generate the private and the public key so just hit enter and it is going to say you that it is going to generate the key at this particular location so I'm just going to copy this part and I'm just going to rename this uh key a bit so I'll just put over here sorry I need to rerun it I'll just clear it once again so I'm just going to run the SSH keyen command once again and I'm just going to paste this part and I'm just going to rename the file so I'll just put uh something like Jenkins uh demo and hit enter uh do you like to associate any prph so I'm just going to keep empty enter enter again and now your keys has been generated so what you need to do you just need to run the command uh LS to verify the keys so LS L and then go to the the location so the location is this this part this is the location where I have generated these keys so copy this one and past it over here and here you will find the two key which is Jenkins demo which is private one without any extension and there is a public key which is Jenkins uh demo uh. Pub Pub is for public key so now you have the public uh sorry the public key and the private key what you need to do you just need to copy the public key part from this public key over here so here is the public key so what you need to do you just need to run the cat command and again same path uh sorry I just need to retype the path so here I'll just use home Ubuntu sorry it should be Rahul uh no no no it should be users and then Rahul and then SSH and then chanin demo and then I'll just use Pub for public key so here you can see this is the content you just copy this content from here and then you just need to go back uh to your terraform code and here you need to paste the content of our public key so here I have already created a variable uh for that public key so if you go to terraform Wars file and I'll I wrap this code so that you can see the complete key over here so here you can see this is the public key which I have already created and pasted over here so just paste that public key content over here and this is the public key variable and if you take a look once again so this is the content which you will see a quite similar one all right so this is the public key part which you need to paste inside this variable for your own use uh remember if you'll clone this repository then this public key will not work for you you need to replace this key with your own public key all right so I'm assuming that you have replaced this key and again go back over here and here you can see the public key will be associated with your ec2 instance and remember that private key is still with you so this is the private key which is residing onto locally onto your laptop so we'll use this key later on to SSH into that ec2 machine all right but for this particular demo I'm using this set of key this is the key which I have generated previously so this key which I'm associating over here uh which you can see over here so this is the public key which I'm associating over here and later on when I'll SSH so I'll use this particular key aw cc2 terraform so these are the key which I generally use for my own purpose but I have shown you like how to create the keys for yourself also all right so let's go back there again and subnet so again uh if you take a look onto the diagram so this ec2 instance is present inside our public subnet which you can see over here so we need to supply the public subnet ID over here so again uh here is the subnet variable which I have created and here I need to provide the subnets so the subnets which I'll be providing is the public subnet so here uh uh since we have created a two public subnet so I'll be providing on only one subnet because we can create an easy to instance into one subnet so that's why I am fetching the zero I'm putting the zero over here so that I can get the whatever first in the list the public subnet I'll just create it over there so that's a subnet then the security group as you remember I have just mentioned you that we need we have created a two Security Group and those Security Group we will be associating with our ec2 instance so here uh this is the security group variable and here you will first uh specify the security group from module so if you go to the security group module over here then this is the one so here this is the first Security Group module uh from where we'll be fetching this one so copy this and you'll find it over there so yeah here it is this is the first one uh but I can show you this is from output actually so this is the first Security Group which I have created and this is the output uh where I'm just using the ID of it so this variable will will be responsible for outputting the ID of this particular Security Group and this output variable will be responsible for outputting the security group ID of port 8080 so these two output variables I'll be using over here so here you can see this first one and this is the second one so I'm just passing both the security group to my chenkin ec2 instances so that's another one enable public IP address so I just wanted to enable the public IP address because I'm just setting this ec2 instance into public subnet so as you can see over here into the public subnet so we need to have the public IP of this ec2 instance okay that's done and then the last part is where we will be installing the genkin so all these things till now we are just setting the ec2 machine that's it and that ec2 machine is just blank there is nothing on it apart from the security groups so here uh this is the Jenkins installation script uh which we will be installing using the user data and if you don't know like what is the user data in AWS cc2 then I will just uh I have already uploaded a video on explaining like how to work with the user data so you can install certain packages onto your E2 machine so we are using the same user data uh concept over here and here I'm supplying this particular installation script for installing the chenkin so I'm just going to show you this installation script of Jenkins to you as well so yeah these are the par which is needed to create an ec2 machine so that it can install the Jenkins along with the proper subnet and the security groups let's take a look onto the genkins installation script so here uh I have uploaded that script inside the genkins runner script directory so here left hand side you'll find the Jenkins Runner script directory and here you will find the installation script so first of all it's just a bash script very simple and here I'll just explain one by one so first of all this is the syntax which is needed for a bash script to work here I'm updating the package manager so that my operating system is up to date with all the latest information on latest packages which is available for that one two after that I'm just installing the jdk11 because jdk 11 is necessary to install the chanin so that's the first requirement after installing I'm just sleeping 30 seconds I'm just waiting for 30 seconds so that my jtk installation finishes properly after that these are the instruction which I have copied from the genkins official installation uh guide so if you go to browser over here and if you click on Google and type Jenkins sorry uh Jenkins installation on Ubuntu then you will find a link this first link and if you go down over here then these are the instructions for installing the chin so I just copied these instructions from here and pasted inside my this installation script so so here it is just uh doing the Jenkins uh package updation and then after that it is updating the package manager and then here it is going to install the genkins and after installing the genkins again I need to wait for little 30 or 60 seconds so that the genkins is install properly and so that it can start the genkin services so that's the uh that's where I'm just stopping it over here so these are the part where we will do the jdk11 and Jenkins installation and once we have installed the genkins then also we need to install the terraform onto that ec2 machine so that we can provision the other infra that I'll show you later but from here these lines of code 15 16 17 and 18 will be responsible for installing the terraform onto my ec2 machine so once the ec2 machine is up and running then we will see Jenkins running over there and we will also have a terraform also installed over there along with our jdk 11 and I would like to show you the Jenkins module code also so in here the genkins directory here is the main.tf file and these are the output variable which is necessary uh which we will use it later but here is the AWS instance here is the Jin C2 instance here is the Ami ID which we are we have passed here is the t2. medium uh this is the tag description which we have provided this is the key uh which I have mentioned that I will be using my own key this is a private key which I have just shown to you which which is this one so this is the key which I'll be using and the subnet which I have already passed the genkins the public IP and these are the some option metadata option which we need to set it over there for my ec2 machine and this is the key pair uh where I'll be passing the public key to my ec2 instance so this public key which we are passing it from here which you can see this is the public key which we are passing it to our ec2 instance let's run the terraform apply command to provision our ec2 instance with our chenin so go to to terminal over here I'm just going to clear the screen over here and I'm just going to run the terraform apply and it might take a couple of minute or more because we are setting up ec2 instance along with the Jenkins and it might take a two or three or 4 minute to come up the Jenkins and we can verify the Jenkins letter so after a couple of minutes so my ec2 instance has been provisioned as you can see over here and if you go back to my ec2 dashboard I'm just going to refresh it over here and here here you can see our ec2 instances running click on this instances and here you will find the instance ID and this is the description which I have mentioned for my instance so click on it and this is the public IP address so just copy this public IP address and remember we have not set up any load balance or any domain yet we need to set it up later so let's just verify the uh genkins installation first of all by using the IP address so I'm just going to copy this one paste it over here and I'm just going to suffix it with the port 8080 and hit enter and here you can see our Jenkins has been installed so now we need to set up our Jenkins so Jenkins is up and running and then we need to set up our own username and password and here you can see it is prompting us that uh your initial password at presented this particular location which is this one where Le Chen can secret an initial admin password so for that we need to SSH into our ec2 machine go to this particular directory copy that password and then we can reset our password so for that I'll open the terminal I'll just go back to my ter over here I'm just going to clear everything and as I mentioned I'll just go to the SSH directory where I have already created my keys so if I run the lslt so here you can see this is the key which I have already generated this is a private key so I'm just going to use the same key to SSH into my machine so how to SSH so for that uh what you need to do I'll just increase the size of this screen so that it's easy for you to view the content I think that's good all right so to SS into this machine what you need to to do you just need to click on connect over here and here you will find the uh option for SSH so just copy this command and remember uh once you generate the key you also need to change the mode of that particular key so just try to run the CH mode 400 command onto that particular key whatever key it is whether it's without Pam extension or with Pam extension otherwise you will not be able to SSH so here I can show that one also uh it's already there so here you can see this is it has only the readon permission if you can see this is the readon permission if it has a read write and executeable then you will not be able to SSH so just keep in mind to run this particular command first of all to change the mode to read only okay so I'm just going to copy this SSH command go to uh my directory I'll just clear this and remember I'm into this SSH directory and key is available into the same directory so just keep in mind so here and the key is without Pam extension so I'm just going to remove this one and just hit enter and type yes are you want to connect yes I just want to connect it and here you can see we are now into our ec2 instance I'll just clear it and uh again go back to the uh our initial screen sorry I just clicked on that link but here this is the path for my initial password so just copy this path uh I'll copy it from here go to my path cat and run the command hit enter uh you need to suffix it with the Pudo and here you can see this is our initial pass password so I'm just going to copy this password from here paste it over here click on continue install suggested plugin so select this one and it might take one or 2 minute to install all the suggested plugins so I'll be back when all the plug-in are installed after all the plugins has been installed then we need to create our admin user so here I'm just going to enter the name so I'm just going to enter username Rahul this is just for demo purpose that's why I'm using the simple username and simple password but for you just try to keep the strong password and good username and the name I'm just going to put Rahul vog and the email address I'm just going to put my own email ID and then save and continue uh yeah Jenkins URL we can keep it or otherwise we can already change it we are going to set up this whole domain later but uh let's let's just keep it uh just keep the way it is right now save and finish start using genkins and here you can see our Jenkins is up and running but still we need to configure our domain which is jenkins. j.org so if you take a look onto the diagram then I'm talking about this particular URL which you can see which is from user so jenkins. j.org which will go to Route 53 then which it will go to load balancer and then it will go to our ec2 instance right now we are just accessing directly this ec2 instance using the IP address all right so let's take a look onto the diagram once again and here you can see this is the complete uh genin setup which we are trying to achieve and now the next component which I'm talking about is this particular load balancer so we need to set up our load balancer so that we can uh Point our domain to this particular load balancer using the uh Route 53 so let's go back to our a uh ID and start setting up our load balancer so here uh this is my uh ID I'll close all the tabs over here and I'll just go to main.tf and I'll just take a look so now we have seen the networking we have seen the security group we have seen the Chan kins we have installed the Chan kins and the next thing is the load balancer Target group so here uh this is not a load balancer yet we need to create a group so load balancer Target group so Target group is like a group of ec2 instances so we'll create a group in that group we will attach multiple we can attach multiple ec2 instances so we are just going to create this logical Target group uh using terraform so that we can later on use that group inside our load balancer so let's take a look onto this module again so here I'm just going to disable the uh uh load balancer module as of now so here this is the code where my load balancer Target group module exists so here if you go into the left navigation so here is the load balancer Target group open the main.tf over here but let's take a look onto the parameter first so first of all this is the path which I have just shown to you this is the name of my target group so yeah this is the Jenkins load balancer Target group load balancer we are going to create it later so Target group Port so as you have seen uh my Jenkins is running on a port 8080 so here you can see this is the port which is 8080 so that's that's why I'm just using the port 8080 over here the protocol which I'm going to use is HTTP for my target group The VPC ID again everything we are working within the VPC so here you can see this uh we are we will be working in the Target group within the VPC load balancer uh we'll set it up later but the Target group will be still inside our VPC okay so here's the VPC for that particular Target group and this is the instance ID so as I have told you this target group can contain multiple ec2 instances so here we need to pass the ec2 inst instance into that Target group so yeah this is the ec2 instance which you can see uh I think I can show you uh like this is the Jenkins go to Main and here is the output variable uh Jenkins E2 instance IP so this is the uh instance IP I'm using it over here so if you go over here so here you can see contr F and here so I'm just passing this instance ID so this will create a Target group for that particular E2 instance and I can show you the code also so here in the main this is the code for Target group here is the name here is the port here is the protocol which I have just shown you here is the VPC ID and here is the health check for that particular genkins so I'll be using the Health Point endpoint URL that is login and after the port 8080 uh where the login URL is accessible so this is the health checkpoint so the target group uh treats that particular uh endpoint as a healthy and it can attach to the load balancer so this is the code for Target group so go to terminal I'll clear the screen over here and then just run terraform apply and yeah just wait for a minute or so and I'll be back once this target group has been created so as you can see our Target group has been created so let's go back to our AWS console I'll go over here I'll go to the homepage and here in the search box type ec2 click on this ec2 over here and in the left navigation go to the load balancing and here you will find option for a Target group so click on target group and here you can see jenin sby Target Group which I have just shown to you click on it and this target group will contain an ec2 instance which is running so here you can see uh this is the instance ID uh which I can open here in the new tab and here you can see this is the instance ID just copy this and copy this IP address and which I have already shown to you U maybe I can re again show to you that this is running 8080 so here you can see the Jenkins is running so yeah so this is the target group and uh this is the instance ID included into that Target group so I'm just going to refresh this page once again and the health status is unused so this target group is up and running now we need to provision our load balancer so that our load balancer can point to this particular Target group all right so now I'm talking about this particular load balancer and this load balancer will point to the Target Group which I have just shown and that Target group will point to this ec2 instance so that's the how the request flow request is Flowing okay so again go back to IDE I'm just going to minimize this one go to our main.tf I'll close everything over here close all all tab go to men. TF and now we are going to enable the application load balancer module this one and I'm just going to enable only this section okay uh certificate manager that's also needed so anyway I'm just going to enable this one I I need to enable all the three modules because there is a dependency between all of it so this is a load balancer module first of all so I'm just going to explain this part first of all so here's the code for load balancer if you go over here into the load balancer then here you will find the code for load balancer so I I'll I'll explain from the parameter that will be easy for you so the load balancer name is this one which is I'm passing over here is it a external load balancer no it is not an external load balancer that's why it's false load balancer type is application because we are deploying it on ec2 instance so that's why I kept it as application Security Group for our load balancer the security group should also have the similar Security Group which we have configured onto our ec2 machine so here I'm just using the same Security Group which we have just configured so here you can see the same Security Group for SG SSH and HTTP so these are the same Security Group I'm using so I have configured the same Security Group for my load balancer subnet ID so here I need to provide the subnet so again my ec2 instance if you can see over here it's into the public subnet so here I have specified the public subnet for my load balancer tag name it's optional just keep whatever you want to keep Target group so again key point so here we have created this target group so we need to use the same Target group over here into this so uh if you go to the Target group module which is this one so here I have created output variable which is this one so this will print out the Arn of this particular Target group so this is the value I'm copying it over here and here I'm pasting uh into my Arn Target group Arn so here we need to attach here I'm attaching the target group to my load balancer so that load balancer can know which Target group it is going to forward the request then again the instance ID we also need to specify the instance ID of my ec2 machine so here this is the instance ID from modules genkins so if you go to genkins main.tf then here you will find a genkins ec2 instance IP so that's there and uh listener Port the port is 80 protocol is HTTP and the default action is forward yes I want to forward the request to that and the 443 is for https because we will be setting up two listeners for our load balancer one is on HTTP and one is on https so that's a difference between those two and then uh the certificate manager so as I told you like as I have shown in the diagram might be previously to you so if you go to this diagram over here so this is Route 53 and this is how the request flow will happen which I'll show you so here this URL will have a https so whenever someone will access this URL then this request will land on Route 53 and Route 53 will forward it to application balancer and then it will go to ec2 machine uh I mean application load balancer will forward it to Target group and Target group will send it to ec2 so for this s https we need uh certificate manager so for that we have cre created a certificate manager and for creating a certificate manager you need a hosted Zone okay so load balancer part is over now but now we are talking about the hosted zone so load balancer uh these many parameters are sufficient enough for you to create a load balancer that I'm going to show you but let's take talk about uh hosted Zone also so here uh if you go to the homepage and if you type here Route 50 this is an important concept if you're managing the domains and if you're pointing the external domain to our AWS so here is the hosted zone so here this is the domain which I have already purchased for myself so if you click on it then this is the domain which I have already purchased so whenever you purchase a domain which is external or whether it's from AWS so you can purchase a domain from AWS or ISO you can purchase it from other services like GoDaddy name chap side ground whatever and uh Al Google domain is also there because I have bought this domain from a Google domain so this is the hosted Zone over here and this hosted Zone I have created for a domain j.org so let's say for example you already purchased a domain and you need to uh point that domain to this hosted zone so you need to create a hosted Zone first of all so go on create hosted Zone and here you need to point type the name not Point type the name so let's assume that you have a test XYZ so you'll just enter the name and create a public hosted Zone and then click on create a hosted Zone and here you will see there will be two hosted Zone uh as you can see so this hosted Zone I have already created so that's why it is there but this test XYZ I have just created although I don't ow the domain but I just wanted to show you the process how you will work with it so create a hosted Zone based on the exact domain name which you have with you so here once you create it then you will get these records which is a name server records for you so now uh this is the text XYZ I'm assuming you will have your own domain you will create your own hosted zone so similarly if you compare with my J hook or then you will also find the same thing so I just wanted to show you that there's nothing superficial I have done over here I just created a hosted Zone and these are the name server settings which I'll be using later so uh to work with this whole setup uh we need to set up this Route 53 and in Route 53 we first need to set up our hosted Zone so we have set it up the hosted Zone just now so I have already explained the application load balancer part to you but here's the hosted Zone uh I know it's it's bit bit complicated but it will not be that complicated once you understand the whole flow so this is a load balancer all the way this is the code for load balancer which will set up our load balancer and this is the hosted zone so let's take a look onto the hosted Zone also so here's the hosted Zone module code to main.tf and here I have already created a data block so why do we need a data block so hosted zone I'm not provisioning with terraform I will create a hosted Zone manually uh using the AWS console which I have just shown to you like I have how I have created for test XYZ so here this data block we will use for any existing resource on AWS so this is the j.org hosted Zone which I have already created so that's there then we need to create a a record a record is just a redirection record so uh whenever we create a hosted Zone and inside that hosted Zone we will create a record and a record will point to the load balancer I can show you that flow also so here's how it works so if a user enters a URL so let's say user ENT j.org or jenkins. j.org then that request will rent on our hosted Zone then hosted Zone which is our j.org and then it will check for the entries so here the default entries are these ones like a name server s so entry which are default one whenever you create a new hosted Zone but in case if you create a a record over here these are the name server record which you can see over here but inside that when you create a a record then this a record is responsible for redirecting the domain request which is like a ww. j.org to our ec2 instance so here you can see this is the a record with the IP but in our case instead of Ip we will be using a load balancer so if someone ENT ww. jenkins. j.org then it will request will land on our hosted Zone and then it will try to find the a record so a record you will find the record and here you will find the url of our load balancer so that that request will forward it to the our load balancer so that's the significance of this particular hosted zone so if you take a look onto this flow again so here is the existing hosted Zone this is the record which is the a record we are trying to create and here we are just trying to pass the load balancer name so here you can see the aliases which we have created over here and here we are passing the load balancer DNS name and load balancer Zone ID where my load balancer DNS exist from which zone and this is the DNS name of my load balancer so that this a record should be created for my load balancer I'll show you once we provision this one okay so let's go back to our main.tf file so this is about hosted Zone how it's working and then this is the certificate manager so hosted Zone will just enable you the HTTP request but it will not enable you the https request so for that we will use the certificate manager and that uh we need to issue the certificate for each domain so here our domain will be jenkins. j.org it will not be j.org but it will be a subdomain jenkins. j.org so whatever subdomain or what how many number of subdomains you create you need to issue the certificate for that particular subdomain or n number of domains so here I'm using this one so I can show you this one also into certificate manager go to men. TF and here you can see this is the resource which will create our uh certificate uh and then after that here is the record for that certificate that again that record I'll show you because once the certificate is issued then those records also need to be inserted into our hosted zone so here uh here in this j.org right now there is no a record so once we will provision our load balancer along with our hosted Z then you will see a record with the load balancer and you will also see a record with our uh ACM which is our application uh a certificate manager so those records will get inserted over here okay and here uh yeah these are the validation and here it also runs the validation because it AWS will try to verify whether I own that domain or not so that's the key thing which you need to keep in mind okay all right so there is also one point which I would like to mention about the domain settings so if I can show you one more slide and flow to you which is this one so right now I have bought this domain from a Google domain and now we are trying to set up our hosted Zone into JH hope.org so what happens is like when uh you create a hosted Zone then you get a new name server setting which is this one uh these are the names server setting from AWS which I can show you over here uh here you can see NS 652 AWS DNS 17 and which is exactly same over here here you can see so these are the name server which is created name server records which is which has been created by AWS but I already bought this domain from Jook do uh Google domain j.org so these also issue some DNS name server record from Google but since we are working on AWS so what what we need to do we just don't need to use this name server record but instead we need to use the name server records from our AWS and put it into my Google domain and so that when someone enters uh j.org or whether he enters jenkins. j.org so any subdomain which belongs to this parent domain will always point to this name server records so instead of using Google's name server record we will be pointing to our own uh name server records from AWS also I can show that part uh before provision load balancer and the hosted Zone certificates so I'll go to Google and here I'll type Google domain go to Google domains go to manage domains and here uh you will find the domains which I own so click on j.org and go to DNS and here you will find a manage name server so manage name server click on it or maybe here you can see the server so here you can see this is NS 652 AWS DNS 17 so I have just manually configured these name server setting which I got from here so this is how the redirection is happening all the way so request is going to the jenkins. j.org from there it was finding the records which is like this so here J hub.org so here it is finding the records of name servers of AWS and then it is going landing on our Route 53 so that's how the request is Flowing okay so that's being said let's go to terminal I'll clear the screen over here and then just I'll check whether I have enabled all the modules or not yeah so I have pretty much enabled everything now so now we should be able to access our genkins using this URL which is jenkins. j.org instead of IP address all right let's run the apply and it might take uh 3 4 minutes because it's a long process because it will create certificate load balancer and then it is trying to validate my domain as well so just be patient with this one uh sorry to interrupt you in the middle of this particular session but I have seen like many of you really enjoy this kind of a Content but you don't consider subscribing to this channel so this is one small request from my side if you really like this kind of a Content then please considering subscribing to this channel because this will help you to grow my channel and to produce similar content in future so here's the uh request from my side just click uh like this uh video and also click on the Subscribe button I would really highly appreciate that one thank you all right so as you can see my load balancer has been provisioned my Route 53 a records has been created my certificate has been issued so let's verify all the details one by one so I'll go to uh my load balancer first of all so I'll just open the tab and here you can see I have not refreshed this Route 53 page yet so I'll refresh it bit later once we verify all those load balancer detail so here go to ec2 click on ec2 and the left navigation menu click on load balancer click on this is the load balancer which we have provisioned and here uh you will find the load balancer details so here you can see the https and HTTP listener for our uh Chan kins application all right but just copy this DN DS name from here and paste it over here so this is the IP address through which we have accessed the chenkin but I'm just going to remove it and put it over here so here you can see we still able to access The Chins using the load balancer DNS name but we still want don't want to use uh the load balancer DNS name we want to use the uh our own subdomain for that one but anyway we are able to verify our domain uh sorry the load balancer the next thing is we need to verify our Route 53 so here uh this was the page uh so I'm just going to refresh this page over here and you will see the couple of entries over here so here you can see uh this is the uh first a record for that one and this a record is pointing to my load balancer so here you can see this is the load balancer URL which I have just verified I can show you once again I'll just close this one I'll close this one I'll close this one also so I'll paste it over here and here you can see this is our Jenkins uh page using the load balancer but we already created the uh a record for our subdomain so that's already exist so just copy this uh subdomain now and I'll open it into incognito mode and hit enter and here you can see we are able to access the genkins so now we are able to access the genkins using the subdomain and here you can find put the password and here you can see this is the homepage of my genkins okay that's been done and all so there is one more thing I need to verify here is the ACM which is like a certificate manager so click on here and here you can see we have issued the certificate for our domain which is jenkins. j.com and if you click on it and here you can see the status is issued and the records has been created here you can see the C name and these are the records which has been created into Route 53 so if you go to Route 53 and they go to hosted Zone and go to j.org and here you will justrl F and find try to find it I think yeah but anyway here it is I think yeah there it is this is the uh record for that particular C name here it is okay so uh I think we are now able to access it I'll just try to re access it jenkins. job.org and yeah it's working I'll just type the username and password and that's been done so now we are able to finish our first part where we have only set up the Jenkins which is running on our EU west region and now we will start setting up our pipeline for our actual application which is our python flask application which will be running into EU central reason so let's take a look onto the second part now now I'm talking about my second repository where I have my infrastructure code for my python flas cap application so this is the GitHub repository where my whole project exist for AWS infra using terraform so here in this uh project I have in this git repository actually I have created a Jenkins file for setting up my whole pipeline so if you take a look over here then from this particular GitHub repository we will use a Jenkins file so here you can see one Jenkin files at the bottom I'll show you the GitHub repository uh very soon so we are going to use this genkin file and with the help of this genkin file we are just going to create our pipeline in Jenkins and that pipeline will execute the command terraform in plan and apply and once we execute that uh commands uh inside on using the Jenkins file then it will provision the whole infra for us and the infra I'm talking about is this whole infra where we'll set up the VPC where we will set up the subnet which is like a public subnet and the private subnet internet gateway uh route table ec2 instance rest API application using Python and then RDS instance with a my SQL so this whole infra we are just going to set up using the our genkins Pipeline and that Jenkins pipeline will only issue the dataform init plan and apply command so let's take a look onto this GitHub repository and try to look into the components which we will be using over there and this is the code repository for my devops project and this is the code repository which contains the infra which is responsible for setting up the whole networking which again includes the same VPC subnet private subnet public subnet internet gateway route table and then your uh Route 53 application load balancer and then pointing our main domain which is J hook. or to our application so here you can see these are the different different module which is like ec2 module certificate manager which we have just seen then we have a hosted Zone load balancer Target group load balancer networking RDS uh our S3 for restoring the remote State file and then the security group and finally the uh main.tf where we will be writing our own modules along with our all the infra so here uh let's take a look onto the flow of this particular whole project the devops project so here this is the main.tf file and again the project structure is exactly same just like a Jenkins so here again it's a networking section where we'll be setting up our VPC public subet and private subnet and I'm just not going to repeat those things again over here because EXA this is going to be the exactly same again the security group for my uh particular ec2 instance here I'm just going to change the port the previously it was running on a port 8080 but this time we it will be running on Port 5,000 because we are using the flask API uh a python flask API which runs on Port 5000 this is the ec2 machine and here ec2 machine again the same Ami ID uh the instance type is 2. micro uh tag name public key again I have already shown you like how to create your public and private key the subnet again I'm putting it into public subnet because our application will resite onto the public subnet if you take a look onto this diagram once again so here you can see this is our public subnet and this is our private subnet so database will be into our private subnet but this E2 instance with our python application will be into public subnet so that's why this C2 instance is belonging to my public subnet over here again the security group which I need to enable which will include the port 80443 our 5,000 Port so these are the essential Port which we need to enable for our ec2 instance and also uh if you go further then here we need to enable the port 5,000 uh so this is the first Security Group this is the second Security Group I have specified over here enable the public IP so yes you need to set the public IP address for your ec2 machine for your python application and then here is the user data for installing the Apache not Apache I have I need to rename this particular file I have just put the wrong name over here but this is the user data section where we'll be running some installation script uh for my ec2 instance let's take a look onto this uh particular installation script file so that we you know like what it's going to do once we start provisioning our ec2 instance so here go to the template and remember the naming is little incorrect over here I need to rename this particular file so it should be like a ec2 python application. s so just just a side note for you all right so as I told you like we will be installing our python application or we will be running our python application inside our ec2 instance so this is the script for that so first of all we'll switch to the directory like a home Ubuntu after installing or after provisioning our ec2 instance then I'll running the update I'll update the package manager then I'll install the python inside my ec2 machine after installing the python I'm just going to clone my project repository so here's the key thing so I had told you we will be working with the three GitHub repository so this GitHub repositor is only responsible for provisioning the infra for my python application so here you can see this is the infra code and there is a one more uh GitHub repository which I have already created so that's with the name python python myc DB project so this is the another repository which we will be cloning inside our ec2 instance and then we will be running that application so if you go to app. this particular repository which you can see over here there is app.py and where it contains all the uh rest apis which I have implemented so here you can see this is the health endpoint to show that our application is up and running this is a create uh this rest API are pretty simple I just leave it to you like how you want to build the rest API because uh once your application is up and running and it is connected to database then you can start implementing your own business logic to it so here's the first rest end point to create a table uh inside my uh uh myql DB then this is the rest end point for inserting a record and here this is the rest end point for fetching the record and this is just a uh default route for index.html so that our uh uh whenever I try to access our API using the j.org then it will land onto the index.html page of my homepage so yeah this is the uh API which we'll try to clone inside our uh ec2 install so here you can see this is the uh here I'm just switching it my work directory then I'm updating the package manager installing the python I'm cloning my GitHub repository then I'm waiting for 20 seconds over here then after cloning then I'll switch to my uh project directory which is like a python myl DB project one and after that I'll run the PIP install requirement. txe so that I can install all the python libraries and after that I'll just wait for a 30 second and then I'll just try to in uh start my python application using the python 3u app.py in the background mode and then again I after I'll try to wait for a 30 seconds so that my application is up and running on a port 5,000 of my ec2 instance moving ahead I'll just go to main.tf file once again so again the load balancer it's exactly the same which we have set up into our ec2 instance for our Chan kins only the difference is the port here I'm using the 5,000 because I'm running my application on a 5,000 Port so that's the basic difference between these again the application load balancer here you will find again exactly same module same everything the port is different over here which is 5 5,000 for HTTP and rest is for https for 443 and again here you will find a certain certificate manager hosted zone so hosted zone I'm using the domain name for j.org here I'm using the main domain I'm not using a subdomain over here so that's the key difference between the old uh ec2 instance genin setup and this setup and here again the certificate manager to issue the certificate for my main domain which is job.org so these are the hosted Zone settings and also you'll find the variables over here so here you will find the details about the variables which I'm passing so my VPC for this Jenkins not Jenkins for for python application is 10.0.0.0 the public IP ranges are 10.0 1.0 and 10.0 2.0 and for private it is 10.0 and and 3.0 and 10.0 4.0 availability zone is EU Central 1A and EU Central 1B but remember Jenkins was into West reason this is the public key which I have shown you like how to generate those public key and put the public key here and keep the private key with you this is is the Ami ID for my ec2 instance and this is the main domain name which I'll be using which is job.org this variable is just empty I can just remove this letter all right and also uh this bucket uh which you can see over here this bucket I have already created manually so that we can store our remote state of my terraform so which I can show you uh if we go to our AWS console over here I think I forgot to mention into the jenin setup but uh just keep in mind these buckets I have created manually because this is going to store the remote state of my terraform so click on S3 over here and here we'll find a couple of buckets so here you can see this is the bucket for Jenkins and this is the remote bucket for my python project so these are the buckets which I have created manually using the console uh and these will store the remote state of my terraform so that's one more point which I forgot to mention earlier all right and uh yeah let's get back to main.tf once again and here this is the RDS instance so this is a new instance or new module uh which we are trying to provision inside this uh python application so here uh the source of the code is here so here you can see this is the module code which is main.tf for my RDS instance and if you take a look over here the location storage is 10 gigs storage type is gp2 uh engine is myass SQL uh engine version is 5.7 uh instance classes t2. micro I'm using for database identifier I'll pass it from uh the module from where I'm calling it so now I can show you the actual module parameter from here so here you can see this is the name of my uh subnet group and this is the subnet Group which I have created so subnet group because this is RDS instance and this RDS instance is going to be created into our private subnet and to assign the private subnet to RDS instance we need to create a subnet group so here I have just created a subnet group and inside that subnet group I'm just trying to pass uh my Subnet ID over here and also uh there's a my SQL Security Group ID because we need to enable the security group for that uh RDS instance because myql Port is bit different over here uh I I'll show you uh in the security group section so probably in the security group if you go to main.tf then you will find a security group here it is Security Group so here the security group which I have created it for my uh RDS instance which is 3306 so that's the security group I'm trying to pass over here and identifier is my DB uh username is DB user password is DB password and DB project uh Dev project DB so that's the DB name for it although there is a One Security uh fallback over here because I'm hard coding the password for demo purpose but in actually we don't hardcode those password we just use some secret manager to store these password so that's just a side note for you uh since I'm doing it for the demo purpose so I just tried to keep it hardcoded over here because I'm just going to destroy the whole environment after the demo so that's a one point just keep in mind okay so this is the actual flow right now so we have just gone through all the modules for our project we have seen the python code of our rest API and now we need to set up our jenin Pipeline and it is going to provision this whole infra for us along with our domains which is point into our rest API so let's go back to our Jenkins and start setting up our pipeline so I'll just open the browser over here and I'll open a new tab and here I'm just going to type uh jenkins. j.org and here I need to enter the name so I'll just username and password I don't want to save it and also before we uh create a genins uh pipeline so just one more point so this genkins will not be able to communicate with our Europe central reason because we need to provide the credentials for our uh AWS so for that what we need to do we need to download one plugin so that we can create our AWS credential inside our chanin so go to manage chanin first of all and here we need to click on plugins uh click on plugins over here go to available plugins here type AWS steps and here we need to install this AWS steps plugin click on install and here you can see it's spending it will install all the plugins soon for AWS steps all right so all the plugins uh for my AWS pipeline steps has been installed and I can go back to my genkins over here again go to manage genkins because we need to store those credentials for my AWS so here you will find a credential section into manage enkin go to system go to Global credential and here click on ADD credentials over here and here in the kind you will find the option for AWS credential and this option will enable after we install the plug-in which we had just did couple of couple of seconds back here I need to use the ID so for that I'll just go to my code first of all so let's just open the Jenkins file over here and this is the jins file and here I'm using I'll I'll go through this Jenkins file very soon but here I need to copy the credential ID because this credential ID we will be using inside my Jenkins file so first of all let's copy this ID here and here we need to enter the access ID and the secret access key so either what you can do either you can go to security credential I'll just close this one in the uh right top corner there is a security credential section and here you can create a new secret create access key which will give you the access ID and the secret key key for your AWS account so just copy those keys from here once you generate those access key and secret key and then you just need to go over here and past the access key and secret access key over here I'm just going to go to my terminal I'll just open it into another tab for the security reasons and I'm just going to copy those details so so first of all I'm just going to paste the access ID and after that I'm just going to paste the secret access key and after that I'm just going to click on Create and here you can see our AWS credentials has been created now the next thing is again go to the homepage and here click on new item and here we need to enter the pipeline name so here let's put the dev project one whatever name you feel comfortable so just keep a suitable name and here select on this pipeline first of all and click on okay and here we need to now Supply the genin file to this particular pipeline so I'll go back to my ID and I can start explaining you the Jenkins file so here in the projects you will find the Jenkins file which I have already pushed it and here you will find options so I'll just uh go and active editor soft rapid so here you will find couple of option so this parameter sections I'll explain it because these are responsible for creating checkbox like when you want to run a terraform plan command apply command or destroy command so these are the check boxes for these parameters I'm just going to run on any agent because in genkins you can have a multiple agent so I just want to choose any available agents for me so here first thing which we will do we will just try to clone our repository so here uh first I'll try to clean the directory then I'm just going to clone the my devops project 1 GitHub repository and once I clone this repository then I'll just switch to that repository and I'm just going to run I'm just going to switch to that repository using the credential which is the credential ID which I have just created which is this one and after that I'm just going to switch through the directory infra so if you take a look onto this particular GitHub repository over here then if I go to devops Project 1 here you will see this is the infr directory and in the infr directory you will have a main.tf file so we need to switch to the infr directory and once we switch there then we need to run the terraform in command so this is how we are going to run the terraform init command and similarly we are going to run the terraform plan command also and here we'll again switch to the infr directory and then we are going to run the terraform plan command uh using the same credential uh which I have just created and same for terraform apply using the same uh infr directory and the same credential but also here we have put the condition if the checkbox is selected then it is going to execute it the plan command and if the checkbox is selected then it is going to execute the up apply command also so this is the simple Jenkins file which I have created for provisioning the whole infra which will set up your networking your uh all the public subnet private subnet internet gateway uh then your ec2 instance RDS instance then your load balancer then your hosted Zone then your certificate manager everything for you okay so now we have seen this genkins file then the next thing which we need to do is again go back to your Chen kins and here we are trying to create the pipeline but we need to point to that particular uh file so if you go down over here then here it's a is it a pipeline script no it is not a pipeline script it's a pipeline from SCM so here we need to select this option SCM like it's for our version management so here which is our git so I'm just going to choose git then it is going to ask the repository URL so just go here and go to the repository here click on this one and copy this URL from here go to again Jenkins and past it over here since my repository is public repository so I'm not going to use credentials I'm just going to keep it blank but in case if you have private repository then just put the correct credential for your repository all right Branch specifier just check the branch so here I'm working on a main branch so I need to change it to main over here and after that repository browser I'm just going to keep Auto script path so again uh if you take a look onto then in the repository I'm just putting the Jenkins file at the same path or at the root of that particular repo so Jenkins files is available right away over here so here I just don't need to change anything over here click on apply and click on Save over here so now our pipeline has been uh created but it has not run yet so let's try to run this pipeline I'll try to zoom a bit over here I feel like the font is little small okay so let's to run the pipeline click on this pipeline click on build now and here you can see our pipeline has just started so click on this uh console output so that we can verify if there is something goes wrong with our pipeline here you can see our terraform in it has been executing it has installed the AWS provider and these are the different different modules which it is trying to execute so here you can see it's showing you success and uh it has only executed the form in it because if you take a look onto the plan and apply so here uh by default all these perams are set to fall so it cannot run the terraform plan it cannot run the terraform apply it cannot run the terraform destroy so for the first time it will just run the terraform in it which we have just seen here you can see it's only run executed the terraform in it again go back to the pipeline click on build with params and here we'll find the checkboxes so now we have seen the terraform in it then the next thing is just run terraform plan and then click on build and here you can see all the steps which we have configured inside our pipeline click on this console output and verify the uh logs so that if there's anything goes wrong then we can uh Point those errors or we can fix those errors actually and here you can see it shows success so let's verify the output over here so here you can see it is planning to add 28 resources which includes everything here it will find all the resources which it will create once we set up the checkbox to terraform apply true so let's trest our pipeline so go back over here click on build with parameter select plan as well as apply and then click on build and now it will take a certain amount of a time because this is a fresh infra which is provisioning which is provisioning everything for us and also here I'll go to homepage and I'll just switch the reason to the EU Central Frankfurt and I'll just click over here you see2 click over here and here we'll just wait for ec2 instance to pop up over here so I'll be back and I'll just keep on monitoring this whole uh logs and I'll show you the exact output how it executed after the run so after waiting for a 4 or 5 minute here you can see my terraform apply has just fished so that's the pipeline run number three which you can see which has been running green now and if you check the console output then you can see there are lots of locks has been generated but we are interested in the finished success and here you can see our resources has been added to our infrastructure so I'll try to refresh this page over here so here you can see our instance is up and running over here and let's check our RDS instances so here you can see our myql DB is also up and running and uh now we need to access our rest API application so what you need to do I'll just open a incognito mode tab over here and here I will just try to Simply access the J hook uh sorry j hook. org to verify whether I'm able to access the application or not so here you can see our flask rest API is accessible this application is really simple I don't try to keep it uh like a very complicated one but the idea is just to make it communicate with the database and then you can take that uh example to implement your own uh Logics for your need all right so here you can see this is our API but the thing is if you take a look onto our code uh which I have shown over here probably I'll just uh move this side over here and if I go to the code then we need to create a table first so go to repository and type the python python my project and here in the ap. py first we need to create a table so here you can see this is the create table which we need to do so what I'll do I'll just this is my browser I'll just copy this URL and go to the table over here and I'll just click on create table so that we can create a table here you can see our table has been successfully created and after that I know it's it's it's not uh the way we should work we have a Flyway and other tools for doing the database migration but since it's a demo that's why I'm just showing you very very plain and simple vanilla version of this DB operation so just keep in mind just try to use some suitable DB migration Tool uh to manage your uh like ddl and DML all right okay so now table has been created I'm just going to show you all the tables in the MySQL also and now we need to insert the record so let's insert test record one I'll just copy the name and click on insert record record insert successfully click okay here you can see the record and then two then click on insert and here you can see so now uh I'm able to access my application and now I'm able to perform some crud database operation using the rest API using the flask uh python app let's try to access our myql database using the myql workbench so here I have already opened my workbench I'm just going to open a new connection over here and since we are using uh our SSH keys and also we need our uh SSH connection to go through via our ec2 instance because that's uh our RDS instance is running into our private subnet so here I'm just going to choose standard TCP over SSH the SSH host name I'm just going to use uh so go to ec2 instance and here we just need to copy the public IP past it over here uh username we are using UB 2 over here SSH key so I'm just going to supply the private SSH key so this is the SSH key which we have created so select that one host name I'm just going to copy the RDS host name from here copy this one and paste it over here Port is 3306 that's okay root and for that I'm just going to use the username uh so this is the username I'm just going to use and then click on test connection and here it is going to ask for the password for our database so I'm just going to supply the password which is this one and paste it over here click okay and successfully made the MySQL connection click okay and click on okay uh we need to supply the connection name so I'm just going to put demo MySQL DB connection so I forgot to type the name click on okay over here and click here and here you can see we are connected to our database and here here if you go to schemas and here you can see our database which is Dev project DB and if you take a look so here it is the DB name which is Dev project DB and if you go over here and the tables and here you can see the example table and here we have the columns which is like ID and name so here we can run the query select uh star from uh Dev I think we can run the example d people uh we need to select the database also so I'm just going to select that database so let's do set as default schema and again I just need to rerun that one and here you can see uh the record I think this is too small but uh I hope you should be able to see it I think the records are quite small over here but here are the records if you are able to see it test one and test two records has been inserted so let's try to insert few more records so I'll just close this one over here and let's insert a three record three and four insert record okay so now I have inserted few more record let's go back over here and rerun the command select start and here you can see the four record 1 2 3 and four so that's how uh we have set up our devops project and the last thing which we need to do is to destroy the everything so here again I'll just close this one and go to your chenkin UI go to your dashboard go to your pipeline go to build pipeline select destroy and click on build and it will destroy the whole infra for you so with that I think we come to the end of this particular end to endend project I hope you have learned something out of this project because this is something which I just wanted to show you uh because uh here you will learn quite a lot of things in starting from terraform from Jenkins from your AWS we have covered a lot of uh topics onto this particular project so if you have any question then please put down into the comment section and if you are interested into similar content than considering being a member to this YouTube channel where I keep on sharing similar more premium content to my YouTube member so see you into the next devops project till then take care and bye-bye

Channel: Rahul Wagh

Views: 35,395

Keywords: aws, devops, terraform, jenkins, devops projects

Id: otQqd7GRVK0

Length: 100min 19sec (6019 seconds)

Published: Thu Dec 21 2023