DevOps Project - Automate deploying to AWS using Terraform with GitLab CICD pipeline

so this is how you can automate using terraform and cicd in this devops project you will learn how to automatically set up infrastructure on AWS using terraform and gitlab cicd throughout the project we will be using industry best practices like using modules S3 backend and lot more to set up a cicd pipeline so if you're interested to learn how you can automate all this make sure you like this video And subscribe to the channel let's start all right so I'm here on my computer screen and let's start with terraform pipeline project but before we do that let me just tell you this is a bit Advanced project that will require you to have knowledge on terraform gitlab and also cicd to help you with that I do have videos already created so make sure you go through this particular video to understand what is terraform how it works also some advanced concepts like meta arguments modules are explained in this video and I highly recommend you going through this video if you're very new to terraform just to make sure that you understand the concepts described in this particular project along with this also check out this particular video which explains how cicd Works what is cicd why we use it just to understand the concept and the next video that I need you to check is the gitlab cicd video which is a 1 hour tutorial explaining jobs stages everything that we are going to be using in this particular project so make sure you go through these three videos before you start doing this project as I told you it's a bit Advanced and you will not be able to understand completely if you don't have prior knowledge to terraform gitlab or cic now moving forward this project is going to be divided into two parts the first part is where we write the terraform code run the commands manually and see if everything is working fine if everything works fine we go to the second part of the video or the project which is to automate all of these using cicd pipeline so we will create a cicd pipeline configuration on gitlab and we will automate all of these process using a pipeline script so this is the second part of the video first part we write the code run the commands and do everything manually to make sure our infrastructure is created on AWS and then we automate it in the second part using cicd pipeline script so this is the overall agenda of the project just to give you an idea now before you get started with this project there are few things you need to have the first is an AWS account if you don't have an account you will not be able to create any resources so make sure you have an AWS account second is to have terraform installed I already have it so if I show you by running terraform D- version command you can see terraform is installed you can simply install it by searching terraform install Windows Mac or whatever operating system you are on next you also need to have gitlab account we are going to create repositories and Pipeline on gitlab so you will need to have gitlab account created as well I already have it uh you can go ahead and create it and also verify it using your credit card or else you will not be able to run pipeline uh there's no charge for you to do it just for verification you will require a card I've explained all this more properly in the the video that I've told you here so make sure you go through it along with this you also need AWS CLI installed we are going to connect our AWS with terraform by running AWS configure command so you will need to have AWS CLI installed I already have it installed and you can check the version is 2.15 for you to install it just simply search for AWS CLI download and you will go to the AWS documentation that can help you install AWS CLI very easy next you also need a code editor we are going to write terraform code in a code editor most likely it's going to be vs code I use vs code so you will also require to have VSS code install if it's not there for you so these are few things to keep in mind before we get started with the project so now we are all set and ready to get started with our project for this particular project I am going to create a folder inside my project directory so let's create a folder named as cicd DF you can name it anything you want this is just creating a new folder if you are new to Linux you might think what is it this command but it's just mkd command which creates a directory or a folder named as cic DTF I'm going to open this up in my vs code to start writing terraform code so now this is opened up in my vs code let me pull it up here this is how it should be inside this I'm going to start writing my terraform code now if you're new to terraform you will have to go through the video that I told you but whenever we start writing code in terraform to create infrastructure on AWS Azure anywhere we start with defining a provider a provider if you don't know what it is you can just search for terraform providers and you will see the list of things or list of cloud providers that terraform supports we are going to create an infrastructure on AWS so we need to Define AWS as a provider terraform supports all these different providers for this project we are deploying stuff on AWS so I will start with defining AWS as a provider inside a file named as provider TF so in this I'm going to define a provider which is going to be AWS and inside this I'm going to Define some configuration settings like region so I want to use a region which is going to be us- east-1 which is the North Virginia region so for now I'm going to put the values but later on to make sure I follow all best practices and and do the things as we do in production I'm going to variabe and also use modules so beforehand I'm going to just create a variables. TF here and aain main.tf file for now okay so I've defined my AWS as a provider I have my main.tf I have variables. TF as I told you in the start of the project we are going to use best practices and follow the same things as we do in production so we are going to deploy a VPC uh Security Group E2 instance and later on when we have a cicd setup we might create more objects as well for now as we are creating VPC and ec2 instance we are going to create two different modules if you don't know what module is again go through the video to understand what is a module modules in terraform is a container for multiple resources that are used together so we are going to use module to create everything about VPC in one module everything about U ec2 instance in another module so let's do that by creating a folder so I'm creating a folder named as VPC which will have everything about VPC in here so VPC subnets uh security groups everything inside this folder I'm going to create three different files first is going to be main.tf the second is is going to be variables. TF and the third file is going to be outputs. TF you're very soon going to understand why we are having all these files and how we will use them along with this I'm also going to create another module for ec2 stuff so I'll create another folder and name this as web module name can be anything it does not has to be the name of the service I'm going to say web inside this I'm going to create three files again main.tf variables. TF and outputs. DF okay all right so we are now ready to get started with writing the code inside this files I'm going to start writing the code for a VPC a subnet and a security group so we will start so we will create uh one VPC one subnet and one Security Group this is not a project where we set up infrastructure for a complete application this is just a project to explain you how you can use terraform with cicd to automate your infrastructure provisioning this is why we are just creating VPC subet and Security Group if you want to see other projects check out the channel I've created many projects using terraform kubernetes and everything else so for this let's start with defining a VPC so I'm going to create a resource and it's going to be awor VPC the name of the VPC I want to give is going to be my VPC inside this we going to add some configurations first is the CER block this VPC is going to have a CER block of 10.0.0 sl6 for now as I said I'm going to put the values directly but later on variabe it to follow best practice along with this I'm going to say enable DNS host name is going to be true and enable DNS support is also going to be true now if you're not sure what is this DNS host name what is this CN support you can always use the official documentation so if you just search for AWS VPC terraform you will go to the official documentation for this particular resource and Below you can find argument reference on what is block and also enable host name you can see DNS host name support is also present so everything every time you are confused about anything feel free to use documentation terraform has one of the best documentation in the devops tools so now along with this I also need to add the name of my VPC so I'm going to say tags and the tag is going to be name equals to my VPC okay this is the code for our VPC along with this I'm also going to create a subnet and a security group as we have discussed here so I'm going to say resource is going to be AWS subnet and this is going to be public subnet so let's say pbor SM and inside this I'm going to pass in some configuration again this subnet belong to this particular VPC so I'll say VPC ID equals to AWS vpcm VPC doid so AWS VPC mybpc do ID next I'm going to add CER block the CER block is going to be 10.0 0.0 sorry 10.1.0 sl24 this is the side block I want to keep for my Subnet again I also have a separate video explaining how how to create VPC Subnet in internet gateway outout table everything so make sure you check that out if you want to learn more about how to create vpcs and subnets in terraform now after the ID blog I also need to add I also need to map public IP on launch so I'll say map public IP on launch equals to true and and this should be in Us East 1A subnet so I'll say availability zone is going to be us- east-1 a okay along with this I also need to add the name for this so I'm going to say tax is going to be name equals to [Music] pbor sn1 public subet one now this is the code for creating AWS subnet next we are going to create resource security group that that is going to be attached to our ec2 instance in this web module so I will say resource is going to be AWS Security Group so Security Group is here and the name of the security group I want to say is going to be SG SG inside this I'm going to first Define where is this Security Group going to be created so it will be created in this VC which is AWS vpcm VPC doid so the VPC ID is AWS VPC my VPC do ID here next the name of this is going to be my SG so I say name is going to be myor SG description if you want you can add it if it is optional so I'll say public Security Group here so public security and then I'm going to Define inbound rules and outbound rules so inbound rules are defined using Ingress and outbound rules are defined using egress for Ingress I'm allowing p 22 which is the SSH port and for ESS I'm allowing everything so I'm going to say Ingress is going to be from Port 22 to Port 22 protocol is going to be TCP so protocol is TCP and CER block which is going to be allowed is going to be 0.0.0 which means everyone can everyone who has the keys can access my AWS E2 instance if this Security Group is allowed so I'll say 0.0.0 sl0 okay now this is side of blocks so we will have to put this in square brackets because because you can have more than one side of blocks in a security group so I'm going to put this in square or else we will get an error okay now this is done for Ingress all let's do the same for egress so I'll say egress is the outbound traffic and I'm going to say from Port zero because I'm allowing everything from Port zero to Port Zer protocol is going to be minus one and then I'm going to say CER block is going to be the same value which is 0.0.0.0 sl0 okay so this is our Security Group which is now ready let's quickly also create an ec2 instance in our web module here so in this I'm going to create an E2 instance which will have Security Group uh that is created here in the main.tf and I'm going to show you how you can use a value from one module to another which is a bit which is a bit complex to many devops Engineers so let's start with creating a resource I'm going to to say resource is going to be AWS _ instance so AWS instance and the name of the instance I'm going to say is word server okay inside this we need to pass in some values like Ami instance type Security Group subnet and so on so I'll put the Ami I want to use Amazon Linux so I'm going to get the Ami for Amazon Linux which is here so you can simply just get it from your account or else it will not work it's account and region based so I'm going to get the from my account in not Virginia because I'm using Us East one now next thing I want to put is going to be instance type so instance type is going to be T2 micro we want to stay in free tier limits so I'll say T2 micro here and I also want to use uh subnet so I'll say subet ID is going to be the one that we want from here so this subnet this is where I want to launch my E2 instance but this is in different module so how can you get one thing from one module to another module which I will show very soon so we will keep this blank as of now similarly I also want to have Security Group so I'll say security groups and this is also something that we will get from this particular model we want to launch we want to use this Security Group for our instance so I will very soon show you how you can use it from one module to another for now let's put in let's get everything else done along with this I also want to have tags I'll say tags is going to be name okay tags equal to name equals to to my server okay now let's get the values of subnet ID and security groups before I show you how to use values from one module to another let's understand the concept of modules you can also learn it from the official documentation here which explains very properly about modules what is root module what is child module how can you use stuff and how to create a module so the concept is this main.tf is the root module or parent module where you will be calling all these child modules this VPC and web is the child module and if you want to pass some values from one child module to another child module you need to First specify the value as an output in the module that you want to transfer it from and then call it as a variable in the module wherever you want to use it if this is not making sense let me show you in this module we want to use subnet ID and security group from the VPC module here so let's get this Security Group and subnet as an output so that web module can use it so I'm going to create output here by SP saying output is going to be pbor SN which is the security group and the value of this is going to be AWS subnet pbsn doid because this is what I want to use here in my main.tf now along with this I also want to have Security Group ID here so I'll also get that so I'll say output SG and then I'm going to hold the value which is going to be a wsor security group. sg. okay now we have created outputs for the things we want to use in this module let's use this by creating as a variable so I'm going to say get the variable so I'm going to say create a variable with the name SG the name could be anything and I'm going to keep this blank for now let me show you how we can reference the value I'm going to create a variable for subnet so I'll say SN and I'm going to keep this value blank as well let's use this in here so the variable for subnet is variable do SN so w. SN and the variable for security group is SG W do SG okay so now we have used we have defined the values in output here and we have defined the values here in variable to start using it we first need to call the module in the root module so I'll say module block the name of the module block is going to be VPC and we need to fill in some details the first is Source where is this module located so this module is located inside this particular directory in a folder named as VPC so I'll say dot SL PPC and then similarly I'm also going to do the same for my second module so I say module ec2 and the sources sources do/ web okay so we have called it now you see a red error here which says there are few things that I need to add the first is the SN and the SG here so what is the SN and what is SN and NG that you have described here in the variable so I'm going to put that now SN is the value that is stored in the main.tf which is a subnet ID and I've defined output here so to call something or to get a value from one module to another you need to Define it by saying module module name output name and you can also check it here accessing child module outputs so you will have to use it somewhere like this module. module name. output name for example if you can go ahead and read it if you want to so it's going to a module do module name is VPC and the output name for SN is pbor SN so the value should be from here pbor SN similarly I'll also do the same for SG so it's going to be module. VPC do output name the output name for security group is just SG so I will also do the same here so pbor SG okay this is how you use modules and this is the way we create infrastructure in terraform we don't just do it in a single file we categorize it using modules as I have done it here when you have a huge infrastructure you will have more modules for databases for networks for VPC and lot more it depends on how you categorize your application but this is how we do it in our production environments so our code is almost ready Let's try to see if it actually works or not and to do that I'm going to run the terraform commands before I actually run the terraform commands you need to connect your AWS with terraform if you haven't done it already and you can do that by running AWS configure command I also have a video showing how you can connect AWS with terraform you can check it out here but it's just adding your keys after running AWS configure so I already have my keys added and I'm also going to show you how you create this keys in the second part of the video because we will require this when we set up our cicd pipeline so if you have done this already it's good if not just go through the video or wait for me to show you how to create keys so you can put it here so I am already connected to my terraform with AWS I'll say terraform init command this is the first command we need to run to initialize our terraform to to install everything to work with AWS provider here so I'll set it up for a minute also make sure you are saving all the files or else you will get an error so I'm going to just save all the files that I created till now and once I save it I am going to then run terraform init command this is going to initialize the back end initialize modules we have a folder created which is having modules and we'll also have provider folder very soon here so this is how terraform works we first init it then we are going to run the next command which is validate plan apply and then if you want to destroy it we are going to destroy it so let's wait for this to be initialized okay so terraform has given us output saying terraform has been successfully initialized let's check if our if our code is proper or not and we can do that by running terraform validate command so I'm going to say let me just terraform validate it says the configuration is valid which means our code is perfect now I'm going to run terraform plan command to see what is going to be created according to the code a VPC subnet Security Group and ec2 instance should be created so four things should be created when I run terraform plan it gives me an error saying string is required okay we have got an error somewhere here so it says security groups v. SG is a string set of strings is required which is true so I'm going to go to my main.tf in this it is asking us to put set of strings rather than just strings so all I need to do is to put this in my square brackets because it's asking for set of strings so let's put that here save it and try terraform plan again these are some of the troubleshooting skills that you need to know as a devops engineer so you can work things around you can see the error is now gone and we have four things that is going to be created which are VPC subnet Security Group and ec2 instance so which means our code is now running properly I can also run terraform apply to actually have this in my infrastructure let's try running it to see if it is going to create it or not so I'll say terap apply hyphen Auto hyphen approve okay when I run this you can see a terraform dotf State file is also present here this is not a good practice to have your terraform TF state in local machine so we are going to very soon set up a back end so that this state file is going to be stored on S3 using Dynamo DP as well so this is how we do it in production we never store it in local machine I will also show you how to do it in fact I already created a video to show you how to create how to set up remote backend on S3 and Dynamo DB so you can check it out here now this is creating uh it has created VPC subnet Security Group is also created and now it is creating an instance to show you let's go to the dashboard and you will very soon see an instance uh in pending state so let's me show you that I'm in North Virginia region and let's remove this tag you will see see a server is now in pending state which is my server as we have defined in here my server so this is this means our code is running properly and everything is working fine so we are now ready to set up back end on S3 once we do that we are going to push the code on gitlab and start with the second part of the project which is to create cicd pipeline to automate all of this so it's working fine let's go ahead and delete everything say so I'm going to go ahead and delete this by running terraform destroy hyphen Auto hyphen approve and this will start deleting everything that we have created okay now I have my code variable IED and it's following all best practices let's go ahead and set up back end using S3 and Dynam DP I I've already created my S3 bucket and also a Dynamo DB table let me show you so this is S3 bucket that I'm going to use along with a Dynamo DB which is used for State locking I have explained how to do this very properly in a video so make sure you check that out or else you can simply just search for uh S3 State backend and you will come across terraform documentation which which explains how you can use S3 and Dynamo DB along with this configuration to store your state on S3 rather than storing it locally so we need to create a back end something like this so I'm going to just copy this I can also optionally type it if you want to but to save time I'm going to copy and say backend. TF inside this I'm going to put the code and fill in my details which is going to be the bucket name that I have uh the bucket that I'm using is going to be the keys where do you want to store inside where do you want to store inside the S3 bucket so I want to store inside a folder named as state so I'll put State here and the name of the bucket is my state terraform 223 something so I'm going to put this here region is Us East one along with this I also want to have Dynamo DB table for State locking State locking means whenever many people are changing something that can affect the state Dynamo DB will lock it for one person at a time to avoid Corruption of State file this is uh this is State locking so I'm going to put the name of the Dynamo DB table that I created which is here so I'm going to tables the name of the table is back in so I'm going to use that here okay now we have backend configure to start using it I will have to run terraform init command again and it should tell me that the back end has configured as S3 so let's see if it does that or not I'm going to save this first and run terraform in it you can see initializing the back end if it and it says successfully configured back end S3 terraform will automatically use this backend unless the back end configuration changes this means we have also followed the best practice to store our state file on S3 this has brought us to the end of the part one of the project where we are doing things manually we have created all this code and also ran terraform commands manually to create all this infrastructure to automate this let's move to the part two where we are going to create gitlab repository push this code on that repository set up a cicd pipeline and automate all this process so to push this code I'm going to first go to gitlab and create a repository so I'm here in my gitlab dashboard and you can see some repositories already created these are the repositories that I used to teach you gitlab cicd in the video that I've mentioned here so make sure you check it out if you want to know more about gitlab cicd because we are going to create a cicd configuration file in this part of the video before we do that let's go ahead and create a repository I'm going to click on new project option here create a blank project let's name this as uh cicd TF if the name is available I'm going to make this public and initialize with a readme and click on create project option here now once it is created I can run this commands here uh to start pushing my code from my local machine to this gitlab account before I do that I first need to initialize my get repository here so I'm going to say get in it this initializes the repository and now you can run the commands present here to push the code now before we push the code there's also one more thing we need to add which is the get ignore file so if I search for get ignore uh Tera form you will get a file that you need to use so that you push only necessary things avoiding anything that you don't want to push which is this thing so I'm going to Simply copy this and create a new file here with the name dogit ignore and I'm going to paste all this here okay now we are going to push this by uh running the commands before I run the command let me connect with my gitlab repository here by running the command git remote add origin test T this so I'm going to run this command and now if I show you get remote- V command gives an output saying this is the repository that you're connected to so you you can fetch and push things here so usually right now we are on the main branch in gitlab the main branch is always protected and that should be the case because we don't push directly on the main branch in production we rather create a separate branch and then if everything is okay we merge it to the main branch so I'm going to do the same I'm going to first create a branch so I'll say get checkout hyphen B and let's create a branch named as Dev so it says switch to Branch Dev now in this Branch I'm going to say get add Dot get commit hyphen name initial commit and then I'm going to say get push hyphen U origin Dev so I want to push the code from my local machine to Dev branch in my gitlab once I do that it's going to ask me for my username and my password and after I enter that you can see it started pushing very soon if I refresh the page we will see a Dev branch has been created and there's an option to create M request so this is how we push the code we pushed the code in the dev Branch or any other Branch apart from Main Branch if everything is okay we create a merge request the senior devops engineer or anyone is going to merge it so that the code can be seen in the main branch here so I'm going to create mer request myself and you can describe it if you want to for now I'm not going to give any information I want the branch to be deleted after it is merged so let's say create merge request uh I am going to review this merge request if it was something someone else let's say a senior devops engineer they would review uh the merge request see what changes have been done and everything else if everything is okay they will just go ahead with click on merge and the branch will be merged every code in the dev so initial commit Dev into main now it is merged you can see here as well uh let's check that in our repository so all the code is now present in the main folder main branch here so this is is the code which is ready now we are ready to start creating our cicd configuration script so to do that I'm going to click on this plus symbol here and click on new file to create a cicd pipeline in gitlab you need to follow a naming format which is gitlab hyen C.L file this file helps gitlab understand that the configuration inside this is cicd configuration I'm again reminding you if you're new to gitlab CI please please check out the previous video it will explain you how it will explain you what is stages what are jobs what are images and also other things that can help you write and understand the cicd pipeline we will be doing very very properly okay so what is the use case of this what why are we creating a cicd pipeline we are creating cicd pipeline to automate terraform commands so that whenever a person makes a change in the infrastructure which can be anything let's say rather than having to one ac2 instance they make it to ac2 instance or they added a new security group or they added a new subnet they don't have to devops engineer don't have to manually run the commands it will be done through cicd cicd will check it plan it apply it destroyed automatically whenever a pipeline is triggered so this is what we want to be done using this script so the first thing we are going to do is use an image that will let us run terraform commands so we need to use an image that has terraform capabilities so I'm going to say image and the image that I want to use is going to be the terraform image which is present on the regist which is present on the gitlab registry so this is the terraform image that I want to use so which will let us run terraform commands along with this I'm also going to set the entry point so this is the entry point and also the path where all my binaries and dependencies are stored so this will let us run terraform commands and it's very important I've explained image much detailed in the gitlab video so make sure you check it out now in this pipeline there are going to be four different stages validate which is going to check the code plan which is going to check what is going to be created or deleted according to the code if everything is okay rent the terraform apply command and then lastly destroy so we are going to have four different stages so I'll say validate plan apply and Destroy okay inside each stage I want uh to run particular commands so here I will run the terraform validate command here I will run plan command I willar apply command and Destroy command so on now let's define jobs uh so jobs I'm going to use the same name for the jobs as well so I'll say validate this job belong to a stage which is validate stage so I'll say validate here inside this job I want to run a command which will be done using a script parameter here so I'll say script and run the command which is going to be terraform terraform terraform validate okay next I'm going to go to the next stage which is the plan stage here so I'll say inside the plan this belongs to the stage which is plan uh in this I'm going to say say script and run the command terraform plan but also output it in a file named as plan file so if you don't know what that command is let's go back to our let's go back to our vs code here and if you have run terraform Plan before you might know when you can also use iph iph out parameter to save your plan into a file and we will do that here so let me show you this is The Hyphen hyphen out equals to path write a plan to a given Path U this can be used as an input to the apply command this is what we are going to do in our cicd configuration here so I'll say terraform terraform plan hyphen out equals to plan file if I want to make sure that the plan that is created here is saved in a file and can be used by another stage which is apply stage here I need to Define this as an artifact so in gitlab whenever you want two stages to use something you need to Define an artifact which is explained again in this particular video so I'm going to Define an artifact saying the save the file and so that it can be used in the next next stage I say artifacts is going to be parts and I want to save uh this particular plan file this is our plan stage and we want the plan to be running only when the code is correct only when the validate stage is passed so this has a dependency saying that plan should run only when the validate is complet so we will add a new parameter saying dependencies C's and we are having dependency with the validate stage here which means if validate passes only then the plan should pass okay now this is our second stage let's go to third stage and create a job for it so I'm going to say apply job apply and the stage it belongs to is apply again apply should also run after the plan is completed so I will say dependencies and it has a dependency with the plan job here okay now inside this I want to run a command which is going to be terraform apply and then use this plan file to apply it so I'll say terraform apply along with this there are also no variables or inputs that you need to add as an user so I'm going to say input equals to false or else sometimes you will see that your terraform or else sometimes you will see that your CD pipeline is stuck because it is asking for some input from the user this is not the case for us I'm saying terraform apply hyph iph input equals to false and then apply whatever is defined in this plan do plan file this will only be possible when we have artifacts because only then we can pass things in different stages this input only has one dash here uh lastly we have destroy job so this job belong to the stage which is destroy and it also it runs a command which is going to be terraform destroy hyphen Auto hyphen approv so I'll say terraform terraform destroy hyphen Auto hyphen approve so approve so these are four stages that we want to be automated using cicd but there's still a problem with this file here before we actually run this commands we need to make sure the terraform is installed and is also initialized so that we can use with AWS or any provider that we have defined so what we need to do is add another set of code here which should be running before the stages are running so we are going to say before underscore script this means run these particular commands before you actually run the pipeline so in this I'm going to run two commands the first is terraform D- version command this will help us know that this particular image has terraform installed in it and we also can see the version in the logs next is terraform init command to initialize our provider backend uh so that it works properly so these are the two commands we need to run before the script is running now in our local machine I have AWS configured properly with my keys and secret access keys but how will gitlab know which account do I want to create infrastructure too so we need to also use our access keys and secret access Keys here in this gitlab cicd so how can we do that we can do that by defining the values directly in the code which is not the best practice so we would rather create a variable we will create two variables actually one for access key and one for secret access key before we do that let's go and get the access keys from I am here so if you don't know how to do it I'll let me show you that very quick I am right now logged in into a user uh logged in with the user named as Nasi which is an IM user so I'm going to use the keys for the same if you are if you don't have an IM user you can simply create it from here after you create it just go to whatever user you want I'm going to go to this user and select security credentials option here in this if I scroll down I have access keys at times you can have two access keys so I have two already I'm going to delete uh one of them so I'm going to deactivate first then copy this and paste it here delete it now it is deleted I'm going to create another set of keys you can use any option here CLI I understand next no need for description next and now I have the keys so I'm going to use this keys to uh make sure that the gitlab will authenticate and also connect to my AWS so that every time the cicd is running it applies on this particular account so I can optionally go ahead and put it directly in the code which is not the good practice and maybe your account might be banned uh because GitHub does GitHub bans your account if it finds access keys or secret access keys in your code uh so we don't do we don't do that here I'm going to open my repository in a new tab here and then create variables for my ACCESS keys and secret access keys so click on settings here go to cicd inside cicd you will find variables here and click on expand variables in this I'm going to add a variable first for my ACCESS Keys second for my secret access keys so this is a description which is optional you can choose to add it if you want or leave it so in here I'm going to name this as mycore AWS uncore accessor key and put the value of that here so this is now my access key similarly I'll do the same for my secret access key as well so I'll say mycore secretor key and then put the value of the secret key here as well do not try to use my keys because I will be deleting this once I complete recording this video U so you have to use your own account you have to use your own Keys now this is done I've created my access key Keys I've have created variables for my ACCESS keys and secret access Keys let's use it in the cicd code here so I'm going to Define variables and I am using two variables one for access key one for secret access keys but I will also use one more for region so I'll say AWS uncore access uncore key uncore ID is the value that is stored inside the variable the variable name is my AWS access key so I'm going to put that here similarly AWS uncore secret access key secret accessor key equals to the value stored here in my secret key okay along with this I also need a region so the region is going to be AWS underscore default underscore region and the value is us - east-1 so us- east-1 okay so this is our code which is almost ready we can also do more things in it but uh for now I'm going to leave it as it is and see if it is actually working or not one more thing to note here is you can see there are four stages here validate plan apply destroy whenever we run this pipeline it will first go through the validate section check if everything is working fine if it is working fine it will go to the next stage which is the plan here and it will create a plan if everything is okay it will go ahead and apply it if everything is created it will also go ahead and destroy it which is not what we want so we want to have validated and plan automatically running but we want to have apply and Destroy manually sometimes uh in many companies you have these manually done because once the plan is created a senior developer or a senior devops engineer will manually run the job to apply or destroy resources which is what we want in our project as well so how to do that I'm going to just add another line here saying when do you want to run this particular job so I want to run it manually so I'll say when manual and do the same for my destroy as well so I'll just add another line here saying when should this job run it should run whenever I do it manually so now this is our code here let's commit this I'm going to say add cicd pipeline file on the main branch I'm going to say commit now once it is committed you will be able to see a symbol here so we got an error saying failed this is because I think the configuration yaml is not correct let's see what is the problem the problem here is I'm using equals to symbols I should be using colons here so I will edit this now U Get lab also has many different features you can see it always gives you what is wrong with your uh what is wrong with your cicd pipeline so I'm going to just change this to colons rather than equals to symbols which is problem here so I'll just change it and now you will be able to see that the pipeline syntax is correct so once it is done I'm going to just go ahead and commit change with the message update gitlab.com so right now we have four jobs uh as you can see here the validate job is running you can also check the logs for it by going inside it if you remember we ran before script which should show us the terraform version and also terraform initialized so let's see if we get that here so we get the version and we get the innate as well but there's something wrong with our code it says uh a manage resource Security Group has not been declared in module. VPC so the validate gave us an error saying there is no uh you don't have a you don't have you don't have a security group with this name which is true because the name of the security group is not SG it's actually pbor SG let me show you that so right now we are inside c.f let's go to VPC inside this VPC we have a security group created with the name pbor SG whereas in the outputs we are referring it as just SG which is not correct so I'm going to edit it real quick just to show you whenever I edit it the pipeline is going to trigger again and again and also update it uh in the infrastructure if everything works fine so let's change that very quick here this is klab's uh own ID in this I'm going to just change this to pbor SG as we have in here so pbor SG once I change this I'm going to commit by saying changed uh SG name commit to main I'm going to click on continue and the commit has been made let's go to the project and you will see a pipeline is going to run again so here is the pipeline symbol which is running right now let's see if once everything is working fine it should bring us to apply where we are going to manually apply it and we should be able to see that ec2 instance Security Group VPC and subnet is created so here we are in the validate stage you can check the logs here uh very soon it will show us the logs if it is correct it will give us a message saying the configuration is valid and it will then move on to the next stage so we have seen that it shows us the version also give us ter a minute it also tells us the configuration is valid let's go and see if it is going to the next job so the next job is running which is the plan so stage is plan here let's see if we get the plan saying four things are going to be created okay so job has been succeeded and you can see plan says four things are going to be created which means our pipeline is working properly now when you go back and you check at the pipelines here these two are manual jobs that you need to run it manually and this we have done it ourself by making the change so I'm just going to say run here when I say it run it will start running the terraform apply command which we have for this particular state P so this is how you can automate using terraform and cicd I hope I hope this project has taught you a lot uh if you have any problems any issues with this project you can let me know in the comment section and feel free to use the code I'm going to paste the link for this repository in the description feel free to create this project yourself share it on LinkedIn because this is this is actually very useful project because this is how we do infrastructure deployment on production so I thought this should be something that you need to know uh as a devops engineer so you can see uh module VPC has been created here Security Group has also been created subnet is also created it's right now creating ec2 instance uh so let's check that in the in our AWS year and you can see job as been succeeded apply complete I can confirm by going to my AWS and checking if the e instance is running or not with the name my server so in the ec2 dashboard we have a server running with the name my server which was created now and it's T2 micro so everything is automated now whenever I make a change which can be adding a new resource or deleting a resource or anything it will automatically update things without me manually running the commands so this is how you can use terraform and gitlab CI let's also see if it can automatically delete things or not so I will go to my uh jobs again let's go to jobs and you will see that the delete is also a manual thing which is actually good you should make it manual or else whenever you make a change you can accidentally delete everything in your account so let's click on run job here and it will start deleting everything because the command it's running is terraform destroy hyphen Auto hyphen approve so this is the project uh let's wait for this to be deleted so it started it has started destroying it has started destroying it will destroy four uh things which are created VPC SG SN and ec2 instance so it has successfully deleted uh everything now so we have we have automated we have learned how to automate terraform deployment uh using gitlab cicd you can do the same thing using GitHub Circle C genkins anything you want but this is the basic architecture I don't want you to learn tools I want you to learn the concept behind the tools so how you can automate it uh if you want you can try out this with gitlab you can if you want you can also try out this with GitHub or genkins or anything you want so this was our terraform pipeline project completing this project will help you understand how we deploy infrastructure on the cloud in production systems I hope this video was informative if you have any questions any doubt do let me know in the comment section try it out share it on LinkedIn and do tag me thank you and have a good day

Channel: Cloud Champ

Views: 21,904

Keywords: terraform project, devops project, terraform pipeline, terraform pipeline gitlab, terraform pipeline github, gitlab pipeline, terraform project aws, devops projects for practice, terraform, gitlab, terraform ci cd pipeline, terraform ci cd pipeline aws, gitlab terraform pipeline, gitlab terraform aws, aws, terraform project for resume, gitlab terraform, gitlab with terraform, terraform with gitlab, terraform with aws, aws terraform tutorial, aws terraform project, aws terraform

Id: oqOzM_WBqZc

Length: 50min 13sec (3013 seconds)

Published: Sat Mar 02 2024