DevOps CI CD Project | Complete DevOps Project from Scratch | CICD DevOps Project | Jenkins CI CD

Hello friends and welcome back to my channel I am ashfak and in today's video we will create a complete automated cicd pipeline project using bit bucket sonar Cube jeffro artifactory Docker Hub and the kubernetes I will explain you in short the of our today's project in our today's project first of all we will build the ec2 instance for the genkins using the terraform and then we will build the pipeline using the genkins in which if user makes any change on the bit bucket repository then that change will trigger the cicd pipelines and then uh our pipeline will first do the maven build uh from the code and then it will perform the sunar cube analysis of the code then it will push the bill to the artifactory jrog artifactory after that it will perform the 3v fs scan of our code then it will perform the build and push of the docker image then it will perform the trivy image scan of the pushed Docker image after that it will deploy the resources using the Manifest file on the kubernetes and then we will have our our application accessible over the internet and for every build we will get the email notification on our Gmail ID and that email will have the trivy fs scan and trivy image scan and the complete build log as the attachment so let's start building our today's [Music] project this is my Windows 10 system on which I'm having the terraform installed you can refer to this video on how to install the terraform on the Windows system I have already installed the AWS CLI on my system you can uh download the AWS CLI from here and you can install it on your system and from this link you can download the visual studio code and you can install it on your local system so the first thing which we are going to do is to create the E2 instance using the terraform so first of all on my system I will create one folder and I will give it the name and in inside this folder I will create one more folder in which I will be keeping all my terraform files and I will give the name to this folder as Jenkins sonar Cube VM on the terap sorry on the visual studio code I will open the folder which I have created just now and under the Jenkin sunar Cube VM folder I will create the main.tf file and uh I will give the content to the main.tf file so I will explain you what we are going to do through the uh terraform this is my Ami you will have to change the Mi as per your region and uh uh the inst type I'm going to create is T2 large and uh this is my key pair name which I'm going to use to access my ec2 instance this one key pair if you don't have you can create uh the key pair for your AWS account and uh this would be the name of the ec2 instance the dis size we are going to create is 40gb and uh we will create the security group uh with this name and note one thing here we are not going to create the VPC because every account has it's default VPC we will create the security group uh and this Security Group will get maap to default VPC of the account so there is no need to create the VPC if I go to my account [Music] so if I go to VPC so this is the default VPC in my account tenc default every account has the default VPC so this should be the name of the Security Group and inside the security group we are going to open these ports inbound ports and uh the user data we are providing this sh file through this sh file we are going to run multiple commands to install the packages in our newly created EC to instance I will save this file and then I will create one more file called provider. TF where we are going to register the provider so this is the AWS provider which we are going to register here and this should be the region of my ec2 instance I will save this file also and then finally I will create the installed dot sh file which terraform is going to run in our newly created ec2 instance so what terraform is going to do to our ec2 instance using this install.sh I will explain you in the short first of all it will update the system then it will install the jdk in the system and then it will install the genkins in our ec2 instance then it will install the docker with these commands and with these commands it will actually give the access to user ubu and Jenkins uh access to the docker Ubuntu user we will use on the terminal to communicate with the docker and Jenkins user will be used by the Jenkins pipeline script to communicate with the docker so it will give the access on the docker to the to these two users and finally it will run the sonar Cube as the container with this command on our act2 instance and with this command it will install the triy in our is2 instance and finally it will install the maven with these commands I will save this file I will go to the uh AWS portal now I will go to IM am users create user I will give the name Cloud admin next I will click here attach policies directly I will give the administrator access next create user okay so user has been created I will go to the user I will go to security credentials and here I will click on create access key I will click here CLI I will click here next I will give the description and create access key I will copy the access key and on the vs code I will right click and click here on open in integrated terminal I will give the command AWS configure I will paste the copied access key and from the portal I will copy the secret key and I will proide it here so I have logged into AWS console through the CLI on the AWS portal I will click on download CSV file and I will click on done on the visual studio code I will give the command terraform in it now I will give the command terraform plan so it is going to create these resources you can review these resources which it is going to create and then I will give the command terraform apply hyphen Auto approve okay so creation completed through the dform if I go to AWS console E2 instances so this is the instance which has been created and if I go to security so these are the inbound rules which has been created inside the security group I will copy the public IP and I will open the terminal on my local system and I will go to session SSH I will provide the public IP username Ubuntu Advance private key I will select my private key [Music] okay if I give the command Jenkins h version so this is the version of the genkins which has been installed in my is2 instance and if I give the command Docker hyphen version so this is the version of the docker which has been installed in my system and if I give the command trivy hyan version so this is the version of the triy which has been installed in the ec2 instance and if I give the command mvn hyph version so mvn could not be installed uh through the install.sh we can install the mvn I will give this command to install the maven so this is the version of the maven which has been installed in my ec2 instance and we are going to run the sonar Cube as the container on our ec2 instance so if I give the command Docker PS so this is the sonar Cube container which is running in our system so sonar cube is running uh as a container on our P2 instance and this is the container for the sunar cube you might be uh practicing this project in multiple settings so if you are uh going to shut down the E2 instance then you'll have to first stop the uh sunar Cube container with this command doer stop and the container ID and once you are back after starting the ec2 instance you'll have to give the command Docker PS h a to get the container ID and then you can run the command Docker start and the container ID to start your container and then you can continue your uh uh [Music] project next task is to set up the genkins I'll copy the public IP of the e to instance and I will browse it in the new tab with the port 80 80 I will go to this place on the terminal I copy the password and I will proide it here install suggested plugins I will create the new user called Cloud user I will set the password okay so we are inside the genkins now it's time to install the required plugins so I will go to manage genkins plugins available plugins and the first plug-in I will search bit bucket I will select this plugin and then I will search artifactory I will select these two plugins and then I will search eclipse and I will select this Eclipse timin installer and then I will search the plugin sunar Cube and I will select this plug-in sunar Cube scanner then I will search here G and I will select this plug-in sonar quality G and the then quality gate after that I will search here Docker I will select Docker Docker commments Docker pipeline Docker API Docker build step and Cloud B Docker build and push so these are the plugins I want to install I will click on install okay so two required plugins got failed artifactory and the docker build step so I will go to homepage I will again go to manage genkins to suppress this warning go to configure which of these warnings are shown and untick these two warnings plugins available plugins and I will search here RT artifactory and then I will search Docker Docker build step and I will again click on install okay this time artifactory and Docker build step got installed I will again go to uh managen kins and this time we need to set up the tools which are required for our cicd pipeline so I will go to tools and I will go to Docker add Docker name I will provide Docker I will tick on install automatically and I will select download from doer. net.com and the second tool which we need is sunar Cube scanner so under the sunar cube Scanner installation I will click on ADD sonar Cube scanner and I will give the name sonar hyen scanner and I will keep the tick on install automatically install from the maven Central and then we need to install the tool called maven so under the maven installation I will click on ADD Maven and the name I will provide the name I will provide as Maven and I will untick this install automatically option because we have already installed the maven in our server so I will untick here then we need to provide the maven Home Path here so I will go to terminal and I will give the command here mvn hyphen hen version so this is the path of the maven in our server where our Maven has been installed I will copy this path and I will provide it here apply and Save [Music] next task is to set up the sunar cube as explained earlier we are running the sonar Cube as a container on our ec2 instance to access the sonar Cube dashboard I will copy the public IP of the ec2 instance and I will browse it on the new tab with the port 9,000 default username and password is admin admin I will set the new password okay so we are inside the sonar Cube dashboard I will go to Administration Tab and under this security I will go to users and I will click here on update tokens I will give the name token for genkins and I will click on generate copy this token and save it on your local system as once you go to the other tab this token will not appear again okay now we have we have to provide this token under the credentials of the Jenkins so I will go to genkins manage Jenkins under security I will go to credentials add credentials and kind I will give secret text and here I will paste the copied secret and ID I will give sunar Cube token and I will click on create now we need to add the sonar Cube server to the genkins so I will go to manage genkins I will go to system and under the sonar Cube servers I will click on ADD sunar Cube and the name I will provide sunar Cube hyphen server and URL I will provide HTTP colon SL slash and I will copy the private IP of the instance and I will provide it here colon 9,000 and do not forget the slash at the end I will select the credentials in the drop- down apply and save I will go to sunar cube dashboard and I will go to Quality Gates so this is the default built-in gate this is actually the threshold so we are not going to create the custom quality gate we are going to use the default quality gate for this project and after the quate we have to set up the web hook between the sunar cube and the genkins so I will go to Administration Tab and under under the configuration drop down I will click on web hooks I will click on create name I will provide Jenkins and the URL I will I will provide in this URL I will replace my private IP address of the2 inst and SL sunar Cube Hy web hook and do not forget the slash at the end otherwise uh you may get the errors during the build and I will click on create [Music] in this section of the video we will configure the jrog artifactory I will go to terminal of my server and first of all I will make one directory for artifactory on my server I will go inside the directory then I will create one file I will provide the access and we are going to run the jrog artifact tree as the container on our ec2 instance similar to sunar cube so to run the docker container for the jrog artifactory on our E2 instance I will give this command if I give the command command Docker PS so this is the container for our jeffro artifactory and this is the container for the sonar Cube I will copy the public IP of the ec2 instance and I will browse it on the new tab with the port 8082 as our jrog artifactory is running on the port 8082 default username is admin and password is password get started I will set the new password next we need to set here the base URL so I will give the base URL as HTTP colon // IP of the E2 instance colon 8082 as our J frog container is running on the port 80 82 I will click on next so this is the proxy option I will skip here create repositories I will select Maven next and finish okay so here you [Music] can see the repositories these are the local repositories and to see the artifacts you will have to go to this option artifactory and artifacts now we need to integrate the jrog artifactory with the genkins I will go to genkins manage Jenkins credentials add credentials kind username with password username admin password I will provide my jro artifactory password ID I will provide jrog description I will provide jrog and I will click on [Music] create okay so in this part of the video we will actually create uh the pipeline script for our cicd so first of all we will create the app password on the bit bucket and then add that uh password to the genkins so this is my repository on the bitbucket.org you can create your account on the bit bucket and you can fog my Repository so under the account you can see here setting symbol is there go to setting personal bit bucket setting app have password and here you can create the app password I have already created the app password in my account for the bit bucket to add the bit bucket app password to the genkins I will go to man genkins credentials add credentials username with password and I will provide my username for the bit bucket this is my bit bucket username and password and I will provide the app password which I have created on the bit bucket and ID I will provide bit bucket and I will click on create okay now we have to add the credentials for the docker Hub into the genkins so if you go to dock Brer Hub and if you go to account my account security and here you can create the access token I have already created the access token for my account and Note One Thing uh here the app password of the bit bucket and the exess token for the uh Docker Hub both are both would be visible once only so copy the token and save it on your local system somewhere because it will not reappear again so to add the dockerhub token to the genkins I will go to managen kins credentials add credentials kind username with password I will provide my username for the dockerhub and password I will provide the access token created on the Docker Hub and ID I will give dockerhub and I will click on create now I will go to my repository on the bit bucket and here I will create the genkins file I will give the name Jenkins file and I will start creating the p P line script here so first of all I will start writing the uh pipeline script and I will type here agent any because we are not having any genins agent and I will Define the tools here so this is the maven tool which we have installed in the genkins and and this was the name given to that tool I'll will close the bracket here after that we will Define the environment variables so the first first variable we are going to use it our sonar scanner and this was the name given to the sonar scanner inside the genkins and this is the name we are giving to our application inside the p pipeline script Java registration app and this is my username on the docker Hub and this is the credential ID for the doer Hub given under the genkins and then I will start creating the stages and the first stage would be clean workspace which will clean our workspace the stage name is clean workpace it will clean the workspace okay so next stage would be check out from git in which it will clone the GitHub sorry bit bucket repository so the stage name is check out from git uh branch is Main and this is the URL of my bit bucket repository if I go to bid bucket repository if you go to clone you will get the URL to clone the repository here the same we have used here okay so next stage would be build package indentation should be correct otherwise your build will fail okay so in the build package stage it will first go to directory web apps if I go to my bid bucket Repository so this is my uh repository and inside this we are having the Webb folder so it will go to that folder web app and then it will run the command mvn package which will create thew file out of our code it looks okay now so the next step would be sunar Cube analysis in our code pipeline so I will give the code for the stage the stage name is sonar Cube analysis and this is the server name which we have given to sonar Cube inside the genkins while defining the sunar cube server under the genkins this name was given thar Cube H server so we are recalling that server details here again it will go to directory web app because all our code are under the directory web app in our repository and then it will run this command for the sonar Cube analysis I will correct the indentations okay uh and then next stage would be quality gate so this is the token which we have added under the genkins for the sunar cube so we are recalling the uh those credential ID this was the name given to the credentials under the genkins so we are recalling the same here next stage in the PIP plan script would be artifactory configuration so I will provide the code for the same so here actually we are first defining the RT server artifactory server and this is the name we are giving to the artifactory server jrog Heaven server and this is the public IP of our instance this one and the credential ID we are recalling here jrog which we have added under the Jenkins for the jrog artifactory and here we are defining the maven deployer and here we are defining the maven resolver and this is the name we are giving MAV deployer and MAV resolver next stage in the pipeline script would be deploy artifact so I will give the next stage in this stage it will run the tool this is the name we have given to the maven tool and the pom file path it is uh given here Webb / pom.xml if I go to my repository inside the Webb folder I'm having the pom.xml the same path we have given here so the goal is clean install and the we are recalling the deployer ID and resolver ID here now the next stage would be publish artifact and this is the server ID which we have defined above jrog hen server this this detail it is recalling here here again with this bracket we are closing the stages and with this bracket we are closing the pipeline ination looks fine now I will click on Commit commit and I will click here on approve I will now go to Jenkins dashboard I will click on new item I will give the name to my job pipeline okay discard all bills Max of bills to keep I will type two GitHub project and here I will Prov write my repository URL I will not set any Bild trigger as of now pipeline pipeline script from SCM SCM is G and here I will provide my bit bucket reposit URL and in the drop down I will select the bit bucket credentials which we have added under the genkins branch I will select Main script path Jenkins file I will click on apply and then save I will now go ahead and click on build now okay so job completed here if I go to the workspaces of my Jenkins job and if I go to this path on my server I will go to web apps I will go to Target directory so inside the target directory you can see this is the package which has been created by the pipeline script and if I go to jrog artifactory console I will click here on the left side and I will go to artifactory artifacts lip 7en release I will expand it and I will go to com Co I will expand it so you can see the web app folder created here I will expand this folder also so this is the release and this is the artifact which has been uploaded to the jeffro artifact Tre through our Jenkins job and this is the URL for the artifact one can copy this URL and download the artifact so the artifact has been downloaded via this link if I go to sonar Cube dashboard if I go to projects so this is the our project name and this is the sonar Cube analysis for our project I will again go to my Repository I will go to Jenkins file I will go to edit and my on my Jenkins file I will add one more stage and the stage name is triy FS scan which will do the trivy scan of our code I will now go ahead and add the next stage in our pipeline script which is build and push Docker image and this stage is actually going to build the docker image and it is going to push the image to the docker Hub so it is recalling the variable Docker path which we have defined at this start and it is recalling the image name it is again the variable which we have defined at the start of the pipeline script and this this is the docker pass this is the actually dockerhub credential ID which we have given under the Jenkins and then image tag which it is creating here I will correct the indentations it looks fine I will now give the one more stage which is 3y image scan and this stage is going to scan the dockerhub image which we have pushed build and push to the toer hub so the stage name is 3v image scan and this is the command which is it is going to [Music] run to scan the dockerhub image this is my username on the dockerhub and this is our application name which we have defined here at the start of the pipeline script so it will give the similar name to the image doab image okay so after the triy image scan I will give the one more stage called cleanup artifact which will actually remove the docker image from local system okay it looks fine now I will click on Commit [Music] approve I will again go to Jenkins Pipeline and I will click on build now job completed if I go to dockerhub and to the refresh here so the image has been pushed to the docker Hub just now on the Jenkins job not down the build number Bild number is two and if I go to my dockerhub image it is having the tag two according to our build job on the genkins it is is having the [Music] tag in this section of the video we will set up the email notification for our Jenkins pipeline I have logged into my Google account and I am under the my account. google.com your Google account should be two Factor authentication enabled I will go to security and I will search here app password I will give the name and I will click on create I will copy the generated password on my local system because it will not appear again I will go to genkins manage genkins credentials add credentials username with password the username I will provide my Gmail ID and the password I will provide the created app password ID I will give Gmail create I will go to manag genkins system and I will go to email notification SMTP server I will provide smtp.gmail.com user suffix I will provide my Gmail ID I will go to advance SSL SMTP Port I will provide 465 use SMTP authentication username I will provide my gmail address and the password I will provide the app password which I have created and I will click on test configuration recipient email address I will again provide my Gmail ID here and I will click on test configuration email sent successfully if I go to Gmail I have received the test email I will now go to extended email notification SMTP server I will provide smtp.gmail.com Port I will give 465 Advance credentials I will select my Gmail credentials use SSL here I will provide my Gmail ID default content type I will select HTML and I will click on default all triggers I will select always and success I will click on apply and save I will now go to bid bucket repository I will go to gen file and I will click on edit at the end of the stages I will add the post here for indentations looks fine now this is the recipient email address and this would be the attachment to our email I will click on Commit I will go to Jenkins Pipeline and I will click on build now job completed not down the build number which is four so this is the sonar Cube analysis for our project and if I go to jrog so the artifact has been updated here and if I go to Gmail I have received the email after the completion of the job from the genkins which is having the 3v scan result attached and complete build log [Music] attached next task is to set up the eks cluster before we install the eks cluster we will have to install the CU CTL aw CLI and ekl in our ec2 instance so first of all I will install the cube CTL I will give the command on the terminal of my2 instance to install the co L then I will download the cube CTL package and then I will execute it now if I give the command Cube C version so this is the version of the cube CTL which has been installed in my ec2 instance now I have to install the AWS CLI so first of all I will download the package for the AWS CI I will install the unzip I will unzip the downloaded package and then I will execute execute the file if I give the command AWS hyph version so this is the AWS C version installed in my server next task is to install the ekl so first of all I will give the command to download the package for the ekl into the temp directory then I will go to /tmp directory and then I will move the ekl file to the /bin directory as all the executable files are under the/ bin now now if I give the command ekl version so the ekl has been installed in my system I will go to AWS dashboard actually now I have to create the role and attach that role to the ec2 instance I will go to IM roles create role AWS service in the drop down I will select E2 next I will give the administrator access to the role next I will give the role name create role role created I will go to ec2 instance I will select the instance I will go to actions security modify IM roll and from the drop down I will select my role update IM am roll I will again go go back to terminal of the instance and now we will install the E cluster so I will give the command this would be the cluster name I will provide the region for my kubernetes cluster then I will give the Noe type and I'm going to use the t2 small then I will provide the node count which is I'm going to create three nodes this cluster creation will take some time so I will pause the recording here and I will resume it once the kubernetes cluster creation completed eks cluster creation completed and note down this path where the cube config file has been saved if I give the command Cube CTL get nodes so these are the three nodes created in my cluster and this is the default service for my cluster I need to now download the secret file for my kubernetes cluster so I will open the duplicate tab and on the left side I will go to do Q folder and this is the config file for my cluster or you can say credential file for the kubernetes cluster I will download this file I will go to the downloaded folder I will open the secret file and I will save a copy and I will give the name as secret. [Music] txt next task is to integrate the kubernetes with our genkins and configure our pipeline script to deploy the resources on the kubernetes so first of all I will go to my Jenkins dashboard I will go to man genkins plugins actually I need to deploy the uh plugins uh required for the kubernetes I will go to available plugins and here I will search kubernetes and I will select these plugins kubernetes kubernetes client API kubernetes credentials kubernetes CLI and I will click on install plugins installation completed I will go to managen kins credentials add credentials here actually I need to add the credential file for the kubernetes to the genkins kind I will select secret file ID I will provide kubernetes I will select the secret file generated in the previous step I will click on create I will now go to my repository on the bit bucket so in my repository this is the kubernetes folder which is having the Manifest files I will open the genkins file and I will add the one more stage called deploy to kubernetes for for for it looks fine now so in this stage it will first go to kubernetes folder where we have kept our manifest files and this is the credential ready for the kubernetes created under the genkins and using this command it will deploy the it will apply the Manifest file for the deployment and through this manifest file it will create the service and using this command it will roll out the deployment if there is change on the dockerhub image and this is the name of the deployment under our deployment manifest file I will commit the changes if I go to kubernetes folder so this is the deployment manifest file uh this is the deployment name and this is the app name this app name we will recall in the service minifies file and under the selector and uh our deployment will create the uh two ports and this is the image name on the dockerhub and this is the container port 8080 if I go to service manifest file so this is the service name and under the selector we have recalled the app name which is the register a app and this is the Target Port which is our container Port 080 and it will open the external port at and the service type is load balancer which means we will be having the public URL [Music] before we verify our cicd pipeline we will first set the web hook trigger I will go to Jenkins configure under the built rer I will click here on build When A change is pushed to bid bucket and on the place of override repository URL I will put my repository uh URL if I go to bit bucket and go to clone so this is the URL I will copy it from here and I will paste it here I will click on apply and save I will go to bit bucket repository and on the left side I will go to repository settings web hooks I will click on ADD web hook title I will give bitbucket web hook and the URL I will provide HTTP colum SL SL public IP of the ec2 instance colon 80 80 slash bit bucket hyphen hook and slash do not forget the slash at the end triggers already selected to push I will click on Save I will now open the git bash in my local system and on the B I will first Define my username with these two commands then I will go to my repository clone I will copy the command and I will provide it here clone completed I will go to my repository I will open the readme file and I will type here test 99 control o and enter to save crol X to exit I will now stage the changes G add dot I will commit the changes and then I will give the I will push the changes to the remote repository while pushing the changes to the remote repository you might need to provide the app password of the bit bucket here which we had created initially I will go to personal access token this is my username on the bit bucket and I will provide the app password here okay changes push to remote repository if I go to Jenkins so you can see here our pipeline has been triggered upon the changes to the remote repository automatically job completed if I go to bit bucket repository and do the refresh here changes are available on the remote repository and if I go to sonar Cube and do the refresh here so this is the sonar Cube analysis for our project if I go to jrog and do the refresh [Music] here so the artifact has been modified just now on the jrog artifactory and if I go to dockerhub and do the refresh here so image has been pushed to dockerhub just now and if I go to the details of the image so note down the tag six here which is similar to our build number on the genins job six if I go to my email I have received the email from the pipeline with the attachment I will go to terminal of my server and if I give the command Cube CTL get ports so these are the ports created uh remember in the deployment manifest file we had mentioned replicas two so two ports are created and if I give the command get service Cube CTL get service so this is the load Balan service and this is the DNS name for our application I will copy the DNS name and I will browse it on the new tab so this is the Apache Tomcat homepage if I give this slash web app so this is my application we will test our pipeline once again we will do the small change on our application and we will see the changes appears on the applic or not so I will go to G bash I will go to web app I will go to SRC folder then I will go to main folder then then I will go to web app [Music] folder and inside the web app folder index.jsp file is available I will open this file and in this file at the end we the thank you I will add this line see you again crl o and enter to save crol X to exit I will stage the changes I will commit the changes and finally I will push these changes to the remote Repository again it is asking for the token to be provided changes has been pushed to remote repository if I go to my Jenkins job so it has triggered automatically upon the changes on the bit bucket Repository job completed I will go to my application and do the refresh here so the changes has been applied to my application by this way you can create the complete automated cicd pipeline using bit bucket terraform genkins sunar Cube jrog artifactory and the kubernetes while prati practicing this project if you face any kind of difficulty you can ask me in the comment section I will definitely answer to your query complete content of the terraform files and all the commands used in this video are available in the description section follow me on the LinkedIn I post multiple useful things on the LinkedIn also after practicing this project post it on the LinkedIn and tag me also hope you find this video helpful a lot of effort went behind this video so do subscribe to my channel to encourage my work like this video share this video with the friends and spread the knowledge I will see you in the next video thank you

Channel: Virtual TechBox

Views: 4,784

Keywords: devops project, ci cd pipeline, jenkins pipeline, devops cicd project, jenkins ci cd pipeline, devops tutorial for beginners full course, devops ci cd project, complete devops project from scratch, devops tutorial for beginners with projects, learn devops with projects, cicd pipeline project, jenkins ci cd pipeline tutorial, jenkins ci cd pipeline explained, ci cd pipeline using docker aws, cicd devops project, devops with projects, devops cicd projec, devops pipeline cicd

Id: E_ZoNZKNsvU

Length: 117min 35sec (7055 seconds)

Published: Sat Feb 17 2024